Supervisor:
Associate Professor Vincent Y. F. Tan

Department of Electrical and Computer Engineering National University of
Singapore

Doctor of Philosophy

# Towards Adversarial Robustness of Deep Vision Algorithms

Hanshu Yan
( M.Sc., NUS; B.Eng.& B.Sc., BUAA )

###### Acknowledgements.

This has been a happy, fortunate, and fruitful journey. It is also long and,
sometimes, arduous. I am very grateful for the help of many individuals. It
would have been much longer without their support. I would like to thank my
supervisors, Prof. Vincent Y. F. Tan and Dr. Jiashi Feng, for teaching me
research thinking and supporting me in pursuing my research interests. They
taught me to think deeply about the value and feasibility of potential
problems. They scheduled weekly meetings to discuss ongoing projects and gave
me suggestions on promising research directions. I have benefitted from their
broad range of knowledge and deep insights into research problems. These help
me establish my long-term research objectives. I also want to thank them for
teaching me to write academic papers rigorously and present ideas cogently.
Vincent always spares no effort to help me with English and mathematical
writing. I am pleased that my writing and presentation skills have improved
significantly during the past four years. Besides, I am grateful that they
help me build connections with many brilliant researchers in academia and
industry. Without the support and encouragement of my supervisors, this thesis
would never have happened. They are excellent supervisors and good friends
(良师益友 in Chinese). I would like to express my sincere gratitude to Dr.
Jingfeng Zhang, Dr. Gang Niu, and Prof. Masashi Sugiyama from RIKEN-AIP and
the University of Tokyo. I met with Jingfeng to discuss the robustness of deep
neural networks when he was doing his Ph.D. in NUS. We were both interested in
developing robust network architectures. After he joined the RIKEN-AIP, we
carried out a series of collaborations to robustify deep vision algorithms. I
truly appreciated that Prof. Sugiyama and Dr. Niu joined the discussion,
helped examine the technical reasonableness, and edited research papers.
Working with the team was a memorable and invaluable experience for my Ph.D. I
would thank my friends in Vincent’s group and Jiashi’s group. They created a
homely atmosphere where we support, inspire, and encourage each other. I
particularly appreciate two individuals, Dr. Junhao Liew and Ms. Haiyun He.
Junhao has always encouraged and comforted me like a big brother when the
going gets tough. He also shared with me his experience and lessons in
research unreservedly. Haiyun and I are in the same class. We helped each
other in curriculum modules and collaborated on research work. The
collaboration has been fruitful and enjoyable. I hope we can continue working
together in the future. I also want to thank my lab mates, Jiawei Du, Mengjiun
Chiou, Kuangqi Zhou, Yifan Zhang, Yanfei Dong, Dapeng Hu, Jian Liang, Kaixin
Wang, Yujun Shi, Pan Zhou, and others, for the inspiring discussions and far
too many things to mention here. I would like to thank many other students
from different departments at NUS. It has been a wonderful experience to
extend my research areas and apply deep learning techniques to scientific and
engineering problems. Lastly, I am extremely indebted to my parents for their
love, and the support in my pursuits and decisions. I am also very grateful to
my girlfriend, Tan Ye, for her deep love, unwavering support, and company.
Words cannot express my love and gratitude to all of them.

###### Contents

  1. ‣ Towards Adversarial Robustness of Deep Vision Algorithms
  2. List of Symbols
  3. 1 Introduction
    1. 1.1 Background
      1. 1.1.1 Adversarial Robustness
      2. 1.1.2 Robustifying Deep Neural Networks
    2. 1.2 Thesis Focus and Main Contributions
    3. 1.3 Organization of Thesis
  4. 2 Towards Adversarially Robust Deep Image Denoising
    1. 2.1 Introduction
    2. 2.2 Notation and Background
    3. 2.3 ObsAtk for Robustness Evaluation
      1. 2.3.1 Observation-based Zero-mean Attack
      2. 2.3.2 Robustness Evaluation via ObsAtk
    4. 2.4 Robust and Generalizable Denoising via HAT
      1. 2.4.1 Hybrid Adversarial Training
      2. 2.4.2 Robustness Enhancement via HAT
      3. 2.4.3 Robustness Benefits Generalization to Unseen Noise
    5. 2.5 Chapter Summary
    6. 2.6 Appendices
      1. 2.6.1 Two-step Projection
      2. 2.6.2 Experiments of Robustness Enhancement on Set12 and Kodak24
      3. 2.6.3 Ablation study
      4. 2.6.4 Visual results of real-world noise removal
  5. 3 On Robustness of Neural Ordinary Differential Equations
    1. 3.1 Introduction
    2. 3.2 Preliminaries on neural ODE
    3. 3.3 An empirical study on the robustness of ODEnets
      1. 3.3.1 Experimental settings
      2. 3.3.2 Robustness of ODEnets trained only on non-perturbed images
      3. 3.3.3 Robustness of ODEnets trained on original images together with Gaussian perturbations
      4. 3.3.4 Insights on the robustness of ODEnets
    4. 3.4 TisODE: boosting the robustness of neural ODEs
      1. 3.4.1 Time-invariant steady neural ODEs
      2. 3.4.2 Evaluating robustness of TisODE-based classifiers
      3. 3.4.3 TisODE - A generally applicable drop-in technique for improving the robustness of deep networks
    5. 3.5 Related works
    6. 3.6 Chapter Summary
    7. 3.7 Appendices
      1. 3.7.1 Networks used on the MNIST, the SVHN, and the ImgNet10 datasets
      2. 3.7.2 The Construction of ImgNet10 dataset
      3. 3.7.3 Gronwall’s Inequality
      4. 3.7.4 More experimental results
  6. 4 Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection
    1. 4.1 Introduction
    2. 4.2 Related Works
    3. 4.3 Channel-wise Importance-based Feature Selection
      1. 4.3.1 Non-robust CNNs vs. Robustified CNNs: A Channel-wise Activation Perspective
      2. 4.3.2 Importance-based Channel Adjustment
      3. 4.3.3 Verification of Hypothesis $\mathcal{H}$ on Robustness Enhancement
    4. 4.4 Experiments
      1. 4.4.1 Robustness Evaluation
      2. 4.4.2 Ablation Study
    5. 4.5 Chapter Summary
    6. 4.6 Appendices
      1. 4.6.1 Details on Visualizing Channel-wise Activations
      2. 4.6.2 Robustness Evaluation on CIFAR10
      3. 4.6.3 Robustness Evaluation on SVHN
      4. 4.6.4 More Results on FMNIST
      5. 4.6.5 More Results on Ablation Study
  7. 5 Conclusion
    1. 5.1 Summary of Contributions
    2. 5.2 Recommendations for Future Research

##

Towards Adversarial Robustness of Deep Vision Algorithms

Hanshu Yan ††Thesis Supervisor: Associate Professor Vincent Y. F. Tan

Submitted to the Department of Electrical and Computer Engineering in May
2022, in partial fulfillment of the requirements for the degree of Doctor of
Philosophy

Summary

Deep learning methods have achieved great successes in solving computer vision
tasks and they have been widely utilized in artificially intelligent systems
for image processing, analysis, and understanding. However, deep neural
networks (DNNs) have been shown to be vulnerable to adversarial perturbations
in input data. The security issues of deep neural networks have thus come to
the fore. It is imperative to comprehensively study the adversarial robustness
of deep vision algorithms. This thesis focuses on the robustness of deep
classification models and deep image denoisers.

For image denoising, we systematically investigate the robustness of deep
image denoisers. Specifically, we propose a novel attack method, observation-
based zero-mean attack (ObsAtk), that considers the zero-mean assumption of
natural noise, to generate adversarial perturbations of noisy input images. We
develop an effective and theoretically-grounded PGD-based optimization
technique to implement ObsAtk. With ObsAtk, we propose the hybrid
adversarially training (HAT) to enhance the robustness of deep image
denoisers. Extensive experiments demonstrate the effectiveness of HAT.
Furthermore, we explore the connection between the adversarial robustness of
denoisers and the adaptivity to unseen types of real-world noise. We find that
deep denoisers that are HAT-trained only with synthetic noisy data can
generalize well to unseen types of noise. The noise removal capabilities are
even comparable to the denoisers trained with true real-world noise.

For image classification, we explore novel robust architectures other than
vanilla convolutional neural networks (CNNs). First, we study the robustness
of neural ordinary differential equations (NODE). We empirically demonstrate
that NODE-based classifiers show superior robustness against input
perturbations in comparison to CNN-based classifiers. To further boost the
robustness of NODE-based models, we introduce the time-invariant property to
NODEs and impose a steady-state constraint to regularize the flow of ODEs on
perturbed data. We demonstrate that the resultant model, dubbed as time-
invariant steady neural ODE (TisODE), is more robust than the vanilla NODE.

Second, we also investigate the robustness of vanilla CNNs from the
perspective of channel-wise activations and propose a feature selection
mechanism to enhance the robustness of vanilla CNNs. In particular, we compare
the channel-wise activations of normally-trained classifiers when dealing with
natural and adversarial data. We observe that adversarial data mislead the
deep classifiers by over-activating negatively-relevant (NR) channels but
under-activating positively-relevant (PR) ones. We also compare the channel-
wise activations of normally-trained models to the adversarially-trained ones
and observe that adversarial training robustifies models by promoting the
under-activated PR channels and suppressing the over-activated NR ones. Thus,
we hypothesize that scaling the activations of channels corresponding to their
relevances to the true categories improves the robustness. To examine this
hypothesis, we develop a novel channel manipulation technique, namely channel-
wise importance-based feature selection (CIFS), that can scale channels’
activations by generating non-negative multipliers based on their relevances.
Extensive experimental results verify the hypothesis and the superior
robustness of CIFS-modified CNNs.

In summary, this thesis systematically studies the robustness of deep vision
algorithms, including robustness evaluation (ObsAtk), robustness improvement
(HAT, TisODE, and CIFS), and the connection between adversarial robustness and
generalization capability to new domains.

###### List of Tables

  1. List of Symbols
  2. 2.1 The average PSNR (in dB) results of DnCNN denoisers on the gray-scale and RGB datasets.
  3. 2.2 Robustness comparison between normally and adversarially-trained DnCNN-B denoisers on the gray-scale Set68 dataset.
  4. 2.3 Robustness comparison between normally and adversarially-trained DnCNN-B denoisers on the RGB BSD68 dataset.
  5. 2.4 Comparison of different methods for denoising real-world noisy images in terms of PSNR (dB).
  6. 2.5 The average PSNR (in dB) results of DnCNN-B denoisers on the gray-scale Set12 dataset.
  7. 2.6 The average PSNR (in dB) results of DnCNN-C denoisers on the RGB Kodak24 dataset.
  8. 3.1 Robustness comparison between CNNs and ODEnets.
  9. 3.2 Robustness comparison of different models trained with Gaussian noisy images.
  10. 3.3 Robustness improvement via TisODE.
  11. 3.4 Robustness improvement via TisODE in conjunction with other techniques.
  12. 3.5 The architectures of the ODEnets on different datasets.
  13. 3.6 The corresponding indexes to each class in the original ImageNet dataset
  14. 3.7 Robustness comparison of different models under adversarial training.
  15. 3.8 Robustness improvement on CIFAR10.
  16. 3.9 The architecture of the ODEnet on CIFAR10.
  17. 4.1 Robustness comparison of defense methods on CIFAR10.
  18. 4.2 Robustness comparison of defense methods on SVHN.
  19. 4.3 Robustness comparison of defense methods trained with FAT on CIFAR10.
  20. 4.4 Top-$k$ accuracies (%) against adversarial attacks on CIFAR10 of an adversarially trained ResNet-18.
  21. 4.5 Robustness comparison (%) of importance assessment based on top-$k$ results.
  22. 4.6 Robustness comparison (AutoAttack) of defense methods on CIFAR10.
  23. 4.7 Robustness comparison (best-epoch) of defense methods on CIFAR10
  24. 4.8 Robust accuracies (%) for values of $\beta_{\text{atk}}$ on CIFAR10.
  25. 4.9 Robustness comparison of vanilla CNNs and their CIFS-modified version under various AT-based strategies.
  26. 4.10 Robustness comparison (AutoAttack) of defense methods on SVHN.
  27. 4.11 Robustness comparison (best-epoch) of defense methods on SVHN.
  28. 4.12 Robustness comparison of defense methods on FMNIST.
  29. 4.13 Robustness accuracies (%) on CIFAR10 for CIFS with various values of $\beta$.
  30. 4.14 Robustness (%) comparison of the positions where CIFS modules are placed.
  31. 4.15 Robustness comparison (%) of the probe architectures in CIFS modules (at P1-P2).

###### List of Figures

  1. 2.1 Illustration of ObsAtk.
  2. 2.2 Vulnerability of normally-trained deep denoisers under adversarial attacks.
  3. (a)  ${\bm{x}}$
  4. (b)  ${\bm{v}}$
  5. (c)  ${\bm{y}}$
  6. (d)  $f^{\gamma}_{\theta}({\bm{y}})$
  7. (e)  ${\bm{v}}+\bm{\delta}$
  8. (f)  ${\bm{y}}+\bm{\delta}$
  9. (g)  $f^{\gamma}_{\theta}({\bm{y}}+\bm{\delta})$
  10. 2.3 Robustness comparison between normally and HAT-trained denoisers.
  11. (a)  Ground-truth
  12. (b)  NT
  13. (c)  HAT
  14. 2.4 Illustration of Theorem 2.6.1.
  15. 2.5 Ablation study on the effect of $\alpha$ in HAT.
  16. 2.6 Ablation study on the effect of $\epsilon$ in HAT.
  17. 2.7 Comparison of different denoisers for denoising SIDD-val set.
  18. (a) Noisy
  19. (b) BM3D
  20. (c) DIP
  21. (d) N2S
  22. (e) NT
  23. (f) vAT
  24. (g) HAT
  25. (h) G.T.
  26. 3.1 The architecture of an ODEnet.
  27. 3.2 No integral curves intersect.
  28. 3.3 An illustration of the time-invariant property of ODEs.
  29. 4.1 Comparison of channel activations of adversarial data between normally and adversarially trained models.
  30. (a)  Normal / “automobile”
  31. (b)  Normal / “cat”
  32. (c)  Adv. / “automobile”
  33. (d)  Adv. / “cat”
  34. 4.2 Illustration of CIFS.
  35. 4.3 Comparison of channel activations of adversarial data between non-CIFS and CIFS modified models.
  36. (a)  non-CIFS
  37. (b)  CIFS-sigmoid
  38. (c)  CIFS-softplus
  39. (d)  CIFS-softmax
  40. 4.4 Comparison of training dynamics of non-CIFS and CIFS modified ResNet-18 models.
  41. 4.5 More comparison of channel activations of adversarial data between normally and adversarially trained models.
  42. (a)  “ship”: non-CIFS
  43. (b)  “ship”: CIFS-sigmoid
  44. (c)  “ship”: CIFS-softplus
  45. (d)  “ship”: CIFS-softmax
  46. 4.6 More comparison of channel activations of adversarial data between normally and adversarially trained models.
  47. (a)  “autmobile”: non-CIFS
  48. (b)  “autmobile”: CIFS-sigmoid
  49. (c)  “autmobile”: CIFS-softplus
  50. (d)  “autmobile”, CIFS-softmax
  51. 4.7 More comparison of channel activations of adversarial data between normally and adversarially trained models.
  52. (a)  “frog”: non-CIFS
  53. (b)  “frog”: CIFS-sigmoid
  54. (c)  “frog”: CIFS-softplus
  55. (d)  “frog”, CIFS-softmax
  56. 4.8 Robust accuracies (%) of PGD-20 adversarial data for various classes of the CIFAR10 dataset.

## List of Symbols

Numbers and Arrays
---
$\displaystyle a$ |  A scalar (integer or real)
$\displaystyle{\bm{a}}$ |  A vector
$\displaystyle{\bm{A}}$ |  A matrix
$\displaystyle{\bm{I}}_{n}$ |  Identity matrix with $n$ rows and $n$ columns
$\displaystyle{\bm{I}}$ |  Identity matrix with dimensionality implied by context
$\displaystyle\text{diag}({\bm{a}})$ |  A square, diagonal matrix with diagonal given by ${\bm{a}}$
a |  A scalar random variable
$\displaystyle{\mathbf{a}}$ |  A vector-valued random variable
$\displaystyle{\mathbf{A}}$ |  A matrix-valued random variable
Sets
$\displaystyle{\mathbb{X}}$ |  A set
$\displaystyle\mathbb{R}$ |  The set of real numbers
$\displaystyle\\{0,1\\}$ |  The set containing 0 and 1
$\displaystyle\\{0,1,\dots,n\\}$ |  The set of all integers between $0$ and $n$
$\displaystyle[a,b]$ |  The real interval including $a$ and $b$
$\displaystyle(a,b]$ |  The real interval excluding $a$ but including $b$
Calculus
$\displaystyle\frac{dy}{dx}$ |  Derivative of $y$ with respect to $x$
$\displaystyle\frac{\partial y}{\partial x}$ |  Partial derivative of $y$ with respect to $x$
$\displaystyle\nabla_{\bm{x}}y$ |  Gradient of $y$ with respect to ${\bm{x}}$
$\displaystyle\nabla_{\bm{X}}y$ |  Matrix derivatives of $y$ with respect to ${\bm{X}}$
$\displaystyle\frac{\partial f}{\partial{\bm{x}}}$ |  Jacobian matrix ${\bm{J}}\in\mathbb{R}^{m\times n}$ of $f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$
$\displaystyle\nabla_{\bm{x}}^{2}f({\bm{x}})\text{ or }{\bm{H}}(f)({\bm{x}})$ |  The Hessian matrix of $f$ at input point ${\bm{x}}$
$\displaystyle\int f({\bm{x}})d{\bm{x}}$ |  Definite integral over the entire domain of ${\bm{x}}$
$\displaystyle\int_{\mathbb{S}}f({\bm{x}})d{\bm{x}}$ |  Definite integral with respect to ${\bm{x}}$ over the set ${\mathbb{S}}$
Probability and Information Theory
$\displaystyle{\textnormal{a}}\sim P$ |  Random variable a has distribution $P$
$\displaystyle\mathbb{E}_{{\textnormal{x}}\sim P}[f(x)]\text{ or }\mathbb{E}f(x)$ |  Expectation of $f(x)$ with respect to $P({\textnormal{x}})$
$\displaystyle\mathrm{Var}(f(x))$ |  Variance of $f(x)$ under $P({\textnormal{x}})$
$\displaystyle\mathrm{Cov}(f(x),g(x))$ |  Covariance of $f(x)$ and $g(x)$ under $P({\textnormal{x}})$
$\displaystyle H({\textnormal{x}})$ |  Shannon entropy of the random variable x
$\displaystyle D_{\mathrm{KL}}(P\|Q)$ |  Kullback–Leibler divergence of $P$ and $Q$
$\displaystyle\mathcal{N}({\bm{x}};{\bm{\mu}},{\bm{\Sigma}})$ |  Gaussian distribution over ${\bm{x}}$ with mean ${\bm{\mu}}$ and covariance ${\bm{\Sigma}}$
Functions
$\displaystyle f:{\mathbb{A}}\rightarrow{\mathbb{B}}$ |  The function $f$ with domain ${\mathbb{A}}$ and range ${\mathbb{B}}$
$\displaystyle f\circ g$ |  Composition of the functions $f$ and $g$
$\displaystyle f({\bm{x}};{\bm{\theta}})$ |  A function of ${\bm{x}}$ parametrized by ${\bm{\theta}}$. (Sometimes we write $f({\bm{x}})$ and omit the argument ${\bm{\theta}}$ to lighten notation)
$\displaystyle\log x$ |  Natural logarithm of $x$
$\displaystyle\sigma(x)$ |  Logistic sigmoid, $\displaystyle\frac{1}{1+\exp(-x)}$
$\displaystyle\zeta(x)$ |  Softplus, $\log(1+\exp(x))$
$\displaystyle||{\bm{x}}||_{p}$ |  $\ell_{p}$ norm of ${\bm{x}}$
$\displaystyle\mathbbm{1}_{\mathrm{condition}}$ |  is 1 if the condition is true, 0 otherwise

##  Chapter 1 Introduction

Intelligent systems have revolutionized various domains, including industry,
research, and our daily lives [lecun2015deep; bedi2019deep; veres2019deep;
wang2019review]. For example, medical diagnostic systems facilitate health
screening and assist doctors to diagnose diseases [shen2017deep]; unmanned
underwater vehicles broaden the horizons of scientists and automatically
perform subsea operations and sample collections [moniruzzaman2017deep];
smartphones have reinvented daily lives and made it easy to exchange
information and connect with people [abdelhamed_high-quality_2018;
ma2019image]. Behind the success of intelligent systems, computer vision and
other technology play vital roles in processing and analyzing information.
Computer vision aims to design algorithms that can enhance and comprehend
digital images effectively and efficiently [krizhevsky2012imagenet;
he2017mask; he_deep_2016]. For example, denoising and deblurring algorithms
can improve the visual quality of medical images that are degraded by unknown
noise and device movement [yan2019unsupervised; amini2021medical];
classification algorithms help to recognize the categories of samples of
marine life [rathi2017underwater]; generative vision models are able to
synthesize fancy objects that do not yet exist in the real world, to create
engaging virtual environments [goodfellow2014generative; kingma2013auto].

In recent years, deep learning has achieved great successes in computer vision
tasks [he_deep_2016; zhao2019object; minaee2021image]. For example, deep
classification networks have accomplished astonishing performances of real-
world image recognition on several benchmark datasets [liu2021swin]; the
performances are comparable to human-level recognition capability. Deep image
denoising models can remove intense RGB noise and reconstruct high-quality
images from extremely low-light shots [zhang_beyond_2017; pang_recorrupted--
recorrupted_2021]. Due to these impressive results, deep learning-based
algorithms have been widely applied to real-world applications, such as
digital cameras [liang2021cameranet], medical imaging systems [shen2017deep],
facial recognition systems [wang2021deep], and autonomous vehicles
[grigorescu2020survey].

Although deep learning algorithms perform very well on various benchmark
datasets, they have been shown to be vulnerable and unstable when confronted
with uncertainties in data. For example, Gaussian noise corruption in input
images [hendrycks2018benchmarking] degrades the classification performance.
Furthermore, Goodfellow [goodfellow_explaining_2015] showed that well-crafted
imperceptible adversarial perturbations in input data can completely mislead a
well-trained classification model and result in unreasonable predictions.
Various sources of uncertainty are ubiquitous and inevitable in real-world
applications, including inherent noise and observation errors. In high-stake
applications like fraud detection [raghavan2019fraud], medical diagnosis
[shen2017deep], and natural disaster prediction [kang2016deep], such high
sensitivity to perturbations will lead to unacceptable economic losses and
other catastrophic consequences. To build reliable and trustworthy intelligent
systems, it is imperative to systematically study the robustness of deep
vision algorithms.

This thesis focuses on the problem of evaluating and enhancing the robustness
of deep vision algorithms against perturbations of input data. In particular,
to examine the sensitivity of deep models against any arbitrary type of input
noise, we develop adversarial attack methods to compute the worst-case
perturbation with a certain limited budget. The resistance against adversarial
perturbations is termed adversarial robustness. Besides, to enhance the
adversarial robustness of deep models, we devise solutions from the
perspectives of network architectures, learning objectives, and training
frameworks. Finally, we also explore the connection between adversarial
robustness and generalization to new domains and empirically demonstrate that
adversarial robustness benefits the models in terms of the ability to adapt to
unseen types of data.

###  1.1 Background

Noise in observed signals is ubiquitous in real-world applications due to
intrinsic noise, measuring errors, numerical errors induced in signal
processing, etc. Reliable intelligent systems ought to exclude the
interference of noise and extract salient task-related information from the
noisy observations. To wit, they should not only effectively process and
analyze the authentic ground truth but also maintain acceptable performances
on corrupted data. The first step toward robust systems is developing methods
to evaluate the robustness of deep models. In Section 1.1.1, we introduce
existing works that proposed attack methods to craft adversarial noise for the
robustness evaluation. We also discuss several metrics for measuring the
robustness level by using the attack methods. Based on these robustness
metrics, we then introduce several state-of-the-art techniques for robustness
improvement in Section 1.1.2.

####  1.1.1 Adversarial Robustness

Consider a deep neural network (DNN) $\\{f_{\theta}:\theta\in\Theta\\}$
mapping an input ${\bm{x}}$ to a target ${\bm{y}}$. The model is trained to
minimize a certain loss function that is measured by particular distance
measure $d(\cdot,\cdot)$ between the output $f_{\theta}({\bm{x}})$ and the
target ${\bm{y}}$. In critical applications, the DNN should resist small
perturbations on the input data and map the perturbed input to a result that
is close to the target. The notion of robustness has been proposed to measure
the resistance of DNNs against slight changes of the input
[szegedy_intriguing_2014; goodfellow_explaining_2015]. The robustness is
characterized by the distance $d(f_{\theta}({\bm{x}}^{\prime}),{\bm{y}})$
between the prediction $f_{\theta}({\bm{x}}^{\prime})$ and the target
${\bm{y}}$, where the perturbed input ${\bm{x}}^{\prime}$ is located within a
small $\rho$-neighborhood of the original input ${\bm{x}}$.

###### Adversarial Examples

It is possible to use various types of random perturbations, such as Gaussian
noise and uniform noise, to evaluate robustness. However, we, in reality, are
incognizant of the true statistics of real-world noise. DNNs that are robust
against random Gaussian noise may be very vulnerable to other types of
perturbations. Thus, to comprehensively examine the robustness of deep models,
we consider the worst-case perturbations of input data, namely
${\bm{x}}^{\prime}$ is synthesized by maximizing the distance between its
output and the target ${\bm{y}}$:

${\bm{x}}^{\prime}=\operatorname*{arg\,max}_{{\bm{x}}^{\prime}:\|{\bm{x}}^{\prime}-{\bm{x}}\|\leq\rho}d(f_{\theta}({\bm{x}}^{\prime}),{\bm{y}}).$
(1.1)

The distance $d(f_{\theta}({\bm{y}}^{\prime}),{\bm{x}})$ is an indication of
the robustness of $f_{\theta}$ around ${\bm{y}}$: a small distance implies
strong robustness and vice versa. In terms of image classification, the
$\rho$-neighborhood is usually defined by the $\ell_{\infty}$-norm and the
distance $d(\cdot,\cdot)$ is measured by the cross-entropy loss
[madry_towards_2018] or a margin loss [carlini_towards_2017]. For image
restoration, such as image denoising and super-resolution, the distance
between images is usually measured by the $\ell_{2}$-norm [zhang_beyond_2017].
In addition to the constraint of the limited budget, there may be other
constraints that are used in the optimization procedure and depend on the
forms of the adversarial perturbations. For example, attackers may add the
$\ell_{0}$-norm to restrict the number of pixels that are allowed to be
changed [croce2019sparse]; attachers may multiply the perturbation with a mask
of certain shapes to generate adversarial perturbations with certain semantic
meanings, such as adversarial T-shirts. Anyone wearing such T-shirts can
protect themselves from malicious person detectors [xu2020adversarial].

###### Adversarial Attack Methods

In general, the worst-case perturbation ${\bm{x}}^{\prime}$ can be
approximated via backpropagation-based methods, such as Fast Gradient Signed
Method (FGSM) [goodfellow_explaining_2015], Iterative-FGSM
[kurakin_adversarial_2017], and Projected Gradient Descent (PGD) attack
[madry_towards_2018], which approximately solve equation 1.1 via gradient
descent methods. However, some deep models may involve randomization modules
or non-differentiable operations [xie_mitigating_2018]. In this case,
backpropagation on its own may not be able to approximate the worst-case
perturbations sufficiently accurately. To solve this problem,
athalye_obfuscated_2018 demonstrated that attackers could circumvent the
gradient obfuscation problem through the Expectation over Transformation (EoT)
technique for randomization methods or the Backward Pass Differentiable
Approximate (BPDA) technique for non-differentiable transformations. EoT
updates perturbations with the average gradients that are computed by
performing multiples times of forward and backward passes; BPDA approximates
the gradients of non-differentiable modules by substituting them with
differentiable modules.

Considering the case that attackers are not allowed to access the inner
information of neural networks, researchers have proposed several black-box
and transfer-based attack methods [guo2019simple; liu2016delving]. Black-box
methods usually iteratively update the adversarial perturbations by querying
the target models for a limited number of times; transfer-based methods,
instead of diagnosing the target models, attack substitute models to craft the
adversarial perturbations. In comparison, gradient-based (white-box) attack
methods, if they are allowed to access the gradients of deep models, are
usually much stronger than other types of attacks [tramer_adaptive_2020]
because they are cognizant of the whole information of deep models. To further
boost the strength and the adaptivity of attacks, croce_reliable_2020 proposed
to use an ensemble of four white-box and black-box attacks, namely AutoAttack,
to reliably evaluate the robustness of deep models.

###### Robustness Metrics

Robustness is measured by the performance of deep models on perturbed input
data. The metrics of the robustness of deep models depend on the tasks
performed. In semantic-level classification tasks, we usually use the accuracy
(or $0$-$1$ loss), $\mathbbm{1}{\left[f_{\theta}(x)=y_{j}\right]}$, to measure
the performance. Thus, the robustness of classification models is quantified
by the average accuracy of adversarial examples over a test set
${\mathbb{S}}$:

$\overline{\text{Acc}}=\frac{1}{|{\mathbb{S}}|}\sum_{({\bm{x}},{\bm{y}})\in{\mathbb{S}}}\mathbbm{1}{\left[f_{\theta}({\bm{x}}^{\prime})={\bm{y}}\right]}.$

In pixel-level vision tasks, such as image reconstruction and image
segmentation, each pixel has a certain ground-truth value. The metrics are
usually defined based on the norm of the difference between pixel-value
vectors. For example, in image denoising and deblurring, we use the peak
signal-to-noise ratio (PNSR) to quantify the reconstruction quality, where
PSNR is defined as:

$\text{PSNR}({\bm{x}},{\bm{y}})=10\cdot\log_{10}(\frac{\text{MAX}^{2}}{\text{MSE}}),$

where MAX represents the maximum possible value of pixels (e.g., 255 for 8-bit
unsigned-integer images and 1 for floating-point images) and MSE represents
the mean-squared-loss between vector ${\bm{x}}$ and ${\bm{y}}$. Thus, we
characterize the robustness via the metric of average PSNR over a test set
${\mathbb{S}}$:

$\overline{\text{PSNR}}=\frac{1}{|{\mathbb{S}}|}\sum_{({\bm{x}},{\bm{y}})\in{\mathbb{S}}}\text{PSNR}({\bm{x}}^{\prime},{\bm{y}}).$

####  1.1.2 Robustifying Deep Neural Networks

Deep learning models have been shown to be vulnerable to adversarial attacks
under normal training (NT) [tramer_adaptive_2020; yan_robustness_2020]. For
example, a well-trained image classification model can achieve above 99%
accuracy on authentic images from the MNIST test set, but the accuracy will
drop to around 0% under some strong attacks like multiple-step PGD and C&W
attacks [madry_towards_2018; carlini_towards_2017]. To robustify DNNs,
researchers have explored a variety of techniques from the perspectives of
model training and network architecture design, among others.

###### Robust Training

Robust training methods aim to devise novel training objectives to impose
robustness regularization to the resultant models. The original idea of robust
training is attributed to goodfellow_explaining_2015, where the authors
propose to use the FGSM adversarial examples for model training. However,
models trained with FGSM adversarial examples cannot effectively defend other
attacks like iteratively-FGSM [kurakin_adversarial_2017] and C&W attacks
[carlini_towards_2017]. The formal statement of PGD-adversarial training (AT)
is proposed in madry_towards_2018, where AT is formulated as the following
min-max optimization problem,

$\min_{\theta\in\Theta}\max_{{\bm{x}}^{\prime}:\|{\bm{x}}^{\prime}-{\bm{x}}\|\leq\rho}d(f_{\theta}({\bm{x}}^{\prime}),{\bm{y}}).$
(1.2)

AT aims to guarantee the performance of models on the worst-case perturbed
data. In practice, the PGD-AT method generates adversarial examples on the fly
for each mini-batch via PGD attacks. Then, PGD adversarial examples are used
to calculate training losses for updating model parameters. The effectiveness
of AT has been verified by extensive empirical and theoretical studies
[yan_cifs_2021; gao_convergence_2019]. For further improvement, many variants
of AT have been proposed in terms of its robustness enhancement
[cai_curriculum_2018; wang_improving_2020], generalization to non-adversarial
data [zhang_theoretically_2019; zhang_attacks_2020], and computational
efficiency [shafahi_adversarial_2019; wong_fast_2019]. wang_improving_2020
proposed the misclassification-aware-adversarial-training (MART) to further
robustify DNNs, where MART simultaneously applies the misclassified natural
data and the adversarial data for model training. To mitigate the trade-off
between the different accuracies on adversarial data and authentic non-
perturbed data, zhang2019theoretically proposed the TRADES method that
minimizes the classification loss on authentic data and the local sensitivity
of deep model around training data at the same time. Additionally, due to the
high extra computational overhead incurred by generating strong adversarial
examples, shafahi_adversarial_2019 proposed Free-AT to accelerate AT by
reusing the gradient information computed when updating model parameters. AT
is a generally applicable framework that can be used in different tasks,
including computer vision and natural language processing [zhu2019freelb;
gan2020large], AT-based methods thus have become the default option for
enhancing the robustness of DNNs.

In addition, some other works introduced various types of regularization for
training models, such as layer-wise feature matching
[sankaranarayanan_regularizing_2018; liao_defense_2018;
kannan_adversarial_2018], low-rank representations [sanyal_robustness_2020;
mustafa_adversarial_2019], attention map alignment [xu_interpreting_2019], and
Lipschitz regularity [Virmaux_lipschitz_2018; cisse_parseval_2017]. These
types of regularization can work in conjunction with AT and benefit the
models’ robustness.

###### Robust Architecture Design

Apart from robust training strategies, researchers also have explored robust
network architectures. A line of works propose to utilize pre-processing
modules to remove adversarial noise from images or introduce randomness to
ruin the adversarial patterns [samangouei2018defense; yang2019me;
xie_mitigating_2018], so that adversarial perturbations will not affect the
pre-trained models. Although these types of methods can defend pure gradient
backpropagation attacks, such as I-FGSM, they can be easily circumvented by
adaptive attacks like BPDA or EoT [athalye_obfuscated_2018]. Another line of
works attempt to modify the basic components of DNNs. For example,
guo_sparse_2018 demonstrated that the robustness of DNNs can be improved by
appropriately designed higher model sparsity. xie_feature_2019 performed
feature denoising at the end of convolutional layers to ameliorate the
adversarial effects on feature maps. zoran_towards_2020 used the spatial
attention mechanism to highlight important regions in feature maps. Existing
empirical works mostly focus on the spatial modification of feature maps in
convolutional neural networks (CNNs). The channel-wise selection of
convolutional layers and other families of models, such as neural ordinary
differential equations [chen2018neural], energy-based models [du2019implicit],
and vision transformers [liu2021swin], have been underexplored.

###  1.2 Thesis Focus and Main Contributions

This thesis systematically studies the adversarial robustness of deep vision
algorithms, from the perspective of robustness evaluation, robustness
enhancement, and the connection between robustness and the adaptation to
unseen data. We solve the robustness evaluation and enhancement problems by
developing a novel PGD-based optimization method to perform attacking,
exploring robust network architectures, and effective adversarial training
strategies.

  * •

Since image denoising algorithms have been extensively used to enhance image
quality in computer vision systems, it is necessary to examine the robustness
of deep image noises against adversarial corruptions. We propose the
observation-based zero-mean attack (ObsAtk), which considers the zero-mean
assumption of natural noise, for robustness evaluation. By using ObsAtk, we
develop a novel adversarial training strategy, hybrid adversarial training
(HAT), to enhance the robustness of deep image denoisers. Besides, we reveal
that deep image denoisers that are adversarially trained with only synthetic
data can generalize or adapt well to unseen types of real-world noise. In this
case, one can train an effective denoiser without the need to collect a large
amount of real-world noisy data for training. This work will be presented at
the International Joint Conference on Artificial Intelligence (IJCAI), 2022
[yan2022towards].

  * •

Neural ordinary differential equations (NODE) [chen2018neural], a new family
of models, approximate nonlinear mappings by using continuous-time ODEs. They
have attracted much attention in various research domains recently due to
their desirable properties, such as invertibility and time-continuity. We
study the robustness property of NODEs and found that NODE-based image
classifiers enjoy superior robustness against random noise and adversarial
perturbations in comparison to CNN-based ones. To further boost the robustness
of NODEs, we propose time-invariant steady neural ODE (TisODE), which
regularizes the flow of neural ODEs via the time-invariant property and a
steady-state constraint. We empirically demonstrate that TisODE outperforms
vanilla NODEs and can work in conjunction with other robustification
techniques. This work appeared at the International Conference on Learning
Representations (ICLR), 2020 [yan_robustness_2020].

  * •

In CNN classifiers, channels in deep layers can extract semantic features, and
the final predictions are made by aggregating information from various
channels. In this case, abnormal activations of channels in deep layers may
result in incorrect predictions. To understand the effects of adversarial data
on each channel, we compare the activations of channels caused by authentic
and adversarial data. We find that natural data tend to over-activate
positively-relevant (PR) channels and under-activate negatively-relevant (NR)
channels, but adversarial data do the opposite. Here, the relevance of each
channel is defined as the gradient of the prediction logit associated with the
true label with respect to the activation level. To investigate the effects of
adversarial training on channels, we also compare the channel activations
between normally-trained and adversarially-trained models. We observe that AT
robustifies CNNs by aligning the channel-wise activations of adversarial data
with those of their natural counterparts. Given these observations, we
hypothesize that suppressing NR channels and promoting PR ones based on their
relevances can further enhance the robustness of CNNs. To examine this
hypothesis, we propose the Channel-wise Importance-based Feature Selection
(CIFS). The CIFS mechanism manipulates channels’ activations of certain layers
by generating non-negative multipliers to these channels based on their
relevances. Extensive experiments on benchmark datasets clearly verify the
hypothesis and CIFS’s effectiveness in robustifying CNNs. This work appeared
at the International Conference on Machine Learning (ICML), 2021
[yan_cifs_2021].

###  1.3 Organization of Thesis

In Chapter 2, we propose the ObsAtk to evaluate the robustness of deep image
denoisers and develop the HAT to enhance the robustness of deep denoisers by
using ObsAtk. In Chapter 3, we demonstrate that NODE-based classifiers are
more robust against input perturbations in comparison to CNN-based ones and
propose TisODE to further boost the robustness of NODEs. In Chapter 4, we
propose the CIFS mechanism to modify CNN-based classifiers and show that CIFS
clearly robustifies CNNs against adversarial corruptions. Finally, Chapter 5
concludes this thesis and suggests several promising topics for future
research.

##  Chapter 2 Towards Adversarially Robust Deep Image Denoising

This work systematically investigates the adversarial robustness of deep image
denoisers (DIDs), i.e, how well DIDs can recover the ground truth from noisy
observations degraded by adversarial perturbations. Firstly, to evaluate DIDs’
robustness, we propose a novel adversarial attack, namely Observation-based
Zero-mean Attack (ObsAtk), to craft adversarial zero-mean perturbations on
given noisy images. We find that existing DIDs are vulnerable to the
adversarial noise generated by ObsAtk. Secondly, to robustify DIDs, we propose
an adversarial training strategy, hybrid adversarial training (HAT), that
jointly trains DIDs with adversarial and non-adversarial noisy data to ensure
that the reconstruction quality is high and the denoisers around non-
adversarial data are locally smooth. The resultant DIDs can effectively remove
various types of synthetic and adversarial noise. We also uncover that the
robustness of DIDs benefits their generalization capability on unseen real-
world noise. Indeed, HAT-trained DIDs can recover high-quality clean images
from real-world noise even without training on real noisy data. Extensive
experiments on benchmark datasets, including Set68, PolyU, and SIDD,
corroborate the effectiveness of ObsAtk and HAT.

###  2.1 Introduction

Image denoising, which aims to reconstruct clean images from their noisy
observations, is a vital part of the image processing systems. The noisy
observations are usually modeled as the addition between ground-truth images
and zero-mean noise maps [dabov_image_2007; zhang_beyond_2017]. Recently, deep
learning-based methods have made significant advancements in denoising tasks
[zhang_beyond_2017; anwar_real_2019] and have been applied in many areas
including medical imaging [gondara_medical_2016] and photography
[abdelhamed_high-quality_2018]. Despite the success of deep denoisers in
recovering high-quality images from a certain type of noisy images, we still
lack knowledge about their robustness against adversarial perturbations, which
may cause severe safety hazards in high-stake applications like medical
diagnosis. To address this problem, the first step should be developing attack
methods dedicated for denoising to evaluate the robustness of denoisers. In
contrast to the attacks for classification [goodfellow_explaining_2015;
madry_towards_2018], attacks for denoising should consider not only the
adversarial budget but also some assumptions of natural noise, such as zero-
mean, because certain perturbations, such as adding a constant value, do not
necessarily result in visual degradation. Although choi_deep_2021;
choi_evaluating_2019 studied the vulnerability for various deep image
processing models, they directly applied the attack from classification. To
the best of our knowledge, no attacks are truly dedicated for the denoising
task till now.

To this end, we propose the observation-based zero-mean attack (ObsAtk), which
crafts a worst-case zero-mean perturbation for a noisy observation by
maximizing the distance between the output and the ground-truth. To ensure
that the perturbation satisfies the adversarial budget and the zero-mean
constraints, we utilize the classical projected-gradient-descent (PGD)
[madry_towards_2018] method for optimization, and develop a two-step operation
to project the perturbation back into the feasible region. Specifically, in
each iteration, we first project the perturbation onto the zero-mean
hyperplane. Then, we linearly rescale the perturbation to adjust its norm to
be less or equal to the adversarial budget. We examine the effectiveness of
ObsAtk on several benchmark datasets and find that deep image denoisers are
indeed susceptible to ObsAtk: the denoisers cannot remove adversarial noise
completely and even yield atypical artifacts, as shown in Figure 2.2(g).

To robustify deep denoisers against adversarial perturbations, we propose an
effective adversarial training strategy, namely hybrid adversarial training
(HAT), to train denoisers by using adversarially noisy images and non-
adversarial noisy images together. The loss function of HAT consists of two
terms. The first term ensures the reconstruction performance from common non-
adversarial noisy images, and the second term ensures the reconstructions
between non-adversarial and adversarial images to be close to each other.
Thus, we can obtain denoisers that perform well on both non-adversarial noisy
images and their adversarial perturbed versions. Extensive experiments on
benchmark datasets verify the effectiveness of HAT.

Moreover, we reveal that adversarial robustness benefits the generalization
capability to unseen types of noise, i.e., HAT can train denoisers for real-
world noise removal only with synthetic noise sampled from common
distributions like Gaussians. That is because ObsAtk searches for the worst-
case perturbations around different levels of noisy images, and training with
adversarial data ensures the denoising performance on various types of noise.
In contrast, other reasonable methods for real-world denoising
[guo_toward_2019; lehtinen_noise2noise_2018] mostly require a large number of
real-world noisy data for the training, which are unfortunately not available
in some applications like medical radiology. We conduct experiments on several
real-world datasets. Numerical and visual results demonstrate the
effectiveness of HAT for real-world noise removal.

In summary, there are three main contributions in this work: 1) We propose a
novel attack, ObsAtk, to generate adversarial examples for noisy observations,
which facilitates the evaluation of the robustness of deep image denoisers. 2)
We propose an effective adversarial training strategy, HAT, for robustifying
deep image denoisers. 3) We build a connection between adversarial robustness
and the generalization to unseen noise, and show that HAT serves as a
promising framework for training generalizable deep image denoisers.

###  2.2 Notation and Background

###### Deep image denoising

During image capturing, unknown types of noise may be induced by physical
sensors, data compression, and transmission. Noisy observations are usually
modeled as the addition between the ground-truth images and certain zero-mean
noise [dabov_image_2007; zhang_poisson_gaussian_2019], i.e.,
${\mathbf{y}}={\mathbf{x}}+{\mathbf{v}}$ with
$\mathbb{E}_{Q}\big{[}\sum_{i=1}^{m}{\mathbf{v}}_{[i]}\big{]}=0$, where
${\mathbf{v}}_{[i]}$ is the $i^{\text{th}}$ element of ${\mathbf{v}}$. The
random vector ${\mathbf{x}}\in{\mathbb{R}}^{m}$ with distribution $P$ denotes
a random clean image and the noise ${\mathbf{v}}\in{\mathbb{R}}^{m}$ with a
distribution $Q$ satisfies the zero-mean constraint. Denoising techniques aim
to recover clean images from their noisy observations [zhang_beyond_2017;
dabov_image_2007]. Suppose we are given a training set
${\mathbb{S}}=\\{({\bm{y}}_{j},{\bm{x}}_{j})\\}^{N}_{j=1}$ of noisy and clean
image pairs sampled from distributions $Q$ and $P$ respectively, we can train
a DNN to effectively remove the noise induced by distribution $Q$ from the
noisy observations. A series of DNNs have been developed for denoising in
recent years, including DnCNN [zhang_beyond_2017], FFDNet [zhang_ffdnet_2018],
and RIDNet [anwar_real_2019].

In real-world applications [abdelhamed_high-quality_2018; xu_multi-
channel_2017], the noise distribution $Q$ is usually unknown due to the
complexity of the image capturing procedures; besides, collecting a large
number of image pairs (clean/noisy or noisy/noisy) for training sometimes may
be unrealistic in safety-critical domains such as medical radiology
[zhang_poisson_gaussian_2019]. To overcome these, researchers developed
denoising techniques by approximating real noise with common distributions
like Gaussian or Poisson [dabov_image_2007; zhang_poisson_gaussian_2019]. To
train denoisers that can deal with different levels of noise, where the noise
level is measured by the energy-density $\|{\bm{v}}\|^{2}_{2}/m$ of noise, the
training set may consist of noisy images sampled from a variety of noise
distributions [zhang_beyond_2017], whose expected energy-densities range from
zero to certain budget $\gamma^{2}$ (the expected $\ell_{2}$-norms range from
zero to $\gamma\sqrt{m}$). For example,
${\mathbb{S}}^{\gamma}=\\{({\bm{y}}_{j},{\bm{x}}_{j})\\}^{N}_{j=1}$ where
${\bm{y}}_{j}={\bm{x}}_{j}+{\bm{v}}_{j}$ and ${\bm{x}}_{j}$ and ${\bm{v}}_{j}$
are sampled from $P$ and $Q$ respectively and where $Q$ is randomly selected
from a set of Gaussian distributions
${\mathbb{Q}}^{\gamma}=\\{\mathcal{N}(\bm{0},\sigma^{2}\mathbf{I})|\sigma\in[0,\gamma]\\}$.
The denoiser $f_{\theta}^{\gamma}(\cdot)$ trained with ${\mathbb{S}}^{\gamma}$
is termed as an $\gamma$-denoiser.

###### On robustness of deep image denoisers

In practice, data storage and transmission may induce imperceptible
perturbations on the original data so that the perturbed noise may be
statistically slightly different from the noise sampled from the specific
original distribution. Although an $\gamma$-denoiser can successfully remove
noise sampled from $Q\in{\mathbb{Q}}^{\gamma}$, the performance of noise
removal on the perturbed data is not guaranteed. Thus, we propose a novel
attack method, ObsAtk, to assess the adversarial robustness of DIDs in Section
2.3. To robustify DIDs, we propose an adversarial training strategy, HAT, in
Section 2.4. HAT-trained DIDs can effectively denoise adversarial perturbed
noisy images and preserve good performance on non-adversarial data.

Besides the adversarial robustness issue, it has been shown that
$\gamma$-denoisers trained with ${\mathbb{S}}^{\gamma}$ cannot generalize well
to unseen real-world noise [lehtinen_noise2noise_2018;
batson_noise2self_2019]. Several methods have been proposed for real-world
noise removal, but most of them require a large number of real noisy data for
training, e.g., CBDNet (clean/noisy pairs) [guo_toward_2019] and Noise2Noise
(noisy pairs) [lehtinen_noise2noise_2018], which is sometimes impractical. In
Section 2.4.3, we show that HAT-trained DIDs can generalize well to unseen
real noise without the need of utilizing real noisy images for training.

###  2.3 ObsAtk for Robustness Evaluation

In this section, we propose a novel adversarial attack, Observation-based
Zero-mean Attack (ObsAtk), to evaluate the robustness of DIDs. We also conduct
experiments on benchmark datasets to demonstrate that normally-trained DIDs
are vulnerable to adversarial perturbations.

####  2.3.1 Observation-based Zero-mean Attack

Fig. 2.1: Illustration of ObsAtk. Left: We perturb a noisy observation
${\bm{y}}$ of the ground-truth ${\bm{x}}$ with an adversarial budget
$\epsilon$ in the $\ell_{2}$-norm. For an $\gamma$-denoiser, we choose a
proper value of $\epsilon$ to ensure the norm of the total noise is bounded by
$\gamma\sqrt{m}$, where $m$ denotes the image size. Right: The perturbation
$\bm{\delta}$ is projected via the two-step operation onto the region defined
by the zero-mean and $\epsilon$-ball constraints.

An $\gamma$-denoiser $f^{\gamma}_{\theta}(\cdot)$ can generate a high-quality
reconstruction $f^{\gamma}_{\theta}({\bm{y}})$ close to the ground-truth
${\bm{x}}$ from a noisy observation ${\bm{y}}={\bm{x}}+{\bm{v}}$. To evaluate
the robustness of $f^{\gamma}_{\theta}(\cdot)$ with respect to a perturbation
on ${\bm{y}}$, we develop an attack to search for the worst perturbation
$\bm{\delta}^{*}$ that degrades the recovered image
$f^{\gamma}_{\theta}({\bm{y}}+\bm{\delta}^{*})$ as much as possible. Formally,
we need to solve the problem stated in equation 2.1. The optimization problem
is subject to two constraints: The first constraint requires the norm of
$\bm{\delta}$ to be bounded by a small adversarial budget $\epsilon$. The
second constraint restricts the mean $M(\bm{\delta})$ of all elements in
$\bm{\delta}$ to be zero. This corresponds to the zero-mean assumption of
noise in real-world applications because a small mean-shift does not
necessarily result in visual noise. For example, a mean-shift in gray-scale
images implies a slight change of brightness. Since the zero-mean perturbation
is added to a noisy observation ${\bm{y}}$, we term the proposed attack as
Observation-based Zero-mean Attack (ObsAtk).

$\displaystyle\bm{\delta}^{*}$
$\displaystyle=\operatorname*{arg\,max}_{\bm{\delta}\in{\mathbb{R}}^{m}}\|f_{\theta}^{\gamma}({\bm{y}}+\bm{\delta})-{\bm{x}}\|_{2}^{2},$
(2.1a) s.t.
$\displaystyle\|\bm{\delta}\|_{2}\leq\epsilon,\;\;\text{~{}}\;\;M(\bm{\delta})=\frac{1}{m}\sum_{i=1}^{m}\bm{\delta}_{[i]}=0.$
(2.1b)

We solve the constrained optimization problem Eq. equation 2.1 by using the
classical projected-gradient-descent (PGD) method. PGD-like methods update
optimization variables iteratively via gradient descent and ensure the
constraints to be satisfied by projecting parameters back to the feasible
region at the end of each iteration. To deal with the $\ell_{2}$-norm and
zero-mean constraints, we develop a two-step operation in Eq. equation 2.2,
that first projects the perturbation $\bm{\delta}$ back to the zero-mean
hyperplane and then projects the result onto the $\epsilon$-neighborhood.

$\displaystyle\bm{\delta}^{\prime}$
$\displaystyle=\bm{\delta}-\frac{\bm{\delta}^{\top}{\bm{n}}}{\|{{\bm{n}}}\|^{2}_{2}}{{\bm{n}}},\quad\mbox{where}\quad{\bm{n}}=[1,1,\ldots,1]^{\top},$
(2.2a) $\displaystyle\bm{\delta}^{\prime\prime}$
$\displaystyle=\min\Big{(}\frac{\epsilon}{\|\bm{\delta}^{\prime}\|_{2}},1\Big{)}\text{~{}}\bm{\delta}^{\prime}.$
(2.2b)

In each iteration, as shown in Figure 2.1, the first step involves projecting
the perturbation $\bm{\delta}$ onto the zero-mean hyperplane. The zero-mean
hyperplane consists of all the vectors ${\bm{z}}$ whose mean of all elements
equals zero, i.e., ${\bm{n}}^{\top}{\bm{z}}=0$, where ${\bm{n}}$ is the
length-$d$ all ones vector. Thus, ${\bm{n}}$ is a normal of the zero-mean
plane. We can project any vector onto the zero-mean plane via equation 2.2a.
The vector $\bm{\delta}$ is first projected along the direction of ${\bm{n}}$,
then its projection $\bm{\delta}^{\prime}$ onto the zero-mean plane equals
itself minus its projection onto ${\bm{n}}$. The second step involves further
projecting $\bm{\delta}^{\prime}$ back to the $\epsilon$-ball via linear
scaling. If $\bm{\delta}^{\prime}$ is already within the $\epsilon$-ball, we
keep $\bm{\delta}^{\prime}$ unchanged. Otherwise, the final projection
$\bm{\delta}^{\prime\prime}$ is obtained by scaling $\bm{\delta}^{\prime}$
with a factor ${\epsilon}/{\|\bm{\delta}^{\prime}\|_{2}}$. For any two sets
$A$ and $B$, although the projection onto $A\cap B$ is, in general, not equal
to the result obtained by first projecting onto $A$, then onto $B$,
surprisingly, the following holds for the two sets in equation 2.1b.

Algorithm 1 ObsAtk

1:Denoiser $f_{\theta}(\cdot)$, ground-truth ${\bm{x}}$, noisy observation
${\bm{y}}$, adversarial budget $\epsilon$, #iterations $T$, step-size $\eta$,
minimum pixel value $p_{\text{min}}$, maximum pixel value $p_{\text{max}}$

2:Adversarial perturbation $\bm{\delta}$

3:$\bm{\delta}\leftarrow\mathbf{0}$

4:for $t=1$ to $T$ do

5:
$\bm{\delta}\leftarrow\bm{\delta}+\eta\nabla_{\bm{\delta}}\|f_{\theta}^{\gamma}({\bm{y}}+\bm{\delta})-{\bm{x}}\|_{2}^{2}$;

6:
$\bm{\delta}\leftarrow\bm{\delta}-(\bm{\delta}^{\top}{\bm{n}}/{\|{{\bm{n}}}\|^{2}_{2}}){{\bm{n}}}$
where ${\bm{n}}$ is in equation 2.2a

7:
$\bm{\delta}\leftarrow\min(\nicefrac{{\epsilon}}{{\|\bm{\delta}\|_{2}}},1)\bm{\delta}$;

8:$\bm{\delta}\leftarrow\text{Clip}({\bm{y}}+\bm{\delta},p_{\text{min}},p_{\text{max}})-{\bm{y}}$

######  Theorem 2.3.1 (Informal)

Given any vector $\bm{\delta}\in{\mathbb{R}}^{m}$, the projection of
$\bm{\delta}$ via the two-step operation in equation 2.2 satisfies the two
constraints in equation 2.1b, and the two-step projection is equivalent to the
exact projection onto the set defined by equation 2.1b.

The formal statement and the proof of Theorem 2.3.1 are provided in Appendix
2.6.1. The complete procedure of ObsAtk is summarized in Algorithm 1.

(a) ${\bm{x}}$

(b) ${\bm{v}}$

(c) ${\bm{y}}$

(d) $f^{\gamma}_{\theta}({\bm{y}})$

(e) ${\bm{v}}+\bm{\delta}$

(f) ${\bm{y}}+\bm{\delta}$

(g) $f^{\gamma}_{\theta}({\bm{y}}+\bm{\delta})$

Fig. 2.2:  Given a normally-trained denoiser $f^{\gamma}_{\theta}(\cdot)$,
from left to right are the ground-truth image ${\bm{x}}$, Gaussian noise
${\bm{v}}$, the Gaussian noisy image ${\bm{y}}={\bm{x}}+{\bm{v}}$, the
reconstruction $f^{\gamma}_{\theta}({\bm{y}})$ from ${\bm{y}}$, adversarial
noise ${\bm{v}}+\bm{\delta}$, the adversarially noisy image
${\bm{y}}+\bm{\delta}$, and the reconstruction
$f^{\gamma}_{\theta}({\bm{y}}+\bm{\delta})$ from ${\bm{y}}+\bm{\delta}$.
Comparing (a), (d) and (g), we observe that $f^{\gamma}_{\theta}(\cdot)$ can
effectively remove Gaussian noise but its performance is degraded when dealing
with the adversarial noise (noise remains on the roof and strange contours
appear in the sky).

####  2.3.2 Robustness Evaluation via ObsAtk

We use ObsAtk to evaluate the adversarial robustness of $\gamma$-denoisers on
several gray-scale and RGB benchmark datasets, including Set12, Set68, BSD68,
and Kodak24. For gray-scale image denoising, we use Train400 to train a
DnCNN-B [zhang_beyond_2017] model, which consists of 20 convolutional layers.
We follow the training setting in Zhang et al.zhang_beyond_2017 and randomly
crop $128\times 3000$ patches in size of $50\times 50$. Noisy and clean image
pairs are constructed by injecting different levels of white Gaussian noise
into clean patches. The noise levels $\sigma$ are uniformly randomly selected
from $[0,\gamma]$ with $\gamma=\nicefrac{{25}}{{255}}$. For RGB image
denoising, we use BSD432 (BSD500 excluding images in BSD68) to train a DnCNN-C
model with the same number of layers as DnCNN-B and but set the input and
output channels to be three. Other settings follow those of the training of
DnCNN-B.

We evaluate the denoising capability of the $\gamma$-denoiser on Gaussian
noisy images and their adversarially perturbed versions. The image quality of
reconstruction is measured via the peak-signal-noise ratio (PSNR) metric. A
large PSNR between reconstruction and ground-truth implies a good performance
of denoising. We denote the energy-density of the noise in test images as
$\hat{\gamma}^{2}$ and consider three levels of noise, i.e.,
$\hat{\gamma}=\nicefrac{{25}}{{255}}$, $\nicefrac{{15}}{{255}}$, and
$\nicefrac{{10}}{{255}}$. For Gaussian noise removal, we add white Gaussian
noise with $\sigma=\hat{\gamma}$ to clean images. For Uniform noise removal,
we generate noise from
$\mathcal{U}(-\sqrt{3}\hat{\gamma},\sqrt{3}\hat{\gamma})$. For denoising
adversarial noisy images, the norm budgets of adversarial perturbation are set
to be $\epsilon=\nicefrac{{5}}{{255}}\cdot\sqrt{m}$ and
$\nicefrac{{7}}{{255}}\cdot\sqrt{m}$ respectively, where $m$ equals the size
of test images. We perturb noisy observations whose noise are generated from
$\mathcal{N}(0,\hat{\gamma}-\nicefrac{{\epsilon}}{{\sqrt{m}}})$, so that the
$\ell_{2}$-norms of total noise in adversarial images are still bounded by
$\hat{\gamma}\cdot\sqrt{m}$ and the energy-density thus are bounded by
$\hat{\gamma}^{2}$ . We use Atk-$\nicefrac{{\epsilon}}{{\sqrt{m}}}$ to denote
the adversarially perturbed noisy images in the size of $m$ with adversarial
budget $\epsilon$. The number of iterations of PGD in ObsAtk is set to be
five.

Table 2.1: The average PSNR (in dB) results of DnCNN denoisers on the gray-
scale and RGB datasets. Four types of noise are used for evaluation, viz.
Gaussian $\mathcal{N}$ and Uniform $\mathcal{U}$ random noise, and ObsAtk with
two different adversarial budgets. The energy-density of noise is bounded by
$\hat{\gamma}^{2}$.

Dataset | $\hat{\gamma}$ | $\mathcal{N}$ | $\mathcal{U}$ | Atk-$\nicefrac{{5}}{{255}}$ | Atk-$\nicefrac{{7}}{{255}}$
---|---|---|---|---|---
Set68 | $\nicefrac{{25}}{{255}}$ | 29.16/0.02 | 29.15/0.01 | 24.26/0.12 | 23.12/0.10
$\nicefrac{{15}}{{255}}$ | 31.68/0.00 | 31.68/0/00 | 26.66/0.04 | 26.08/0.02
Set12 | $\nicefrac{{25}}{{255}}$ | 30.39/0.01 | 30.41/0.01 | 24.32/0.18 | 22.96/0.13
$\nicefrac{{15}}{{255}}$ | 32.78/0.00 | 32.81/0.00 | 26.91/0.05 | 26.25/0.01
BSD68 | $\nicefrac{{25}}{{255}}$ | 31.25/0.11 | 31.17/0.11 | 27.44/0.08 | 26.08/0.06
$\nicefrac{{15}}{{255}}$ | 33.98/0.11 | 33.93/0.12 | 29.31/0.08 | 27.84/0.04
Kodak24 | $\nicefrac{{25}}{{255}}$ | 32.20/0.13 | 32.13/0.14 | 27.87/0.08 | 26.37/0.07
$\nicefrac{{15}}{{255}}$ | 34.77/0.13 | 34.73/0.14 | 29.55/0.07 | 28.00/0.04

From Tables 2.1, we observe that ObsAtk clearly degrades the reconstruction
performance of DIDs. In comparison to Gaussian or Uniform noisy images with
the same noise levels, the recovered results from adversarial images are much
worse in the sense of the PSNR. For example, when removing
$\hat{\gamma}=\nicefrac{{15}}{{255}}$ noisy images in Set12, the average PSNR
of reconstructions from Gaussian noise can achieve 32.78 dB, whereas the PSNR
drops to 26.25 dB when dealing with Atk-$\nicefrac{{7}}{{255}}$ adversarial
images. We observe the consistent phenomenon that a normally-trained denoiser
$f^{\gamma}_{\theta}(\cdot)$ cannot effectively remove adversarial noise from
visual results in Figure 2.2.

###  2.4 Robust and Generalizable Denoising via HAT

The previous section shows that existing deep denoisers are vulnerable to
adversarial perturbations. To improve the adversarial robustness of deep
denoisers, we propose an adversarial training method, hybrid adversarial
training (HAT), that uses original noisy images and their adversarial versions
for training. Furthermore, we build a connection between the adversarial
robustness of deep denoisers and their generalization capability to unseen
types of noise. We show that HAT-trained denoisers can effectively remove
real-world noise without the need to leverage the real-world noisy data.

####  2.4.1 Hybrid Adversarial Training

AT has been proved to be a successful and universally applicable technique for
robustifying deep neural networks. Most variants of AT are developed for the
classification task specifically, such as TRADES [zhang_theoretically_2019]
and GAIRAT [zhang_geometry-aware_2020]. Here, we propose an AT strategy, HAT,
for robust image denoising:

$\displaystyle\min_{\theta\in\Theta}\mathbb{E}_{{\mathbf{x}}\sim P}$
$\displaystyle\mathbb{E}_{Q\sim\mathcal{U}({\mathbb{Q}}^{\gamma})}\mathbb{E}_{{\mathbf{v}}\sim
Q}\frac{1}{2}\Big{(}\frac{1}{1+\alpha}\|f_{\theta}^{\gamma}({\mathbf{y}})-{\mathbf{x}}\|_{2}^{2}$
$\displaystyle+\frac{\alpha}{1+\alpha}\|f_{\theta}^{\gamma}({\mathbf{y}})-f_{\theta}^{\gamma}({\mathbf{y}}^{\prime})\|_{2}^{2}\Big{)},$
(2.3)

where ${\mathbf{y}}={\mathbf{x}}+{\mathbf{v}}$ and
${\mathbf{y}}^{\prime}={\mathbf{y}}+\bm{\delta}^{*}$. Note that
$\bm{\delta}^{*}$ is the adversarial perturbation obtained by solving ObsAtk
in Eq. equation 2.1.

As shown in Eq. equation 2.3, the loss function consists of two terms. The
first term measures the distance between ground-truth images ${\bm{x}}$ and
reconstructions $f_{\theta}^{\gamma}({\bm{y}})$ from non-adversarial noisy
images ${\bm{y}}$, where ${\bm{y}}$ contains noise ${\bm{v}}$ sampled from a
certain common distribution $Q$, such as Gaussian. This term encourages a good
reconstruction performance of $f_{\theta}^{\gamma}$ from common distributions.
The second term is the distance between $f_{\theta}^{\gamma}({\bm{y}})$ and
the reconstruction $f_{\theta}^{\gamma}({\bm{y}}^{\prime})$ from the
adversarially perturbed version ${\bm{y}}^{\prime}$ of ${\bm{y}}$. This term
ensures that the reconstructions from any two noisy observations within a
small neighborhood of ${\bm{y}}$ have similar image qualities. Minimizing
these two terms at the same time controls the worst-case reconstruction
performance $\|f_{\theta}^{\gamma}({\bm{y}}^{\prime})-{\bm{x}}\|$.

The coefficient $\alpha$ balances the trade-off between reconstruction from
common noise and the local continuity of $f_{\theta}^{\gamma}$. When $\alpha$
equals zero, HAT degenerates to normal training on common noise. The obtained
denoisers fail to resist adversarial perturbations as shown in Section 2.3.
When $\alpha$ is very large, the optimization gradually ignores the first term
and completely aims for local smoothness. This may yield a trivial solution
that $f_{\theta}^{\gamma}$ always outputs a constant vector for any input. A
proper value of $\alpha$ thus ensures a denoiser that performs well for common
noise and the worst-case adversarial perturbations simultaneously. We perform
an ablation study on the effect of $\alpha$ for the robustness enhancement and
unseen noise removal in Appendix 2.6.3.

To train a denoiser applicable to different levels of noise with an energy-
density bounded by $\gamma^{2}$, we randomly select a noise distribution $Q$
from a family of common distributions ${\mathbb{Q}}^{\gamma}$.
${\mathbb{Q}}^{\gamma}$ includes a variety of zero-mean distributions whose
variance are bounded by $\gamma^{2}$. For example, we define
${\mathbb{Q}}^{\gamma}_{\mathcal{N}}=\\{\mathcal{N}(\bm{0},\sigma^{2}\mathbf{I}))|\sigma\sim\mathcal{U}(0,\gamma)\\}$
for the experiments in the remaining of this paper.

####  2.4.2 Robustness Enhancement via HAT

(a) Ground-truth

(b) NT

(c) HAT

Fig. 2.3:  From left to right are the ground-truth, the reconstruction of a
normally-trained denoiser against attack, and the reconstruction of a HAT-
trained denoiser against attack.

We follow the same settings as those in Section 2.3 for training and
evaluating $\gamma$-deep denoisers. The highest level of noise used for
training is set to be $\gamma=\nicefrac{{25}}{{255}}$. Noise is sampled from a
set of Gaussian distributions ${\mathbb{Q}}^{\gamma}_{\mathcal{N}}$. We train
deep denoisers with the HAT strategy and set $\alpha$ to be $1$, and use one-
step Atk-$\nicefrac{{5}}{{255}}$ to generate adversarially noisy images for
training. We compare HAT with normal training (NT) and the vanilla adversarial
training (vAT) used in Choi et al.choi_deep_2021 that trains denoisers only
with adversarial data. The results on Set68 and BSD68 are provided in this
section. More results on Set12 and Kodak24 (in Tables 2.5 and 2.6) are
provided in Appendix 2.6.2.

Table 2.2: The average PSNR (in dB) results of DnCNN-B denoisers on the gray-
scale Set68 dataset. NT and HAT are compared in terms of the noise removal of
Gaussian noise and adversarial noise. We repeat the training for three times
and report the mean and standard deviation (mean/std).

Method | $\hat{\gamma}$ | $\mathcal{N}$ | Atk-$\nicefrac{{3}}{{255}}$ | Atk-$\nicefrac{{5}}{{255}}$ | Atk-$\nicefrac{{7}}{{255}}$
---|---|---|---|---|---
NT | $\nicefrac{{25}}{{255}}$ |  29.16/0.02 | 26.20/0.07 | 24.26/0.12 | 23.12/0.10
$\nicefrac{{15}}{{255}}$ |  31.68/0.00 | 27.98/0.05 | 26.66/0.04 | 26.08/0.02
vAT | $\nicefrac{{25}}{{255}}$ | 29.05/0.07 | 27.02/0.15 | 25.51/0.32 | 24.34/0.34
$\nicefrac{{15}}{{255}}$ | 31.53/0.09 | 28.74/0.16 | 27.43/0.19 | 26.68/0.15
HAT | $\nicefrac{{25}}{{255}}$ | 28.88/0.04 |  27.48/0.10 |  26.40/0.16 |  25.32/0.17
$\nicefrac{{15}}{{255}}$ | 31.36/0.03 |  29.52/0.01 |  28.34/0.03 |  27.34/0.03

From Tables 2.2 and 2.3, we observe that HAT obviously improves the
reconstruction performance from adversarial noise in comparison to normal
training. For example, on the Set68 dataset (Table 2.2), when dealing with
$\nicefrac{{15}}{{255}}$-level noise, the normally-trained denoiser achieves
31.68 dB for Gaussian noise removal, but the PSNR drops to 26.08 dB against
Atk-$\nicefrac{{7}}{{255}}$. In contrast, the HAT-trained denoiser achieves a
PSNR of 27.34 dB (1.26 dB higher) against Atk-$\nicefrac{{7}}{{255}}$ and
maintains a PSNR of 31.36 dB for Gaussian noise removal. In Figure 2.3, we can
see that when dealing with adversarially noisy images, the HAT-trained
denoiser can recover high-quality images while the normally-trained denoiser
preserves noise patterns in the output. Besides, we observe that, similar to
image classification tasks [zhang_theoretically_2019], AT-based methods (HAT
and vAT) robustify deep denoisers at the expense of the performance on non-
adversarial data (Gaussian denoising). Nevertheless, the degraded
reconstructions are still reasonably good in terms of the PSNR.

Table 2.3: The average PSNR (in dB) results of DnCNN-C denoisers on the RGB
BSD68 dataset.

Method | $\hat{\gamma}$ | $\mathcal{N}$ | Atk-$\nicefrac{{3}}{{255}}$ | Atk-$\nicefrac{{5}}{{255}}$ | Atk-$\nicefrac{{7}}{{255}}$
---|---|---|---|---|---
NT | $\nicefrac{{25}}{{255}}$ |  31.25/0.11 | 28.93/0.08 | 27.44/0.08 | 26.08/0.06
$\nicefrac{{15}}{{255}}$ |  33.98/0.11 | 31.09/0.10 | 29.31/0.08 | 27.84/0.04
vAT | $\nicefrac{{25}}{{255}}$ | 30.64/0.02 | 28.81/0.03 | 27.67/0.01 | 26.64/0.03
$\nicefrac{{15}}{{255}}$ | 33.45/0.06 | 31.10/0.05 | 29.79/0.02 | 28.63/0.08
HAT | $\nicefrac{{25}}{{255}}$ | 30.98/0.03 |  29.18/0.03 |  28.02/0.02 |  26.93/0.04
$\nicefrac{{15}}{{255}}$ | 33.67/0.04 |  31.38/0.04 |  30.03/0.02 |  28.80/0.01

Table 2.4: Comparison of different methods for denoising real-world noisy
images in terms of PSNR (dB). We repeat the experiments of each denoising
method for three times and report the mean/standard deviation of PSNR values.

Dataset | BM3D | DIP | N2S(1) | NT | vAT | HAT | N2C
---|---|---|---|---|---|---|---
PolyU | 37.40/0.00 | 36.08/0.01 | 35.37/0.15 | 35.86/0.01 | 36.77/0.00 |  37.82/0.04 | – / –
CC | 35.19/0.00 | 34.64/0.06 | 34.33/0.14 | 33.56/0.01 | 34.49/0.10 |  36.26/0.06 | – / –
SIDD | 25.65/0.00 | 26.89/0.02 | 26.51/0.03 | 27.20/0.70 | 27.08/0.28 |  33.44/0.02 |  33.50/0.03

####  2.4.3 Robustness Benefits Generalization to Unseen Noise

It has been shown that denoisers that are normally trained on common synthetic
noise fail to remove real-world noise induced by standard imaging procedures
[xu_multi-channel_2017; abdelhamed_high-quality_2018]. To train denoisers that
can handle real-world noise, researchers have proposed several methods which
can be roughly divided into two categories, namely dataset-based denoising
methods and single-image-based denoising methods. High-performance dataset-
based methods require a set of real noisy data for training, e.g., CBDNet
requiring pairs of clean and noisy images [guo_toward_2019] and Noise2Noise
requiring multiple noisy observations of every single image
[lehtinen_noise2noise_2018]. However, a large number of paired data are not
available in some applications, such as medical radiology and high-speed
photography. To address this, single-image-based methods are proposed to
remove noise by exploiting the correlation between signals across pixels and
the independence between noise. This category of methods, such as DIP
[ulyanov_deep_2018] and N2S [batson_noise2self_2019], are adapted to various
types of signal-independent noise, but they optimize the deep denoiser on each
test image. The test-time optimization is extremely time-consuming, e.g., N2S
needs to update a denoiser for thousands of iterations to achieve good
reconstruction performance.

Here, we point out that HAT is a promising framework to train a generalizable
deep denoiser only with synthetic noise. The resultant denoiser can be
directly applied to perform denoising for unseen noisy images in real-time.
During training, HAT first samples noise from common distributions (Gaussian)
with noise levels from low to high. ObsAtk then explores the
$\epsilon$-neighborhood for each noisy image to search for a particular type
of noise that degrades the denoiser the most. By ensuring the denoising
performance of the worst-case noise, the resultant denoiser can deal with
other unknown types of noise within the $\epsilon$-neighborhood as well. To
train a robust denoiser that generalizes well to real-world noise, we need to
choose a proper adversarial budget $\epsilon$. When $\epsilon$ is very small
and close to zero, the HAT reduces to normal training. When $\epsilon$ is very
much larger than the norm of basic noise ${\bm{v}}$, the adversarially noisy
image may be visually unnatural because the adversarial perturbation
$\bm{\delta}$ only satisfies the zero-mean constraint and is not guaranteed to
be spatially uniformly distributed as other types of natural noise being. In
practice, we set the value of $\epsilon$ of ObsAtk to be
$\nicefrac{{5}}{{255}}\cdot\sqrt{m}$, where $m$ denotes the size of image
patches. The value of $\alpha$ of HAT is kept unchanged as $2$.

###### Experimental Settings

We evaluate the generalization capability of HAT on several real-world noisy
datasets, including PolyU [xu_real-world_2018], CC [xu_multi-channel_2017],
and SIDD [abdelhamed_high-quality_2018]. PolyU, CC, and SIDD contain RGB
images of common scenes in daily life. These images are captured by different
brands of digital cameras and smartphones, and they contain various levels of
noise by adjusting the ISO values. For the PolyU and CC, we use the clean
images in BSD500 for training an adversarially robust $\gamma$-denoiser with
$\gamma=\nicefrac{{25}}{{255}}$. We sample Gaussian noise from a set of
distributions ${\mathbb{Q}}^{\gamma}_{\mathcal{N}}$ and add the noise to clean
images to craft noisy observations. HAT trains the denoiser jointly with
Gaussian noisy images and their adversarial versions. For the SIDD, we use
clean images in the SIDD-small set for training and test the denoisers on the
SIDD-val set. The highest level of noise used for HAT is set to be
$\gamma=\nicefrac{{50}}{{255}}$. In each case, we only use clean images for
training denoisers without the need of real noisy images

###### Results

We compare HAT-trained denoisers with the NT and vAT-trained ones. From Table
2.4, we observe that HAT performs much better than both competitors. For
example, on the SIDD-val dataset, the HAT-trained denoiser achieves an average
PSNR value of 33.44 dB that is 6.24 dB higher than the NT-trained one. We also
compare HAT-trained denoisers with single-image-based methods, including DIP,
N2S, and the classical BM3D [dabov_image_2007]. For DIP and N2S,111The
officially released codes of DIP and N2S are used here.  the numbers of
iterations for each image are set to be 2,000 and 1,000, respectively. N2S
works in two modes, namely single-image-based denoising and dataset-based
denoising. Here, we use N2S in the single-image-based mode, denoted as N2S(1),
due to the assumption that no real noisy data are available for training. We
observe that HAT-trained denoisers consistently outperform these baselines.
Visual comparisons are provided in Appendix 2.6.4. Besides, since the SIDD-
small provides a set of real noisy and ground-truth pairs, we train a
denoiser, denoted as Noise2Clean (N2C), with these paired data and use the N2C
denoiser as the oracle for comparison. We observe that HAT-trained denoisers
are comparable to the N2C one for denoising images in SIDD-val (a PSNR of
33.44dB vs 33.50dB).

###  2.5 Chapter Summary

Normally-trained deep denoisers are vulnerable to adversarial attacks. HAT can
effectively robustify deep denoisers and boost their generalization capability
to unseen real-world noise. In the future, we will extend the adversarial-
training framework to other image restoration tasks, such as deblurring. We
aim to develop a generic AT-based robust optimization framework to train deep
models that can recover clean images from unseen types of degradation.

###  2.6 Appendices

####  2.6.1 Two-step Projection

###### Theorem 2.6.1

For any arbitrary vector $\bm{\delta}\in{\mathbb{R}}^{m}$, its projection onto
the region defined by the intersection of the norm-bounded and zero-mean
constraints is equivalent to the projection first onto the zero-mean
hyperplane followed by the projection onto the $\epsilon$-ball ($\epsilon>0$),
i.e.,

$\text{Proj}_{A\cap
B}(\bm{\delta})=\text{Proj}_{B}(\text{Proj}_{A}(\bm{\delta})),$ (2.4)

where

$\displaystyle A$
$\displaystyle=\big{\\{}{\bm{z}}\in{\mathbb{R}}^{m}\,\big{|}\,{\bm{n}}^{\top}{\bm{z}}=0\big{\\}},$
(2.5a) $\displaystyle B$
$\displaystyle=\big{\\{}{\bm{z}}\in{\mathbb{R}}^{m}\,\big{|}\,\|{\bm{z}}\|_{2}^{2}\leq\epsilon^{2}\big{\\}},$
(2.5b)

and ${\bm{n}}=[1,1,\ldots,1]^{\top}$.

Fig. 2.4: Illustration of Theorem 2.6.1. In the case of
$\|\text{Proj}_{A}(\bm{\delta})\|>\epsilon$, the red-dot lines show that the
perturbation $\bm{\delta}$ is projected onto the region defined by the zero-
mean and $\epsilon$-ball constraints sequentially. The blue-dot line shows the
exact projection of $\bm{\delta}$ on to $A\cap B$.

###### Proof

Let us consider the RHS of Eq. equation 2.4 first. It is easy to derive the
projections onto $A$ and $B$ seperately:

$\displaystyle\text{Proj}_{A}(\bm{\delta})$
$\displaystyle=\bm{\delta}-\frac{{\bm{n}}^{\top}\bm{\delta}}{\|{{\bm{n}}}\|^{2}_{2}}{{\bm{n}}},$
(2.6a) $\displaystyle\text{Proj}_{B}(\bm{\delta})$
$\displaystyle=\min\Big{(}\frac{\epsilon}{\|\bm{\delta}\|_{2}},1\Big{)}\text{~{}}\bm{\delta}.$
(2.6b)

Thus, we have

$\text{Proj}_{B}(\text{Proj}_{A}(\bm{\delta}))=\begin{cases}\text{Proj}_{A}(\bm{\delta}),&\text{if
$\|\text{Proj}_{A}(\bm{\delta})\|\leq\epsilon$};\\\
\frac{\epsilon}{\|\text{Proj}_{A}(\bm{\delta})\|_{2}}\text{Proj}_{A}(\bm{\delta}),&\text{if
$\|\text{Proj}_{A}(\bm{\delta})\|>\epsilon$.}\end{cases}$ (2.7)

Now let us consider the LHS of Eq. equation 2.4. The projection onto $A\cap B$
can be formulated as the solution of the following convex optimization
problem:

$\min_{{\bm{z}}}\frac{1}{2}\|\bm{\delta}-{\bm{z}}\|^{2}_{2},\quad\text{s.t.}\quad{\bm{n}}^{\top}{\bm{z}}=0,\quad\|{\bm{z}}\|^{2}_{2}\leq\epsilon^{2},$
(2.8)

where ${\bm{z}}\in{\mathbb{R}}^{m}$. We can write the Lagrangian,
$L:{\mathbb{R}}^{m}\times{\mathbb{R}}\times{\mathbb{R}}\to{\mathbb{R}}$,
associated with the problem equation 2.8 as

$L({\bm{z}},\lambda,\nu)=\frac{1}{2}\|\bm{\delta}-{\bm{z}}\|^{2}_{2}+\lambda(\|{\bm{z}}^{*}\|^{2}_{2}-\epsilon^{2})+\nu{\bm{n}}^{\top}{\bm{z}}.$
(2.9)

Since there exists an ${\bm{z}}\in{\mathbb{R}}^{m}$, e.g.,
${\bm{z}}=[0,\ldots,0]^{\top}\in{\mathbb{R}}^{m}$, such that
${\bm{n}}^{\top}{\bm{z}}=0$ and $\|{\bm{z}}\|^{2}_{2}<\epsilon^{2}$, the
problem equation 2.8 is strictly feasible, i.e., it satisfies the Slater’s
condition boyd2004convex. Besides, the objective and the constraints are all
differentiable, thus the KKT conditions in Eq. equation 2.10 provide necessary
and sufficient conditions for optimality.

$\displaystyle\|{\bm{z}}^{*}\|^{2}_{2}-\epsilon^{2}\leq 0,$ (2.10a)
$\displaystyle{\bm{n}}^{\top}{\bm{z}}^{*}=0,$ (2.10b)
$\displaystyle\lambda\geq 0,$ (2.10c)
$\displaystyle\lambda(\|{\bm{z}}^{*}\|^{2}_{2}-\epsilon^{2})=0,$ (2.10d)
$\displaystyle\frac{\partial
L}{\partial{\bm{z}}}=(1+2\lambda){\bm{z}}^{*}-\bm{\delta}+\nu{\bm{n}}=0.$
(2.10e)

We obtain the optimal solution by considering the following two cases
separately, i.e., $\|\text{Proj}_{A}(\bm{\delta})\|\leq\epsilon$ and
$\|\text{Proj}_{A}(\bm{\delta})\|>\epsilon$.

Case-(1): $\|\text{Proj}_{A}(\bm{\delta})\|>\epsilon$.
If $\lambda>0$, then Eq. equation 2.10 reduces to the following equation:

$\displaystyle{\bm{n}}^{\top}{\textnormal{z}}^{*}=0,$ (2.11a)
$\displaystyle\|{\bm{z}}^{*}\|^{2}_{2}-\epsilon^{2}=0,$ (2.11b)
$\displaystyle(1+2\lambda){\bm{z}}^{*}-\bm{\delta}+\nu{\bm{n}}=0.$ (2.11c)

We can easily solve these equations and obtain that

$\displaystyle(1+2\lambda)$
$\displaystyle=\text{Proj}_{A}(\bm{\delta})/\epsilon,$ (2.12a)
$\displaystyle\nu$ $\displaystyle={\bm{n}}^{\top}\bm{\delta}/m,$ (2.12b)
$\displaystyle{\bm{z}}^{*}$
$\displaystyle=\frac{\epsilon}{\|\text{Proj}_{A}(\bm{\delta})\|_{2}}\text{Proj}_{A}(\bm{\delta}).$
(2.12c)

If $\lambda=0$, then Eq. equation 2.10 reduces to the following set of
equations:

$\displaystyle\|{\bm{z}}^{*}\|^{2}_{2}-\epsilon^{2}$ $\displaystyle\leq 0,$
(2.13a) $\displaystyle{\bm{n}}^{\top}{\textnormal{z}}^{*}$ $\displaystyle=0$
(2.13b) $\displaystyle{\bm{z}}^{*}-\bm{\delta}+\nu{\bm{n}}$ $\displaystyle=0.$
(2.13c)

According to equation 2.13b and equation 2.13c, we obtain that
${\bm{z}}^{*}=\bm{\delta}-\frac{{\bm{n}}^{\top}\bm{\delta}}{m}{\bm{n}}=\text{Proj}_{A}(\bm{\delta})$
with a norm strictly larger than $\epsilon$, which contradicts the constraint
$\|{\bm{z}}^{*}\|^{2}_{2}\leq\epsilon^{2}$. Thus, for the case of
$\|\text{Proj}_{A}(\bm{\delta})\|>\epsilon$, we have that
${\bm{z}}^{*}=\frac{\epsilon}{\|\text{Proj}_{A}(\bm{\delta})\|_{2}}\text{Proj}_{A}(\bm{\delta})$
which is equal to $\text{Proj}_{B}(\text{Proj}_{A}(\bm{\delta}))$ in Eq.
equation 2.7.

Case-(2): $\|\text{Proj}_{A}(\bm{\delta})\|\leq\epsilon$.
Since $\|\text{Proj}_{A}(\bm{\delta})\|\leq\epsilon$ and
$\|\text{Proj}_{A}(\bm{\delta})\|\in A$, we have
$\text{Proj}_{A}(\bm{\delta})\in A\cap B$. For any other point
${\bm{z}}^{\prime}\in A\cap B$ and
${\bm{z}}^{\prime}\neq\text{Proj}_{A}(\bm{\delta})$, we have
$\|\bm{\delta}-{\bm{z}}^{\prime}\|>\|\bm{\delta}-\text{Proj}_{A}(\bm{\delta})\|$,
where the strict inequality holds because $A$ is the set of points from a
hyperplane. Thus, ${\bm{z}}^{\prime}$ is not the $\text{Proj}_{A\cap
B}(\bm{\delta})$. Therefore, $\text{Proj}_{A\cap
B}(\bm{\delta})=\text{Proj}_{A}(\bm{\delta})=\text{Proj}_{B}(\text{Proj}_{A}(\bm{\delta}))$.

In summary, we show that $\text{Proj}_{A\cap
B}(\bm{\delta})=\text{Proj}_{B}(\text{Proj}_{A}(\bm{\delta}))$ for any
arbitrary $\bm{\delta}\in{\mathbb{R}}^{m}$.

####  2.6.2 Experiments of Robustness Enhancement on Set12 and Kodak24

We compare the robustness of deep denoisers trained via three strategies,
i.e., NT, vAT and HAT. The results on Set 12 and Kodak24 are provided in Table
2.5 and Table 2.6 respectively. We observe that HAT can effectively robustify
deep denoisers. The reconstruction quality of HAT-trained denoisers from
adversarially noisy images is clearly better than that of the NT and vAT-
trained ones.

Table 2.5: The average PSNR (in dB) results of DnCNN-B denoisers on the gray-
scale Set12 dataset.

Training | $\hat{\gamma}$ | $\mathcal{N}$ | Atk-$\nicefrac{{3}}{{255}}$ | Atk-$\nicefrac{{5}}{{255}}$ | Atk-$\nicefrac{{7}}{{255}}$
---|---|---|---|---|---
NT | $\nicefrac{{25}}{{255}}$ |  30.39/0.01 | 26.51/0.14 | 24.32/0.18 | 22.96/0.13
$\nicefrac{{15}}{{255}}$ |  32.78/0.00 | 28.50/0.08 | 26.91/0.05 | 26.25/0.01
vAT | $\nicefrac{{25}}{{255}}$ | 30.25/0.08 | 27.56/0.06 | 25.82/0.04 | 24.33/0.04
$\nicefrac{{15}}{{255}}$ | 32.63/0.09 | 29.37/0.17 | 27.83/0.15 | 26.91/0.08
HAT | $\nicefrac{{25}}{{255}}$ | 30.01/0.06 |  27.96/0.15 |  26.46/0.20 |  25.13/0.19
$\nicefrac{{15}}{{255}}$ | 32.47/0.04 |  29.95/0.03 |  28.45/0.04 |  27.20/0.03

Table 2.6: The average PSNR (in dB) results of DnCNN-C denoisers on the RGB
Kodak24 dataset.

Training | $\hat{\gamma}$ | $\mathcal{N}$ | Atk-$\nicefrac{{3}}{{255}}$ | Atk-$\nicefrac{{5}}{{255}}$ | Atk-$\nicefrac{{7}}{{255}}$
---|---|---|---|---|---
NT | $\nicefrac{{25}}{{255}}$ |  32.20/0.13 | 29.57/0.09 | 27.87/0.08 | 26.37/0.07
$\nicefrac{{15}}{{255}}$ |  34.77/0.13 | 31.54/0.11 | 29.55/0.07 | 28.00/0.04
vAT | $\nicefrac{{25}}{{255}}$ | 31.44/0.01 | 29.41/0.05 | 28.13/0.06 | 26.98/0.02
$\nicefrac{{15}}{{255}}$ | 34.14/0.08 | 31.53/0.11 | 30.06/0.08 | 28.78/0.06
HAT | $\nicefrac{{25}}{{255}}$ | 31.83/0.04 |  29.85/0.02 |  28.56/0.02 |  27.34/0.05
$\nicefrac{{15}}{{255}}$ | 34.36/0.06 |  31.84/0.05 |  30.37/0.02 |  29.05/0.01

####  2.6.3 Ablation study

##### Effect of $\alpha$ on Robustness Enhancement and Generalization to Real-
world noise

Here, we evaluate the effect of $\alpha$ in HAT on the adversarial robustness
and the generalization capability to real-world noise. We train deep denoisers
on the RGB BSD500 (except 68 images for test) dataset. The obtained denoisers
are tested on the BSD68 dataset for Gaussian and adversarial noise removal.
The generalization capability is evaluated on two datasets of real-world noisy
images, i.e., PolyU and CC. Experimental settings follow those in Section
2.4.2.

Figure 2.5 corroborates the analysis in Section 2.4.1 that the coefficient
$\alpha$ balances the trade-off between reconstruction from common noise and
the adversarial robustness. We also find that the generalization capability to
real-world noise is correlated to the adversarial robustness. Specifically,
good adversarial robustness usually implies good generalization to real-world
noise. In Figure 2.5, the best robustness and the best performance on real-
world noise appear around $\alpha=1$ or $2$. When $\alpha$ is too large or too
small, the robustness and generalization worsen. For the noise sampled from
Gaussian distributions, increasing $\alpha$ degrades the denoising
performance. In summary, we set $\alpha$ to $1$ or $2$ to achieve a good
balance between the denoising performance on common noise and the adversarial
robustness as well as real-world generalization.

Fig. 2.5: Ablation study on the effect of $\alpha$ in HAT. Green lines show
the denoising results on non-adversarial noise sampled from common
distributions. The legend $\hat{\gamma}$-w-$\mathcal{N}$ denotes the Gaussian
noise ($\sigma=\nicefrac{{w}}{{255}}$) with a energy-density bounded by
$\hat{\gamma}^{2}=\nicefrac{{w^{2}}}{{255^{2}}}$. Blue lines show that
denoising results on adversarially perturbed noisy images.
$\hat{\gamma}$-w-ObsAtk-5 denotes the adversarial noise crafted by ObsAtk-5
with a energy-density bounded by $\hat{\gamma}^{2}$. Red lines show the
denoising results on real-world noisy images. Fig. 2.6: Ablation study on the
effect of $\epsilon$ in HAT.

(a) Noisy

(b) BM3D

(c) DIP

(d) N2S

(e) NT

(f) vAT

(g) HAT

(h) G.T.

Fig. 2.7:  Comparison of different denoisers for denoising SIDD-val set. From
left to right are the input noisy images, reconstructions of different
denoisers, and the ground-truth (G.T.) images We can see that the HAT-trained
denoiser performs the best in comparison to other baseline methods.

##### Effect of $\epsilon$ on Generalization to Real-world Noise

Here, we evaluate the effect of $\epsilon$ used in HAT on the generalization
capability to real-world noise. We train deep denoisers on the RGB BSD500
(except 68 images for test) dataset and evaluate the generalization capability
on two real-world datasets, namely PolyU and CC. The $\alpha$ is set to be
$2$. The adversarial budget $\epsilon$ of
ObsAtk-$\nicefrac{{\epsilon}}{{\sqrt{m}}}$, that generates adversarially noisy
images for HAT, is set to be values from
$[0,\sqrt{m},3\sqrt{m},\ldots,11\sqrt{m}]$ for comparison, where $m$ denotes
the size of images. Other experimental settings follow those in Section 2.4.2.

Figure 2.6 corroborates the analysis in Section 2.4.3. When $\epsilon$ is very
small and close to zero, the HAT reduces to normal training. The resultant
denoisers cannot effectively remove real-world noise. When $\epsilon$ is very
much larger than the norm of basic noise ${\bm{v}}$, the statistics of
adversarial noise may be very unnatural because the adversarial perturbation
$\bm{\delta}$ might concentrate on a certain region, like edges or texture,
and not be spatially uniformly distributed as other types of natural noise
being. We can see that, when $\epsilon>\nicefrac{{7}}{{255}}\sqrt{m}$, the
denoising performance on real-world datasets starts to decrease. In practice,
we set the value of $\epsilon$ of ObsAtk to be
$\nicefrac{{5}}{{255}}\cdot\sqrt{m}$ to train generalizable denoisers.

####  2.6.4 Visual results of real-world noise removal

We show the denoising results on SIDD-val set in Figure 2.7. We observe that
HAT-trained denoiser can effectively remove the real-world noise while the
normally-trained one retains much noise in the reconstructions. Besides, the
HAT-trained denoiser outperforms other baseline methods and produces much
cleaner results.

##  Chapter 3 On Robustness of Neural Ordinary Differential Equations

Neural ordinary differential equations (ODEs) have been attracting increasing
attention in various research domains recently. There have been some works
studying optimization issues and approximation capabilities of neural ODEs,
but their robustness is still yet unclear. In this work, we fill this
important gap by exploring robustness properties of neural ODEs both
empirically and theoretically. We first present an empirical study on the
robustness of the neural ODE-based networks (ODEnets) by exposing them to
inputs with various types of perturbations and subsequently investigating the
changes of the corresponding outputs. In contrast to conventional
convolutional neural networks (CNNs), we find that the ODEnets are more robust
against both random Gaussian perturbations and adversarial attack examples. We
then provide an insightful understanding of this phenomenon by exploiting a
certain desirable property of the flow of a continuous-time ODE, namely that
integral curves are non-intersecting. Our work suggests that, due to their
intrinsic robustness, it is promising to use neural ODEs as a basic block for
building robust deep network models. To further enhance the robustness of
vanilla neural ODEs, we propose the time-invariant steady neural ODE (TisODE),
which regularizes the flow on perturbed data via the time-invariant property
and the imposition of a steady-state constraint. We show that the TisODE
method outperforms vanilla neural ODEs and also can work in conjunction with
other state-of-the-art architectural methods to build more robust deep
networks.

###  3.1 Introduction

Neural ordinary differential equations [chen2018neural] form a family of
models that approximate nonlinear mappings by using continuous-time ODEs. Due
to their desirable properties, such as invertibility and parameter efficiency,
neural ODEs have attracted increasing attention recently [dupont2019augmented;
liu2019neural]. For example, grathwohl2018ffjord proposed a neural ODE-based
generative model—the FFJORD—to solve inverse problems;
quaglino2019accelerating used a higher-order approximation of the states in a
neural ODE, and proposed the SNet to accelerate computation. Along with the
wider deployment of neural ODEs, robustness issues come to the fore. However,
the robustness of neural ODEs is still yet unclear. In particular, it is
unclear how robust neural ODEs are in comparison to the widely-used CNNs.
Robustness properties of CNNs have been studied extensively. In this work, we
present the first systematic study on exploring the robustness properties of
neural ODEs.

To do so, we consider the task of image classification. We expect that results
would be similar for other machine learning tasks such as regression. Neural
ODEs are dimension-preserving mappings, but a classification model transforms
a high-dimensional input—such as an image—into an output whose dimension is
equal to the number of classes. Thus, we consider the neural ODE-based
classification network (ODEnet) whose architecture is shown in Figure 3.1. An
ODEnet consists of three components: the feature extractor (FE) consists of
convolutional layers which maps an input datum to a multi-channel feature map,
a neural ODE that serves as the nonlinear representation mapping (RM), and the
fully-connected classifier (FCC) that generates a prediction vector based on
the output of the RM.

Fig. 3.1: The architecture of an ODEnet. The neural ODE block serves as a
dimension-preserving nonlinear mapping.

The robustness of a classification model can be evaluated through the lens of
its performance on perturbed images. To comprehensively investigate the
robustness of neural ODEs, we perturb original images with commonly-used
perturbations, namely, random Gaussian noise [szegedy2013intriguing] and
harmful adversarial examples [goodfellow_explaining_2015; madry_towards_2018].
We conduct experiments in two common settings—training the model only on
authentic non-perturbed images and training the model on authentic images as
well as the Gaussian perturbed ones. We observe that ODEnets are more robust
compared to CNN models against all types of perturbations in both settings. We
then provide an insightful understanding of such intriguing robustness of
neural ODEs by exploiting a certain property of the flow
[dupont2019augmented], namely that integral curves that start at distinct
initial states are non-intersecting. The flow of a continuous-time ODE is
defined as the family of solutions/paths traversed by the state, starting from
different initial points, and an integral curve is a specific solution for a
given initial point. The non-intersecting property indicates that an integral
curve starting from some point is constrained by the integral curves starting
from that point’s neighborhood. Thus, in an ODEnet, if a correctly classified
datum is slightly perturbed, the integral curve associated to its perturbed
version would not change too much from the original one. Consequently, the
perturbed datum could still be correctly classified. Thus, there exists
intrinsic robustness regularization in ODEnets, which is absent from CNNs.

Motivated by this property of the neural ODE flow, we attempt to explore a
more robust neural ODE architecture by introducing stronger regularization on
the flow. We thus propose a _T ime-Invariant Steady neural ODE_ (TisODE). The
TisODE removes the time dependence of the dynamics in an ODE and imposes a
steady-state constraint on the integral curves. Removing the time dependence
of the derivative results in the time-invariant property of the ODE. To wit,
given a solution ${\bm{z}}_{1}(t)$, another solution
$\widetilde{{\bm{z}}}_{1}(t)$, with an initial state
$\widetilde{{\bm{z}}}_{1}(0)={\bm{z}}_{1}(T^{\prime})$ for some
$T^{\prime}>0$, can be regarded as the $-T^{\prime}$\- shift version of
${\bm{z}}_{1}(t)$. Such a time-invariant property would make bounding the
difference between output states convenient. To elaborate, let the output of a
neural ODE correspond to states at time $T>0$. By the time-invariant property,
the difference between outputs,
$\|\widetilde{{\bm{z}}}_{1}(T)-{\bm{z}}_{1}(T)\|$, equals to
$\|{\bm{z}}_{1}(T+T^{\prime})-{\bm{z}}_{1}(T)\|$. To control this distance, a
steady-state regularization term is introduced to the overall objective to
constrain the change of a state after time exceeds $T$. With the time-
invariant property and the steady-state term, we show that TisODE even is more
robust. We do so by evaluating the robustness of TisODE-based classifiers
against various types of perturbations and observe that such models are more
robust than vanilla ODE-based models.

In addition, some other effective architectural solutions have also been
recently proposed to improve the robustness of CNNs. For example,
xie_mitigating_2018 randomly resizes or pads zeros into test images to destroy
the specific structure of adversarial perturbations. Besides, the model
proposed by xie_feature_2019 contains feature denoising filters to remove the
feature-level patterns of adversarial examples. We conduct experiments to show
that our proposed TisODE can work seamlessly and in conjunction with these
methods to further boost the robustness of deep models. Thus, the proposed
TisODE can be used as a generally applicable and effective component for
improving the robustness of deep models.

In summary, our contributions are as follows. Firstly, we are the first to
provide a systematic empirical study on the robustness of neural ODEs and find
that the neural ODE-based models are more robust compared to conventional CNN
models. This finding inspires new applications of neural ODEs in improving
robustness of deep models, a problem that concerns many deep learning
theorists and practitioners alike. Secondly, we propose the TisODE method,
which is simple yet effective in significantly boosting the robustness of
neural ODEs. Moreover, the proposed TisODE can also be used in conjunction
with other state-of-the-art robust architectures. Thus, TisODE can serve as a
drop-in module to improve the robustness of deep models effectively.

###  3.2 Preliminaries on neural ODE

It has been shown that a residual block [he_deep_2016] can be interpreted as
the discrete approximation of an ODE by setting the discretization step to be
one. When the discretization step approaches zero, it yields a family of
neural networks, which are called neural ODEs [chen2018neural]. Formally, in a
neural ODE, the relation between input and output is characterized by the
following set of equations:

$\displaystyle\frac{\mathrm{d}{\bm{z}}(t)}{\mathrm{d}t}=f_{\theta}({\bm{z}}(t),t),\quad{\bm{z}}(0)={\bm{z}}_{\text{in}},\quad{\bm{z}}_{\text{out}}={\bm{z}}(T),$
(3.1)

where $f_{\theta}:{\mathbb{R}}^{d}\times[0,\infty)\rightarrow{\mathbb{R}}^{d}$
denotes the trainable layers that are parameterized by weights $\theta$ and
${\bm{z}}:[0,\infty)\rightarrow{\mathbb{R}}^{d}$ represents the
$d$-dimensional state of the neural ODE. We assume that $f_{\theta}$ is
continuous in $t$ and globally Lipschitz continuous in ${\bm{z}}$. In this
case, the input ${\bm{z}}_{\text{in}}$ of the neural ODE corresponds to the
state at $t=0$, and the output ${\bm{z}}_{\text{out}}$ is associated to the
state at some $T\in(0,\infty)$. Because $f_{\theta}$ governs how the state
changes with respect to time $t$, we also use $f_{\theta}$ to denote the
dynamics of the neural ODE.

Given input ${\bm{z}}_{\text{in}}$, the output ${\bm{z}}_{\text{out}}$ can be
computed by solving the ODE in (3.1). If $T$ is fixed, the output
${\bm{z}}_{\text{out}}$ only depends on the input ${\bm{z}}_{\text{in}}$ and
the dynamics $f_{\theta}$, which also corresponds to the weighted layers in
the neural ODE. Therefore, the neural ODE can be represented as the
$d$-dimensional function $\phi_{T}(\cdot,\cdot)$ of the input
${\bm{z}}_{\mathrm{in}}$ and the dynamics $f_{\theta}$, i.e.,

${\bm{z}}_{\text{out}}={\bm{z}}(T)={\bm{z}}(0)+\int_{0}^{T}f_{\theta}({\bm{z}}(t),t)\,\mathrm{d}t=\phi_{T}({\bm{z}}_{\text{in}},f_{\theta}).$

The terminal time $T$ of the output state ${\bm{z}}(T)$ is set to be $1$ in
practice. Several methods have been proposed for training neural ODEs, such as
the adjoint sensitivity method [chen2018neural], SNet
[quaglino2019accelerating], and the auto-differentiation technique
[paszke2017automatic]. In this work, we use the most straightforward
technique, i.e., updating the weights $\theta$ with the auto-differentiation
technique in the PyTorch framework.

###  3.3 An empirical study on the robustness of ODEnets

Robustness of deep models has gained increased attention, as it is imperative
that deep models employed in critical applications, such as healthcare, are
robust. The robustness of a model is measured by the sensitivity of the
prediction with respect to small perturbations on the inputs. In this study,
we consider three commonly-used perturbation schemes, namely random Gaussian
perturbations, FGSM [goodfellow_explaining_2015] adversarial examples, and PGD
[madry_towards_2018] adversarial examples. These perturbation schemes reflect
noise and adversarial robustness properties of the investigated models
respectively. We evaluate the robustness via the classification accuracies on
perturbed images, in which the original non-perturbed versions of these images
are all correctly classified.

For a fair comparison with conventional CNN models, we made sure that the
number of parameters of an ODEnet is close to that of its counterpart CNN
model. Specifically, the ODEnet shares the same network architecture with the
CNN model for the FE and FCC parts. The only difference is that, for the RM
part, the input of the ODE-based RM is concatenated with one more channel
which represents the time $t$, while the RM in a CNN model has a skip
connection and serves as a residual block. During the training phase, all the
hyperparameters are kept the same, including training epochs, learning rate
schedules, and weight decay coefficients. Each model is trained _three_ times
with different random seeds, and we report the average performance
(classification accuracy) together with the standard deviation.

####  3.3.1 Experimental settings

Dataset: We conduct experiments to compare the robustness of ODEnets with CNN
models on three datasets, i.e., the MNIST [lecun1998gradient], the SVHN
[netzer2011reading], and a subset of the ImageNet datset [deng2009imagenet].
We call the subset ImgNet10 since it is collected from 10 synsets of ImageNet:
dog, bird, car, fish, monkey, turtle, lizard, bridge, cow, and crab. We
selected 3,000 training images and 300 test images from each synset and
resized all images to $128\times 128$.

Architectures: On the MNIST dataset, both the ODEnet and the CNN model
consists of four convolutional layers and one fully-connected layer. The total
number of parameters of the two models is around 140k. On the SVHN dataset,
the networks are similar to those for the MNIST; we only changed the input
channels of the first convolutional layer to three. On the ImgNet10 dataset,
there are nine convolutional layers and one fully-connected layer for both the
ODEnet and the CNN model. The numbers of parameters is approximately 280k. In
practice, the neural ODE can be solved with different numerical solvers such
as the Euler method and the Runge-Kutta methods [chen2018neural]. Here, we use
the easily-implemented Euler method in the experiments. To balance the
computation and the continuity of the flow, we solve the ODE initial value
problem in equation (3.1) by the Euler method with step size $0.1$. Our
implementation builds on the open-source neural ODE
codes.111https://github.com/rtqichen/torchdiffeq. Details on the network
architectures are included in the Appendix.

Training: The experiments are conducted using two settings on each
dataset—training models only with original non-perturbed images and training
models on original images together with their perturbed versions. In both
settings, we added a weight decay term into the training objective to
regularize the norm of the weights, since this can help control the model’s
representation capacity and improve the robustness of a neural network
[sokolic2017robust]. In the second setting, images perturbed with random
Gaussian noise are used to fine-tune the models, because augmenting the
dataset with small perturbations can possibly improve the robustness of models
and synthesizing Gaussian noise does not incur excessive computation time.

####  3.3.2 Robustness of ODEnets trained only on non-perturbed images

The first question we are interested in is how robust ODEnets are against
perturbations if the model is only trained on original non-perturbed images.
We train CNNs and ODEnets to perform classification on three datasets and set
the weight decay parameters for all models to be 0.0005. We make sure that
both the well-trained ODEnets and CNN models have satisfactory performances on
original non-perturbed images, i.e., around 99.5% for MNIST, 95.0% for the
SVHN, and 80.0% for ImgNet10. Here, to calculate the robust accuracy, we first
identify and select the samples whose clean versions are correctly classified
by the obtained classifiers. Then, we compute the classification accuracy of
the adversarial examples on the set of the selected samples.

Since Gaussian noise is ubiquitous in modeling image degradation, we first
evaluated the robustness of the models in the presence of zero-mean random
Gaussian perturbations. It has also been shown that a deep model is vulnerable
to harmful adversarial examples, such as the FGSM
[goodfellow_explaining_2015]. We are also interested in how robust ODEnets are
in the presence of adversarial examples. The standard deviation $\sigma$ of
Gaussian noise and the $l_{\infty}$-norm $\epsilon$ of the FGSM attack for
each dataset are shown in Table 3.1.

Table 3.1: Robustness comparison of different models. We report their mean
classification accuracies (%) and standard deviations (mean $/$ std) on
perturbed images from the MNIST, the SVHN, and the ImgNet10 datasets. Two
types of perturbations are used—zero-mean Gaussian noise and FGSM adversarial
attack. The results show that ODEnets are much more robust in comparison to
CNN models.

| Gaussian noise | Adversarial attack
---|---|---
MNIST | $\sigma=\nicefrac{{50}}{{255}}$ | $\sigma=\nicefrac{{75}}{{255}}$ | $\sigma=\nicefrac{{100}}{{255}}$ | FGSM-0.15 | FGSM-0.3 | FGSM-0.5
CNN |  98.1$/$0.7 |  85.8$/$4.3 |  56.4$/$5.6 |  63.4$/$2.3 |  24.0$/$8.9 |  8.3$/$3.2
ODEnet |  98.7$/$0.6 |  90.6$/$5.4 |  73.2$/$8.6 |  83.5$/$0.9 |  42.1$/$2.4 |  14.3$/$2.1
SVHN | $\sigma=\nicefrac{{15}}{{255}}$ | $\sigma=\nicefrac{{25}}{{255}}$ | $\sigma=\nicefrac{{35}}{{255}}$ | FGSM-3/255 | FGSM-5/255 | FGSM-8/255
CNN |  90.0$/$1.2 |  76.3$/$2.7 |  60.9$/$3.9 |  29.2$/$2.9 |  13.7$/$1.9 |  5.4$/$1.5
ODEnet |  95.7$/$0.7 |  88.1$/$1.5 |  78.2$/$2.1 |  58.2$/$2.3 |  43.0$/$1.3 |  30.9$/$1.4
ImgNet10 | $\sigma=\nicefrac{{10}}{{255}}$ | $\sigma=\nicefrac{{15}}{{255}}$ | $\sigma=\nicefrac{{25}}{{255}}$ | FGSM-5/255 | FGSM-8/255 | FGSM-16/255
CNN |  80.1$/$1.8 |  63.3$/$2.0 |  40.8$/$2.7 |  28.5$/$0.5 |  18.1$/$0.7 |  9.4$/$1.2
ODEnet |  81.9$/$2.0 |  67.5$/$2.0 |  48.7$/$2.6 |  36.2$/$1.0 |  27.2$/$1.1 |  14.4$/$1.7

From the results in Table 3.1, we observe that the ODEnets demonstrate
superior robustness compared to CNNs for all types of perturbations. On the
MNIST dataset, in the presence of Gaussian perturbations with a large $\sigma$
of $\nicefrac{{100}}{{255}}$, the ODEnet produces much higher accuracy on
perturbed images compared to the CNN model (73.2% vs. 56.4%). For the FGSM-0.3
adversarial examples, the accuracy of ONEnet is around twice as high as that
of the CNN model. On the SVHN dataset, ODEnets significantly outperform CNN
models, e.g., for the FGSM-5/255 examples, the accuracy of the ODEnet is
43.0%, which is much higher than that of the CNN model (13.7%). On the
ImgNet10, for both cases of $\sigma=\nicefrac{{25}}{{255}}$ and FGSM-8/255,
ODEnet outperforms CNNs by a large margin of around 9%.

####  3.3.3 Robustness of ODEnets trained on original images together with
Gaussian perturbations

Training a model on original images together with their perturbed versions can
improve the robustness of the model. As mentioned previously, Gaussian noise
is commonly assumed to be present in real-world images. Synthesizing Gaussian
noise is also fast and easy. Thus, we add random Gaussian noise into the
original images to generate their perturbed versions. ODEnets and CNN models
are both trained on original images together with their perturbed versions.
The standard deviation of the added Gaussian noise is randomly chosen from
$\\{\nicefrac{{50}}{{255}},\nicefrac{{75}}{{255}},\nicefrac{{100}}{{255}}\\}$
on the MNIST dataset,
$\\{\nicefrac{{15}}{{255}},\nicefrac{{25}}{{255}},\nicefrac{{35}}{{255}}\\}$
on the SVHN dataset, and
$\\{\nicefrac{{10}}{{255}},\nicefrac{{15}}{{255}},\nicefrac{{25}}{{255}}\\}$
on the ImgNet10. All other hyperparameters are kept the same as above.

Table 3.2: Robustness comparison of different models. We report their mean
classification accuracies (%) and standard deviations (mean $/$ std) on
perturbed images from the MNIST, the SVHN, and the ImgNet10 datsets. Three
types of perturbations are used—zero-mean Gaussian noise, FGSM adversarial
attack, and PGD adversarial attack. The results show that ODEnets are more
robust compared to CNN models.

| Gaussian noise | Adversarial attack
---|---|---
MNIST | $\sigma=\nicefrac{{100}}{{255}}$ | FGSM-0.3 | FGSM-0.5 | PGD-0.2 | PGD-0.3
CNN | 98.7$/$0.1 | 54.2$/$1.1 | 15.8$/$1.3 | 32.9$/$3.7 | 0.0$/$0.0
ODEnet |  99.4$/$0.1 |  71.5$/$1.1 |  19.9$/$1.2 |  64.7$/$1.8 |  13.0$/$0.2
SVHN | $\sigma=\nicefrac{{35}}{{255}}$ | FGSM-5/255 | FGSM-8/255 | PGD-3/255 | PGD-5/255
CNN | 90.6$/$0.2 | 25.3$/$0.6 | 12.3$/$0.7 | 32.4$/$0.4 | 14.0$/$0.5
ODEnet |  95.1$/$0.1 |  49.4$/$1.0 |  34.7$/$0.5 |  50.9$/$1.3 |  27.2$/$1.4
ImgNet10 | $\sigma=\nicefrac{{25}}{{255}}$ | FGSM-5/255 | FGSM-8/255 | PGD-3/255 | PGD-5/255
CNN | 92.6$/$0.6 | 40.9$/$1.8 | 26.7$/$1.7 | 28.6$/$1.5 | 11.2$/$1.2
ODEnet | 92.6$/$0.5 |  42.0$/$0.4 |  29.0$/$1.0 |  29.8$/$0.4 |  12.3$/$0.6

The robustness of the models is evaluated under Gaussian perturbations, FGSM
adversarial examples, and PGD [madry_towards_2018] adversarial examples. The
latter is a stronger attacker compared to the FGSM. The $l_{\infty}$-norm
$\epsilon$ of the PGD attack for each dataset is shown in Table 3.2. Based on
the results, we observe that ODEnets consistently outperform CNN models on
both two datasets. On the MNIST dataset, the ODEnet outperforms the CNN
against all types of perturbations. In particular, for the PGD-0.2 adversarial
examples, the accuracy of the ODEnet (64.7%) is much higher than that of the
CNN (32.9%). Besides, for the PGD-0.3 attack, the CNN is completely misled by
the adversarial examples, but the ODEnet can still classify perturbed images
with an accuracy of 13.0%. On the SVHN dataset, ODEnets also show superior
robustness in comparison to CNN models. For all the adversarial examples,
ODEnets outperform CNN models by a margin of at least 10 percentage points. On
the ImgNet10 dataset, the ODEnet also performs better than CNN models against
all forms of adversarial examples.

####  3.3.4 Insights on the robustness of ODEnets

Fig. 3.2: No integral curves intersect. The integral curve starting from
$\widetilde{A_{1}}$ is always sandwiched between two integral curves starting
from $A_{1}$ and $A_{3}$.

From the results in Sections 3.3.2 and 3.3.3, we find ODEnets are more robust
compared to CNN models. Here, we attempt to provide an intuitive understanding
of the robustness of the neural ODE. In an ODEnet, given some datum, the FE
extracts an informative feature map from the datum. The neural ODE, serving as
the RM, takes as input the feature map and performs a nonlinear mapping. In
practice, we use the weight decay technique during training which regularizes
the norm of weights in the FE part, so that the change of feature map in terms
of a small perturbation on the input can be controlled. We aim to show that,
in the neural ODE, a small change on the feature map will not lead to a large
deviation from the original output associated with the feature map.

######  Theorem 3.3.1 (ODE integral curves never intersect.)

Let ${\bm{z}}_{1}(t)$ and ${\bm{z}}_{2}(t)$ be two solutions of the ODE in
(3.1) with different initial conditions, i.e.
${\bm{z}}_{1}(0)\neq{\bm{z}}_{2}(0)$. In (3.1), $f_{\theta}$ is continuous in
$t$ and globally Lipschitz continuous in ${\bm{z}}$. Then, it holds that
${\bm{z}}_{1}(t)\neq{\bm{z}}_{2}(t)$ for all $t\in[0,\infty)$.

To illustrate Theorem 3.3.1 [coddington1955theory; edwards1993elementary],
considering a simple 1-dimensional system in which the state is a scalar. As
shown in Figure 3.2, equation (3.1) has a solution $z_{1}(t)$ starting from
$A_{1}=(0,z_{1}(0))$, where $z_{1}(0)$ is the feature of some datum. Equation
(3.1) also has another two solutions $z_{2}(t)$ and $z_{3}(t)$, whose starting
points $A_{2}=(0,z_{2}(0))$ and $A_{3}=(0,z_{3}(0))$, both of which are close
to $A_{1}$. Suppose $A_{1}$ is between $A_{2}$ and $A_{3}$. By Theorem 3.3.1,
we know that the integral curve $z_{1}(t)$ is always sandwiched between the
integral curves $z_{2}(t)$ and $z_{3}(t)$.

Now, let $\epsilon<\min\\{|z_{2}(0)-z_{1}(0)|,|z_{3}(0)-z_{1}(0)|\\}$.
Consider a solution $\widetilde{z}_{1}(t)$ of equation (3.1). The integral
curve $\widetilde{z}_{1}(t)$ starts from a point
$\widetilde{A}_{1}=(0,\widetilde{z}_{1}(0))$. The point $\widetilde{A}_{1}$ is
in the $\epsilon$-neighborhood of $A_{1}$ with
$|\widetilde{z}_{1}(0)-z_{1}(0)|<\epsilon$. By Theorem 3.3.1, we know that
$|\widetilde{z}_{1}(T)-z_{1}(T)|\leq|z_{3}(T)-z_{2}(T)|$. In other words, if
any perturbation smaller than $\epsilon$ is added to the scalar $z_{1}(0)$ in
$A_{1}$, the deviation from the original output $z_{1}(T)$ is bounded by the
distance between $z_{2}(T)$ and $z_{3}(T)$. In contrast, in a CNN model, there
is no such bound on the deviation from the original output. Thus, we opine
that due to this non-intersecting property, ODEnets are intrinsically robust.

###  3.4 TisODE: boosting the robustness of neural ODEs

In the previous section, we presented an empirical study on the robustness of
ODEnets and observed that ODEnets are more robust compared to CNN models. In
this section, we explore how to boost the robustness of the vanilla neural ODE
model further. This motivates the proposal of _time-invariant steady neural
ODEs_ (TisODEs).

####  3.4.1 Time-invariant steady neural ODEs

Fig. 3.3: An illustration of the time-invariant property of ODEs. We can see
that the curve $\widetilde{{\bm{z}}}_{1}(t)$ is exactly the horizontal
translation of ${\bm{z}}_{1}(t)$ on the interval $[T^{\prime},\infty)$.

From the discussion in Section 3.3.4, the key to improving the robustness of
neural ODEs is to control the difference between neighboring integral curves.
By Grownall’s inequality [howard1998gronwall] (see Theorem 3.7.1 in the
Appendix), we know that the difference between two terminal states is bounded
by the difference between initial states multiplied by the exponential of the
dynamics’ Lipschitz constant. However, it is very difficult to bound the
Lipschitz constant of the dynamics directly. Alternatively, we propose to
achieve the goal of controlling the output deviation by following two steps:
(i) removing the time dependence of the dynamics and (ii) imposing a certain
steady-state constraint.

In the neural ODE characterized by equation (3.1), the dynamics
$f_{\theta}({\bm{z}}(t),t)$ depends on both the state ${\bm{z}}(t)$ at time
$t$ and the time $t$ itself. In contrast, if the neural ODE is modified to be
time-invariant, the time dependence of the dynamics is removed. Consequently,
the dynamics depends only on the state ${\bm{z}}$. So, we can rewrite the
dynamics function as $f_{\theta}({\bm{z}})$, and the neural ODE is
characterized as

$\begin{cases}\displaystyle\frac{\mathrm{d}{\bm{z}}(t)}{\mathrm{d}t}=f_{\theta}({\bm{z}}(t));\\\
{\bm{z}}(0)={\bm{z}}_{\text{in}};\\\ {\bm{z}}_{\text{out}}={\bm{z}}(T).\\\
\end{cases}$ (3.2)

Let ${\bm{z}}_{1}(t)$ be a solution of (3.2) on $[0,\infty)$ and $\epsilon>0$
be a small positive value. We define the set
${\mathbb{M}}_{1}=\\{({\bm{z}}_{1}(t),t)|t\in[0,T],\|{\bm{z}}_{1}(t)-{\bm{z}}_{1}(0)\|\leq\epsilon\\}$.
This set contains all points on the curve of ${\bm{z}}_{1}(t)$ during $[0,T]$
that are also inside the $\epsilon$-neighborhood of ${\bm{z}}_{1}(0)$. For
some element $({\bm{z}}_{1}(T^{\prime}),T^{\prime})\in{\mathbb{M}}_{1}$, let
$\widetilde{{\bm{z}}}_{1}(t)$ be the solution of (3.2) which starts from
$\widetilde{{\bm{z}}}_{1}(0)={\bm{z}}_{1}(T^{\prime})$. Then we have

$\widetilde{{\bm{z}}}_{1}(t)={\bm{z}}_{1}(t+T^{\prime})$ (3.3)

for all $t$ in $[0,\infty)$. The property shown in equation (3.3) is known as
the _time-invariant property_. It indicates that the integral curve
$\widetilde{{\bm{z}}}_{1}(t)$ is the $-T^{\prime}$ shift of ${\bm{z}}_{1}(t)$
(Figure 3.3).

We can regard $\widetilde{{\bm{z}}}_{1}(0)$ as a slightly perturbed version of
${\bm{z}}_{1}(0)$, and we are interested in how large the difference between
$\widetilde{{\bm{z}}}_{1}(T)$ and ${\bm{z}}_{1}(T)$ is. In a robust model, the
difference should be small. By equation (3.3), we have
$\|\widetilde{{\bm{z}}}_{1}(T)-{\bm{z}}_{1}(T)\|=\|{\bm{z}}_{1}(T+T^{\prime})-{\bm{z}}_{1}(T)\|$.
Since $T^{\prime}\in[0,T]$, the difference between ${\bm{z}}_{1}(T)$ and
$\widetilde{{\bm{z}}}_{1}(T)$ can be bounded as follows,

$\displaystyle\|\widetilde{{\bm{z}}}_{1}(T)\\!-\\!{\bm{z}}_{1}(T)\|\\!$
$\displaystyle=\\!\left\|\int_{T}^{T+T^{\prime}}f_{\theta}({\bm{z}}_{1}(t))\,\mathrm{d}t\right\|$
$\displaystyle\\!\leq\\!\left\|\int_{T}^{T+T^{\prime}}|f_{\theta}({\bm{z}}_{1}(t))|\,\mathrm{d}t\right\|$
$\displaystyle\\!\leq\\!\left\|\int_{T}^{2T}|f_{\theta}({\bm{z}}_{1}(t))|\,\mathrm{d}t\right\|,$
(3.4)

where all norms are $\ell_{2}$ norms and $|f_{\theta}|$ denotes the element-
wise absolute operation of a vector-valued function $f_{\theta}$. That is to
say, the difference between $\widetilde{{\bm{z}}}_{1}(T)$ and
${\bm{z}}_{1}(T)$ can be bounded by only using the information of the curve
${\bm{z}}_{1}(t)$. For any $t^{\prime}\in[0,T]$ and element
$({\bm{z}}_{1}(t^{\prime}),t^{\prime})\in{\mathbb{M}}_{1}$, consider the
integral curve that starts from ${\bm{z}}_{1}(t^{\prime})$. The difference
between the output state of this curve and ${\bm{z}}_{1}(T)$ satisfies
inequality (3.4).

Therefore, we propose to add an additional term $L_{\mathrm{ss}}$ to the loss
function when training the time-invariant neural ODE:

$L_{\mathrm{ss}}=\sum_{i=1}^{N}\left\|\int_{T}^{2T}|f_{\theta}({\bm{z}}_{i}(t))|\,\mathrm{d}t\right\|,$
(3.5)

where $N$ is the number of samples in the training set and ${\bm{z}}_{i}(t)$
is the solution whose initial state equals to the feature of the
$i^{\mathrm{th}}$ sample. The regularization term $L_{\mathrm{ss}}$ is termed
as the steady-state loss. This terminology “steady state” is borrowed from the
dynamical systems literature. In a stable dynamical system, the states
stabilize around a fixed point, known as the steady-state, as time tends to
infinity. If we can ensure that $L_{\mathrm{ss}}$ is small, for each sample,
the outputs of all the points in ${\mathbb{M}}_{i}$ will stabilize around
${\bm{z}}_{i}(T)$. Consequently, the model is robust. This modification of the
neural ODE is dubbed _Time-invariant steady neural ODE_.

####  3.4.2 Evaluating robustness of TisODE-based classifiers

Table 3.3: Classification accuracy (mean $/$ std in %) on perturbed images
from MNIST, SVHN and ImgNet10. To evaluate the robustness of classifiers, we
use three types of perturbations, namely zero-mean Gaussian noise with
standard deviation $\sigma$, FGSM attack and PGD attack. From the results, the
proposed TisODE effectively improve the robustness of the vanilla neural ODE.

| Gaussian noise | Adversarial attack
---|---|---
MNIST | $\sigma=\nicefrac{{100}}{{255}}$ | FGSM-0.3 | FGSM-0.5 | PGD-0.2 | PGD-0.3
CNN | 98.7$/$0.1 | 54.2$/$1.1 | 15.8$/$1.3 | 32.9$/$3.7 | 0.0$/$0.0
ODEnet | 99.4$/$0.1 | 71.5$/$1.1 | 19.9$/$1.2 | 64.7$/$1.8 | 13.0$/$0.2
TisODE |  99.6$/$0.0 |  75.7$/$1.4 |  26.5$/$3.8 |  67.4$/$1.5 |  13.2$/$1.0
SVHN | $\sigma=\nicefrac{{35}}{{255}}$ | FGSM-5/255 | FGSM-8/255 | PGD-3/255 | PGD-5/255
CNN | 90.6$/$0.2 | 25.3$/$0.6 | 12.3$/$0.7 | 32.4$/$0.4 | 14.0$/$0.5
ODEnet |  95.1$/$0.1 | 49.4$/$1.0 | 34.7$/$0.5 | 50.9$/$1.3 | 27.2$/$1.4
TisODE | 94.9$/$0.1 |  51.6$/$1.2 |  38.2$/$1.9 |  52.0$/$0.9 |  28.2$/$0.3
ImgNet10 | $\sigma=\nicefrac{{25}}{{255}}$ | FGSM-5/255 | FGSM-8/255 | PGD-3/255 | PGD-5/255
CNN | 92.6$/$0.6 | 40.9$/$1.8 | 26.7$/$1.7 | 28.6$/$1.5 | 11.2$/$1.2
ODEnet | 92.6$/$0.5 | 42.0$/$0.4 | 29.0$/$1.0 | 29.8$/$0.4 | 12.3$/$0.6
TisODE |  92.8$/$0.4 |  44.3$/$0.7 |  31.4$/$1.1 |  31.1$/$1.2 |  14.5$/$1.1

Here, we conduct experiments to evaluate the robustness of our proposed
TisODE, and compare TisODE-based models with the vanilla ODEnets. We train all
models with original non-perturbed images together with their Gaussian
perturbed versions. The regularization parameter for the steady-state loss
$L_{\mathrm{ss}}$ is set to be $0.1$. All other hyperparameters are exactly
the same as those in Section 3.3.3. We ensure that the well-trained CNNs,
ODEnets, and the TisODE-based classifiers all have similar performances on the
orginal non-perturbed images, i.e., around 99.5% for MNIST, 95.0% for SVHN,
and 80.0% for ImgNet10.

From the results in Table 3.3, we can see that our proposed TisODE-based
models are clearly more robust compared to vanilla ODEnets. On the MNIST
dataset, when combating FGSM-0.3 attacks, the TisODE-based models outperform
vanilla ODEnets by more than $4$ percentage points. For the FGSM-0.5
adversarial examples, the accuracy of the TisODE-based model is $6$ percentage
points better. On the SVHN dataset, the TisODE-based models perform better in
terms of all forms of adversarial examples. On the ImgNet10 dataset, the
TisODE-based models also outperform vanilla ODE-based models on all types of
perturbations. In the presence of FGSM and PGD-5/255 examples, the accuracies
are enhanced by more than $2$ percentage points.

####  3.4.3 TisODE - A generally applicable drop-in technique for improving
the robustness of deep networks

In view of the excellent robustness of the TisODE, we claim that the proposed
TisODE can be used as a general drop-in module for improving the robustness of
deep networks. We support this claim by showing the TisODE can work in
conjunction with other state-of-the-art techniques and further boost the
models’ robustness. These techniques include the feature denoising (FDn)
method [xie_feature_2019] and the input randomization (IR) method
[xie_mitigating_2018]. We conduct experiments on the MNIST and SVHN datasets.
All models are trained with original non-perturbed images together with their
Gaussian perturbed versions. We show that models using the FDn/IRd technique
becomes much more robust when equipped with the TisODE. In the FDn
experiments, the dot-product non-local denoising layer [xie_feature_2019] is
added to the head of the fully-connected classifier.

Table 3.4: Classification accuracy (mean $/$ std in %) on perturbed images
from MNIST and SVHN. We evaluate against three types of perturbations, namely
zero-mean Gaussian noise with standard deviation $\sigma$, FGSM attack and PGD
attack. From the results, upon the CNNs modified with FDn and IRd, using
TisODE can further improve the robustness.

| Gaussian noise | Adversarial attack
---|---|---
MNIST | $\sigma=\nicefrac{{100}}{{255}}$ | FGSM-0.3 | FGSM-0.5 | PGD-0.2 | PGD-0.3
CNN | 98.7$/$0.1 | 54.2$/$1.1 | 15.8$/$1.3 | 32.9$/$3.7 | 0.0$/$0.0
CNN-FDn | 99.0$/$0.1 | 74.0$/$4.1 | 32.6$/$5.3 | 58.9$/$4.0 | 8.2$/$2.6
TisODE-FDn |  99.4$/$0.0 | 80.6$/$2.3 | 40.4$/$5.7 | 72.6$/$2.4 | 28.2$/$3.6
CNN-IRd | 95.3$/$0.9 | 78.1$/$2.2 | 36.7$/$2.1 | 79.6$/$1.9 | 55.5$/$2.9
TisODE-IRd | 97.6$/$0.1 |  86.8$/$2.3 |  49.1$/$0.2 |  88.8$/$0.9 |  66.0$/$0.9
SVHN | $\sigma=\nicefrac{{35}}{{255}}$ | FGSM-5/255 | FGSM-8/255 | PGD-3/255 | PGD-5/255
CNN | 90.6$/$0.2 | 25.3$/$0.6 | 12.3$/$0.7 | 32.4$/$0.4 | 14.0$/$0.5
CNN-FDn | 92.4$/$0.1 | 43.8$/$1.4 | 31.5$/$3.0 | 40.0$/$2.6 | 19.6$/$3.4
TisODE-FDn |  95.2$/$0.1 | 57.8$/$1.7 | 48.2$/$2.0 | 53.4$/$2.9 | 32.3$/$1.0
CNN-IRd | 84.9$/$1.2 | 65.8$/$0.4 | 54.7$/$1.2 | 74.0$/$0.5 | 64.5$/$0.8
TisODE-IRd | 91.7$/$0.5 |  74.4$/$1.2 |  61.9$/$1.8 |  81.6$/$0.8 |  71.0$/$0.5

From Table 3.4, we observe that both FDn and IRd can effectively improve the
adversarial robustness of vanilla CNN models (CNN-FDn, CNN-IRd). Furthermore,
combining our proposed TisODE with FDn or IRd (TisODE-FDn, TisODE-IRd), the
adversarial robustness of the resultant model is significantly enhanced. For
example, on the MNIST dataset, the additional use of our TisODE increases the
accuracies on the PGD-0.3 examples by at least 10 percentage points for both
FDn (8.2% to 28.2%) and IRd (55.5% to 66.0%). However, on both MNIST and SVHN
datasets, the IRd technique improves the robustness against adversarial
examples, but its performance is worse on random Gaussian noise. With the help
of the TisODE, the degradation in the robustness against random Gaussian noise
can be effectively ameliorated.

###  3.5 Related works

In this section, we briefly review related works on the neural ODE and works
concerning improving the robustness of deep neural networks.

Neural ODE: The neural ODE [chen2018neural] method models the input and output
as two states of a continuous-time dynamical system by approximating the
dynamics of this system with trainable layers. Before the proposal of neural
ODE, the idea of modeling nonlinear mappings using continuous-time dynamical
systems was proposed in weinan2017proposal. lu2017beyond also showed that
several popular network architectures could be interpreted as the
discretization of a continuous-time ODE. For example, the ResNet
[he_deep_2016] and PolyNet [zhang2017polynet] are associated with the Euler
scheme and the FractalNet [larsson2016fractalnet] is related to the Runge-
Kutta scheme. In contrast to these discretization models, neural ODEs are
endowed with an intrinsic invertibility property, which yields a family of
invertible models for solving inverse problems [ardizzone2018analyzing], such
as the FFJORD [grathwohl2018ffjord].

Recently, many researchers have conducted studies on neural ODEs from the
perspectives of optimization techniques, approximation capabilities, and
generalization. Concerning the optimization of neural ODEs, the auto-
differentiation techniques can effectively train ODEnets, but the training
procedure is computationally and memory inefficient. To address this problem,
chen2018neural proposed to compute gradients using the adjoint sensitivity
method [pontryagin2018mathematical], in which there is no need to store any
intermediate quantities of the forward pass. Also in quaglino2019accelerating,
the authors proposed the SNet which accelerates the neural ODEs by expressing
their dynamics as truncated series of Legendre polynomials. Concerning the
approximation capability, dupont2019augmented pointed out the limitations in
approximation capabilities of neural ODEs because of the preserving of input
topology. The authors proposed an augmented neural ODE which increases the
dimension of states by concatenating zeros so that complex mappings can be
learned with simple flow. The most relevant work to ours concerns strategies
to improve the generalization of neural ODEs. In liu2019neural, the authors
proposed the neural stochastic differential equation (SDE) by injecting random
noise to the dynamics function and showed that the generalization and
robustness of vanilla neural ODEs could be improved. However, our improvement
on the neural ODEs is explored from a different perspective by introducing
constraints on the flow. We empirically found that our proposal and the neural
SDE can work in tandem to further boost the robustness of neural ODEs.

Robust Improvement: A straightforward way of improving the robustness of a
model is to smooth the loss surface by controlling the spectral norm of the
Jacobian matrix of the loss function [sokolic2017robust]. In terms of
adversarial examples [chen2017zoo; carlini_towards_2017], researchers have
proposed adversarial training strategies [elsayed2018adversarial;
tramer2017ensemble; madry_towards_2018] in which the model is fine-tuned with
adversarial examples generated in real-time. However, generating adversarial
examples is not computationally efficient, and there exists a trade-off
between the adversarial robustness and the performance on original non-
perturbed images [yan2018deep; tsipras_robustness_2018]. In wang2019resnets,
the authors model the ResNet as a transport equation, in which the adversarial
vulnerability can be interpreted as the irregularity of the decision boundary.
Consequently, a diffusion term is introduced to enhance the robustness of the
neural nets. Besides, there are also some works that propose novel