# An Analytic Framework for Robust Training of
Artificial Neural Networks

Ramin Barati
Department of Computer Engineering
Amirkabir University of Technology
Tehran, Iran
<EMAIL_ADDRESS>
&Reza Safabakhsh
Department of Computer Engineering
Amirkabir University of Technology
Tehran, Iran
<EMAIL_ADDRESS>
Mohammad Rahmati
Department of Computer Engineering
Amirkabir University of Technology
Tehran, Iran
<EMAIL_ADDRESS>

###### Abstract

The reliability of a learning model is key to the successful deployment of
machine learning in various industries. Creating a robust model, particularly
one unaffected by adversarial attacks, requires a comprehensive understanding
of the adversarial examples phenomenon. However, it is difficult to describe
the phenomenon due to the complicated nature of the problems in machine
learning. Consequently, many studies investigate the phenomenon by proposing a
simplified model of how adversarial examples occur and validate it by
predicting some aspect of the phenomenon. While these studies cover many
different characteristics of the adversarial examples, they have not reached a
holistic approach to the geometric and analytic modeling of the phenomenon.
This paper propose a formal framework to study the phenomenon in learning
theory and make use of complex analysis and holomorphicity to offer a robust
learning rule for artificial neural networks. With the help of complex
analysis, we can effortlessly move between geometric and analytic perspectives
of the phenomenon and offer further insights on the phenomenon by revealing
its connection with harmonic functions. Using our model, we can explain some
of the most intriguing characteristics of adversarial examples, including
transferability of adversarial examples, and pave the way for novel approaches
to mitigate the effects of the phenomenon.

##  1 Introduction

The state-of-the-art neural models are shown to suffer from the phenomenon of
adversarial examples, where a machine learning model is fooled to return an
undesirable output on particular inputs that an adversary carefully crafts.
The phenomenon is peculiar because it seems to affect neural networks in every
application globally and that the adversarial examples are invariant to
changes in network architecture, transferring from one network to another.
From the perspective of learning theory, the existence of these samples is
paradoxical since the nonrobust networks show acceptable, even super-human,
performance on the natural samples. In the literature, many attempts at
resolving this paradox have been made, each revealing a different facet of the
phenomenon.

Szegedy et al. [2014] explained the adversarial examples as small pockets in
the domain of the hypothesis, where the hypothesis fails to be correct due to
its highly nonlinear nature. In contrast, Goodfellow et al. [2015] proposed
that the phenomenon is a side-effect of a linear hypothesis in high
dimensions. Conversely, Ilyas et al. [2019] blamed useful nonrobust features
that are effective in dealing with natural samples; but are a hindrance when
the model is tested on adversarial examples. Barati et al. [2021] proposes
that all of these different and opposing perspectives unite under the banner
of pointwise convergence of the hypothesis to the optimal hypothesis. On a
separate thread, Tanay and Griffin [2016], Shamir et al. [2021] propose
geometrical descriptions of the phenomenon in which adversarial examples are
attributed to gemetrical interactions between the manifold of natural samples
and the decision boundary.

In this paper, we take the first steps for the analysis the phenomenon through
the lens of complex analysis. The use of complex variables and functions
enables us to consider the algebraic and the geometric properties of the
phenomenon simultaneously, leading to a rigorous framework to study the
phenomenon and a possible remedy for its effects. For this reason, we
introduce a novel approach in describing the phenomenon using complex-valued
hypotheses. We then introduce the space of holomorphic hypotheses and show
that, in the limit of infinite samples, all holomorphic hypotheses are forced
to converge to the same holomorphic hypotheses, explaining the transfer of
adversarial examples between analytic hypotheses. Finally, we will generalize
the results to real-valued hypothesis through the methods of calculus of
variation and differential equations.

##  2 Preliminaries

In this section, we provide a summary of the relevant concepts in learning
theory, and formally describe the objects of our study. A goal of learning
theory is to provide a way to decide on the learnability of a hypothesis class
$\mathcal{H}$ and quantize the hardness of a learning task for a learning rule
$\mathcal{A}$ with respect to $\mathcal{H}$. The complexity of a learning task
is decided by its sample complexity function. The sample complexity is an
integer-valued function that determines a lower bound on the size of the
training set relative to a desired error magnitude $\epsilon$ and an error
probability $\delta$. The training set is sampled from a domain set
$\mathcal{X}$ and is labeled by a labeling function
$t:\mathcal{X}\rightarrow\\{+1,-1\\}$.

A restricted and strong form of learning is the probably approximately correct
(PAC) learning. PAC learnability of $\mathcal{H}$ means that a learning rule
$\mathcal{A}$ exists that its sample complexity with respect to $\mathcal{H}$
is only dependent on $\epsilon$ and $\delta$. In other words, not matter the
task, the training set size is only proportional to the desired generalization
error.

A more relaxed notion of learnability is nonuniform learnability. The sample
complexity in nonuniform learning is dependent on the optimal hypothesis. In
other words, some hypotheses need a larger training set for $\mathcal{A}$ to
successfully pass the learning task requirements.

It is possible to further weaken and relax nonuniform learnability and allow
$\mathcal{A}$ to be consistent with respect to $\mathcal{H}$. Consistency is
characterized by the dependency of the sample complexity on the sample
distribution. Consequently, the learner needs an even larger training set to
deal with complications in sampling from $\mathcal{X}$ on top of the
complexity of a complicated learning goal.

The first account for the existence of optimal training sets and their
relation with adversarial examples phenomenon is due to Ilyas et al. [2019].
It was later shown by Barati et al. [2021] that the output of a consistent
learning rule could be weak against adversarial attacks in unexpectedly simple
learning tasks. Consequently, we argue that if $\mathcal{A}$ realizes
nonuniform learnability with respect to $\mathcal{H}$, then $\mathcal{A}$ is a
robust learner with respect to $\mathcal{H}$.

######  Theorem 2.1.

If a learning rule $\mathcal{A}$ is a nonuniform learner with respect to
$\mathcal{H}$, then it is robust.

######  Definition 2.2 (generalized ANN).

The hypothesis class of generalized artificial neural networks (ANNs)
$\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}$ consists of
functions from a domain set $\mathcal{X}$ to either $\mathbb{C}$ or
$\mathbb{R}$ which has a representation of the form

$f(x)=\int_{\Omega}h(\omega)s(x;\omega)\,dV(\omega)\quad
h\in\mathcal{H}(\Omega),s\in\sigma(\mathcal{X})$ (1)

in which $dV(\omega)$ is the volume differential of $\Omega$,
$\mathcal{H}(\Omega)\subseteq L^{2}(\Omega)$ is a hypothesis class on the
domain set $\Omega$, and $\sigma(\mathcal{X})\subset L^{2}(\mathcal{X})$ is a
family of activation functions on $\mathcal{X}$ parameterized by
$\omega\in\Omega$.

######  Lemma 2.3.

$\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}\subseteq\mathcal{H}(\mathcal{X})$
with equality being true when $\sigma(\mathcal{X})$ has the universal
approximation property.

The largest hypothesis class that we consider here is $L^{2}(\mathcal{X})$.
The main characteristic of functions in $L^{2}(\mathcal{X})$ is that they are
square-integrable. Formally, for a function $f\in L^{2}(\mathcal{X})$,

$\|f\|_{L^{2}(\mathcal{X})}=\Big{(}\int_{\mathcal{X}}|f(x)|^{2}\,dV(x)\Big{)}^{\frac{1}{2}}<\infty.$
(2)

######  Theorem 2.4.

$L^{2}(\mathcal{X})$ is nonuniform learnable.

######  Definition 2.5 (SVC learning rule).

Consider a training set $S=\\{(x_{n}\in\mathcal{X},t(x_{n}))\\}_{n=1}^{N}$ and
a hypothesis $f\in\mathcal{N}\big{(}L^{2}(\Omega),\sigma(\mathcal{X})\big{)}$.
The support vector classifier (SVC) learner solves the following program,
v12∥h∥^2_L^2(Ω) t_nf(x_n)≥1

######  Theorem 2.6.

The SVC learning rule is not nonuniform with respect to
$\mathcal{N}\big{(}L^{2}(\Omega),\sigma(\mathcal{X})\big{)}$, and the optimal
solution $h^{*}$ is,

$h^{*}(\omega)=\sum_{n=1}^{N}\lambda_{n}t_{n}\sigma(x_{n};\omega),$ (3)

in which $\\{\lambda_{n}\\}_{n=1}^{N}$ are the Lagrange multipliers of program
(2.5).

Considering theorem 2.6, we expect that the output of the SVC learner would be
weak against adversarial attacks in general.

######  Proposition 2.7.

The SVC learner of a hypothesis class
$\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}$ would
return the labeling function $t$ as the output in the infinite sample limit
unless it is agnostic to $t$.

Based on proposition 2.7, one may suggest that SVC learners with respect to
$L^{2}(\mathcal{X})$ are weak against black box attacks as well. We see that
if we want to take a full account of the adversarial examples phenomenon, we
need to formalize the conditions for transferability of adversarial examples.
Demontis et al. [2019] have proposed three metrics for measuring
transferability between a target $f\in\mathcal{H}(\mathcal{X})$ and a
surrogate $\hat{f}\in\hat{\mathcal{H}}(\mathcal{X})$. The first metric is the
size of the gradient of the loss function with respect to input
$\|\nabla_{x}\ell\|$ for the target hypothesis. $\|\nabla_{x}\ell\|$ basically
measures the robustness of the target model. The second metric is the cosine
distance between the gradient of the loss of the target $\nabla_{x}\ell$ and
of the surrogate $\nabla_{x}\hat{\ell}$. In other words, how aligned the
gradients are. The third metric is the variance of the loss landscape of the
surrogate hypothesis class

$\mathbb{E}_{\mathcal{X}}\big{[}\ell\big{(}t(x),x,\hat{f}\big{)}^{2}\big{]}-\mathbb{E}_{\mathcal{X}}\big{[}\ell\big{(}t(x),x,\hat{f}\big{)}\big{]}^{2},$
(4)

which compensates for the fact that the surrogate is not trained on the same
training set as the target. Inspired by the definition of a normal family of
functions, we introduce normal hypothesis classes.

######  Definition 2.8 (normal hypothesis class).

$\mathcal{H}(\mathcal{X})$ is a normal hypothesis class if for every
universally consistent learning rule $\mathcal{A}$ and all
$\epsilon,\delta\geq 0$ a number $N\in\mathbb{N}$ exists where for all sets of
training points $S\subseteq\mathcal{X}$ with size larger than $N$ we have with
probability $1-\delta$ that

$|\mathcal{A}(S)(x)-\mathcal{A}(\mathcal{X})(x)|\leq\epsilon.$ (5)

Based on the aforementioned metrics, we can see that if both
$\mathcal{H}(\mathcal{X})$ and the class of its derivatives
$\partial\mathcal{H}(\mathcal{X})$ are normal, and if
$\hat{f}\in\mathcal{H}(\mathcal{X})$, then an attacker would be confident that
the adversarial examples of $\hat{f}$ would transfer to a nonrobust $f$.

######  Theorem 2.9.

Consider a nonrobust target
$f\in\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma_{1}(X)\big{)}$ and a
surrogate
$\hat{f}\in\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma_{2}(X)\big{)}$. If
$\mathcal{H}(\Omega)$ and $\partial\mathcal{H}(\Omega)$ are normal and
$\sigma_{1}(\mathcal{X})$ and $\sigma_{2}(\mathcal{X})$ have the universal
approximation property, then the adversarial examples of $\hat{f}$ will
transfer to $f$.

######  Theorem 2.10.

$L^{2}(\mathcal{X})$ is not a normal hypothesis class.

Even though theorem 2.4 shows that it is possible to find a robust learning
rule for $L^{2}(\mathcal{X})$, we argue that this hypothesis class is not
appropriate for the analysis of the adversarial examples phenomenon in ANNs.
Since it is not a normal hypothesis class, the hypotheses in
$L^{2}(\mathcal{X})$ are not clustered together and it does not model the
transferability of the adversarial examples between ANNs with different
activation functions.

##  3 The space of holomorphic hypotheses

In section 2 we introduced
$\mathcal{N}\big{(}L^{2}(\Omega),\sigma(\mathcal{X})\big{)}$ and generalized
the SVC learning rule to this hypothesis class. However, we could infer that
$\mathcal{N}\big{(}L^{2}(\Omega),\sigma(\mathcal{X})\big{)}$ is not a normal
hypothesis class in general. In order to overcome this obstacle, we will turn
to a special subset of $L^{2}(\mathcal{X})$.

######  Definition 3.1 (the Bergman space).

Consider a compact and simply connected domain set
$\mathcal{X}\subset\mathbb{C}^{d}$. The Bergman space
$A^{2}(\mathcal{X})\subset L^{2}(\mathcal{X})$ is a reproducing kernel Hilbert
space defined as

$A^{2}(\mathcal{X})=\\{f\in\mathcal{O}(\mathcal{X})\,|\,\Big{(}\int_{\mathcal{X}}|f(z)|^{2}dV(z)\Big{)}^{\frac{1}{2}}<\infty\\}.$
(6)

######  Theorem 3.2.

$A^{2}(\mathcal{X})$ and $\partial A^{2}(\mathcal{X})$ are normal hypothesis
classes.

Theorem 3.2 is exactly the result that we are looking for.
$\mathcal{O}(\mathcal{X})$ is the space of holomorphic functions on $\Omega$.
There are different equivalent ways to characterize holomorphic functions. The
holomorphic functions are the solutions to the homogeneous Cauchy-Riemann
equations, or its counterpart in higher dimensions $\overline{\partial}$ (del-
bar) equations. Equivalently, the holomorphic functions are the functions that
are complex differentiable in each dimension of $z\in\mathcal{X}$. A third
characterization of $\mathcal{O}(\mathcal{X})$ is that these functions are
complex analytic and have a power series representation.
$\mathcal{O}(\mathcal{X})$ is also special because, unlike real analytic
functions, it is closed under uniform convergence. In one dimension,
holomorphic functions are also known as the conformal maps. However, it could
be shown that holomorphic functions can only be defined over the field of
complex numbers. Nevertheless, we argue that the simplicity of analysis in
$A^{2}(\mathcal{X})$ relative to other normal hypothesis classes, and its
unique set of properties would justify the switch to complex numbers, as long
as it results into methods that could be translated to a counterpart for real
numbers. Consequently, we have to extend the SVC learning rule to accommodate
for complex-valued hypotheses. We have replaced $x$ with $z$ to symbolize the
transition from the real to the complex number system.

######  Definition 3.3 (complex-valued classifier).

A complex-valued classifier $f:\Omega\rightarrow\mathbb{C}$ is a function
$f(z)=u(z)+iv(z)$ in which the real part $\Re[f(z)]=u(z)=0$ encodes the
geometrical position of the decision boundary and the imaginary part
$\Im[f(z)]=v(z)=0$ regresses through the geometrical position of the training
samples.

######  Definition 3.4 (the Bergman kernel).

The Bergman kernel $K_{\mathcal{X}}(z,\zeta)$ of a compact and simply
connected domain $\mathcal{X}$ is the unique function with the reproducing
property

$f(z)=\int_{\mathcal{X}}f(z)K(z,\zeta)dV(\zeta),\quad\forall f\in
A^{2}(\mathcal{X}).$ (7)

The reproducing property of the Bergman kernel of $\mathcal{X}$ in conjunction
with the fact that the optimal Bayes classifier achieves the minimum of the
0-1 loss function provides the means to define the infinite sample limit of
any learning rule on $A^{2}(\Omega)$ that is minimizing the complex 0-1 loss
function,

$\ell_{\mathbb{C}}(t,z,f)=\ell_{0-1}(t,z,\Re[f])+\ell_{MSE}(0,z,\Im[f]),$ (8)

independently from the details of the implementation or the training process.

######  Definition 3.5 (holomorphic optimal Bayes classifier).

The holomorphic optimal Bayes classifier is the orthogonal projection of the
optimal Bayes classifier $f_{\mathcal{D}}:\mathcal{X}\rightarrow\mathbb{R}$
into $A^{2}(\mathcal{X})$,

$o_{\mathcal{D}}(z)=\int_{\mathcal{X}}f_{\mathcal{D}}(\zeta)K_{\mathcal{X}}(z,\zeta)\,dV(\zeta),$
(9)

Following Shamir et al. [2021] and Tanay and Griffin [2016], we are interested
to know how would $\mathcal{S}(f)=\\{z\in\mathcal{X}\,|\,\Im[f(z)]=0\\}$ and
$\mathcal{C}(f)=\\{z\in\mathcal{X}\,|\,\Re[f(z)]=0\\}$ interact with the
geometrical position of training samples. Geometrical properties of
holomorphic functions would enable us to infer the geometrical relation
between the manifold of the samples and the decision boundary by studying the
image of of the manifold in the range space of the hypothesis. This property
of the holomorphic hypotheses would prove to be key to finding a robust
learning rule for $A^{2}(\mathcal{X})$.

######  Definition 3.6 (complex-valued SVC learning rule).

Consider a dataset
$S=\\{(z_{n},t_{n})\in\mathcal{X}\times\\{-1,+1\\}\\}_{n=1}^{N}$ and a
hypothesis $f\in\mathcal{N}\big{(}A^{2}(\Omega),\sigma(\mathcal{X})\big{)}$.
The complex-valued SVC learner solves the following program,
h,ξ12∥h∥_A^2(Ω)^2+C∑_n=1^Nξ_n ξ_n≥0 t_nℜ[f(¯z_n)]≥1 - ξ_n, ξ_n≥ℑ[f(¯z_n)])
ℑ[f(¯z_n)]≥-ξ_n in which $\\{\xi_{n}\\}_{n=1}^{N}$ are slack variables
introduced to allow for soft margins. If we use the training sample $z$
instead of its complex conjugate $\overline{z}$ then the solution of the
program would be anticonformal in $z$.

(a) graph of the complex-valued hypotheses on the unit disk $\mathbb{D}$.

(b) graph of the complex-valued hypotheses on the unit circle
$\partial\mathbb{D}$.

(c) image of $\mathcal{X}=\partial\mathbb{D}$ under the complex-valued
hypotheses.

Figure 1: Visualizations of $o_{\mathcal{D}}$ and the optimal hypotheses of
the hinge loss function for $S_{30}$. We can see that even though the
hypotheses are optimizing a different loss function than the 0-1 loss, they
show certain similarities with $o_{\mathcal{D}}$.

##  4 Robust classification for holomorphic hypotheses

In this section, we will develop a robust learning rule with respect to
$A^{2}(\mathcal{X})$. To do so, we first examine a toy problem to get an
intuition for how adversarial examples occur in holomorphic hypotheses, and
then continue to introduce the proposed robust learning rule.

To start, we will try to classify the unit disk $\mathbb{D}$ into distinct
halves in which

$t(z)=\mathrm{sign}(\Re[z])\quad z\in\mathbb{D},$ (10)

is the labeling function. We will choose our training samples to be the set

$S_{n}=\Big{\\{}\big{(}z,t(z)\big{)}\,|\,z=e^{i\frac{k}{n}2\pi}\quad
k=0,\cdots,n-1\Big{\\}}.$ (11)

We will use the orthonormal polynomial basis of the unit disk as features,

$\varphi_{k}(z)=\sqrt{\frac{k+1}{\pi}}z^{k}.$ (12)

We will visualize a holomorphic function in three ways. First, we use a domain
coloring technique and graph the hypotheses on $\mathbb{D}$. The hue of a
color represent the angle, and the saturation represent the magnitude of a
complex number. We will also plot the contours of the real and imaginary parts
of the hypotheses in the same graph using white and black lines, respectively.
Second, we will graph the real and the imaginary parts of the hypotheses on
the unit circle $\mathbb{T}=\partial\mathbb{D}$. $\mathbb{T}$ is the set in
which all the training points of $S_{n}$ are sampled from. Third, we will
graph the image of $\mathbb{T}$ in the range space of the hypotheses. In the
range space of a complex-valued hypothesis, the decision boundary
$\mathcal{C}(f)$ is represented by the imaginary axis, and the real axis
represent the curve that has regressed through the training points
$\mathcal{S}(f)$.

We have visualized the analogue of holomorphic optimal Bayes classifier
$o_{\mathcal{D}}$ in the left column of figure 1. However, we have made use of
the Szegő kernel to project the labeling function $t$ instead of the Bergman
projection of $t$. The middle column of figure 1 depicts the output of
applying the complex-valued SVC learning rule to $S_{30}$ using the
orthonormal basis of $A^{2}(\mathbb{D})$ as features. As we have demonstrated
in figure 1, the solution to program (3.6) is not robust with respect to
$A^{2}(\mathbb{D})$. Nevertheless, due to the normality of the holomorphic
functions, we can see that the learned hypothesis resembles $o_{\mathcal{D}}$,
even though they are not learned through the same learning rule, or make use
of the same hypothesis class for that matter.

Looking at figure 1, we can see that $o_{\mathcal{D}}$ has two logarithmic
branch points on $i$ and $-i$. With this observation in mind, it is no wonder
that approximating $o_{\mathcal{D}}$ is troublesome for learning rules.
Furthermore, we see that the natural boundary of the nonrobust hypothesis has
advanced inside of $\mathbb{D}$. This advancement has caused the learners
output to be nonrobust in a neighborhood of $\mathbb{T}$. This observation is
common to nonrobust learning rules with respect to $A^{2}(\mathcal{X})$.

######  Theorem 4.1.

Let $f\in A^{2}(\mathcal{X}\subset\mathbb{C}^{d})$ be the output of a
nonrobust learning rule, then $f$ is robust on a dense open subset of
$\mathcal{X}$.

Theorem 4.1 shows that even though our choice of $A^{2}(\mathcal{X})$ as a
replacement for $L^{2}(\mathcal{X})$ was mainly derived by the transferability
of adversarial examples, it also correctly predicts the apparent paradox in
the existence of the adversarial examples in ANNs, where the nonrobust
hypothesis appears to be robust almost everywhere.

Figure 1(c) graphs the image of $\mathbb{T}$ in the range space of the
hypotheses $f(\mathbb{T})$. Due to the singularities of the holomorphic
projection of $t$, $o_{\mathcal{D}}(\mathbb{T})$ passes through the point at
the infinity. Consequently, $o_{\mathcal{D}}(\mathbb{T})$ consists of two
parallel lines; the trapezoidal shape in the figure is caused by the fact that
we cannot in practice reach the point at the infinity. Looking at the
nonrobust hypothesis in Figure 1(c), we can see that the image of $\mathbb{T}$
passes through the decision boundary quite a few times. The figure shows that
$f(\mathbb{T})$ of the nonrobust hypothesis is longer than necessary. In other
words, it is possible to find a holomorphic hypothesis that achieves the same
loss on the training set with $f(\mathbb{T})$ not passing through the decision
boundary so many times. We can repeat the same argument for any other curve
inside $\mathbb{D}$. Thus, we argue that if the output $f$ of $\mathcal{A}$
with respect to $A^{2}(\mathcal{X})$ minimizes the area covered by the image
of $\mathcal{X}$ under $f$, then it is robust.

######  Theorem 4.2.

A learning rule $\mathcal{A}$ with respect to $A^{2}(\mathcal{X})$ that
minimizes the Dirichlet energy of its output,

$E[f]=\int_{\mathcal{X}}\|\nabla f(z)\|^{2}\,dV(z),$ (13)

is robust.

######  Definition 4.3 (robust (complex-valued) SVC learning rule).

The robust (complex-valued) SVC learning rule is the same as the (complex-
valued) SVC learning rule, but minimizes $\|\nabla
h\|_{A^{2}(\mathcal{X})}^{2}$ ($\|\nabla h\|_{L^{2}(\mathcal{X})}^{2}$)
instead of $\|h\|_{A^{2}(\mathcal{X})}^{2}$
($\|h\|_{L^{2}(\mathcal{X})}^{2}$).

######  Definition 4.4 (harmonic features).

A set of non-constant features $\\{\varphi_{k}\\}_{1}^{\infty}$ is harmonic if

$\int_{\mathcal{X}}\nabla\varphi_{j}(z)\cdot\nabla\varphi_{k}(z)\,dV(z)=\delta_{jk},$
(14)

in which $\delta_{jk}$ is the Kronecker delta.

Given the tuning matrix

$\Sigma_{jk}=\int_{\mathcal{X}}\nabla\varphi_{j}(z)\cdot\nabla\varphi_{k}(z)\,dV(z),$
(15)

we can transform any set of features to its harmonic counterpart,

$\varphi^{*}=\Sigma^{-\frac{1}{2}}\varphi.$ (16)

We emphasize that $\Sigma$ is positive definite by definition. Thus, a unique
square root of $\Sigma$ exists and it is invertible.

######  Theorem 4.5.

$\ell^{2}$ regularization of a harmonic hypothesis minimizes its Dirichlet
energy.

We have repeated the toy experiment using the harmonic basis functions of
$A^{2}(\mathcal{X})$, and reported the results in the right column of figure
1. We can see that the harmonic hypothesis is robust as expected. Thus, we
have succeeded in finding a robust learning rule for $A^{2}(\mathcal{X})$.

##  5 Robust learning of ANNs

In this section, we will apply the results of section 4 to ANNs in a limited
manner to demonstrate the feasibility and applicability of our approach.
First, we put the proposed framework to use and describe how adversarial
examples occur in ANNs.

Figure 2: The output of the robust and nonrobust SVC learning rules for
$\mathcal{N}\big{(}A^{2}(\mathbb{D}),\mathrm{ReLU}([0,1])\big{)}$.

We already know that the orthonormal basis for $A^{2}(\Omega)$ would not be
robust on the boundary of $\Omega$. In other words, the learning rule would
fail to find the robust coefficients of neurons on the boundary of $\Omega$.
When we use the harmonic bases on the other hand, the region of convergence of
$h(\omega)$ would cover $\partial\Omega$ and the output of the learning rule
would be robust. We have computed the first 30 orthonormal and the harmonic
projection $\psi_{n}$ of the activation function
$\mathrm{ReLU}\big{(}\Re[\omega]x+\Im[\omega]\big{)}$ for $x\in[0,1]$, and
applied the complex-valued SVC learning rule of
$\mathcal{N}\big{(}A^{2}(\mathbb{D}),\mathrm{ReLU}(\mathcal{X})\big{)}$ to the
toy example of Barati et al. [2021]. We have reported the results of the
experiment in figure 2. It is clear from the figure that the harmonic ANN is
very robust. However, it is also clear that the output of the learner has not
maximized the margin. The problem persists for
$\mathcal{N}\big{(}L^{2}(\mathbb{D}),\sigma(\mathcal{X})\big{)}$ as well. We
believe that the most likely reason is that we have not chosen the best domain
set $\Omega$ to parameterize the neurons and as a result the learners output
is biased. However, we believe that a satisfying study of the parameter domain
sets $\Omega$ of generalized ANNs is out of the scope of the current paper,
and we leave it for future studies.

######  Theorem 5.1.

The output of the robust SVC learning rule with respect to
$\mathcal{N}\big{(}L^{2}(\Omega),\sigma(\mathcal{X})\big{)}$ satisfies the
following partial differential equation (PDE)

$-\Delta h(\omega)=\sum_{n=1}^{N}\lambda_{n}t_{n}s(x_{n};\omega),$ (17)

in which $\Delta$ is the Laplace operator.

Theorem 5.1 shows that training a robust ANN is the same as solving a PDE.
Based on the fact that we are minimizing the magnitude of the gradient of $h$,
it is natural to assume the Neumann boundary condition. This observation
suggests that robust classifiers are a normal family of functions. The
fundamental solution $\Phi$ of the Laplace operator could be used to find $h$
as a function of $\\{\lambda_{n}\\}_{n=1}^{N}$,

$h(\omega)=\sum_{n=1}^{N}\lambda_{n}t_{n}\int_{\Omega}\Phi(\omega-w)s(x_{n};w)\,dV(w).$
(18)

######  Definition 5.2 (harmonic activation function).

A family of activation functions $\mathfrak{S}(\mathcal{X})$ is harmonic if

$-\Delta\mathfrak{s}(x;\omega)=s(x;\omega)\quad\mathfrak{s}\in\mathfrak{S}(\mathcal{X}),s\in\sigma(\mathcal{X}).$
(19)

in which $\sigma(\mathcal{X})$ is a family of activation functions.

######  Theorem 5.3.

Assume that

$-\Delta\mathfrak{s}(x;\omega)=\mathfrak{s}(x;\omega)\quad\mathfrak{s}\in\mathfrak{S}(\mathcal{X}).$
(20)

Then, $L^{2}$ regularization of $h(\omega)$ of a generalized ANN
$f\in\mathcal{N}\big{(}L^{2}(\Omega),\mathfrak{S}(\mathcal{X})\big{)}$ is the
same as minimizing its Dirichlet energy.

Theorem 5.3 is the final result of this paper. We have managed to find a
robust learning rule for a generalized notion of ANNs. Nevertheless, finding a
family of activation functions that both play well with $\Phi$ and is not a
family of radial functions needs further study.

##  6 Conclusion

In this paper we introduced a general framework for training robust ANN
classifiers. Through holomorphicity, our framework provides a geometrical
interpretations for the adversarial examples phenomenon. In doing so, we have
made a plethora of theorems that described and explained the unique features
of the phenomenon. To the best of our knowledge, we have provided the first
method that makes use of calculus of variations and differential equations to
tackle the challenge of robust training of ANNs.

Nevertheless, the analysis proved that training robust classifiers is not an
easy task, and we need to solve an intermediate PDE, either implicitly or
explicitly. As a result, we have left the actual implementation of the
proposed method to future, when an in depth understanding of the appropriate
parameter domain sets and harmonic activation functions in generalized ANNs is
achieved.

The main point of distinction between the proposed framework and the competing
proposals is in the use of complex analysis to simplify the analysis. Even
though analytic properties of the hypotheses has been exploited before in the
literature to derive various methods of defence and attack, e.g. Wen [2022],
the use of holomorphic functions and complex analysis in machine learning have
been sparse Barkatou and Jaroschek [2018], Heilman et al. [2015], Sarma et al.
[2019]. We believe the reason to be that applying complex analysis to machine
learning is an interdisciplinary effort by nature.

Goodfellow et al. [2015] were the first to show that nonrobust ANNs are weak
to analytic attacks, and attributed the phenomenon to something that they
called linearity. Tanay and Griffin [2016] refuted this claim by showing that
linear classifiers could be robust. We answer the apparent paradox between
linear and nonlinear nature of the phenomenon by proposing that both positions
are in some sense correct, and that the phenomenon would be better described
by the analytic properties of the robust hypothesis.

Hein and Andriushchenko [2017] proposed a certificate of robustness for
differentiable classifiers. The same certificate could be used for analytic
functions. The certificate could be improved by considering the Taylor series
expansion of the classifier around a test sample. Due to the Abel-Ruffini
theorem, it would be impossible to devise a finite algorithm for computing a
certificate that is tighter than the certificate for the 4th degree Taylor
approximation. Nevertheless, it is still possible to approximate the
certificate using an infinite algorithm, e.g. perturbation theory.

From a geometric point of view, our proposal is very well aligned with the
dimpled manifold model of Shamir et al. [2021]. Regarding the boundary tilting
perspective of Tanay and Griffin [2016], it seems that the nonrobust
classifier in figure 1(c) has managed to cross the decision boundary at right
angles, and there seems to be exceptions to the boundary tilting perspective
in nonlinear cases. Paknezhad et al. [2021] hypothesises that the robust
classifier would maximize the margin. We did not make an attempt to analyze
the learning rule from the perspective of maximum margin classification.
Nonetheless, the peculiar shape of the image of $\mathbb{T}$ under the robust
hypothesis in figure 1(c) suggests that the proposed robust SVC learning rule
also maximizes the margin in some sense.

Our model for the transfer of adversarial examples is mostly similar to the
proposal of Goodfellow et al. [2015], in which the reason for the transfer of
adversarial examples are deemed to be that the models converge to the optimal
linear hypothesis. Papernot et al. [2016] shows that existing machine learning
approaches are in general vulnerable to systematic black-box attacks
regardless of their structure, showing that transferable adversarial examples
are common in machine learning models. Here, we have defined the holomorphic
optimal Bayes classifier and given a formal definition of how transfer occurs.
Ilyas et al. [2019] moves the blame to hidden patterns in the input, which our
proposal does not align with. Inkawhich et al. [2019] relates the transfer of
the adversarial examples to the learned features as well.

There are multiple instances of the use of gradient information through out
the literature Ros and Doshi-Velez [2018], Paknezhad et al. [2021]. The main
contrast between our proposal is that we manage to relate gradient
regularization with the geometry of $f(\mathcal{X})$ and further reveal how
gradient regularization is related with the Dirichlet energy of the
hypothesis.

## References

  * Szegedy et al. [2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  In Yoshua Bengio and Yann LeCun, editors, _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , 2014.  URL http://arxiv.org/abs/1312.6199.
  * Goodfellow et al. [2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In Yoshua Bengio and Yann LeCun, editors, _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6572.
  * Ilyas et al. [2019] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry.  Adversarial examples are not bugs, they are features.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 125–136, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/e2c420d928d4bf8ce0ff2ec19b371514-Abstract.html.
  * Barati et al. [2021] Ramin Barati, Reza Safabakhsh, and Mohammad Rahmati.  Towards explaining adversarial examples phenomenon in artificial neural networks.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pages 7036–7042, 2021.  doi: 10.1109/ICPR48806.2021.9412367.
  * Tanay and Griffin [2016] Thomas Tanay and Lewis D. Griffin.  A boundary tilting persepective on the phenomenon of adversarial examples.  _CoRR_ , abs/1608.07690, 2016.  URL http://arxiv.org/abs/1608.07690.
  * Shamir et al. [2021] Adi Shamir, Odelia Melamed, and Oriel BenShmuel.  The dimpled manifold model of adversarial examples in machine learning.  _CoRR_ , abs/2106.10151, 2021.  URL https://arxiv.org/abs/2106.10151.
  * Demontis et al. [2019] Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli.  Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks.  In _28th USENIX Security Symposium (USENIX Security 19)_ , pages 321–338, Santa Clara, CA, August 2019. USENIX Association.  ISBN 978-1-939133-06-9.  URL https://www.usenix.org/conference/usenixsecurity19/presentation/demontis.
  * Wen [2022] Zerui Wen.  Fourier attack – a more efficient adversarial attack method.  In _The 6th International Conference on Control Engineering and Artificial Intelligence_ , CCEAI 2022, page 125–130, New York, NY, USA, 2022\. Association for Computing Machinery.  ISBN 9781450385916.  doi: 10.1145/3522749.3523078.  URL https://doi.org/10.1145/3522749.3523078.
  * Barkatou and Jaroschek [2018] Moulay A. Barkatou and Maximilian Jaroschek.  Desingularization of first order linear difference systems with rational function coefficients.  In _Proceedings of the 2018 ACM International Symposium on Symbolic and Algebraic Computation_ , ISSAC ’18, page 39–46, New York, NY, USA, 2018. Association for Computing Machinery.  ISBN 9781450355506.  doi: 10.1145/3208976.3208989.  URL https://doi.org/10.1145/3208976.3208989.
  * Heilman et al. [2015] Steven Heilman, Elchanan Mossel, and Joe Neeman.  Standard simplices and pluralities are not the most noise stable.  In _Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science_ , ITCS ’15, page 255, New York, NY, USA, 2015. Association for Computing Machinery.  ISBN 9781450333337.  doi: 10.1145/2688073.2688076.  URL https://doi.org/10.1145/2688073.2688076.
  * Sarma et al. [2019] Sankar Das Sarma, Dong-Ling Deng, and Lu-Ming Duan.  Machine learning meets quantum physics.  _Physics Today_ , 72(3):48–54, mar 2019.  doi: 10.1063/pt.3.4164.  URL https://doi.org/10.1063%2Fpt.3.4164.
  * Hein and Andriushchenko [2017] Matthias Hein and Maksym Andriushchenko.  Formal guarantees on the robustness of a classifier against adversarial manipulation.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 2266–2276, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/e077e1a544eec4f0307cf5c3c721d944-Abstract.html.
  * Paknezhad et al. [2021] Mahsa Paknezhad, Cuong Phuc Ngo, Amadeus Aristo Winarto, Alistair Cheong, Chuen Yang Beh, Jiayang Wu, and Hwee Kuan Lee.  Explaining adversarial vulnerability with a data sparsity hypothesis.  2021\.  doi: 10.48550/ARXIV.2103.00778.  URL https://arxiv.org/abs/2103.00778.
  * Papernot et al. [2016] Nicolas Papernot, Patrick D. McDaniel, and Ian J. Goodfellow.  Transferability in machine learning: from phenomena to black-box attacks using adversarial samples.  _CoRR_ , abs/1605.07277, 2016.  URL http://arxiv.org/abs/1605.07277.
  * Inkawhich et al. [2019] Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen.  Feature space perturbations yield more transferable adversarial examples.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7059–7067, 2019.  doi: 10.1109/CVPR.2019.00723.
  * Ros and Doshi-Velez [2018] Andrew Slavin Ros and Finale Doshi-Velez.  Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence_ , AAAI’18/IAAI’18/EAAI’18. AAAI Press, 2018.  ISBN 978-1-57735-800-8.

## Checklist

The checklist follows the references. Please read the checklist guidelines
carefully for information on how to answer these questions. For each question,
change the default [TODO] to [Yes] , [No] , or [N/A] . You are strongly
encouraged to include a justification to your answer, either by referencing
the appropriate section of your paper or providing a brief inline description.
For example:

  * •

Did you include the license to the code and datasets? [Yes]

  * •

Did you include the license to the code and datasets? [No] The code and the
data are proprietary.

  * •

Did you include the license to the code and datasets? [N/A]

Please do not modify the questions and only use the provided macros for your
answers. Note that the Checklist section does not count towards the page
limit. In your paper, please delete this instructions block and only keep the
Checklist section heading above along with the questions/answers below.

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [Yes]

    2. (b)

Did you include complete proofs of all theoretical results? [Yes]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [N/A]

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [N/A]

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [N/A]

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [N/A]

    2. (b)

Did you mention the license of the assets? [N/A]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [N/A]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  Appendix A Proof of the theorems

###  A.1 Theorem 2.1

###### Proof.

Consider a nonuniform learner $\mathcal{A}$ with respect to
$\mathcal{H}(\mathcal{X})$. Suppose that we have a sequence
$\\{S_{n}\\}_{n=1}^{\infty}$ of training points $S_{n}$ in which

$\lim_{n\rightarrow\infty}\,S_{n}=\mathcal{X}.$ (21)

Then, $\\{f_{n}=\mathcal{A}(S_{n})\\}_{n=1}^{\infty}$ converges uniformly to
$f=\mathcal{A}(\mathcal{X})$. By Cauchy’s criterion for uniform convergence,
for any $\epsilon>0$ a natural number $N<\infty$ exists in which

$\sup_{x\in\mathcal{X}}|f_{m}(x)-f_{n}(x)|\leq\epsilon\quad m,n\geq N.$ (22)

Now suppose that an adversarial test point $x\notin S_{n}$ for $f_{n}$ exists.
We know that for some $m>n$ it is true that $x\in S_{m}$. Thus,

$\sup_{x\in\mathcal{X}}|f_{m}(x)-f_{n}(x)|>\epsilon,$ (23)

which contradicts our assumption. Consequently, it must be true that no
adversarial examples for $f_{n}$ exists. ∎

###  A.2 Lemma 2.3

###### Proof.

Since $\sigma(\mathcal{X})$ has the universal approximation property, then for
all $\alpha\in\mathbb{N}$ an $\omega\in\Omega$ exists where

$\psi_{\alpha}(\omega)=\int_{\mathcal{X}}\varphi_{\alpha}(x)s(x;\omega)\,dV(\mathcal{X})\neq
0.$ (24)

$\\{\varphi_{\alpha}\\}_{\alpha=0}^{\infty}$ is the complete orthonormal basis
for $\mathcal{H}(\mathcal{X})$. It is easy to check that any
$f\in\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}$ has a
representation

$f(x)=\sum_{\alpha=0}^{\infty}a_{\alpha}\psi^{\prime}_{\alpha}(x),$ (25)

in which the projected features
$\\{\psi^{\prime}_{\alpha}\\}_{\alpha=0}^{\infty}$ are computed by

$\psi_{\alpha}^{\prime}(x)=\int_{\Omega}\varphi^{\prime}_{\alpha}(\omega)s(x;\omega)\,dV(\omega),$
(26)

where $\\{\varphi^{\prime}_{\alpha}\\}_{\alpha=0}^{\infty}$ is the complete
orthonormal basis for $\mathcal{H}(\Omega)$.

Consider the inner product of $\psi^{\prime}_{\alpha}$ with $\varphi_{\beta}$,

$\displaystyle\psi^{\prime}_{\alpha}\cdot\varphi_{\beta}$
$\displaystyle=\int_{\mathcal{X}}\psi^{\prime}_{\alpha}(x)\varphi_{\beta}(x)\,dVx,$
(27)
$\displaystyle=\int_{\mathcal{X}}\int_{\Omega}\varphi_{\beta}(x)\varphi^{\prime}_{\alpha}(\omega)s(x;\omega)\,dV(\omega)dV(x),$
(28)
$\displaystyle=\int_{\Omega}\varphi^{\prime}_{\alpha}(\omega)\int_{\mathcal{X}}\varphi_{\beta}(x)s(x;\omega)\,dV(x)dV(\omega),$
(29)
$\displaystyle=\int_{\Omega}\varphi^{\prime}_{\alpha}(\omega)\psi_{\beta}(\omega)\,dV(\omega),$
(30) $\displaystyle=\varphi_{\alpha}^{\prime}\cdot\psi_{\beta}.$ (31)

Suppose that some $\varphi_{\beta}$ is orthogonal to all
$\psi^{\prime}_{\alpha}$. Then, we can deduce that $\psi_{\beta}$ is
orthogonal to all $\varphi^{\prime}_{\alpha}$. Given that
$\\{\varphi^{\prime}_{\alpha}\\}_{\alpha=0}^{\infty}$ is a complete basis for
$\mathcal{H}(\Omega)$, this would lead to a contradiction $\psi_{\beta}=0$.
Thus,
$\mathcal{H}(\mathcal{X})\subseteq\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}$.

To see that
$\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}\subseteq\mathcal{H}(\mathcal{X})$
is also true, we have to consider a few cases. Since $\sigma(\mathcal{X})$ has
the universal approximation property, we know that the dimension of $\Omega$
is greater than or equal to the dimension of $\mathcal{X}$.

First, suppose that the dimension of $\Omega$ is greater than $\mathcal{X}$.
This would mean that it for each $\varphi_{\beta}$ one or more
$\psi^{\prime}_{\alpha}$ exists. Given that
$\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}\subseteq
L^{2}(\mathcal{X})$, it must be true that
$\\{\psi^{\prime}_{\alpha}\\}_{\alpha=0}^{\infty}$ is redundant. Thus, we can
find a sequence $\\{\psi"_{\beta}\\}_{\beta=0}^{\infty}$ which has the same
index as $\\{\varphi^{\prime}_{\beta}\\}_{\beta=0}^{\infty}$ by performing
linear transformations on $\\{\psi^{\prime}_{\alpha}\\}_{\alpha=0}^{\infty}$.
As a result, this case could be transformed to an equivalent situation in
which the dimensions of $\Omega$ and $\mathcal{X}$ are equal.

Now, suppose that the the two domain sets have the same dimension. Then, a
bijection between $x\in\mathcal{X}$ and $\omega\in\Omega$ exists. Otherwise,
$\mathcal{N}\big{(}L^{2}(\Omega),\sigma(\mathcal{X})\big{)}$ would not have
enough degrees of freedom to universally approximate $L^{2}(\mathcal{X})$.
Consequently, a bijection between $\mathcal{H}(\mathcal{X})$ and
$\mathcal{H}(\Omega)$ exists. A bijection between $\mathcal{H}(\Omega)$ and
$\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}$ exists by
definition as well. In other words,
$|\mathcal{H}(\mathcal{X})|=|\mathcal{H}(\Omega)|=|\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}|$.
However, we just showed
$\mathcal{H}(\mathcal{X})\subseteq\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}$.
Thus, it must be true that
$\mathcal{H}(\mathcal{X})=\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}$.

If on the other hand $\sigma(\mathcal{X})$ does not have the universal
approximation property, then it is possible that $\psi_{\alpha}=0$ for none or
all $\alpha$. ∎

###  A.3 Theorem 2.4

###### Proof.

$L^{2}(\mathcal{X})$ would be nonuniform learnable if and only if it is a
countable union of PAC learnable hypothesis classes
$\mathcal{H}_{\alpha}(\mathcal{X})$. Since $L^{2}(\mathcal{X})$ is a Hilbert
space, it has a orthonormal basis
$\\{\varphi_{\alpha}\\}_{\alpha=0}^{\infty}$. Then, we can choose

$\mathcal{H}_{\alpha}(\mathcal{X})=\big{\\{}\sum_{k=0}^{\alpha}a_{k}\varphi_{k}(x)|a_{k}\in\mathbb{R}\big{\\}},$
(32)

and the theorem would follow. ∎

###  A.4 Theorem 2.6

###### Proof.

The Lagrangian of program (2.5) is

$\mathcal{L}=\frac{1}{2}\int_{\Omega}|h(\omega)|^{2}\,dV(\omega)+\sum_{n=1}^{N}\lambda_{n}\big{(}1-t_{n}\int_{\Omega}h(\omega)s(x_{n};\omega)\,dV(\omega)\big{)}.$
(33)

We can rearrange the Lagrangian into

$\displaystyle\mathcal{L}$
$\displaystyle=\int_{\Omega}\frac{1}{2}|h(\omega)|^{2}+\sum_{n=1}^{N}\lambda_{n}\big{(}1-t_{n}h(\omega)s(x_{n};\omega)\big{)}\,dV(\omega),$
(34) $\displaystyle=\int_{\Omega}L(\omega,h,\nabla h)\,dV(\omega).$ (35)

By the Euler-Lagrange equations for a function of several variables, $h^{*}$
must satisfy the following PDE

$\frac{\delta L}{\delta h}-\sum_{j=1}^{d}\frac{\partial}{\partial
x_{j}}\frac{\delta L}{\delta h_{j}}=0,$ (36)

in which $h_{j}=\frac{\partial h}{\partial x_{j}}$, and $\frac{\delta}{\delta
h}$ is the usual differentiation with the exception that it treats $h$ like a
symbolic variable. Thus,

$h^{*}(x)=\sum_{n=1}^{N}\lambda_{n}t_{n}s(x_{n};\omega).$ (37)

Consider the Dirac’s delta

$\delta(x;\omega)=\begin{cases}\infty\quad x=\omega\\\
0\quad\mathrm{otherwise}\end{cases}.$ (38)

We can see that the output of the SVC learning rule for
$\mathcal{N}\big{(}L^{2}(\mathcal{X}),\mathrm{Dirac}(\mathcal{X})\big{)}=L^{2}(\mathcal{X})$
would be the same as the memorising learning rule, which is the text book
example of a consistent learner that is not nonuniform. Thus, the SVC learning
rule is not a nonuniform learner with respect to
$\mathcal{N}\big{(}L^{2}(\Omega),\sigma(\mathcal{X})\big{)}$ in general. ∎

###  A.5 Proposition 2.7

###### Proof.

Consider
$\mathcal{N}\big{(}L^{2}(\mathcal{X}),\mathrm{Dirac}(\mathcal{X})\big{)}$.
According to theorem 2.6, in the infinite sample limit we would have

$\displaystyle f(x)$
$\displaystyle=\int_{\mathcal{X}}\lambda(\omega)t(\omega)\delta(x;\omega)\,dV(\omega),$
(39) $\displaystyle=\lambda(x)t(x).$ (40)

Since the norm of $f$ should be minimal and $f$ should satisfy $t(x)f(x)\geq
1$, we can deduce that $\lambda(x)=1$ for all $x$. We emphasize that when $x$
is on the decision boundary, it would not be participating in the optimization
since we have no constraint in the program for those points. Consequently, if
$t\in\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}$, then
$t$ would remain feasible and would dominate all other
$f\in\mathcal{N}\big{(}\mathcal{H}(\Omega),\sigma(\mathcal{X})\big{)}$ in
terms of the training loss and the regularization score. ∎

###  A.6 Theorem 2.9

###### Proof.

Using lemma 2.3, we know that both classes are equal to
$\mathcal{H}(\mathcal{X})$. Since $\mathcal{H}(\mathcal{X})$ is normal, we
know that with probability $1-\delta$ it is true that

$|f(x)-\hat{f}(x)|<\epsilon.$ (41)

Moreover, since $\partial\mathcal{H}(\mathcal{X})$ is normal as well, it is
true that

$\|\nabla f(x)-\nabla\hat{f}(x)\|<\epsilon^{\prime}.$ (42)

Given both conditions, we can infer that a gradient based attack would follow
a similar path for both $f$ and $\hat{f}$. ∎

###  A.7 Theorem 2.10

###### Proof.

It is enough to find a learning rule $\mathcal{A}$ with respect to
$L^{2}(\mathcal{X})$ that does not satisfy the condition for being normal. The
SVC learning rule for
$\mathcal{N}\big{(}L^{2}(\mathcal{X}),\mathrm{Dirac}(\mathcal{X})\big{)}$ is
such a learning rule. To see why, imagine a set of training points
$S\subset\mathcal{X}$. Then $f$ would be a point mass function. Since the
point masses are not measurable inside $\mathcal{X}$, then it is almost surely
true that

$|f(x)-t(x)|\geq 1.$ (43)

Thus, $L^{2}(\mathcal{X})$ is not a normal hypothesis class. ∎

###  A.8 Theorem 3.2

###### Proof.

First, assume that $\mathcal{A}$ is a nonuniform learner with respect to
$A^{2}(\Omega)$. By the definition of nonuniform learnability, for any
$\epsilon,\delta>0$ a natural number $N$ exists that for any training set
$S_{n}$ with size $n\geq N$ and with probability $1-\delta$ we have that

$\ell(t(x),x,\mathcal{A}(S_{n}))\leq\epsilon.$ (44)

Consider all sequences $\\{S_{n}\\}_{n=1}^{\infty}$ of training points $S_{n}$
in which

$\lim_{n\rightarrow\infty}\,S_{n}=\mathcal{X}.$ (45)

Then every $\\{\mathcal{A}(S_{n})\\}_{n=1}^{\infty}$ is converging uniformly
to $\mathcal{A}(\mathcal{X})$. We can see that any training set would appear
in at least one of the sequences that we are considering. Consequently, for
any training set $S_{n}$ larger than $N$ and with probability $1-\delta$ we
have that

$|\mathcal{A}(S_{n})(x)-\mathcal{A}(\mathcal{X})(x)|\leq\epsilon.$ (46)

######  Proposition A.1.

Suppose $\\{f_{n}\\}_{n=1}^{\infty}\subset\mathcal{O}(\mathcal{X})$ converges
uniformly on compact subsets of $\mathcal{X}$ to the function
$f:\mathcal{X}\rightarrow\mathbb{C}$. Then $f\in\mathcal{O}(\mathcal{X})$ and
for each $\alpha\in\mathbb{N}^{n}$,

$\lim_{n\rightarrow\infty}\partial^{\alpha}f_{n}=\partial^{\alpha}f$ (47)

compactly in $\Omega$. $\partial^{\alpha}$ is a shorthand notation for

$\frac{\partial^{|\alpha|}}{\partial x_{1}^{\alpha_{1}}\partial
x_{2}^{\alpha_{2}}\cdots\partial
x_{d}^{\alpha_{d}}},\quad|\alpha|=\sum_{j=1}^{d}\alpha_{j}.$ (48)

Thus, we can deduce that

$|\frac{\partial\mathcal{A}(S_{n})}{\partial
x_{j}}(x)-\frac{\partial\mathcal{A}(\mathcal{X})}{\partial
x_{j}}(x)|\leq\epsilon\quad j=1,\cdots,d.$ (49)

Equation (46) and equation (49) show that $A^{2}(\mathcal{X})$ and $\partial
A^{2}(\mathcal{X})$ are normal hypothesis classes with respect to nonuniform
learners.

######  Proposition A.2.

Let $\\{f_{n}\\}_{n=1}^{\infty}$ be a sequence of holomorphic functions on a
domain $\mathcal{X}\subset\mathbb{C}^{d}$. Assume that the sequence converges
pointwise to a limit function $f$ on $\mathcal{X}$. Then $f$ is holomorphic on
a dense open subset of $\mathcal{X}$. Also the convergence is uniform on
compact subsets of the dense open set.

Proposition A.2 shows that when $\mathcal{A}$ is a consistent learner for
$A^{2}(\mathcal{X})$ and $\\{\mathcal{A}(S_{n})\\}_{n=1}^{\infty}$ is
converging pointwise to $\mathcal{A}(\mathcal{X})$, then for some dense open
$K\subset\mathcal{X}$ and with probability $1-\delta$ we have that

$\displaystyle|\mathcal{A}(S_{n})(x)-\mathcal{A}(\mathcal{X})(x)|\leq\epsilon$
$\displaystyle\quad x\in K$ (50)
$\displaystyle|\frac{\partial\mathcal{A}(S_{n})}{\partial
x_{j}}(x)-\frac{\partial\mathcal{A}(\mathcal{X})}{\partial
x_{j}}(x)|\leq\epsilon$ $\displaystyle\quad j=1,\cdots,d.$ (51)

Consequently, we can find a similar bound for all $x\in\mathcal{X}$ with some
error probability $\delta^{\prime}\propto\delta$. As a result,
$A^{2}(\mathcal{X})$ and $\partial A^{2}(\mathcal{X})$ are normal hypothesis
classes with respect to universally consistent learners. ∎

###  A.9 Theorem 4.1

###### Proof.

The theorem is a direct consequence of proposition A.2. ∎

###  A.10 Theorem 4.2

###### Proof.

Consider a smooth path $\gamma:[0,1]\rightarrow\mathcal{X}$. The length of the
image of this curve under $f\in A^{2}(\mathcal{X})$ is

$\|f(\gamma)\|=\int_{0}^{1}\|\nabla f(\gamma(t))\|\gamma^{\prime}(t)\,dt,$
(52)

Hence, when $f$ has minimized $E[f]$, then it has minimized $\|f(\gamma)\|$
for all $\gamma$. Consequently, the image of all of the paths that start from
a test sample $z$ would stay as close to $f(z)$ as possible. It follows that
$f$ must be robust unless the training samples are poisoned. ∎

###  A.11 Theorem 4.5

###### Proof.

$f$ has a representation

$f(z)=\sum_{\alpha\geq 0}a_{\alpha}\varphi_{\alpha}(z)=a^{H}\varphi(z).$ (53)

We can prove the theorem by simply calculating $E[f]$,

$\displaystyle\int_{\Omega}\|\nabla f(z)\|^{2}\,dV(z)$
$\displaystyle=\int_{\Omega}a^{H}J(z)J(z)^{H}a\,dV(z),$ (54)
$\displaystyle=a^{H}\int_{\Omega}J(z)J(z)^{H}\,dV(z)a,$ (55)
$\displaystyle=a^{H}\Sigma a=a^{H}a,$ (56)

in which $J(z)$ is the Jacobian of the feature vector $\varphi(z)$. ∎

###  A.12 Theorem 5.1

###### Proof.

The Lagrangian for the real-valued robust SVC learning rule is

$\displaystyle\mathcal{L}$ $\displaystyle=\frac{1}{2}\int_{\Omega}\|\nabla
h(\omega)\|^{2}\,dV(\omega)+\sum_{n=1}^{N}\lambda_{n}\big{(}1-t_{n}\int_{\Omega}h(\omega)s(x_{n};\omega)\,dV(\omega)\big{)},$
(57)
$\displaystyle=\int_{\Omega}\frac{1}{2}\sum_{j=1}^{d}h_{j}(\omega)^{2}+\sum_{n=1}^{N}\lambda_{n}\big{(}1-t_{n}h(\omega)s(x_{n};\omega)\big{)}\,dV(\omega),$
(58) $\displaystyle=\int_{\Omega}L(\omega,h,\nabla h)\,dV(\omega).$ (59)

By the Euler-Lagrange equations for a function of several variables, $h$
satisfies the following equation

$\frac{\delta L}{\delta
h}-\sum_{j=1}^{d}\frac{\partial}{\partial\omega_{j}}\frac{\delta L}{\delta
h_{j}}=0.$ (60)

Consequently, $h$ must satisfy the following PDE

$-\Delta h(\omega)=\sum_{n=1}^{N}\lambda_{n}t_{n}s(x_{n};\omega).$ (61)

∎

###  A.13 Theorem 5.3

###### Proof.

We know that

$\displaystyle h(\omega)$
$\displaystyle=\sum_{n=1}^{N}a_{n}\mathfrak{s}_{n}(\omega),$ (62)
$\displaystyle-\Delta\mathfrak{s}(x;\omega)$ $\displaystyle=s(x;\omega).$ (63)

Computing $E[h]$, we would have

$\displaystyle E[h]$ $\displaystyle=\int_{\Omega}\nabla h(\omega)\cdot\nabla
h(\omega)\,dV(\omega),$ (64)
$\displaystyle=\sum_{n=1}^{N}\sum_{m=1}^{N}a_{n}a_{m}\int_{\Omega}\nabla\mathfrak{s}_{n}(\omega)\cdot\nabla\mathfrak{s}_{m}(\omega)\,dV(\omega).$
(65)

Since $\mathfrak{s}$ is twice differentiable, we may apply the divergence
theorem to get

$\displaystyle\int_{\Omega}\nabla\cdot\big{(}\mathfrak{s}_{n}(\omega)\nabla\mathfrak{s}_{m}(\omega)\big{)}\,dV(\omega)$
$\displaystyle=\int_{\Omega}\nabla\mathfrak{s}_{n}(\omega)\cdot\nabla\mathfrak{s}_{m}(\omega)-\mathfrak{s}_{n}(\omega)s_{m}(\omega)\,dV(\omega),$
(66)
$\displaystyle=\int_{\partial\Omega}\mathfrak{s}_{n}(\omega)\frac{\partial\mathfrak{s}_{m}(\omega)}{\partial
n}\,dt,$ (67)

Given $\mathfrak{s}_{m}(\omega)$ satisfies the Newman boundary condition, the
right hand side of equation (66) must vanish. Consequently,

$\int_{\Omega}\nabla\mathfrak{s}_{n}(\omega)\cdot\nabla\mathfrak{s}_{m}(\omega)\,dV(\omega)=\int_{\Omega}\mathfrak{s}_{n}(\omega)s_{m}(\omega)\,dV(\omega).$
(68)

By replacing the expression in $E[h]$ we would have

$E[h]=\sum_{n=1}^{N}\sum_{m=1}^{N}a_{n}a_{m}\int_{\Omega}\mathfrak{s}_{n}(\omega)s_{m}(\omega)\,dV(\omega).$
(69)

Similarly, if we compute $\|h\|^{2}_{L^{2}(\Omega)}$, we have

$\|h\|^{2}_{L^{2}(\Omega)}=\sum_{n=1}^{N}\sum_{m=1}^{N}a_{n}a_{m}\int_{\Omega}\mathfrak{s}_{n}(\omega)\mathfrak{s}_{m}(\omega)\,dV(\omega).$
(70)

Thus, we can compute the difference between the norm and the energy as

$\|h\|^{2}_{L^{2}(\Omega)}-E[h]=\sum_{n=1}^{N}\sum_{m=1}^{N}a_{n}a_{m}\int_{\Omega}\mathfrak{s}_{n}(\omega)\big{(}\mathfrak{s}_{m}(\omega)-s_{m}(\omega)\big{)}\,dV(\omega).$
(71)

Consequently, if we further have that

$-\Delta\mathfrak{s}(x;\omega)=\mathfrak{s}(x;\omega),$ (72)

the two regularization functions become equivalent. ∎