# Convolutional Normalization

Massimiliano Esposito  Nader Ganaba

###### Abstract

As the deep neural networks are being applied to complex tasks, the size of
the networks and architecture increases and their topology becomes more
complicated too. At the same time, training becomes slow and at some instances
inefficient. This motivated the introduction of various normalization
techniques such as Batch Normalization and Layer Normalization. The
aforementioned normalization methods use arithmetic operations to compute an
approximation statistics (mainly the first and second moments) of the layer’s
data and use it to normalize it. The aforementioned methods use plain Monte
Carlo method to approximate the statistics and such method fails when
approximating the statistics whose distribution is complex. Here, we propose
an approach that uses weighted sum, implemented using depth-wise convolutional
neural networks, to not only approximate the statistics, but to learn the
coefficients of the sum.

###### Contents

  1. 1 Introduction
  2. 2 Background
  3. 3 Weighted Mean
    1. 3.1 Implementation of Weighted Mean
      1. 3.1.1 Initialising values
  4. 4 Experiments
  5. 5 Normalization Using Learned Statistics
    1. 5.1 Numerical Experiments
  6. 6 Conclusion
  7. 7 Future Work
  8. A Importance Sampling View of DWCK
  9. B Details of section 3.1
  10. C Results Weighted Mean First Architecture CIFAR 10

##  1 Introduction

During the past few years, there has been considerable success of applying
deep learning to complex problems ranging from speech synthesis (cite WaveNet)
to deep reinforcement learning achieving victories in complex games such as Go
[19]. This, in turn, has fueled massive interest in the field. At the core of
deep learning are training algorithms which allow neural networks to learn
from the data they are presented or learn how to extremize a target quantity.
One such algorithm is the stochastic gradient descent (SGD) (cite) and it is
one of most widely used training algorithms.

However, this does not come without issues as many training algorithms suffer
from being slow at converging to the optimal state and lack stability when
approaching the optimum (citation needed).One common approach is to transform
the data into a manageable form through centring and scaling the data and it
is the base of many normalization techniques. In fact, the success of “Batch
Normalization” by Sgzedy and Ioffe [8] sparked an interest in such techniques.
In [8], they noticed that the learning procedure, which uses stochastic
gradient descent (SGD), or derived adaptive algorithms like Adam [12], can be
hindered and they claimed that a phenomenon known as internal covariance shift
is the cause. This phenomenon depends on the change of the network’s weights
during backpropagation that can lead to the shift of data-distribution towards
the saturation regime of the activation functions. Although the authors of
[18] claim otherwise and that internal covariance shift does not cause slow
convergence. In any case, to avoid this pathology, in [8] they proposed re-
scaling of the data in a channel-wise fashion. The increase in performance
motivated research into normalizations techniques, which lead to the
introduction of methods such as Layer Normalization [1], Instance
Normalization [21] to name a few. In this paper, we present a new type of
normalization procedures that apply, specifically, to images and directly
generalise Batch Normalization.

The standard normalization techniques rely on an arithmetic method, which are
in essence is a bare basic Monte Carlo method, to compute the statistics from
data and then uses it to transform the tensor, and in our case the image. That
poses an issue, mainly, it can output statistics that are fictitious. In
addition to that, the aforementioned normalization methods do not tackle data
decorrelation, and that is because it does not approximate the covariance
matrix. Indeed, naive application of Monte Carlo methods can fail when the
data belong to a probability distribution that is difficult to sample from. In
order to tackle such issue, we propose to use neural networks to render
normalization methods more flexible and general. Further, using arithmetic
methods do not take into consideration the hidden and underlying structure of
the images, whereas using neural networks can uncover such structures and
adjust the normalization accordingly.

That said, there are many ways in which we can introduce neural networks and
here we present one where a neural network is used to implement a weighted
sum. It is discussed in section 3, the neural network takes the data as the
input and then outputs a weighted sum and contrary to arithmetic weighted
sums, the neural network learns the coefficients of the weighted sum. In other
words, the network learns from the data what the weighted sum should be to
accelerate the learning process. It is similar to importance sampling [10,
11], where it acts a change of measure to better sample from the intractable
distributions. However, one issue with this approach is that we do not have
great control on the number of parameters used. In addition to that, the
quality of the approximation of the mean and standard deviation depends on the
number of elements in the training dataset and, in some applications, the
availability of a large dataset for training is not possible.

One possible solution is presented in section 5, where it is inspired by
variational methods for Bayesian inference [2], two neural networks
approximate the statistics of the hidden layers activation. In other words,
the networks learn the mean and standard deviation. The parametrisation of
statistics using neural networks is a widely used in application of
variational Bayesian methods and, to the best of our knowledge, here it is the
first time applied for normalization purposes.

The advantage of this approach is that it can approximate statistics from
complex and multimodel distributions and it gives freedom in choosing a larger
variety of architectures. This is the case where in the section 5 the
numerical experiments show that networks with a fraction of the number of
parameters compared to Batch Normalization achieves acceleration that, in some
cases, can rival and exceed it. Furthermore, as the learning is done through
solving an optimisation problem and this approach is used for principal
component analysis applied on manifolds [3] known as principal geodesic
analysis. Thus, it has the potential to learn statistics from data that belong
to a non-Gaussian distribution and also data defined on non-Euclidean
manifolds. This is one of the reasons why such approach is appealing, however,
a study of cases of data on non-Euclidean manifolds is currently beyond the
scope of this paper.

Main contributions:

  * •

Implement weighted sum using depth-wise convolutional neural networks for
computing the mean and standard deviation. This allows for approximating the
statistics from data whose underlying distribution is too difficult to sample
from. Depth-wise convolutional neural networks are specifically used to
perform normalization for every feature map seperately.

  * •

Introduce normalization method, where the mean and standard deviation are two
neural networks. Despite its infancy, we show that this approach can
accelerate converge of training of classification problem by introducing less
learnable parameters than the standard normalization methods.

##  2 Background

As we are primarily dealing with images, the data structure for a single image
is the so-called $(0,3)$ tensor. It can be thought of as a stack of matrices
and each matrix corresponds to a different channel. When we have a set of
images, this enlarges to $(0,4)$ tensor. As using tensor notation can be
taxing, a convention we follow here is that we denote tensors by variables
such as $x$ for the sake of simplification and space of images by $I$. The set
of images is called a batch, denoted by $\mathcal{B}$, and it is a subset of
the input component of training set $\mathcal{D}$. The batch can be defined as
$\mathcal{B}:=\\{x^{(i_{1})},x^{(i_{1})}\in I|i_{1}=0,\dots,N_{b}-1\\}$, where
$(i_{1})$ in parenthesis indicates element of the batch with size $N_{b}$.

A neural network is described abstractly as

$\displaystyle x^{l}=f\left(x^{l-1},\theta\right),\quad l=1,\dots N_{l},$ (1)

where $l$ denotes the index of the layer and $\theta$ denotes the network’s
parameters. In this article, we consider only convolutional neural networks,
and the major component of this type of networks is the kernel, denoted by $K$
and whose size is smaller than the input image. The way it works is performs a
weighted sum over the image, and the coefficients of the this weighted sum are
the elements of the kernel $K$. We can have multiple kernels where each kernel
extracts certain information from the image. A simple example is when the
kernel is set to

$\displaystyle K=\begin{bmatrix}-1&-1&-1\\\ -1&8&-1\\\ -1&-1&-1\end{bmatrix},$
(2)

then the extracted information are the edges found in the image. Generally, we
do not set the kernels beforehand, we choose arbitrary values and then through
training we update these values so that the kernel would give useful
information.

Consider the training set
$\mathcal{D}:=\\{(x_{i_{1}},y_{i_{1}})\,|\,x_{i_{1}}\in
I,\,y_{i_{1}}\in\mathcal{C}\\}$, where $\mathcal{C}$ is the space of classes.
The training problem can then be stated as finding the parameters $\theta$
that would minimise the cost functional

$\displaystyle
S(\theta)=\sum_{i_{1}}L\left(f^{N_{l}}(x_{i_{1}},\theta),y_{i_{1}}\right),$
(3)

where $L$ is the cost function and it is a map $L:I\to\mathbb{R}$. There are
many choices for $L$ and for linear regression it is chosen as the $L^{2}$
norm. For classification problems, cross entropy is the preferred choice. To
solve this minimisation problem, optimisation algorithms such as gradient
descent are used. However, such methods can be inefficient and slow to
converge and that can be due to the magnitude of the gradient of the loss
function with respect to the network’s parameters. One way to remedy this is
to introduce transformation that renders the data such the gradient would not
have large values that hamper the convergence rate.

The most celebrated of these methods is the Batch Normalization [8] and it
works by centering the data to zero and scaling it so that the standard
deviation becomes unity. Since this article’s focus is on images, we recall
how specifically Batch Normalization operates on them. In a deep neural
network for image classification, Batch Normalization operates per
convolutional layer re-scaling each element $x_{ij}^{(k)}$ in the
convolutional filter $\alpha$ as depitcted in the image below 1. Here $i,j$
denote the position of the element with respect to height and width, while $k$
denotes the $k^{th}$ element of batch $\mathcal{B}$, as we have already said.

Figure 1: How Batch Norm operates on images.

The element $x_{ij}^{(k)}$ of the $\alpha^{th}$ convolutional layer is
rescaled by the average and standard deviation of all the elements in the
batch that correspond to this convolutional layer.
If we define the average and standard deviation per layer $\alpha$ and given
batch $\mathcal{B}$ as

$\mu^{\alpha}=\frac{\sum_{ijk}x_{ij}^{(k)}}{|N_{b}|\times|N_{h}|\times|N_{w}|},$
(4)
${(\sigma^{\alpha})}^{2}={\frac{\sum_{ijk}\left(x_{ij}^{(k)}-\mu^{\alpha}\right)^{2}}{{|N_{b}|\times|N_{h}|\times|N_{w}|}}},$
(5)

the generical element $x_{ij}^{(k)}$ is transformed into

$\gamma\cdot\frac{x_{ij}^{(k)}-\mu^{\alpha}}{\sigma^{\alpha}}+\beta,$

where $\gamma$ and $\beta$ are two learnable parameters.

The average $\mu^{\alpha}$ can be seen as a particular weighted average, and
thus as a convolution. Indeed, if we apply a kernel $K$ like (2), of size
$(N_{h},N_{w})$ with each entry equal to $1/(N_{h}\cdot N_{w})$, to a image
with elements $x_{ij}^{(k)}$, we obtain equation (4) modulo $1/N_{b}$.
Similarly, we can construct a new set of data whose elements are tensors of
sides $(N_{h},N_{w})$ and whose entries are $(x_{ij}^{(k)}-\mu^{\alpha})^{2}$
and we obtain equation (5), again modulo $1/N_{b}.$

##  3 Weighted Mean

As mentioned at the end of the previous section, both $\mu^{\alpha}$ and
$\sigma^{\alpha}$ are a particular case of weighted averages. Therefore, they
can be computed via a fixed convolutional kernel of the same size of the image
$(N_{h},N_{w})$ and with each entry equal to $1/(N_{h},N_{w}).$

Instead of having weighted sum with predefined coefficients as mentioned at
the end of the previous section, we use substitute $K$ with a general kernel,
$K_{p}$ to be learned (see figure 2).

Figure 2: A possible generalization of Batch Norm

The weights of the general kernel $K_{p}$ are chosen such that they satisfy
the following constraint:

$\displaystyle\sum\limits_{i,j}p_{i,j}=1,$ $\displaystyle\quad\text{and}\quad
p_{i,j}\geq 0,$ (6)

for all $i\in\\{1,\dots,N_{h}\\}$ and $j\in\\{1,\dots,N_{w}\\}$. Thus
equations (4) and (5) then become:

$\displaystyle\mu^{\alpha}=\frac{1}{N_{b}}\sum\limits_{i,j,k}^{N_{h},N_{w},N_{b}}p_{i,j}x^{(k)}_{i,j},$
(7)

and similarly, the square of the standard deviation is

$\displaystyle\left(\sigma^{\alpha}\right)^{2}=\frac{1}{N_{b}}\sum\limits_{i,j,k}^{N_{h},N_{w},N_{b}}p_{i,j}\left(x^{{(k)}}_{i,j}-\mu^{\alpha}\right)^{2}.$
(8)

Notice we have suppressed the index $\alpha$ in the data $x_{ij}^{(k)}$ for
the sake of clarity. We again remark that the elements under analysis belong
to the given convolutional layer $\alpha.$
We then substitute Batch Normalization with the steps mentioned here using
Convolutional Kernel procedure, i.e.

$\displaystyle\widehat{x_{ij}^{(k)}}=\frac{x_{ij}^{(k)}-\mu^{\alpha}}{\sigma^{\alpha}},$
(9)

being this latter a proper generalisation of the former.

######  Remark 3.1.

One justification of this approach is that the arithmetic mean used can lead
to false statistics when the data belong to complex spaces. A possible
solution is to use importance sampling, where it uses weighted arithmetic
mean. The main message is that it can be viewed as a mapping of the data to a
different space which is easier to compute the mean and the variance. One
thing to point out is the difference between our approach and importance
sampling, as the latter uses coefficients determined beforehand while the
former learns the coefficients. A more detailed exposition on this argument
can be found in Section A of the Appendix.

###  3.1 Implementation of Weighted Mean

The goal is to apply the described convolutional kernel to tensor separately
for each feature map. To do this, we use Depthwise Convolutional Kernels (DWCK
henceforth) [5] instead of Convolutional Kernels. This latter, indeed,
operates along several input channels and is not desirable for our purposes.
The weights of each DWCK have to be non-negative to verify the second
condition of (6). Also, using DWCK of the size $height\times width$ can be
computationally intensive due to the high numbers of weights we are
introducing. To avoid this, we use a stack of several smaller DWCK that
emulate a similar result of one kernel of size $height\times width$. In doing
this we put attention on the fact we want for each $x_{ij}$ a differnt weight
$p_{ij}$. Let’s consider the example in the picture 3 to see how we can
achieve these two latter purposes when we process a $4\times 4$ image: we
initizialy apply a DWCK (in orange in the picture) with kernel size 2x2 and
stride 2x2 to the image of shape 4x4 (in green in the picture). In this way we
get a 2x2 block, that we call A, where each entry of the block contains the
following values:

$\displaystyle\mbox{entry A1:}\qquad
p_{11}x_{11}+p_{12}x_{12}+p_{21}x_{21}+p_{22}x_{22},$ (10)
$\displaystyle\mbox{entry A2:}\qquad
p_{11}x_{13}+p_{12}x_{14}+p_{21}x_{23}+p_{22}x_{24},$ (11)
$\displaystyle\mbox{entry A3:}\qquad
p_{11}x_{31}+p_{12}x_{32}+p_{21}x_{41}+p_{42}x_{22},$ (12)
$\displaystyle\mbox{entry A4:}\qquad
p_{11}x_{33}+p_{12}x_{34}+p_{21}x_{43}+p_{22}x_{44}.$ (13)

Figure 3: How Conv Norm operates

As it can be seen there are some elements that share the same $p-weight.$ For
example $x_{11}$ and $x_{13}$. To avoid this we use another 2x2 DWCK with
weights $q$ getting as a final result:

$\displaystyle
q_{11}\cdot(p_{11}x_{11}+p_{12}x_{12}+p_{21}x_{21}+p_{22}x_{22})+$
$\displaystyle
q_{12}\cdot(p_{11}x_{13}+p_{12}x_{14}+p_{21}x_{23}+p_{22}x_{24})+$
$\displaystyle
q_{21}\cdot(p_{11}x_{31}+p_{12}x_{32}+p_{21}x_{41}+p_{42}x_{22})+$
$\displaystyle
q_{22}\cdot(p_{11}x_{33}+p_{12}x_{34}+p_{21}x_{43}+p_{22}x_{44}).$

In this way all the $x_{ij}^{\prime}s$ element have different weights and we
have reduced the amount of weights by a factor of 2.
We can compute the standard deviation similarly. In general we can apply this
procedure to all kind of images as far as we use some padding. For further
details see the appendix B.

####  3.1.1 Initialising values

If we have just a DWCK of size $N_{h}\times N_{w}$, to be unbiased, we have to
set all the entries of the DWCK equal to $1/(N_{h}\times N_{h})$. However,
initializing parameters of a neural network to the same value can prevent it
from learning. Therefore we uniformly and randomly assign weights to each
kernel element in a small neighborhood of $1/(N_{h}\times N_{w})$. Using this
methodology, we can both train the net and staying close to our initial
hypothesis. According to the length of the stack of DWCK we use, we might need
to change this procedure a bit. In the example in the previous section, we
noticed that each $x_{ij}^{(k)}s$ had a different weight, and it was equal to
a $p_{ij}$ coming from the first kernel, multiplied by $q_{ab}$ coming from
the second kernel. Therefore, $p_{ij}\cdot q_{ab}$ should be equal to
$1/(N_{h}\times N_{w})$. Assuming both $p_{ij}$ and $q_{ab}$ carry the same
amount of information we can set each of them equal to $\sqrt{1/(N_{h}\times
N_{w})}$. Indeed if $p_{ij}$ and $q_{ab}$ are similar we have

$p_{ij}\cdot p_{ij}\sim p_{ij}\cdot q_{ab}=1/(N_{h}\times N_{w}),$
$q_{ab}\cdot q_{ab}\sim p_{ij}\cdot q_{ab}=1/(N_{h}\times N_{w}),$

which gives the desired result. If we have longer stacks of DWCK of length $n$
we apply a similar procedure but using $n-th$ root.

##  4 Experiments

Here we perform a number of experiments to validate the theory and perform
quantitative and qualitative analysis of the presented normalization
techniques in 3. The experiments are classification and that entails finding a
classifier, which can be described mathematically as
$g:\mathcal{X}\to\mathcal{C}$ and takes a tensor $x\in\mathcal{X}$ to a label
$c\in\mathcal{C}$, where $\mathcal{X}$ denotes the space of inputs of a
dataset and for all experiments here $\mathcal{X}\subset\mathcal{B}$ and
$\mathcal{C}$ is the space of labels and it is a subset of $\mathbb{R}^{10}$.
In most cases the labels are scalar, or text string, but here we use hot-
encoding and all experiments have $10$ classes, hence each label is a $10$
dimensional vector.

Since all experiments are classification problems, the learning paradigm is
supervised learning, and it requires the training data set
$\\{(x_{i_{1}},c_{i_{1}})\\}_{i_{1}}^{N_{t}}$, the goal is to find a
classifier $g$ such taht it minimises the loss functional

$\displaystyle\mathbb{S}=\sum_{i_{1}=1}^{N_{t}}L(g(x_{i_{1}}),c_{i_{1}})$ (14)

where $L$ is the categorical cross-entropy defined by

$\displaystyle
L(x_{i_{1}},c_{i_{1}})=-\sum_{i_{2}}c_{i_{1},i_{2}}\mathrm{log}(x_{i_{1},i_{2}}).$

There are many datasets the can be used for classification problem, but here
we limit ourselves the CIFAR-10 dataset [15]. For this dataset we use two
different architectures, one of which is based on all a classifier known as
ALL-CNN-C presented in [20]. The reason for this choice is that the classifier
uses solely convolutional neural networks and avoids using any extra layers
such as dropout and normalization. And because of that, it presents a good
candidate for a classifier where we can study how different normalization
methods can improve. The base classifier consists of three two-dimensional CNN
with $96$ filters and kernels of the size $3\times 3$, where the third layer
has a $(2,2)$ strides. That followed by four two-dimensional CNN with $192$
filters and $3\times 3$ kernel and the third one has a $(2,2)$ strides. Then
followed by a two dimensional CNN with $192$ filters with $1\times 1$ kernel.
The penultimate layer is a two dimensional CNN with $10$ filters and $1\times
1$ kernel and then fed into an average pooling layer, which outputs a $10$
dimensional vector. Mathematically, the classifier is written as

$\displaystyle x^{l}_{i_{1},i_{2}}$ $\displaystyle=(\phi\circ
f^{l}_{N})\left(w^{l}_{i_{2}}\ast x^{l-1}_{i_{1},i_{2}}+b^{l}_{i_{2}}\right),$
(15)

for $l=1,\dots,N_{h}$ to denote the layer and $w^{l}_{i_{2}}$ and
$b^{l}_{i_{2}}$ are the kernel and bias, respectively, of the $i_{2}^{th}$
feature map of the $l^{th}$ layer. The index $i_{1}=0,\dots,N_{b}$ denotes the
$i_{1}^{th}$ element of $\mathcal{X}$. The $f_{N}$ denotes the normalizing
method. Starting with Batch Normalization, reviewed in section 2, the
normalization method $f_{N}$ is here denoted by $BN$. The method does not use
neural networks per se, but it learns two parameters, $\gamma$ and $\beta$,
for each feature. This renders the method with a low number of parameters.

In addition to Batch Normalization, we apply the a weighted normalization
method, described in section 3, where the weighted sum is implemented using
depth-wise convolutional neural networks. here we can set $f_{N}=DWCK(a,b)$,
where $a$ is the kernel size and $b$ is the stride. The normalization for the
first two layers of ALL-CNN-C network is implemented using four successive
layers of depth-wise CNN. For the third, fourth and fifth layers of ALL-CNN-C,
the normalization consists of three layers of depth-wise CNN. For the sixth
layer, two layers of depth-wise CNN are utilised. The dimensions and strides
of the DWCK are a mix of (4,4) and (2,2) as shown in the table below.

The following table summarises the the classifier used for the numerical
experiments presented in this section. One thing to note, in order to include
as much as information in the table, we use the notation $(c,a,b)$ to indicate
the $c$ number of channels in the CNN with kernel of size $a$ and stride $b$.
Whenever the third number is not included, it means the stride is set to $1$.

Classifier
---
ALL-CNN-C | ALL-CNN with BN | ALL-CNN with DWCK
$(96,3\times 3)$ with $ReLU$ |  $(96,3\times 3)$ with $ReLU$ |  $(96,3\times 3)$ with $ReLU$
| $f^{1}=BN$ | $f^{1}=1\ DWCK(4,4)$
|  | $+3DWCK(2,2)$
$(96,3\times 3)$ with $ReLU$ |  $(96,3\times 3)$ with $ReLU$ |  $(96,3\times 3)$ with $ReLU$
| $f^{2}=BN$ | $f^{2}=1\ DWCK(4,4)$
|  | $+3DWCK(2,2)$
$(96,3\times 3,2)$ with $ReLU$ |  $(96,3\times 3,2)$ with $ReLU$ |  $(96,3\times 3,2)$ with $ReLU$
| $f^{3}=BN$ | $f^{3}=1\ DWCK(4,4)$
|  | $+2DWCK(2,2)$
$(192,3\times 3)$ with $ReLU$ |  $(192,3\times 3)$ with $ReLU$ |  $(192,3\times 3)$ with $ReLU$
| $f^{4}=BN$ | $f^{4}=DWCK(4,4)$
|  | $+2DWCK(2,2)$
$(192,3\times 3)$ with $ReLU$ |  $(192,3\times 3)$ with $ReLU$ |  $(192,3\times 3)$ with $ReLU$
| $f^{5}=BN$ | $f^{5}=1\ DWCK(4,4)$
|  | $+2DWCK(2,2)$
$(192,3\times 3,2)$ with $ReLU$ |  $(192,3\times 3,2)$ with $ReLU$ |  $(192,3\times 3,2)$ with $ReLU$
| $f^{6}=BN$ | $f^{6}=1\ DWCK(2,2)$
|  | $+1DWCK(2,2)$
$(192,3\times 3)$ with $ReLU$ |  $(192,3\times 3)$ with $ReLU$ |  $(192,3\times 3)$ with $ReLU$
| $f^{7}=BN$ | $f^{7}=1\ DWCK(2,2)$
|  | $+1DWCK(2,2)$
$(192,1\times 1)$ with $ReLU$ |  $(192,1\times 1)$ with $ReLU$ |  $(192,1\times 1)$ with $ReLU$
| $f^{7}=BN$ | $f^{7}=DWCK(2,2)$
$(10,1\times 1)$
$\mathrm{AveragePooling}$ with $\mathrm{SoftMax}$

The results for ALL-CNN with BN and ALL-CNN with DWCK were comparable for
batch sizes of $256,128,$ and $64$. This for the learning rates we have tried:
$0.1,0.05,0.01,0.005,0.001.$ However, for most of the learning rates for batch
sizes of $32$ and $16$ ALL-CNN with DWCK resulted superior and more stable
(see pics below) and in appendix C. It is also worth mentioning that we used a
simple standard deviation since a weighted one was computationally unstable.

##  5 Normalization Using Learned Statistics

In the previous sections, there are two issues that arise: the first, concerns
the number of extra parameters added to the classifier thanks to DWCK is large
in comparison to Batch Normalization and that could be a potential issue if
bigger DWCK and smaller hardware memory are used. The second issue concerns
the use of an arithmetic sum to approximate the mean and standard deviation,
it does not take into consideration the space in which the data lives and thus
can lead to statistics that do not reflect the true ones. Furthermore, for it
to give an accurate approximation, it requires a training dataset with a very
large number of elements and in some application there are challenges with the
availability of such training datasets [16].

One possible way to address the issues mentioned, is to consider a
normalization method with the mean and standard deviation represented using
neural networks. This in turn gives freedom of choosing a suitable
architecture with a significantly fewer number of parameters. This fact is
demonstrated in section 5.1, where such normalization method is shown its
ability to accelerate training using a fraction of the parameters that Batch
Normalization would require. That was achieved using two networks each
composed of global average pooling layer, so that the result is a vector
representing the mean of each channel, and a series of one-dimensional
convolutional neural layers followed by another average pooling layer.

Regarding the issue with approximating the mean and the standard deviation,
one way is to use a method that is inspired by Bayesian inference. The reason
for that is because Bayesian methods have the ability to approximate the
parameters of distribution without the need of a large training set, and
obtain statistics from data with non-Gaussian distribution [9, 2, 14]. One
obstacle to applying many Bayesian inference techniques is that they suffer
from intractability issues, meaning some quantities cannot be computed.
However, neural networks can be used to instead learn the intractable
quantities [13]. The same two networks mentioned above are used to approximate
the mean and the standard deviation.

###  5.1 Numerical Experiments

Similar to section 4, we add normalizing layers that use CNN neural networks
to learn the statistics to ALL-CNN-C classifier. For the first layer of ALL-
CNN-C, the networks for both the mean and standard deviation are implemented
by first applying a global averaging pool layer, then followed by three one-
dimensional CNN layers with $4$, $4$, and $1$ channels respectively and kernel
sizes of $4$, $3$ and $3$ respectively. Then followed by an average pooling
layer. The reason for having a global pooling layer is to reduce the
dimensionality of the input, i.e. $x\in I$ is mapped to a vector in
$\mathbb{R}^{N_{f}}$, where $N_{f}$ is the number of features maps of $x$. The
remaining layers of ALL-CNN-C all have their statistics parametrised by a
sequence of global average pooling followed by four one-dimensional CNN layers
with $4$, $4$, $2$ and $1$ channels and each having kernels with sizes $4$,
$3$, $2$ and $3$, respectively. Similar to the normalization used for the
first layer in ALL-CNN-C, an average pooling layer is used as a final layer.

When used with CIFAR10 classification problem, described in section 4, in
cases with a learning rate in the range of $[10^{-5},10^{-3}]$, this
normaliziation approach does not match the validation accuracy of Batch
Normalization, yet making gains with respect to the plain classifier. An
example of such behaviour can be seen in Figure 4.

However, when the learning rate is in the range $[0.05,0.01]$, we notice that
the method initially out performs Batch Normalization when it comes to
validation accuracy and validation loss. This is shown to be the case in
Figure 5 where the accuracy of the CNN normalization is noticibly higher than
than Batch Normalization for the first $25$ epochs. However, for the same
classifiers, the loss of the CNN normalization is lower and less oscillating
than Batch Normalization throughtout the duration of the training as shown in
Figure 6.

Figure 4: (This one I have to redo it) In this figure, the result of running
CIFAR $10$ classification for $50$ epochs and with a learning rate set to
$10^{-4}$ and the classifiers used are Batch Normalization and one-dimensional
CNN’s. The dashed lines are the accuracy of the validation set and the solid
lines are the training set, while the blue coloured lines are the results
plain ALL-CNN-C classifier, the blue lines for Batch Normalized classifier and
the green lines are for the learned classifier.  Figure 5: In this figure, the
result of running CIFAR $10$ classification for $75$ epochs and with a
learning rate set to $0.025$ and the classifiers used are Batch Normalization
and one-dimensional CNN’s. The dashed lines are the accuracy of the validation
set and the solid lines are the training set, while the blue coloured lines
are the results plain ALL-CNN-C classifier, the blue lines for Batch
Normalized classifier and the green lines are for the learned classifier.
Figure 6: In this figure, the result of running CIFAR $10$ classification for
$75$ epochs and with a learning rate set to $0.025$ and the classifiers used
are Batch Normalization and one-dimensional CNN’s. The dashed lines are the
loss of the validation set and the solid lines are the training set, while the
blue coloured lines are the results plain ALL-CNN-C classifier, the blue lines
for Batch Normalized classifier and the green lines are for the learned
classifier.

##  6 Conclusion

We have presented a normalizing method that helps increasing the convergence
rate of deep convolutional neural network training. At the core of the method
a weighted arithmetic sum that is used to compute the mean and the standard
deviation used for centring and scaling the output of the hidden layers of the
deep convolutional neural network. Usually weighted arithmetic sum has fixed
weights, but here the weights are not fixed and instead they are learned from
the data. This is realised using depth-wise convolutional neural networks,
whose kernel weights fulfil the role of the weights of the arithmetic sum. The
experimental results on CIFAR10 dataset show that his method can indeed match
the performance of Batch Normalization, which here is used as a benchmark.

##  7 Future Work

Thus far we explored normalization techniques which can be summarised as
mapping the images of a dataset to another set whose images all have a mean
zero and standard deviation equal to one. However this does not result in a
decorrelated data and to address this, in [7], a Zero Phase Component Analysis
(ZCA) was introduced to Batch Normalization process. However, ZCA requires a
number of linear algebra operations, including inverting a matrix, which can
be a potential bottleneck when it comes to higher dimensional data. One
solution was introduced in [6], where an iterative scheme was used to
approximate the inverse of the covariance matrix. One avenue of research is to
investigate the whether the addition of decorrelation step to DWCK described
in section 3 can yield improved results.

Another research question one can ask is whether or not the addition of
normalizing flows of [17] can help improve the performance of our method. A
Normalizing flow, in short, is a series of simple invertible transformations
applied iteratively to an image and it learns to map an image into a two
dimensional simple distribution, such as the Normal distribution. This is
helpful as the mean and standard deviation of the normal distribution are
known, so we can use the inverse mapping to obtain the mean and standard
deviation of an image. To answer why this is a subject of interest is due to
the ability of normalizing flows to map between simple distribution and
complex non-Gaussian ones.

Finally, we aim to see if it is possible to resolve the computational problems
of introducing weighted standard deviation in our first methodology of section
3. This could indeed enhance further our results.

## References

  * [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [2] Matthew J Beal.  Variational algorithms for approximate Bayesian inference.  PhD thesis, UCL (University College London), 2003.
  * [3] P Thomas Fletcher, Conglin Lu, Stephen M Pizer, and Sarang Joshi.  Principal geodesic analysis for the study of nonlinear statistics of shape.  IEEE transactions on medical imaging, 23(8):995–1005, 2004.
  * [4] I. V. Girsanov.  On transforming a certain class of stochastic processes by absolutely continuous substitution of measures.  Theory of Probability & Its Applications, 5(3):285–301, 1960.
  * [5] Yunhui Guo, Yandong Li, Liqiang Wang, and Tajana Rosing.  Depthwise convolution is all you need for learning multiple visual domains.  In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 8368–8375. AAAI Press, 2019\.
  * [6] Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao.  Iterative normalization: Beyond standardization towards efficient whitening.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4874–4883, 2019.
  * [7] L. Huangi, L. Huangi, D. Yang, B. Lang, and J. Deng.  Decorrelated batch normalization.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 791–800, 2018.
  * [8] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 448–456. JMLR.org, 2015.
  * [9] Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul.  An introduction to variational methods for graphical models.  Machine learning, 37(2):183–233, 1999.
  * [10] Herman Kahn.  Random sampling (monte carlo) techniques in neutron attenuation problems. i.  Nucleonics (US) Ceased publication, 6(See also NSA 3-990), 1950\.
  * [11] Herman Kahn.  Random sampling (monte carlo) techniques in neutron attenuation problems. ii.  Nucleonics (US) Ceased publication, 6(See also NSA 4-3795), 1950\.
  * [12] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [13] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [14] Rahul Krishnan, Uri Shalit, and David Sontag.  Structured inference networks for nonlinear state space models.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
  * [15] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * [16] David JC MacKay and David JC Mac Kay.  Information Theory, Inference and Learning Algorithms.  Cambridge university press, 2003.
  * [17] Danilo Rezende and Shakir Mohamed.  Variational inference with normalizing flows.  In International Conference on Machine Learning, pages 1530–1538. PMLR, 2015.
  * [18] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry.  How does batch normalization help optimization?  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, pages 2483–2493. Curran Associates, Inc., 2018.
  * [19] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  nature, 529(7587):484–489, 2016.
  * [20] J.T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller.  Striving for simplicity: The all convolutional net.  In ICLR (workshop track), 2015.
  * [21] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6924–6932, 2017.

##  Appendix A Importance Sampling View of DWCK

Treating the hidden layers’ activations as independent and identically
distributed random variables
$X^{\mu}:(\Omega,\mathcal{F},\mathbb{P})\to\mathbb{R}^{N^{l}_{h}}_{+}$, where
${N^{l}_{h}}$ is the number of nodes in the $l^{th}$ layer and
$(\Omega,\mathcal{F},\mathbb{P})$ is the probability space adapted to
filtration $\mathcal{F}$ and with probability space $\mathbb{P}$.

To justify our approach, we look towards statistics, as the method used to
compute the statistics is through Monte Carlo method and according to the Law
of Large Numbers, i.e. $N_{b}\to\infty$, we have

$\displaystyle\widehat{\mu}=\frac{1}{N_{b}}\sum\limits_{i=0}^{N_{b}}X_{i}\to\mathbb{E}_{\mu}[x]=\int
x\,d\mu=\int x\rho(x)dx,$ (16)

where $\rho(x)$ is the probability density function and $\mu$ is the
probability measure on the probability space $\mathbb{P}$. In a way, Monte
Carlo method can be viewed as Riemann summation of the continuous integral on
(16) with $\rho(x)$ set to unity for all $x$. And similar to other quadrature
methods, Monte Carlo method fails to approximate the mean of random variables
whose underlying probability distribution is difficult to sample from and it
leads to having a numerical method with a very large variance. In order to
decrease the variance and obtain an approximation close to the true mean, we
sample from a different probability space $\mathbb{Q}$ with probability
measure $d\nu$ and not $\mathbb{P}$.

Consider the Brownian motions $B_{t}$ and $Y_{t}=B_{t}+\theta\,t$, where
$\theta$ is the drift of the stochastic process $Y_{t}$. According to
Girsanov’s theorem [4], the processes $B_{t}$ and $Y_{t}$ are equivalent when
the probability space $\mathbb{P}$ is changed to $\mathbb{Q}$. In other words,
Girsanov’s theorem tells us how random variables on probability space
$(\Omega,\mathcal{F},\mathbb{P})$ are defined when the probability space is
changed to $\mathbb{Q}$. One manifestation of Girsanov’s theorem is the method
of importance sampling, which is used in the numerical calculation of expected
values. This takes us back to (16), we insert $\frac{\xi(x)}{\xi(x)}$ in the
integrand, where $\xi(x)$ is a probability density function of the probability
distribution of $\mathbb{Q}$, and we obtain

$\displaystyle\mathbb{E}_{\nu}[x]=\int x\rho(x)\frac{\xi(x)}{\xi(x)}dx=\int
x\frac{\rho(x)}{\xi(x)}\xi(x)dx=\mathbb{E}_{\mu}\left[x\frac{\rho(x)}{\xi(x)}\right]$
(17)

or in terms of measures

$\displaystyle\mathbb{E}_{\nu}[x]=\int x\frac{d\mu}{d\nu}d\nu.$

The term $\frac{d\mu}{d\nu}$ is known as the Radon-Nikodym derivative [16].
With that, the approximation of the mean is

$\displaystyle\widehat{\mu}=\frac{1}{N_{b}}\sum\limits_{i=0}^{N_{b}}x_{i}p_{i}\approx\mathbb{E}_{\nu}[x],$
(18)

where the coefficients $p_{i}=\frac{\rho(x_{i})}{\xi(x_{i})}$ and
$x\simeq\mathbb{Q}$ and this is known as importance sampling [16]. For the
case of the stochastic processes $B_{t}$ and $Y_{t}$, the Radon-Nikodym
derivative is computed as $\mathrm{exp}\left(-x\mu+\frac{\mu^{2}}{2}\right)$.
Notice that equation (18) is the same as equation (7).

##  Appendix B Details of section 3.1

Before explaining the procedure in general, let’s examine another example.
Let’s assume we have an image $I$ $32\times 32$. We can factor $32$ in
$4\times 2\times 2\times 2$. We can stack DWCK of sizes $(4,4)$, $(2,2)$,
$(2,2)$, $(2,2)$, with strides respectively $(4,4)$, $(2,2)$, $(2,2)$,
$(2,2)$, If we let pass $I$ through the stack of DWCK we get a single number
which can be interpret, according to the cases, as the sandard deviation or
the mean. In general if we have an immage $I$ of sides $a\times b$ we can
factor $a$ and $b$ as $a=a_{1}\times\dots\times a_{n}$ and
$a_{1}\times\dots\times a_{n}$ (if $a$ and $b$ are primes we can pad the
image). We can then construct DWCK stack of kernels
$(a_{1},b_{1}),\dots(a_{n},b_{n})$ and strides
$(a_{1},b_{1}),\dots(a_{n},b_{n})$, getting again a single value which can be
itnerpreted as the mean or standard deviation.

##  Appendix C Results Weighted Mean First Architecture CIFAR 10