Kernel function impact on convolutional neural networks

M.Amine Mahmoudia (mohamed.mahmoudi@univ-mascara.dz), Aladine Chetouanib,
Fatma Bouferaa, Hedi Tabiac

a Mustapha Stambouli University of Mascara, Algeria
b PRISME laboratory, University of Orleans, Orleans, France
c Université Paris Saclay, IBISC, Univ Evry, Evry, France

Corresponding Author:
M.Amine Mahmoudi
Email<EMAIL_ADDRESS>
###### Abstract

This paper investigates the usage of kernel functions at the different layers
in a convolutional neural network. We carry out extensive studies of their
impact on convolutional, pooling and fully-connected layers. We notice that
the linear kernel may not be sufficiently effective to fit the input data
distributions, whereas high order kernels prone to over-fitting. This leads to
conclude that a trade-off between complexity and performance should be
reached. We show how one can effectively leverage kernel functions, by
introducing a more distortion aware pooling layers which reduces over-fitting
while keeping track of the majority of the information fed into subsequent
layers. We further propose Kernelized Dense Layers (KDL), which replace fully-
connected layers, and capture higher order feature interactions. The
experiments on conventional classification datasets i.e. MNIST, FASHION-MNIST
and CIFAR-10, show that the proposed techniques improve the performance of the
network compared to classical convolution, pooling and fully connected layers.
Moreover, experiments on fine-grained classification i.e. facial expression
databases, namely RAF-DB, FER2013 and ExpW demonstrate that the discriminative
power of the network is boosted, since the proposed techniques improve the
awareness to slight visual details and allows the network reaching state-of-
the-art results.

###### keywords:

kernel functions , convolution neural network , fine-grained classification ,
facial expression recognition , emotion recognition.

††journal: Expert Systems with Applications

##  1 Introduction

Image classification has always been the core operation of computer vision.
Several methods have been proposed in the literature addressing this problem,
which consist in efficiently assigning the correct label to an image.
Recently, with the emergence of Convolutional Neural Network (CNN), the
computer vision community has witnessed an era of blossoming result thanks to
the use of very large training databases. These databases contain a very large
number of different images (i.e. objects, animals…etc). This advance
encouraged the computer vision community to go beyond classical image
classification that recognizes basic-level categories. The new challenge
consists of discriminating categories that were considered previously as a
single category and have only small subtle visual differences. This new sub-
topic of image classification, called fine-grained image classification, is
receiving a special attention from the computer vision community [Liu et al.,
2020, Tang et al., 2020, Chen et al., 2020, Ji et al., 2020, Wang et al.,
2020, Huang & Li, 2020, Gao et al., 2020, Zhuang et al., 2020]. Such methods
aims at discriminating between classes in a sub-category of objects like birds
species [Welinder et al., 2010], models of cars [Krause et al., 2013], and
facial expressions, which makes the classification more difficult due to the
high intra-class and low inter-class variations. State-of-the-art approaches
typically rely on convolutional neural network as classification backbone and
propose a method to improve its awareness to subtle visual details.

A CNN is mainly a stack of three different types of layers: convolution
layers, pooling layers and fully-connected layers. Each of these types of
layers perform specific task. Convolution layers are the core building block
of a CNN by leveraging the fact that an input image is composed of small
details, or features, and create a mechanism for analyzing each feature in
isolation, which makes a decision about the image as a whole. Pooling layers,
on the other hand, are used for the gradual spatial down-sampling of the
feature map by reducing the number of parameters and thus decreases both the
consumption of the memory and the complexity of computing . In addition,
pooling layers widen the receptive field size of the intermediate neurons
which allow the latter to receive data from a larger area of the image. These
two layers are usually used in alternation until getting the most size-
effective representative feature which is finally fed into a fully connected
neural network in order to take a final classification decision.

CNNs have been used for a multitude of visual tasks. They showed to perform
very competitive results while linear operations are used at different layers
of the network. Linear functions are efficient, particularly, when the
original data is linearly separable, which should have, in general, a high
dimensional representation. In such a case, the decision boundary can be
representable as a linear combination of the original features. It is worth
noting that not every high dimensional problems are linearly separable
[Robert, 2014]. For instance, images may have a high dimensional
representation, but individual pixels are not very informative. Moreover,
taking in consideration only small regions of the image, dramatically reduces
their dimension, which makes linear functions less sensitive to subtle changes
in input data. The ability of detecting such differences is crucial
essentially for fine-grained recognition.

To overcome these limitations, some researches investigated different ways to
include non-linear functions in CNNs. Starting from non linear activation
functions like ReLU, eLU [Clevert et al., 2015], SeLU [Klambauer et al., 2017]
and more recently [Sitzmann et al., 2020]. Moreover, some recent work intended
to replace the underlying linear function of a CNN by non linear kernel
function without resorting to activation functions. For instance, some of them
replaced convolution layers ([Zoumpourlis et al., 2017], [Wang et al., 2019]),
while others replaced the pooling layers ([Lin et al., 2015], [Tenenbaum &
Freeman, 2000], [Lin & Maji, 2017], [Gao et al., 2016], [Cui et al., 2017],
[Gao et al., 2019], [Hyun et al., 2019]). The higher order kernel function
are, the more susceptible they are to fit slight changes in data. We leverage
this kernel function property to find the best use of these functions in
different level of the CNN.

In this paper, we investigate the usage of kernel functions at the different
layers in a CNN. We carry out extensive studies of their impact on
convolutional, pooling and fully-connected layers. For this purpose, we first
replace the convolution operation in CNNs by a non-linear kernel function
similarly to Kervolution [Wang et al., 2019]. We further used a novel pooling
layer, based on kernel functions, that keeps the down-scaling aspect of the
standard pooling function and brings various new features. This pooling layer
relay on learnable weights that generalize ordinary pooling operations (i.e.
average pooling and max pooling). Furthermore, it encodes patch-wise non-
linearity. In this manner, the discrimination power of the full network is
enhanced. The novel pooling, called learnable weights pooling, can be used at
any level of the network and is fully differentiable, which allows the network
to be trained in an end-to-end fashion. We also use a novel fully-connected
layer in which we use kernel functions to create a neuron unit that uses a
higher degree kernel function on its inputs rather than computing the weighted
sum. The used layer, called Kernelized Dense Layers (KDL) [Mahmoudi et al.,
2020, 2021c, 2022], is also differentiable and demonstrates its usefulness in
the improvement of the discrimination power of the full network. We explore
different dispositions and configurations of these layers to find which
configuration gives the best accuracy without over-fitting

The remainder of this paper is organized as follow: Section 2 reviews similar
work that tried to incorporate kernel functions in CNN models. Section 3 gives
more details about the methods (i.e. kervolution, learnable weights pooling
and KDL) we used to incorporate kernel functions in a CNN model. Section 4
presents our experiments, datasets and results; and Section 5 discusses the
obtained results and concludes the paper.

##  2 Related work

Kernel-based learning machines, like kernel Support Vector Machines (SVMs)
[Burges et al., 1999], kernel Fisher Discriminant (KFD) [Mika et al., 1999],
and kernel Principal Component Analysis (KPCA) [Schölkopf et al., 1998], have
been widely used in literature for various tasks. Their ability to operate in
a high-dimensional feature space, allow them to be more accurate than linear
models. These approaches have shown practical relevance for some specific
fields like computer vision. Even-though CNNs have boosted dramatically the
pattern recognition field, some recent works intended to take advantage of
kernel-based learning machines to further boost CNN performance.

Recently, Zoumpourlis et al. [Zoumpourlis et al., 2017] developed a second-
order convolution method by exploring quadratic forms through the Volterra
kernels. Constituting a richer function space, this method is used as
approximation of the response profile of visual cells. However, in this method
the number of parameters increases the training complexity exponentially. Wang
et al. [Wang et al., 2019] proposed kervolution layers to replace the
convolution layers in a CNN. This proposed layer uses non-linear kernel
function instead of linear convolution function as core operation. These
layers allow the model to capture higher order features at the convolutional
level.

In [Lin et al., 2015] a bilinear pooling method for fine-grained recognition
has been proposed. This method is based on the second order pooling model
proposed by [Tenenbaum & Freeman, 2000]. This method applies, first, two CNNs
to extract features from the input data. After that, it multiplies using the
outer product each location of the outcoming feature maps. At the end, an
image descriptor is obtained by sum-pooling the previous product of the
outcoming feature maps. The author have further enhanced this model later [Lin
& Maji, 2017] by applying matrix function normalization. For instance, the
matrix square-root, and the matrix logarithm. These methods have also been
used for facial expression recognition [Mahmoudi et al., 2021b] and enhanced
the overall accuracy to nearly 3% over an ordinary CNN. However, these methods
have high dimensions and they might be unsuitable for several of image
analysis applications, where reduced complexity is needed. To alleviate this
issue, Gao et al. [Gao et al., 2016] proposed a compact representation of
these methods. This new method has only a few thousand dimensions, yet it has
a similar discriminative power as the full bilinear model. Cui et al [Cui et
al., 2017] have further generalized [Gao et al., 2016] method in the form of
Taylor series kernel. This method captures non-linear and higher order
features. This method is fully differentiable, and can be trained with a CNN
in an end-to-end fashion. The methods cited above are used between the
convolution layers and the fully connected layers. They expand the basis of
the data, which enhances the discrimination power of the fully connected
layers. This enhancement in discrimination is then back-propagated to the
convolution layers. Even-thought the methods cited above enhances the CNN
capacity, they do not carry learning in themselves and rely completely on the
underlying CNN. Finally, we have also conducted some works Mahmoudi et al.
[2021a], Mahmoudi [2022] that studies the differences between our methods and
the ordinary kernel methods proposed before (like Kernel SVM and Kervolution).

To the best of our knowledge, kernel functions have never been used in fully-
connected layers. These layers are always used in their basic linear form.
Although non-linearity can be reached by stacking multiple dense layers, it
makes the model prone to overfit. Thus many recent architectures omit entirely
the fully-connected layers. In this paper, we investigate the usage of kernel
functions at the different layers of a convolutional neural network, in
particular, convolution, pooling and fully-connected layers. We build upon the
previously cited works and propose novel non-linear functions specific for
each type of layer. These functions replace the underlying linear functions of
the specific layer, perform the same tasks and bring new features. By doing
so, these new layers will be able to extract linear and non-linear relations
between features.

##  3 Study design

In this section, we present the different proposed kernel based techniques
which can be plugged into a CNN. These techniques consist in new feature
extraction, pooling and classification layers that can be used in the same
manner as the usual CNN layers. Furthermore, these novel layers can be used
solely or jointly with the usual CNN layers. This flexibility makes them
usable in any architecture or even plugged at any level of a pre-trained CNN
model. The novelty in this work is the use of higher order kernel function to
replace the underlying function of each layer. These kernel functions allow
them to perform the same task and brings additional features.

Figure 1: The different configurations to replace each layer type of an
ordinary CNN with a higher order kernel layer, notably Kervolution, Learnable
weights pooling and kernelized dense layers KDL.

In the following, we give details on how kernel functions can be employed at
each level of a CNN. As shown in Figure 1, we replace each layer type of an
ordinary CNN (Fig 1-(a)) with a higher order kernel layer (Fig 1-(b)). More
precisely, we replace convolution layers with Kervolution layers, Max/AVG
pooling layers with Learnable weights pooling layers, and fully connected
layers with Kernelized dense layers KDL.

###  3.1 Kervolution

Kervolution has been proposed by Wang et al [Wang et al., 2019]. It extends
the convolution operation which computes the dot product between an input
vector $X$ and a weight vector $W$, and adds eventually a bias term, according
to Equation 1:

$C_{o,i,j}=(X\times W)_{i,j}=\sum_{g}\sum_{h}X_{g+i,h+j}W_{o,g,h}+B_{o},$ (1)

Where $o$ corresponds to the output size, $i$ and $j$ are specific locations
in the input, $g$ and $h$ are respectively the width and height of the
convolution filter and $B$ is a bias term.

Convolution is a linear operation that usually requires adding an activation
function to introduce non-linearity. Without these activation functions the
CNN performance drops dramatically. Kervolution leverages this fact and
proposes to replace the convolution operation in CNNs by a non-linear function
that performs the same task as convolution without resorting to activation
functions. In this work, we use Kervolution with three kernel functions:

  1. 1.

Linear kervolution which corresponds to the convolution function (Equation 1);

  2. 2.

Polynomial kervolution:

$K_{o,i,j}=\langle
X,W\rangle_{i,j}=\sum_{g}\sum_{h}(X_{g+i,h+j}W_{o,g,h}+C)^{n}+B_{o}$ (2)

where $C$ ($C\in\mathbb{R^{+}}$) is a learnable constant and $n$
($n\in\mathbb{Z^{+}}$) is the polynomial order, it extends the feature space
to $n$ dimensions;

  3. 3.

Gaussian RBF kervolution:

$K_{o,i,j}=e^{-{\frac{\|X_{g+i,h+j}-W_{o,g,h}\|^{2}}{2\sigma^{2}}}}$ (3)

where $\sigma$ ($\sigma\in\mathbb{R^{+}}$) is a hyperparameter to control the
smoothness of decision boundary. It extends kervolution to infinite
dimensions.

The linear kernel measures the similarity between the filter weight vector $W$
and the feature map vector $X$. However, when $n>1$ in Equation 2, the
polynomial kernel encodes the non-linear relations between both $X$ and $W$
vectors, in addition to the linear relation between them. In the case of the
Gaussian kernel (Equation 3), the non-linearity is expanded to the infinity.
For this purpose, we can also use other kernels, such as the the Abel kernel
defined as $\mathcal{K}(X,W)=e^{-\alpha|X-W|},$ or the Laplacian kernel
defined by $\mathcal{K}(X,W)=e^{-\alpha\|X-W\|}$, $\text{ where, }\alpha>0$ in
both kernels.

Kervolution layers are as flexible as convolution layers and can be plugged at
any level of a CNN. In this paper, we use three configurations to study the
kervolution layer impact on CNNs. As shown in Figure 2, we use one kervolution
layer at the beginning of the network (Fig 2-(a)), one kervolution layer at
the end of the network (Fig 2-(b)), and an end-to-end kervolution network (Fig
2-(c)).

Figure 2: The three study configurations of kervolution layers, namely: (a)
one kervolution layer at the beginning of the network, (b) one kervolution
layer at the end of the network and (c) an end-to-end kervolution network.

###  3.2 Learnable Weights Pooling

Learnable weights pooling has been proposed in [Mahmoudi et al., 2020]. It
extends the three conventional pooling methods that have usually been employed
with CNNs, namely:

  1. 1.

Max pooling $P_{i,j}=Max(C_{i+g,j+h})$ where $C$ is the convolution layer
output, $g,h$ is the pooling size and $i,j$ is a specific location.

  2. 2.

Average pooling $P_{i,j}=\frac{\sum_{g}\sum_{h}(C_{i+g,j+h})}{gh}$

  3. 3.

Strided convolution (Equation 1).

These pooling methods exclude key details that are crucial for fine-grained
classification. Particularly for FER, in which, the detection of specific
distortions of facial regions is more important than simply detecting it in a
given location (which is the case in max-pooling [Acharya et al., 2018a]).

Learnable weights pooling [Mahmoudi et al., 2020] rely on a kernel function
that down-scale the input while keeping track of the most important
information instead of deleting part of it. This gives pooling layers the
ability to detect more sensitive and subtle details in data than the usual
pooling methods. This is done by incorporating learnable weights to the
pooling layers. In this manner, the usual pooling methods can be considered as
a spaecial case of learnable weights pooling that have fixed (non-learnable)
weights.

Figure 3: The processing of the learnable weights pooling layer is similar to
usual pooling layer in the manner that it down-scales the spatial dimensions
of the input. Learnable weights pooling rely on learnable weights to encode
important relations between features through kernel function.

Learnable weights pooling layer is the same as a usual pooling layer in the
manner that it down-scales the spatial dimensions of feature map, on a
specific location and a specific stride. It differs from other pooling
strategies in the way that it has the capability to extract more important
features from the input map. This is specially achieved by learning specific
pooling weights at each location of the feature map. These weights are learned
similarly to convolutional weights yet with a single depth output (see Figure
3). Finally, a combination between the input and the out-coming weights is
calculated through a function chosen precisely to capture non linear and
linear correlations between the weights and the input.

Here, we use three kernel functions as follow:

  1. 1.

Linear pooling is able to learn a suitable pooling from a continuum of methods
that ranges from average to max (or extremum) pooling;

$P_{i,j}=\sum_{g}\sum_{h}(C_{g+i,h+j}W_{g,h}+C)$ (4)

  2. 2.

Similarly to polynomial kervolution, the polynomial pooling does not only
encode linear relation between inputs and weights, but linear relations too.
It extends the feature space to $n$ dimension, yet reduces the size;

$P_{i,j}=\sum_{g}\sum_{h}(C_{g+i,h+j}W_{g,h}+C)^{n}$ (5)

  3. 3.

Gaussian RBF pooling expands the pooling non-linearity to the infinity;

$P_{i,j}=e^{-{\frac{\|C_{g+i,h+j}-W_{g,h}\|^{2}}{2\sigma^{2}}}}$ (6)

This method is able to detect the suitable pooling strategy at a patch-wise
level. In other words, in the same pooling layer, our method can apply a
complex mixture of pooling method ranging from average pooling to max pooling
(with a linear kernel) and beyond (with higher order kernel). Yet remaining as
simple as an ordinary pooling layer. In order to efficiently study the impact
of our proposed pooling layer on a CNN, we test it following the three
configurations. As shown in Figure 4, we study the impact of one Learnable
weights pooling layer after the first convolution layer (Fig 4-(a)), one
Learnable weights pooling layer after the last convolution layer (Fig 4-(b)),
and an end-to-end Learnable weights pooling network (Fig 4-(c)).

Figure 4: The three study configurations of Learnable weights pooling layers,
namely: (a) one Learnable weights pooling layer after the first convolution
layer, (b) one Learnable weights pooling layer after the last convolution
layer, and (c) an end-to-end Learnable weights pooling network.

###  3.3 Kernelized dense layer

The inputs of the fully-connected layers consist of the result of the
subsequent alternation of convolution and pooling layers. These input values
goes through the first fully-connected layer where we multiply them by
weights. After that, we apply an activation function (i.e ReLU). Finally, they
goes forward through the output layer, in which each neuron represents a
classification category. The fully-connected layers back-propagates the most
accurate weights where every neuron gets weights that prioritize the most
relevant category.

In this paper, we use a novel dense layer composed of neuron that uses a
kernel function instead of the usual dot product. Kernelized Dense Layer
(KDL), proposed in [Mahmoudi et al., 2020]. In contrary to a classical neuron
layer where it computes a dot product between an input vector and a vector of
weights, add a bias vector ($b\geq 0$), KDL applies higher degree kernel
function, which permits to the latter to map the input data to a higher space
and therefore be more discriminative than a usual linear layer.

Figure 5: The basic unit of KDL is a kernel based neuron. It applies a kernel
function on a vector of weights $w=\\{w_{1},w_{2},\dots,w_{n}\\}$ and an input
vector $x=\\{x_{1},x_{2},\dots,x_{n}\\}$ and adds a bias term.

Figure 5 shows the processing of a core unit (kernel based neuron) of KDL.
Formally, the output $Y$ is calculated using a kernel function $K$ on an input
vector $x=\\{x_{1},x_{2},\dots,x_{n}\\}$ and the corresponding vector of
weights $W=\\{w_{1},w_{2},\dots,w_{n}\\}$ and, adding the bias vector ($b\geq
0$).

  1. 1.

Linear KDL which correspond to the ordinary dense layer :

$Y_{i}=\sum_{j}X_{j}W_{i,j}+B_{i}$ (7)

  2. 2.

Polynomial KDL :

$Y_{i}=\sum_{j}(X_{j}W_{i,j}+C)^{n}+B_{i}$ (8)

where $C$ ($C\in\mathbb{R^{+}}$) is a learnable constant and $n$
($n\in\mathbb{Z^{+}}$) is the polynomial order, it extends the KDL to $n$
dimensions;

  3. 3.

Gaussian RBF KDL:

$Y_{i}=e^{-{\frac{\|X_{j}-W_{i,j}\|^{2}}{2\sigma^{2}}}}$ (9)

where $\sigma$ ($\sigma\in\mathbb{R^{+}}$) is a hyperparameter to control the
smoothness of decision boundary. It extends KDL to infinite dimensions.

##  4 Experiments

In this section, we evaluate the performance of the kernel-based layers
described above, in terms of accuracy rate and convergence speed. Several
experiments have been conducted on six well-known datasets, following the
configuration shown in Figure 2 and 4. Note that for KDL, only one
configuration can be tested, since they can be plugged only at the end of the
network. In the following, we detail our experiments process. First we
describe the datasets used to evaluate our approach (sec.4.1). After that we
define the training process of our networks (sec.4.2). Then we discuss the
obtained results (sec.4.3). Finally, we compare our results to state-of-the-
art results (sec. 4.4).

###  4.1 Datasets

Our experiments have been conducted on six well-known datasets. In order to
study different characteristics of kernel functions in CNN, these datasets
have been grouped in two categories. The first category is used to study the
accuracy enhancement and convergence speed brought by the kernel function. It
is composed of the following three datasets: MNIST, Fashion-MNIST [Xiao et
al., 2017] and CIFAR-10 [Krizhevsky et al., 2009]. The second category is
specially used to study the awareness of kernel function to subtle visual
details. It is composed of three well-known fine-grained facial expression
datasets: RAF-DB [Li et al., 2017], ExpW [Zhang et al., 2018] and FER2013
[Goodfellow et al., 2013]. Facial expression datasets contain few classes that
are nearly identical, which makes the recognition process more challenging.

  * 1.

The RAF-DB [Li et al., 2017] or Real-world Affective Face DataBase is composed
of 29,672 facial in the wild images. This images are categorized in either
seven basic classes or eleven compound classes.

  * 2.

The ExpW [Zhang et al., 2018] or EXPression in-the-Wild dataset is composed of
91,793 facial in the wild images. The annotation was done manually on
individual images.

  * 3.

The FER2013 database was used for the ICML 2013 Challenges in Representation
Learning [Goodfellow et al., 2013]. It is composed of 28,709 training images
and 3,589 images for both validation and test.

###  4.2  Training process

Figure 6: Base model-1 architecture: it is composed of two blocks. Each one of
these blocks is composed of a convolution layer, a batch normalization layer,
a dropout layer and ReLU activation. At the end, two fully-connected layers
are added with respectively 320 units and ReLU activation and 10 units with
softmax activation.

The datasets we have used for our experiments are not similar in terms image
complexity. MNIST and Fashion-MNIST contain small and very simple images,
while Cifar-10 and FER datasets contain larger complex images. We have thus
decided to use two model architectures, as shown in Figures 6 and 7. These
architectures are quite simple and can effectively run on cost-effective GPUs.
Model-1 (Figure 6) is used for MNIST and Fashion-MNIST. It is composed of two
blocks. Each one of these blocks is composed of a convolution layer, a batch
normalization layer, a dropout layer and ReLU activation. At the end, two
fully-connected layers are added with respectively 320 units and ReLU
activation and 10 units with softmax activation. On the other hand, model-2
(Figure 7) is used for CIFAR-10 and FER datasets. It is composed of five
blocks. Each one of these block is composed of a convolution layer, a batch
normalization layer, a dropout layer and ReLU activation. At the end, two
fully-connected layers are added with respectively 128 units and ReLU
activation and 10 units, for CIFAR-10, or 7 units for FER datasets with
softmax activation.

We have used Adam optimiser with a learning rate starting from 0.001 decreased
by a factor of 0.5 if the validation accuracy does not increase over two
epochs for Model-1 and five epochs for Model-2. We trained Model-1 for 50
epochs and Model-2 for 100 epochs. To avoid over-fitting, for FER datasets, we
used data augmentation with a shear intensity of 0.2, a range degree for
random rotations of 20, randomly flip inputs horizontally and a range for
random zoom of 0.2. We have also cropped the face region on FER datasets and
resize the resulting images to $100\times 100$ pixels.

Figure 7: Base model-2 architecture: it is composed of five blocks. Each one
of these block is composed of a convolution layer, a batch normalization
layer, a dropout layer and ReLU activation. At the end, two fully-connected
layers are added with respectively 128 units and ReLU activation and 10 or 7
units with softmax activation.

###  4.3 Ablation Study

In this section, we explore the impact of the three kernel-based layers in
terms of accuracy rate and convergence speed. We study the impact of each
layer solely, in a full non-linear configuration network. After that, we test
them jointly with the usual CNN layer, plugging them either at the beginning
or at the end of the network. The kernel functions we have used are: (1) the
linear kernel functions, (2) polynomial kernel functions from second degree to
degree five and (3) the Gaussian RBF kernel with $\sigma=0.9$. The accuracy
rate results are illustrated in tables 1\- 7 and convergence speed results are
illustrated in figures 8\- 14. In order to distinguish clearly the difference
in convergence among kernel, we chose to plot only the first 20 epochs for
MNIST, Fashion-MNIST and Cifar-10 datasets.

####  4.3.1 Kervolution

In this section we present the obtained results obtained with our two base
models with a full Kervolution configuration (Table 1, Figure 8), a single
Kervolution layer at the beginning of the network (Table 2, Figure 9), and a
single Kervolution layer at the end of the network (Table 3, Figure 10).

Table 1: Accuracy rates of full kervolution networks

| Model-1 | Model-2
---|---|---
Layers configuration | MNIST | Fashion-MNIST | Cifar10 | RAF-DB | FER2013 | ExpW
Convolution (Linear kernel) | 98.50% | 88.29% | 86.87% | 87.05% | 70.49% | 75.91%
2nd-order Poly |  99.05% |  90.06 % | 87.92% |  87.77% |  70.68% |  76.25%
3rd-order Poly |  99.16% |  90.04 % |  88.64% |  87.93% |  70.95% |  76.32%
4nd-order Poly |  99.02% | 89.57% |  88.41% |  87.31% |  70.82% |  76.16%
5rd-order Poly | 98.99% | 89.71% | 87.13% | 86.04% | 69.65% | 75.80%
Gaussian RBF $\sigma=0.9$ | 98.61% | 88.78% | 87.51% |  87.33% |  70.78% |  76.23%

(a) MNIST

(b) Fashion-MNIST

(c) Cifar-10

(d) RAF-DB

(e) Fer2013

(f) ExpW

Figure 8: Convergence of full kervolution networks

Table 1 illustrates the accuracy rates obtained using our two base models with
a full kervolution configuration. In table 1 one can clearly notice that the
use of kervolution layers enhance the overall accuracy of the network,
especially with second, third and fourth order polynomial kernels. In
comparison to full convolution network, the full kervolution network enhances
the accuracy of Model-1 on MNIST dataset by 0.65% and 1.77% on Fashion-MNIST.
On the other hand, the accuracy of Model-2 increases by 1.73% on CIFAR-10,
0.88% on RAF-DB, 0.46% on FER2013 and 0.41% on ExpW. The best accuracy rates
are, in most cases, reached with third order polynomial kernel and decreases
with higher order polynomial kernels. We have also noticed that Gaussian RBF
kernel is more beneficial on fine-grained FER dataset than other datasets.
Therefore, we can deduce that Gaussian RBF kernels are more sensitive to
subtle details.

Figure 8 shows the validation accuracy rates evolution of our base models with
full convolution and kervolution configurations. We can clearly notice that
kervolution networks, especially second and third order polynomial kernels,
converge in fewer epochs than convolution network. On the other hand, Gaussian
RBF kernel takes more time to converge than the other kernels. We have also
noticed that, for fine-grained FER datasets, the accuracy curve fluctuates
accordingly to the kernel degree. This can be explained by the fact that
kernels are more sensitive to subtle changes in input data. Therefore, any
small change in the weights during the training phase can cause a big change
in the final classification decision.

Table 2: Accuracy rates with a single kervolution layer at the begining

| Model-1 | Model-2
---|---|---
Layers configuration | MNIST | Fashion-MNIST | Cifar10 | RAF-DB | FER2013 | ExpW
Convolution (Linear kernel) | 98.50% | 88.29% | 86.87% | 87.05% | 70.49% | 75.91%
2nd-order Poly | 98.87% | 89.74% | 87.81% | 87.62% | 70.82% | 76.64%
3rd-order Poly |  99.04% |  90.26% |  88.73% |  88.06% |  71.06% |  76.85%
4nd-order Poly |  99.04% |  90.61% |  88.52% |  87.88% | 70.88% |  76.53%
5rd-order Poly | 98.99% |  90.35% | 86.48% | 86.71% | 69.89% | 75.56%
Gaussian RBF $\sigma=0.9$ | 98.74% | 89.53% |  88.48% |  87.89% |  70.98% |  76.75%

Table 2 presents the accuracy rate results of our two base models, in which we
replaced the first convolution layer by a kervolution layer. In table 2 we can
also notice that the use of one kervolution layer at the beginning of the
network increases the accuracy of the network comparing to full convolution
network. Furthermore, the use of only one kervolution layer in the beginning
of the network allows to reach better results than full-kervolution network.
Compared to full convolution network, the accuracy rate increases up to 1.97%
on Fashion-MNIST, 1.86% on Cifar-10, 2% on RAF-DB, 0.57% on FER2013 and 0.94
on ExpW. Gaussian RBF kernels are slightly less accurate than third and fourth
polynomial kernels, yet they also increase the accuracy rate compared to full-
convolution network. They enhance the accuracy on Cifar-10 by 1.61%, 0.84% on
REF-DB, 0.49% on FER2013 and 0.84% on ExpW. Another remark is that higher
order kernels are more accurate than lower order kernel when used only at the
beginning of the network. This may be explained by the fact that higher order
kernels can fit to subtle details more efficiently than lower order kernels.
Therefore, the network learns to detect more useful information than full-
convolution network and will be less prone to over-fitting than full-
kervolution network.

(a) MNIST

(b) Fashion MNIST

(c) Cifar-10

(d) RAF-DB

(e) FER2013

(f) A ExpW

Figure 9: Convergence of networks with one kervolution layer at the begining

Figure 9 shows the accuracy convergence of our two base models, in which we
replaced the first convolution layer by a kervolution layer. As illustrated in
figure 9, using one kervolution layer at the beginning of the network allows
the latter to converge in less time than the full convolution network.
Similarly to full-kervolution configuration, second and third degree
polynomial are the fastest to converge in all cases. Higher degree polynomial
layers are also faster to converge than full convolution configuration except
for FER datasets. On the other hand, Gaussian RBF layers are the slowest to
converge, though they surpass convolution on MNIST, Fashion MNIST and
Cifar-10.

Table 3: Accuracy rates with a single kervolution layer at the end

| Model-1 | Model-2
---|---|---
Layers configuration | MNIST | Fashion-MNIST | Cifar10 | RAF-DB | FER2013 | ExpW
Convolution (Linear kernel) | 98.50% | 88.29% | 86.87% | 87.05% | 70.49% | 75.91%
2nd-order Poly | 98.89% | 88.37% | 87.11% | 87.63% | 70.81% | 76.24%
3rd-order Poly | 98.84% | 88.45% | 87.23% | 88.03% | 70.95% | 76.82%
4nd-order Poly | 98.62% | 87.76% | 87.09% | 87.85% | 70.75% | 76.66%
5rd-order Poly | 98.72% | 87.45% | 86.97% | 87.15% | 69.76% | 75.45%
Gaussian RBF $\sigma=0.9$ |  98.75% |  89.33% |  90.13% |  89.36% |  71.15% |  77.21%

Table 3 presents the accuracy rate results of our two base models, in which we
replaced the last convolution layer by a kervolution layer. In table 3 one can
notice that the use of polynomial kervolution layers at the end of the network
decreases its accuracy compared to full-kervolution configuration and
kervolution at the beginning. Yet, it performs better than full-convolution
configuration. Indeed, we could surpass full-convolution configuration by
0.34% on MNIST, 0.16% on Fashion MNIST, 0.36% on Cifar-10, .0.98% on RAF-DB,
0.46% on FER2013 and 0.91% on ExpW. The best accuracy rates where reached with
third degree polynomial kernel. Other polynomial kernels also perform slightly
better than full-convolution configuration, but are less accurate than the
other kervolution configurations. On the other hand, Gaussian RBF kernels
increase remarkably the overall accuracy of the network compared to full-
convolution configuration and other Gaussian RBF kervolution configurations.
Indeed, it increases the overall accuracy rates by 0.25% on MNIST, 1.04% on
Fashion MNIST, 3.26% on Cifar-10, 2.31% on RAF-DB, 0.66% on FER2013 and 1.30%
on ExpW. With this has been said, we deduce that polynomial kervolution layers
are more suited for feature extraction which explains why they are more
accurate when plugged at the beginning of the network. On the other hand,
Gaussian RBF kervolution layers are more beneficial when plugged at the end of
the network than the beginning. This can be explained by the fact that
Gaussian RBF kernels are more accurate for classification than for feature
extraction.

(a) MNIST

(b) Fashion MNIST

(c) Cifar-10

(d) RAF-DB

(e) FER2013

(f) A ExpW

Figure 10: Convergence of networks with one kervolution layer at the end

Figure 10 shows the accuracy convergence of our two base models, in which we
replaced the last convolution layer by a kervolution layer. We can clearly
notice that the use of one kervolution layer at the end of the network quicken
remarkably its convergence. This impact on convergence speed is valid with all
kernels. On the other hand, even-though Gaussian RBF kernel reaches the
highest accuracy rates among all kernels, it is still the slowest kernel to
converge. Polynomial kernels, on the other hand, are still the fastest in
convergence.

According to these results, we can say that the use of kervolution layers have
clearly a beneficial impact on the accuracy rate and convergence speed of the
network. As we have seen above, this positive impact is noticeable wherever
kervolution layers are used. In terms of accuracy, using one kervolution layer
at the beginning of the network shown to be the most efficient configuration.
Whereas, in terms of convergence speed, using one kervolution layer at the end
of the network shown to be the fastest to converge. Full-kervolution
configuration is, for its part, average in both accuracy and convergence speed
compared with the above configurations. In addition, full-kervolution
configuration is more prone to overfitting than the other kervolution
configurations.

####  4.3.2 Learnable weights pooling

Similarly to kervolution, we explore the impact of using learnable pooling
with three configurations, namely: full learnable pooling network (Table 4,
Figure 11), learnable pooling after the first convolution block (Table 5,
Figure 12) and learnable pooling at the end of the network (Table 6, Figure
13). This time, we compare the performance of the network with the usual
pooling methods, namely: Max and average pooling.

Table 4: Accuracy rates of full learnable pooling network

| Model-1 | Model-2
---|---|---
Layers configuration | MNIST | Fashion-MNIST | Cifar10 | RAF-DB | FER2013 | ExpW
Max |  99.05% |  91.34% | 88.42% | 87.05% | 70.49% | 75.91%
AVG |  99.17% |  91.37% | 88.12% | 86.89% | 70.13% | 75.74%
Linear kernel | 98.85% |  90.05% |  90.25% |  90.81% | 70.96% | 76.28%
2nd-order Poly |  99.26% |  91.42% |  90.62% |  92.87% | 70.88% |  76.64%
3rd-order Poly |  99.35% |  91.79% |  90.97% |  93.21% |  71.35% |  76.81%
4nd-order Poly | 98.73% | 88.56% | 89.88% |  93.03% |  71.13% |  76.73%
5rd-order Poly | 98.52% | 87.91% | 89.61% |  92.64% | 69.91% | 75.56%
Gaussian RBF $\sigma=0.9$ |  99.11% |  91.32% |  90.45% |  92.74% | 70.74% | 76.42%

Table 4 presents the results of our base models with learnable pooling only.
One can clearly notice that learnable pooling enhance the accuracy of the
network with all kernel functions. Second and third order polynomial kernels
are the most accurate kernels for pooling with all datasets. With third order
polynomial kernel Model-1 surpassed usual pooling methods by 0.18% on MNIST
and 0.42% on Fashion-MNIST. On the other hand, Model-2 outperformed usual
pooling method by 2.55% on Cifar-10, 6.15% on RAF-DB, 0.86% on FER2013 and
0.90% on ExpW. We can also notice that for fine-grained FER datasets, higher
order polynomial kernels are also beneficial for the model for the same
reasons cited for kervolution at the beginning of the network. Indeed,
learnable pooling uses less weights than convolution or kervolution which
allows it to fit subtle details without over-fitting. Finally, Gaussian RBF
learnable pooling performs better than usual pooling methods even-though it
does not outperform polynomial kernels. Although it did not surpass usual
pooling method on model-1, it outperformed ordinary pooling methods with
Model-2 by 2.03% on Cifar-10, 5.69% on RAF-DB, 0.25% on FER2013 and 0.51 on
ExpW.

(a) MNIST

(b) Fashion MNIST

(c) FER20132

(d) RAF-DB

(e) FER2013

(f) A ExpW

Figure 11: Convergence of networks with learnable pooling

The convergence progression of full learnable pooling networks is shown in
Figure 11. We can notice that second and third order polynomial kernels are
the fastest to converge. On the other hand, higher order polynomial kernels
convergence is the slowest among all kernels, even if they reach higher
accuracy rates than Max and average pooling for FER datasets. Finally,
Gaussian RBF kernel has similar accuracy progression to Max and average
pooling.

Table 5: Accuracy rates networks with a single learnable pooling layer at the
begining

| Model-1 | Model-2
---|---|---
Layers configuration | MNIST | Fashion-MNIST | Cifar10 | RAF-DB | FER2013 | ExpW
Max |  99.05% |  91.34% | 88.42% | 87.05% | 70.49% | 75.91%
AVG |  99.17% |  91.37% | 88.12% | 86.89% | 70.13% | 75.74%
Linear kernel |  99.11% |  90.58% |  88.15% |  87.19% |  70.26% |  75.95%
2nd-order Poly | 98.97% | 89.80% | 87.71% | 86.92% | 70.08% | 75.82%
3rd-order Poly |  99.24% | 89.88% | 87.54% | 86.79% | 69.92% | 75.23%
4nd-order Poly |  99.12% |  90.32% | 87% | 86.57% | 69.79% | 74.84%
5rd-order Poly |  99.10% |  90.24% | 86.89% | 86.38% | 69.54% | 74.88%
Gaussian RBF $\sigma=0.9$ | 98.43% | 88.93% | 85.71% | 84.12% | 68.63% | 73.21%

In table 5 we notice that using one learnable pooling layer after the first
convolution block does not perform as well as the full learnable pooling
network. Even-though linear kernel outperforms the usual pooling methods. As
stated before, linear kernel pooling can learn a suitable pooling from a
continuum of methods that ranges from average to max pooling. This is the
reason why linear kernel pooling performs well wherever it is plugged.

(a) MNIST

(b) Fashion MNIST

(c) Cifar-10

(d) RAF-DB

(e) FER2013

(f) A ExpW

Figure 12: Networks convergence with learnable pooling at the beginning

The convergence progression of learnable pooling at the beginning of the
networks is shown in Figure 12. We can notice that polynomial kernels are, in
general, the fastest to converge. Second and third order polynomial kernels
are still the most efficient among these kernels. On the other hand, Gaussian
RBF kernel is the slowest kernel to converge, specially for the biggest FER
datasets (i.e. FER2013 and ExpW). This kernel is more sensitive to the
learning rate used. As we can see in Figure 12, Gaussian RBF kernel does not
start to converge until reaching a specific learning rate.

Table 6: Accuracy rates networks with a single learnable pooling layer at the
end

| Model-1 | Model-2
---|---|---
Layers configuration | MNIST | Fashion-MNIST | Cifar10 | RAF-DB | FER2013 | ExpW
Max |  99.05% |  91.34% | 88.42% | 87.05% | 70.49% | 75.91%
AVG |  99.17% |  91.37% | 88.12% | 86.89% | 70.13% | 75.74%
Linear kernel | 98.93% | 90.42% | 87.52% |  87.12% |  70.06% |  75.87%
2nd-order Poly | 98.85% | 89.94% | 87.28% | 86.89% | 69.87% | 75.66%
3rd-order Poly | 98.78% | 89.72% | 87.04% | 86.75% | 69.71% | 75.48%
4nd-order Poly | 98.62% | 89.37% | 86.79% | 86.69% | 69.58% | 75.36%
5rd-order Poly | 98.90% | 89.10% | 86.63% | 86.42% | 69.22% | 74.81%
Gaussian RBF $\sigma=0.9$ |  99.03% |  90.36% |  88.09% |  86.94% |  70.11% |  75.79%

Similarly to learnable pooling at the beginning of the network (Table 5),
using one learnable pooling layer after the last convolution layer and before
fully connected layers (Table 6) does not perform as well as the full
learnable pooling network. One exception is the Gaussian RBF kernel which
slightly performs better at the end than at the beginning. This last remark
strengthens the deduction made on kervolution at the end of the network which
states that Gaussian RBF kernel is best suited for classification than feature
extraction or even pooling.

The convergence speed of learnable pooling at the end of the network is
illustrated in Figure 13. One can clearly notice that kernel based pooling
does not perform as well as when plugged at the beginning or in a full
configuration. It performs similarly to the usual pooling methods. On the
other hand, Gaussian RBF kernels have a faster convergence when used at the
end of the network. Moreover, it outperforms other pooling method, specially
with FER dataset. This confirms the assumption made above that Gaussian RBF
kernel is more efficient when used just before the fully connected layers.

(a) MNIST

(b) Fashion MNIST

(c) Cifar-10

(d) RAF-DB

(e) FER2013

(f) ExpW

Figure 13: Networks convergence with a single learnable pooling layer at the
end

####  4.3.3 kernelized Dense Layers

kernelized Dense Layers are only plugged at the end of the network since they
replace the fully connected layers. Table 7 shows the result of using
kernelized Dense Layers instead of the usual fully-connected layers. One can
clearly notice that using kernel function for classification improves the
accuracy of the network. For instance, polynomial kernel enhances the accuracy
of the network for about 1% with fine-grained FER datasets. Third and Fourth
order polynomial kernels are those who gave the best result with all datasets.
On the other hand, Gaussian RBF kernel results confirm the deduction made
trough this experimental section that it is best suited for classification
than feature extraction. Whenever it is used at the end of the network it
performs better than when used in the beginning.

Table 7: Accuracy rates of networks with KDL

| Model-1 | Model-2
---|---|---
Layers configuration | MNIST | Fashion-MNIST | Cifar10 | RAF-DB | FER2013 | ExpW
Linear kernel | 99.07% |  90.07% | 86.92% |  87.05% | 70.49% | 75.91%
2nd-order Poly | 99.12% |  90.73% | 87.61% |  87.64% | 70.85% | 76.13%
3rd-order Poly |  99.11% |  90.63% |  89.37% |  88.12% |  71.28% |  76.64%
4nd-order Poly | 99.09% |  90.30% |  89.09% |  87.83% |  71.13% |  76.42%
5rd-order Poly |  99.01% |  91.07% | 88.95% | 86.93% | 70.62% | 75.86%
Gaussian RBF $\sigma=0.9$ |  99.23% |  90.79% |  89.11% |  88.03% |  71.06% |  76.51%

(a) MNIST

(b) Fashion MNIST

(c) Cifar 10

(d) RAF-DB

(e) FER2013

(f) A ExpW

Figure 14: Networks convergence with KDL

Figure 14 shows the convergence of networks with KDL. We can say that
polynomial kernels converge faster than other kernels. Even-though second and
third order polynomial reach better accuracy rates than other polynomial
kernels, they have not better convergence. On the other hand, Gaussian RBF
kernel shows the slowest convergence even if it gives the best accuracy rates.

We have also tried to combine these kernel-based methods together in the same
network. The number of all possible configurations is very big, therefore we
only show the configuration which gave the best results. These results are
shown in Table 8. All the combinations tested shown a dramatically decreasing
in performance. The only reasonable performances we could attain were using
one kervolution layer at the beginning of the network and KDL at the end. Yet
we have noticed that this configuration shows a clear over fitting.

Table 8: Accuracy rates of networks with best combinations

| Model-1 | Model-2
---|---|---
Layers configuration | MNIST | Fashion-MNIST | Cifar10 | RAF-DB | FER2013 | ExpW
Single Kervolution layer at the beginning and KDL
2rd-order Poly (1kerv-KDL) |  99.16% |  91.19% | 65.86% | 72.81% | 62.41% | 58.72%
3rd-order Poly (1kerv-KDL) | 98.96% | 90.23% | 65.75% | 72.64% | 61.73% | 57.96%
4nd-order Poly (1kerv-KDL) | 98.88% | 89.64% | 61.87% | 69.58% | 59.86% | 57.41%
Gaussian RBF $\sigma=0.9$ (1kerv-KDL) | 98.57% | 88.92% | 61.20% | 68.12% | 59.12% | 56.87%

###  4.4 Comparison with the State-of-the-Art

In this section, we compare the performance of the proposed methods, namely:
kervolution, learnable pooling and KDL, with respect to several state-of-the-
art methods. According to table 9, we obtained state-of-the-art results on two
FER datasets (RAF-DB and ExpW) with all the proposed methods. For the
remaining datasets namely MNIST, Fashion MNIST, Cifar-10 and FER2013, we could
not surpass state-of-the-art results, yet we obtained very close results. In
the following, we give the best results of each configuration.

With full kervolution network (Table 1), the highest accuracy rate we reached,
was using third order polynomial kernel. It gave 99.16% on MNIST, 87.93% on
RAF-DB and 76.32% on ExpW. Kervolution attains better results when used at the
beginning of the network (Table 2). With this configuration, we got 99.04 on
MNIST, 88.73% on RAF-DB and 76.85% on ExpW using third order polynomial
kernel. On the other hand, when using kervolution at the end of the network
(Table 3), the Gaussian RBF reaches the best results. It attains 89.36% on
RAF-DB and 77.21% on ExpW.

Learnable pooling is the method that allowed to reach the best results. With
the full learnable pooling configuration (Table 4), we could reach the best
results on all datasets using third order polynomial kernel. We got very close
results to state-of-the-art with 99.35% on MNIST, 91.79% on Fashion MNIST,
90.97% on Cifar-10 and 71.35% on FER2013. We have also reached state-of-the-
art result on RAF-DB and ExpW with respectively 93.21% and  76.81%. Using
learnable pooling after the first convolution block (Table 5), we could not
surpass full learnable pooling network. The best results were obtained with
linear kernel with 87.19% on RAF-DB and 75.95% on Expw. The same results were
obtained when using learnable pooling at the end of the network, with a slight
improvement for Gaussian RBF kernel. Yet linear kernel remains the more
accurate with 87.12% for RAF-DB and 75.87% for ExpW.

Finally, KDL (Table 7) gives very good results with all the used kernels. On
MNIST dataset, all the results are above 99% and reached 99.23% with Gaussian
RBF kernel. It also reached 88.03% on RAF-DB and 76.51 on ExpW. The best KDL
results were obtained with third order polynomial kernel. It reached 88.12% on
RAF-DB and 76.64% on ExpW.

##  5 Discussion

In this paper, we extensively studied the impact of using kernel functions on
different levels of a CNN, in particular, convolution, pooling and fully-
connected layers. We replaced convolution layer by a non-linear layer as
introduced by Wang et al. [Wang et al., 2019]. We tested the performance of
this layer following three network configurations. We first tested them
solely, in a full configuration network. After that, we tested them jointly
with the usual convolution layers, plugging them either at the beginning of
the network or at the end.

We have also used a novel pooling layer, based on kernel functions, that keeps
the down-scaling aspect of the standard pooling function and brings various
new features. This pooling layer relay on learnable weights that generalize
ordinary pooling operations (i.e. average pooling and max pooling).
Furthermore, it encodes patch-wise non-linearity. In this manner, the
discrimination power of the full network is enhanced. The novel pooling,
called learnable weights pooling, can be used at any level of the network and
is fully differentiable, which allows the network to be trained in an end-to-
end training. We have also explored their impact following the same network
configuration as stated above.

We also use a novel fully-connected layer in which we use kernel functions to
create a neuron unit that uses a higher degree kernel function on its inputs
rather than computing the weighted sum. The used layer, called Kernelized
Dense Layers (KDL), is also differentiable and demonstrates its usefulness in
the improvement of the discrimination power of the full network.

For the three proposed layers, we explored their impact on the overall
accuracy and convergence speed of the network. The results illustrated in
Tables 1\- 7 and Figures 8\- 14 allow us to deduce the following. Kervolution
layer is best suited for feature extraction and gives the best of its results
when used at beginning of the network conjointly with convolution layers. It
is more accurate with fine-grained FER datasets which means that it is more
sensitive to subtle details. Adding more kervolution layers only increases
over-fitting. While using kervolution at the end of the network decreases its
performance. On the other hand, learnable pooling works best when used in full
network configuration. It has the same advantages of kervolution in term of
being more sensitive to subtle details with fewer parameters, which prevents
it from over-fitting. The performance of learnable pooling decreases when used
only at the beginning of the network or at the end, compared to the full
network configuration. Finally, KDL is the most stable proposed layer. It
shows good performance with all kernels and datasets.

Table 9: Accuracy rates of networks with best combinations

| Model-1 | Model-2
---|---|---
Layers configuration | MNIST | Fashion-MNIST | Cifar10 | RAF-DB | FER2013 | ExpW
Full kervolution networks
3rd-order Poly |  99.16% |  90.04 % |  88.64% |  87.93% |  70.95% |  76.32%
Gaussian RBF $\sigma=0.9$ | 98.61% | 88.78% | 87.51% |  87.33% |  70.78% |  76.23%
Single kervolution layer at the beginning
3rd-order Poly |  99.04% |  90.26% |  88.73% |  88.06% |  71.06% |  76.85%
Gaussian RBF $\sigma=0.9$ | 98.74% | 89.53% |  88.48% |  87.89% |  70.98% | 76.75%
Single kervolution layer at the end
3rd-order Poly | 98.84% | 88.45% | 87.23% | 88.03% | 70.95% | 76.82%
Gaussian RBF $\sigma=0.9$ |  98.75% |  89.33% |  90.13% |  89.36% |  71.15% |  77.21%
Full learnable pooling
3rd-order Poly |  99.35% |  91.79% |  90.97% |  93.21% |  71.35% |  76.81%
Gaussian RBF $\sigma=0.9$ |  99.11% |  91.32% |  90.45% |  92.74% | 70.74% | 76.42%
Learnable pooling at the beginning
3rd-order Poly |  99.24% | 89.88% | 87.54% | 86.79% | 69.92% | 75.23%
Gaussian RBF $\sigma=0.9$ | 98.74% | 89.53% |  88.48% |  87.89% |  70.98% |  76.75%
Learnable pooling at the end
Linear kernel | 98.93% | 90.42% | 87.52% |  87.12% |  70.06% |  75.87%
Gaussian RBF $\sigma=0.9$ |  99.03% |  90.36% |  88.09% |  86.94% |  70.11% |  75.79%
Kernelized Dense Layer
3rd-order Poly |  99.11% |  90.63% |  89.37% |  88.12% |  71.28% |  76.64%
Gaussian RBF $\sigma=0.9$ |  99.23% |  90.79% |  89.11% |  88.03% |  71.06% |  76.51%
State-of-the-art results
Byerly et al [Byerly et al., 2020] | 99.84% | – | – | – | – | –
Assiri [Assiri, 2020] | 99.83% | – | – | – | – | –
Jayasundara et al [Jayasundara et al., 2019] | 99.71% | 96.36% | – | – | – | –
Kolesnikov et al [Kolesnikov et al., 2019] | – | – | 99.30% | – | – | –
Huang et al [Huang et al., 2019] | – | – | 99% | – | – | –
Ridnik et al [Ridnik et al., 2020] | – | – | 99% | – | – | –
Tang et al. [Tang, 2013] | – | – | – | – | – | 71.16%
Guo et al. [Guo et al., 2016] | – | – | – | – | – | 71.33%
Kim et al. [Kim et al., 2016] | – | – | – | – | – | 73.73%
Bishay et al. [Bishay et al., 2019] | – | – | – | – | 73.10% | –
Lian et al. [Lian et al., ] | – | – | – | – | 71.90 % | –
Acharya et al. [Acharya et al., 2018b] | – | – | – | 87% | – | –
S Li et al. [Li & Deng, 2018] | – | – | – | 74.20% | – | –
Z.Liu et al. [Liu et al., 2017] | – | – | – | 73.19% | – | –

In terms of kernels, we have also noticed that they may perform differently
when used at different levels. For instance, polynomial kernels work best as
feature extractors. They work better when used at the beginning of the network
only; as stated in [Wang et al., 2019]. On the other hand, Gaussian RBF
kernels work best when used at the end of the network. Either as feature
extraction or classification layer.

##  6 Conclusion

In light of the results obtained in the experiments, we have conducted in this
work, we deduce that kernel functions impact positively the CNN performance.
Indeed, the experimental results prove that the use of kernel functions,
instead of the linear functions used in CNN layers, improves the accuracy rate
of the whole model and its convergence speed. Furthermore, we concluded that
the impact of these kernel functions varies according to the level in which
they are plugged into and the specific kernel in use. We have also concluded
that high order kernels prone to overfitting, whereas low order kernels might
not be sufficiently effective to fit the input data distributions. Therefore,
a trade-off between complexity and performance should be reached.

## References

  * Acharya et al. [2018a] Acharya, D., Huang, Z., Pani Paudel, D., & Van Gool, L. (2018a).  Covariance pooling for facial expression recognition.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
  * Acharya et al. [2018b] Acharya, D., Huang, Z., Pani Paudel, D., & Van Gool, L. (2018b).  Covariance pooling for facial expression recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 367–374).
  * Assiri [2020] Assiri, Y. (2020).  Stochastic optimization of plain convolutional neural networks with simple methods.  arXiv preprint arXiv:2001.08856, .
  * Bishay et al. [2019] Bishay, M., Palasek, P., Priebe, S., & Patras, I. (2019).  Schinet: Automatic estimation of symptoms of schizophrenia from facial behaviour analysis.  IEEE Transactions on Affective Computing, .
  * Burges et al. [1999] Burges, C. J., Scholkopf, B., & Smola, A. J. (1999).  Advances in kernel methods: support vector learning.  MIT press Cambridge, MA, USA:.
  * Byerly et al. [2020] Byerly, A., Kalganova, T., & Dear, I. (2020).  A branching and merging convolutional network with homogeneous filter capsules.  arXiv preprint arXiv:2001.09136, .
  * Chen et al. [2020] Chen, S., Zhao, Y., Jin, Q., & Wu, Q. (2020).  Fine-grained video-text retrieval with hierarchical graph reasoning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10638–10647).
  * Clevert et al. [2015] Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015).  Fast and accurate deep network learning by exponential linear units (elus).  arXiv preprint arXiv:1511.07289, .
  * Cui et al. [2017] Cui, Y., Zhou, F., Wang, J., Liu, X., Lin, Y., & Belongie, S. (2017).  Kernel pooling for convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2921–2930).
  * Gao et al. [2016] Gao, Y., Beijbom, O., Zhang, N., & Darrell, T. (2016).  Compact bilinear pooling.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 317–326).
  * Gao et al. [2020] Gao, Y., Han, X., Wang, X., Huang, W., & Scott, M. (2020).  Channel interaction networks for fine-grained image categorization.  In AAAI (pp. 10818–10825).
  * Gao et al. [2019] Gao, Z., Xie, J., Wang, Q., & Li, P. (2019).  Global second-order pooling convolutional networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3024–3033).
  * Goodfellow et al. [2013] Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H. et al. (2013).  Challenges in representation learning: A report on three machine learning contests.  International Conference on Neural Information Processing, Springer, (pp. 117–124).
  * Guo et al. [2016] Guo, Y., Tao, D., Yu, J., Xiong, H., Li, Y., & Tao, D. (2016).  Deep neural networks with relativity learning for facial expression recognition.  In 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) (pp. 1–6).  IEEE.
  * Huang et al. [2019] Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y. et al. (2019).  Gpipe: Efficient training of giant neural networks using pipeline parallelism.  In Advances in Neural Information Processing Systems (pp. 103–112).
  * Huang & Li [2020] Huang, Z., & Li, Y. (2020).  Interpretable and accurate fine-grained recognition via region grouping.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8662–8672).
  * Hyun et al. [2019] Hyun, J., Seong, H., & Kim, E. (2019).  Universal pooling–a new pooling method for convolutional neural networks.  arXiv preprint arXiv:1907.11440, .
  * Jayasundara et al. [2019] Jayasundara, V., Jayasekara, S., Jayasekara, H., Rajasegaran, J., Seneviratne, S., & Rodrigo, R. (2019).  Textcaps: Handwritten character recognition with very small datasets.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 254–262).  IEEE.
  * Ji et al. [2020] Ji, R., Wen, L., Zhang, L., Du, D., Wu, Y., Zhao, C., Liu, X., & Huang, F. (2020).  Attention convolutional binary neural tree for fine-grained visual categorization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10468–10477).
  * Kim et al. [2016] Kim, B.-K., Dong, S.-Y., Roh, J., Kim, G., & Lee, S.-Y. (2016).  Fusing aligned and non-aligned face information for automatic affect recognition in the wild: a deep learning approach.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, (pp. 48–57).
  * Klambauer et al. [2017] Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017).  Self-normalizing neural networks.  In Advances in neural information processing systems (pp. 971–980).
  * Kolesnikov et al. [2019] Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2019).  Large scale learning of general visual representations for transfer.  arXiv preprint arXiv:1912.11370, .
  * Krause et al. [2013] Krause, J., Stark, M., Deng, J., & Fei-Fei, L. (2013).  3d object representations for fine-grained categorization.  In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13).  Sydney, Australia.
  * Krizhevsky et al. [2009] Krizhevsky, A., Hinton, G. et al. (2009).  Learning multiple layers of features from tiny images, .
  * Li & Deng [2018] Li, S., & Deng, W. (2018).  Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition.  IEEE Transactions on Image Processing, 28, 356–370.
  * Li et al. [2017] Li, S., Deng, W., & Du, J. (2017).  Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild.  In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on (pp. 2584–2593).  IEEE.
  * [27] Lian, Z., Li, Y., Tao, J.-H., Huang, J., & Niu, M.-Y. ().  Expression analysis based on face regions in read-world conditions.  International Journal of Automation and Computing, (pp. 1–12).
  * Lin & Maji [2017] Lin, T.-Y., & Maji, S. (2017).  Improved bilinear pooling with cnns.  arXiv preprint arXiv:1707.06772, .
  * Lin et al. [2015] Lin, T.-Y., RoyChowdhury, A., & Maji, S. (2015).  Bilinear cnn models for fine-grained visual recognition.  In Proceedings of the IEEE international conference on computer vision (pp. 1449–1457).
  * Liu et al. [2017] Liu, Z., Li, S., & Deng, W. (2017).  Boosting-poof: boosting part based one vs one feature for facial expression recognition in the wild.  In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) (pp. 967–972).  IEEE.
  * Liu et al. [2020] Liu, Z., Lu, F., Wang, P., Miao, H., Zhang, L., Yang, R., & Zhou, B. (2020).  3d part guided image editing for fine-grained object understanding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11336–11345).
  * Mahmoudi et al. [2021a] Mahmoudi, M., Chetouani, A., Boufera, F., & Tabia, H. (2021a).  Deep kernelized network for fine-grained recognition.  In International Conference on Neural Information Processing (pp. 100–111).  Springer.
  * Mahmoudi [2022] Mahmoudi, M. A. (2022).  Deep learning for emotion recognition.  Ph.D. thesis.
  * Mahmoudi et al. [2020] Mahmoudi, M. A., Chetouani, A., Boufera, F., & Tabia, H. (2020).  Kernelized dense layers for facial expression recognition.  In 2020 IEEE International Conference on Image Processing (ICIP) (pp. 2226–2230).
  * Mahmoudi et al. [2020] Mahmoudi, M. A., Chetouani, A., Boufera, F., & Tabia, H. (2020).  Learnable pooling weights for facial expression recognition.  Pattern Recognition Letters, 138.
  * Mahmoudi et al. [2021b] Mahmoudi, M. A., Chetouani, A., Boufera, F., & Tabia, H. (2021b).  Improved bilinear model for facial expression recognition.  In Pattern Recognition and Artificial Intelligence. MedPRAI 2020. Communications in Computer and Information Science (pp. 47–59).  Springer volume 1322.
  * Mahmoudi et al. [2021c] Mahmoudi, M. A., Chetouani, A., Boufera, F., & Tabia, H. (2021c).  Taylor series kernelized layer for fine-grained recognition.  In 2021 IEEE International Conference on Image Processing (ICIP) (pp. 1914–1918).  IEEE.
  * Mahmoudi et al. [2022] Mahmoudi, M. A., Chetouani, A., Boufera, F., & Tabia, H. (2022).  Kernel-based convolution expansion for facial expression recognition.  Pattern Recognition Letters, 160, 128–134.
  * Mika et al. [1999] Mika, S., Ratsch, G., Weston, J., Scholkopf, B., & Mullers, K.-R. (1999).  Fisher discriminant analysis with kernels.  In Neural networks for signal processing IX: Proceedings of the 1999 IEEE signal processing society workshop (cat. no. 98th8468) (pp. 41–48).  Ieee.
  * Ridnik et al. [2020] Ridnik, T., Lawen, H., Noy, A., & Friedman, I. (2020).  Tresnet: High performance gpu-dedicated architecture.  arXiv preprint arXiv:2003.13630, .
  * Robert [2014] Robert, C. (2014).  Machine learning, a probabilistic perspective.
  * Schölkopf et al. [1998] Schölkopf, B., Smola, A., & Müller, K.-R. (1998).  Nonlinear component analysis as a kernel eigenvalue problem.  Neural computation, 10, 1299–1319.
  * Sitzmann et al. [2020] Sitzmann, V., Martel, J. N., Bergman, A. W., Lindell, D. B., & Wetzstein, G. (2020).  Implicit neural representations with periodic activation functions.  arXiv preprint arXiv:2006.09661, .
  * Tang et al. [2020] Tang, L., Wertheimer, D., & Hariharan, B. (2020).  Revisiting pose-normalization for fine-grained few-shot recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14352–14361).
  * Tang [2013] Tang, Y. (2013).  Deep learning using linear support vector machines.  arXiv preprint arXiv:1306.0239, .
  * Tenenbaum & Freeman [2000] Tenenbaum, J. B., & Freeman, W. T. (2000).  Separating style and content with bilinear models.  Neural computation, 12, 1247–1283.
  * Wang et al. [2019] Wang, C., Yang, J., Xie, L., & Yuan, J. (2019).  Kervolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 31–40).
  * Wang et al. [2020] Wang, Z., Wang, S., Yang, S., Li, H., Li, J., & Li, Z. (2020).  Weakly supervised fine-grained image classification via guassian mixture model oriented discriminative learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9749–9758).
  * Welinder et al. [2010] Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., & Perona, P. (2010).  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001 California Institute of Technology.
  * Xiao et al. [2017] Xiao, H., Rasul, K., & Vollgraf, R. (2017).  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv preprint arXiv:1708.07747, .
  * Zhang et al. [2018] Zhang, Z., Luo, P., Loy, C. C., & Tang, X. (2018).  From facial expression recognition to interpersonal relation prediction.  International Journal of Computer Vision, Springer, 126, 550–569.
  * Zhuang et al. [2020] Zhuang, P., Wang, Y., & Qiao, Y. (2020).  Learning attentive pairwise interaction for fine-grained classification.  In AAAI (pp. 13130–13137).
  * Zoumpourlis et al. [2017] Zoumpourlis, G., Doumanoglou, A., Vretos, N., & Daras, P. (2017).  Non-linear convolution filters for cnn-based learning.  In Proceedings of the IEEE International Conference on Computer Vision (pp. 4761–4769).