#  Deep Neural-Kernel Machines

Siamak Mehrkanoon  Department of Data Science and Knowledge Engineering,
Maastricht University, The Netherlands

###### Abstract

In this chapter we review the main literature related to the recent
advancement of deep neural-kernel architecture, an approach that seek the
synergy between two powerful class of models, i.e. kernel-based models and
artificial neural networks. The introduced deep neural-kernel framework is
composed of a hybridization of the neural networks architecture and a kernel
machine. More precisely, for the kernel counterpart the model is based on
Least Squares Support Vector Machines with explicit feature mapping. Here we
discuss the use of one form of an explicit feature map obtained by random
Fourier features. Thanks to this explicit feature map, in one hand bridging
the two architectures has become more straightforward and on the other hand
one can find the solution of the associated optimization problem in the
primal, therefore making the model scalable to large scale datasets. We begin
by introducing a neural-kernel architecture that serves as the core module for
deeper models equipped with different pooling layers. In particular, we review
three neural-kernel machines with average, maxout and convolutional pooling
layers. In average pooling layer the outputs of the previous representation
layers are averaged. The maxout layer triggers competition among different
input representations and allows the formation of multiple sub-networks within
the same model. The convolutional pooling layer reduces the dimensionality of
the multi-scale output representations. Comparison with neural-kernel model,
kernel based models and the classical neural networks architecture have been
made and the numerical experiments illustrate the effectiveness of the
introduced models on several benchmark datasets.

###### keywords:

Neural Networks , Kernel machines , convolutional neural network , explicit
feature mapping

††journal: arXiv

##  1 Introduction

Deep Learning techniques have recently achieved the state-of-the-art
performances in different application domains. Thanks to their representation
learning power, they can analyse complex tasks by learning from subtasks [1,
2] . In particular, it learns multiple levels of hierarchical representation
from the given raw input data by means of successive nonlinear modules that
are stacked in a hierarchical architecture. Thanks to the the staked nonlinear
layers, the learnt representation (features) at one level is transformed into
a slightly more abstract representation at a higher level [3]. Recent years
have witnessed the significant impact of various deep learning architectures
including for instance Restricted Boltzmann Machines [4, 5, 6], Stacked
Denoising Autoencoders [7, 8], , Convolutional Neural Networks [9, 10], Long
Short Term Memories [11] among others. Deep architectures have shown their
prominent superiority over other machine learning in several application
domains [1, 12]. However, it should be noted that training deep neural
networks is a labeled data demanding process and involves optimizing an
expensive non-convex optimization problem. A deep artificial neural networks
model often posses thousands to millions of parameters to be learned, which
heavily influence its generalization performance. In addition, searching the
proper architecture such as the type of activation functions, the number of
hidden units/layers, the networks hyper-parameters, the learning rates among
others, become a laborious task when the complexity of deep model increases.
Moreover training deep networks demand lots of computing and memory resources.
On the other kernel based models such as support vector machines (SVM) have
also made a large impact in a wide range of application domains [13, 14, 15].
kernel based models are well established with strong foundations in learning
theory and optimization. They are well suited for problems with limited
training instances and are able to extend linear methods to nonlinear ones
with theoretical guarantees [3]. However, in their classical formulation, they
cannot learn features from raw data and do not scale well to the size of the
training datasets. One often requires to choose a relevant kernel amongst the
well known ones a priori, or defines/learns an appropriate kernel for the data
under study. Deep learning models and Kernel machines have mostly been
investigated separately. Furthermore, most of the developed successful deep
models are based on artificial neural networks (ANN) architecture. On the
other hand, deep kernel machines have not yet been explored in great detail.
Both deep learning models and kernel machines have their strengths and
weaknesses and can potentially be considered as complementary family of
methods with respect to the settings where they are most relevant. Therefore,
exploring hybridization between ANN and kernel machines appears worthwhile to
pursue and can lead to the development of models that benefit from the
advantages of the two frameworks.

The literature has already witnessed such research direction for instance one
can refer to kernel models for deep learning [16], deep Gaussian processes
[17, 18], convolutional kernel networks [19] and a convex deep learning model
using normalized kernels [20]. The iterated compositions of Gaussian kernels
are investigated in [21]. The authors in [19] proposed a kernel based
convolutional neural network. Other deep kernel models that allow working with
a hierarchy of representations are discussed in [16, 22, 23, 24]. A brief
overview of existing research works on hybridization techniques for bridging
artificial neural networks and kernel machines are discussed in [25].

Mehrkanoon et al. in [26, 3, 27, 28] introduced a new line of research that
aims at combining the kernel machines and artificial neural networks
architectures in order to achieve the best of two class of models. To this
end, they introduced hybrid layers constructed by means of kernel functions
that are used in a neural networks architecture. In particular, the authors in
[3] introduced neural-kernel model by means of deploying hybrid layers
constructed by explicit feature map using random Fourier features in the
neural network architectures. In related study, the author in [27] explored
the use of different choices of merging layers and introduced three types of
kernel blocks. i.e. average, maxout and convolutional kernel blocks.

It is the purpose of this chapter to give a detailed overview of the recent
advancements on the neural-kernel studies. Precisely, we start by presenting
the basics of kernel as well as artificial neural networks based modelling and
their differences. Next the use of an explicit and implicit feature mapping in
kernel models are discussed and in particular an overview of two popular
approximation techniques for constructing the explicit feature mapping is
given. Afterwards, the neural-kernel model formulation and its various choices
of architectures are explained. Lastly, the performance of the described
models on several real-life benchmark datasets are analysed.

##  2 Kernel Machines vs Neural Networks

Kernel machines framework and in particular Support Vector Machine (SVM) is a
powerful methodology for solving complex pattern recognition and function
estimation tasks. In this method, one often works with a primal-dual setting
by expressing the solution in the primal in terms of feature maps and seeking
the optimal representation of the model in the dual. In the primal, the input
data are projected into a high dimensional feature space by means of an
implicit feature mapping. The projected data points are then mapped to a
target space using an inner product with a weight matrix [3], see Fig 1. The
feature mapping $\varphi(\cdot)$ is not in general explicitly known and can be
infinite dimensional. Therefore, the kernel trick is applied and the solution
is sought in the dual by solving a quadratic programming problem [29]. A
variant of SVM is the Least Squares SVM (LS-SVM) where the problem formulation
involves equality instead of inequality constraints. This leads to a system of
linear system of equations at the dual level, in the context of regression,
classification, semi-supervised learning and domain adaptation [29, 30, 31].
Thanks to the primal-dual setting, the prior knowledge and side information
can be incorporated in the primal formulation using proper regularization
terms. For instance the authors in [32, 33, 34] proposed a systematic approach
based on LS-SVM for learning the solutions of different types of differential
equations. Furthermore, in [35, 36] an extension of LS-SVM formulation have
been proposed for parameter estimation of dynamical systems governed by
ordinary as well as delay differential equations. It should be noted that in
the kernel based model the size of dual problem depends on the number of
training samples. In case of large scale problems solving the problem becomes
memory demanding. Therefore, when the number of training instances is much
larger than that of feature dimensions, we can avoid solving the problem in
the dual by constructing an explicit feature map and seeking the solution in
the primal. In contrary, if the number of variables (feature dimensions) is
much larger than the that of training instances then thanks the kernel trick
one can seek the solution by solving the dual optimization problem.

Figure 1:  Schematic illustration of Kernel based modelling

The neural networks architecture differs from kernel based models. In neural
networks modelling, the input data is projected into a new space via linear
transformation (multiplication with a weight matrix) followed by applying a
nonlinear activation function. One can consider the described operations in a
module and form a deep architecture by stacking several of these modules.
Thus, depending on the configurations of stacking these modules as well as the
used connection scheme, one can generate several choices of architectures
suitable for different modeling tasks. It should be noted that the projection
space of an implicit feature mapping in kernel machines corresponds to a
hidden layer in a neural network with infinite number of neurons [29]. The
bridge between neural network architectures and kernel models becomes even
closer when an explicit feature map is used. More precisely, the number of
hidden units in the hidden layer of a neural network architecture corresponds
to the dimension of the explicit feature map used in kernel machines.

###  2.1 Implicit and Explicit Feature Mapping

There are various kernel based models which follow different approaches. For
instance in Reproducing Kernel Hilbert Spaces (RKHS) [37] the problem of
function estimation is treated as a variational problem. The Gaussian
Processes (GP) [38] follow a probabilistic Bayesian setting. Here we focus on
the Least Squares Support vector machines (LS-SVM) which follows a primal-dual
formulation and any additional constraints can be added to the formulation and
therefore makes it straightforward to integrate prior knowledge into the model
[29]. Given training data points $\mathcal{D}=\\{x_{1},...,x_{n}\\}$, where
$\\{x_{i}\\}_{i=1}^{n}\in\mathbb{R}^{d}$ with targets $\\{y_{i}\\}_{i=1}^{n}$,
LS-SVM framework for regression problems assumes that the underlying function
describing the complex input-output relation of the system can be written in
the primal as follows [29]:

$y(x)=w^{T}\varphi(x)+b,$ (1)

where $\varphi(\cdot):\mathbb{R}^{d}\rightarrow\mathbb{R}^{h}$ is the feature
map and $h$ is the dimension of the feature space which can be infinite.
Thanks to the nonlinear feature map, the input data are embedded into a
feature space where the optimal solution is sought by minimizing the residual
between the real measurements and the model outputs via solving the following
optimization problem [29]:

$\displaystyle\underset{w,b,e}{\text{minimize}}$
$\displaystyle\frac{1}{2}w^{T}w+\frac{\gamma}{2}e^{T}e$ (2) subject to
$\displaystyle y_{i}=w^{T}\varphi(x_{i})+b+e_{i},\ \ i=1,...,n,$

where $\gamma\in\mathbb{R}^{+},b\in\mathbb{R}$, $w\in\mathbb{R}^{h}$. Here,
one may choose to work with implicit or explicit feature maps. Implicit
feature maps are usually unknown and infinite dimensional. Therefore, after
obtaining the Karush-Kuhn-Tucker (KKT) optimality conditions, applying the
kernel trick as well as eliminating the primal variables $e_{i}$ and $w$, the
solution is obtained in the dual as follows [29]:

$\left[\begin{array}[]{c|c}\Omega+\gamma^{-1}\mathbf{\mathit{I}}_{n}&1_{n}\\\
\hline\cr 1_{n}^{T}&0\\\ \end{array}\right]\left[\begin{array}[]{c}\alpha\\\
\hline\cr b\end{array}\right]=\left[\begin{array}[]{c}y\\\ \hline\cr
0\end{array}\right]$ (3)

where $\Omega_{ij}=K(x_{i},x_{j})=\varphi(x_{i})^{T}\varphi(x_{j})$ is the
$ij$-th entry of the positive definite kernel matrix.
$1_{n}=[1,\ldots,1]^{T}\in\mathbb{R}^{n}$,
$\alpha=[\alpha_{1},\ldots,\alpha_{n}]^{T}$, $y=[y_{1},\ldots,y_{n}]^{T}$ and
$I_{n}$ is the identity matrix. The LSSVM model for regression problem in the
dual form becomes [29]:

$y(x)=w^{T}\varphi(x)+b=\sum_{i=1}^{n}\alpha_{i}K(x,x_{i})+b.$ (4)

For large-scale problems, the cost of storing as well as computing the
solution vectors of (3) can be prohibitive due to the size of the kernel
matrix. An alternative approach would be for instance to use a low rank
approximation of the kernel matrix. Among kernel approximation techniques are
for instance Incomplete Cholesky Factorization [39], Nyström method [40],
randomized low-dimensional feature space [41] or reduced kernel technique [42,
43]. In this way one can construct an explicit feature map, and rewrite the
optimization problem (2) in the primal as follows:

$\displaystyle\underset{\hat{w},b}{\min}J(\hat{w},b)=\hat{w}^{T}\hat{w}+\frac{\gamma}{2}\sum_{i=1}^{n}(y_{i}-\hat{w}^{T}\hat{\varphi}(x_{i})-b)^{2},$
(5)

where $\hat{\varphi}(\cdot)$ is an explicit finite dimensional feature map.
The primal and dual LSSVM formulations corresponding to explicit and implicit
feature maps are depicted in Fig. 2.

Figure 2:  LS-SVM model formulation is the primal and dual. (Image adapted
from [3]).

Next, let us analyse the difference between a single layer neural network
model and a shallow kernel based model shown in Fig. 3. In artificial neural
network architectures the non-linearity is explicitly imposed by means of non-
linear activation functions which operate on the weighted sum of the input
layer. Among activation functions are hyperbolic tangent, sigmoid or Rectified
Linear Units (ReLU). Considering a single-layer with activation function $f$,
one can formulate the input-output relation as $y(x)=f(Wx+b)$. Here $x\in
x\in\mathbb{R}^{d}$, $W\in x\in\mathbb{R}^{d_{h}\times d}$ and $b$ is the bias
vector. On the contrary, in the kernel machines the nonlinear feature maps are
implicit, usually unknown and infinite dimensional. They directly operate on
the input instances and project them to a feature space. The model output then
is formulated as a weighted sum over the dimension of the projected sample. In
the case of explicit feature maps $\hat{\varphi}(\cdot)$, its dimension can be
larger or smaller than the dimension of the training instances. Moreover, a
kernel based model is linear in the weight matrix $W$, and therefore convex
optimization techniques can be applied to obtain an optimal values of $W$. In
what follows we discuss some of the approximation techniques for constructing
the explicit feature maps.

---

Figure 3:  (a) A single module of a neural network model. (b) A single module
of kernel based model with explicit feature mapping. (Images adapted from
[26]).

In the context of kernel machines, several methodologies are introduced in the
literature to scale up the Kernel models for analysing large scale problems.
Among them are kernel approximation techniques such greedy basis selection
approach [44], incomplete Cholesky decomposition [45], and Nyström method
[46], reduced kernel technique [42], random features approximation [47]. Among
the exiting methods, Nyström approximation [46] and random features
approximation [41] are the two popular ones. The former aims at approximating
the kernel matrix using a low-rank decomposition. The latter is based on
mapping the input features into a feature space where dot products between
elements of that space can well approximate the kernel function. In what
follows these two successful techniques will be discussed.

###  2.2 Nyström method

Consider the previously defined training data points
$\mathcal{D}=\\{x_{1},...,x_{n}\\}$. One can use an eigenvalue decomposition
of the kernel matrix $\Omega$ to obtain an explicit expression for feature map
$\varphi$. In particular, in Nyström approximation method, for the given
instance $x\in\mathbb{R}^{d}$, the $i$-th component of the $n$-dimensional
feature map $\hat{\varphi}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{n}$ can be
obtained as follows [29]:

$\hat{\varphi}_{i}(x)=\frac{1}{\sqrt{\lambda_{i}}}\sum_{k=1}^{n}u_{ki}\,K(x_{k},x),$
(6)

where $K(\cdot,\cdot)$ is the kernel function. The eigenvalues and
eigenvectors of the kernel matrix $\Omega_{n\times n}$ whose $(i,j)$-th
element is defined as $K(x_{i},x_{j})$ are denoted by $\lambda_{i}$ and
$u_{i}$. The $k$-th element of the $i$-th eigenvector is also denoted by
$u_{ki}$ denotes. It should be noted that when the sample size $n$ is large,
one can alternatively work with a subsample of size $m\ll n$. In this case,
the $m$-dimensional feature map
$\hat{\varphi}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{m}$ is approximated as
follows [29, 3]:

$\hat{\varphi}(x)=[\hat{\varphi}_{1}(x),\ldots,\hat{\varphi}_{m}(x)]^{T},$ (7)

where

$\hat{\varphi}_{i}(x)=\frac{1}{\sqrt{\lambda_{i}}}\sum_{k=1}^{m}u_{ki}\,K(x_{k},x),i=1,\ldots,m,$
(8)

where $\lambda_{i}$ and $u_{i}$ are eigenvalues and eigenvectors of the kernel
matrix $\Omega_{m\times m}$ which is constructed using the selected subsamples
(prototype vectors). The subsamples can be selected by several approaches
including incomplete Cholesky factorization [45], clustering, entropy based
methods and random selection. In the context of semi-supervised learning, the
authors in [48] empirically compared the performance of three kernel
approximation techniques obtained based on Nyström approximation, the reduced
kernel techniques and random Fourier features. In particular, it was shown in
[48] that the model with random Fourier features demands less training
computation times while the keeping the test accuracy comparable to that of
other two techniques.

###  2.3 Random Fourier features

One of the most popular approaches for scaling up kernel based models is
random Fourier features (RFF) proposed in [41]. In this method one
approximates the original kernel by mapping the input features into a new
space spanned by a small number of Fourier basis. The basic Principe of Random
Fourier features boils down to computing feature map $\hat{\varphi}(\cdot)$,
such that for the given two data points $x_{i}$ and $x_{j}$ the inner product
$<\hat{\varphi}(x_{i}),\hat{\varphi}(x_{j})>$ can well approximate the kernel
function $K(x_{i},x_{j})$. The authors in [41] exploited the classical
Bochner’s theorem in harmonic analysis and introduced Random Fourier features
in the field of kernel methods. A continuous kernel $K(x,y)$ defined on
$\mathbb{R}^{d}$ is positive definite if and only if $K$ is the Fourier
transform of a non-negative measure [41]. If a shift-invariant kernel $k$ is
properly scaled, its Fourier transform $p(\zeta)$ is a proper probability
distribution [3]. Thanks to this property, kernel functions can be
approximated using linear projections on $D$ random features as follows [41]:

$K(x-y)=\int_{\mathbb{R}^{d}}p(\zeta)e^{j\zeta^{T}(x-y)}d\zeta=\mathbb{E}_{\zeta}[z_{\zeta}(x)z_{\zeta}(y)^{*}],$
(9)

where $z_{\zeta}(x)=e^{j\zeta^{T}x}$. Here $z_{\zeta}(x)z_{\zeta}(y)^{*}$ is
an unbiased estimate of $K(x,y)$ when $\zeta$ is drawn from $p(\zeta)$ (see
[41]). To obtain a real-valued random feature for $K$, one replaces the
$z_{\zeta}(x)$ by the mapping
$z_{\zeta}(x)=[\cos(\zeta^{T}x),\sin(\zeta^{T}x)]$. The random Fourier
features $\hat{\varphi}(x)$, for the sample $x$, are then defined as

$\hat{\varphi}(x)=\frac{1}{\sqrt{D}}[z_{\zeta_{1}}(x),\ldots,z_{\zeta_{D}}(x)]^{T}\in\mathbb{R}^{2D},$
(10)

where $\frac{1}{\sqrt{D}}$ is used as a normalization factor to reduce the
variance of the estimate and $\zeta_{1},\ldots,\zeta_{D}\in\mathbb{R}^{d}$ are
sampled from $p(\zeta)$. If the variables $\zeta$ are drawn according to a
Gaussian distribution then the method is shown to approximate the Gaussian
kernel. In this case, they are drawn from a Normal distribution
$\mathcal{N}(0,\sigma^{2}I_{d})$ where $\sigma$ is the parameter of the
Gaussian distribution that generates the random features which at the same
time is related to the kernel bandwidth parameter. It should be mentioned in
practice there exists two approaches, i.e. data-independent and data-
dependent, to select the variable $\zeta$. In standard RFF, they are selected
independently from training data, so they are randomly sampled from
$p(\zeta)$. In the second approach one uses the training data to guide the
generation of random Fourier features. (see [49, 50]). The authors in [16, 51]
showed that by replacing the $\cos(\cdot)$ and $\sin(\cdot)$ activation
functions by a Rectified Linear Unit (ReLU), one can approximate the arc-
cosine kernel instead of the radian basis function kernel. The quality of the
random Fourier features for shift-invariant kernels are analysed in [52].

##  3 Neural-Kernel Machines

In this section, we will give a detailed overview of deep neural-kernel
machines. We start by introducing the deep neural-kernel networks which will
further be used as first building blocks for developing neural-kernel models
equipped with multi-scale representations and different choices of pooling
Layers.

###  3.1 Deep neural-kernel networks

Bridging the neural networks and kernel machines frameworks can potentially
lead to new model design that have the best of the two worlds. Authors in [26,
3] proposed a deep hybrid architecture that aims at bridging the artificial
neural networks and kernel models. To this end, an explicit feature mapping
based on random Fourier features is used for the kernel model. It should
however be noted that other possible methods such as greedy based selection
techniques [44] or Nyström methods [46] can alternatively be used for
obtaining explicit feature maps. Assuming that there are $Q$ class in the
given datasets, a two layer hybrid architecture using explicit feature map
shown in Fig. 4 is formulated as follows [26]:

$h_{1}=W_{1}x+b_{1},\;\;\;h_{2}=\hat{\varphi}(h_{1}),\;\;\;s=W_{2}h_{2}+b_{2}.$
(11)

where $W_{1}\in\mathbb{R}^{d_{1}\times d}$ and $W_{2}\in\mathbb{R}^{Q\times
d_{2}}$ are weight matrices and the bias vectors are denoted by $b_{1}$ and
$b_{2}$.

Figure 4:  Two-layer hybrid neural kernel network architecture. (Image adapted
from [26]).

The exiting connections with the standard neural networks architecture becomes
more clear, if one re-writes (11) as
$s=W_{2}\hat{\varphi}(W_{1}x+b_{1})+b_{2}$. The dimensions of the hidden
variables $h_{1}$ and $h_{2}$ are user defined parameters which here are set
to $d_{1}$ and $d_{2}$. In order to learn the model parameters, one can
formulate the following objective optimization problem:

$\underset{W_{1},W_{2},b_{1},b_{2}}{\min}\;\;J(W_{1},W_{2},b_{1},b_{2})=\frac{\gamma}{2}\sum_{j=1}^{2}Tr(W_{j}W_{j}^{T})+\frac{1}{n}\sum_{i=1}^{n}L(x_{i},y_{i}).$
(12)

It consists of a misclassification loss function $L(\cdot)$ and the
regularization terms. The added regularization terms aim at keeping the
weights of the model small to avoid overfitting. The emphasis given to the
regularization terms is controlled by $\gamma$ where needs to be properly
selected as too low values can result in neglecting the regularization term
and too high values encourage zero weights. Here we use the cross-entropy loss
function (softmax function) which provides probabilistic membership
assignments to each instance by minimizing the negative log likelihood of the
correct class. The cross-entropy loss for the instance $x_{i}$ whose class
scores are denoted by $s^{\ell}_{i}$ for $\ell=1,\ldots,Q$ are calculated as
follows:

$L(x_{i},y_{i})=-\log\bigg{(}\frac{\exp(s_{i}^{y_{i}})}{\sum_{j=1}^{Q}\exp(s_{i}^{j})}\bigg{)},$
(13)

where $s_{i}^{j}$ denotes the $j$-th class score for the $x_{i}$ instance with
ground truth label $y_{i}$. Given the instance $x_{i}$, the softmax classifier
outputs a normalized probability assigned to the correct label $y_{i}$. The
model can be trained using (stochastic) gradient descent algorithm. The class
label for the test instance $x_{\text{test}}$ is computed as follows [3]:

$\hat{y}_{\text{test}}=\underset{\ell=1,\ldots,Q}{\arg\\!\max}(s_{\text{test}}),$
(14)

where
$s_{\text{test}}=W_{2}\;\tilde{\varphi}(W_{1}x_{\text{test}}+b_{1})+b_{2}$. It
is shown in [3] that one can develop deeper models by staking the hybrid model
(11), where in this case the input data can pass through several hybrid layers
before reaching the output layer. (See Fig. 5).

Figure 5:  Deep hybrid neural kernel network architecture. (Image adapted from
[3]).

One can formulate the stacked hybrid deep neural kernel model as follows:

$\displaystyle h_{1}=W_{1}x+b_{1},$ (15) $\displaystyle
h_{2}=\hat{\varphi_{1}}(h_{1}),$ $\displaystyle h_{3}=W_{2}h_{2}+b_{2},$
$\displaystyle h_{4}=\hat{\varphi_{2}}(h_{3}),$ $\displaystyle
s=W_{3}h_{4}+b_{3}.$

Here, the input $x\in\mathbb{R}^{d}$, $\\{W_{i}\\}_{i=1}^{3}$ are weight
matrices and the bias vectors are denoted by $b_{1},b_{2}$ and $b_{3}$. The
dimension of the hidden layers $h_{1},h_{2},h_{3}$ and $h_{4}$ are denoted by
$d_{1},d_{2},d_{3}$ and $d_{4}$ respectively. Similar type of objective
functions can be defined to learn the parameters of the model (15). Several
strategies can potentially be applied for training the stacked model. Similar
to the idea related to transfer learning, one can use the previously learned
weights and transfer them from model (12) to the new model (15) and keep it
unchanged. This helps in reducing the training computation time as there are
less number of parameters to be learned. Alternatively, the learned weights of
the model (12) can be used for the initialization of the first two layers in
(15). Here, we start by training the two-layer hybrid neural kernel networks
(12). Then we drop the last fully connected and the Softmax layer from model
(11) and build the new stacked model, see Fig. 5. In training the new model,
the parameters of the first two layers are not trained (are frozen). The
advantage of this approach can be more noticed when analyzing large scale
dataset.

Next we explore the use of competitive pooling layers in the context of
neural-kernel architectures proposed in the previous subsection. In
particular, here we discuss three types of neural-kernel blocks proposed in
[27] with average, maxout and convolutional pooling layers. The deep neural-
kernel blocks are built using a stacking strategy which enables learning
multi-scale representations of the data corresponding to different levels in
network hierarchy.The maxout neural-kernel architecture uses the maxout layer
which receives as input multiple linear transformations of its previous layer.
The output of the maxout flows through an explicit feature mapping constructed
by random Fourier features. The convolutional neural-kernel architecture uses
a pointwise convolutional layer which receives the concatenated multiple
linear transformations of the previous layer. Choosing less number of
(1x1)-filters than the dimension of the input of a pointwise convolutional
layer results in obtaining an output that has less dimension compared to the
input dimension. The average kernel block on the other hand is obtained by
using an average layer which can be shown that it is a special case of the
pointwise convolutional kernel block.

###  3.2 Deep maxout neural-kernel network

Among important changes in classical artificial neural networks that the
literature has witnessed is the use of piece-wise linear activation functions
such as ReLU [6, 53], maxout [54] and their variants.

The piece-wise linear activation units partitions the input space into several
and consequently simpler sub-problems can potentially be addressed by training
several sub-networks jointly. In particular, the input space is partitioned
into as many regions as the number of inputs when the the maxout activation
units are applied [54, 27]. As stated in [54], the maxout activation function
can approximate any convex function of the input by computing the maximum over
a set of input linear functions. Here, we first begin by describing the maxout
kernel block. The deep maxout neural-kernel networks is then built by stacking
several maxout kernel blocks. The maxout neural-kernel block consists of
constructing multiple linear transformations of the input and sending them
through the maxout units. The explicit feature maps then are then applied on
the output of the maxout units [27]. Based on the definition of the maxout
unit, it returns the largest value of its inputs. In other words, the
activation function of each hidden unit is learned in the maxout layer. Assume
that $h^{(\ell-1)}$ denotes the representation of the $(\ell-1)$-th layer,
$V^{(\ell)}_{k}\in\mathbb{R}^{d_{\ell}\times d}$ and
$b^{(\ell)}_{k}\in\mathbb{R}^{d_{\ell}}$ are the weight matrices and the bias
vectors of the $(\ell)$-th layer for $k=1,\ldots,m$ respectively where $m$ is
the number of linear transformations. Then the $(\ell)$-th maxout kernel block
can be formulated as as follows [27]:

$\displaystyle
h^{(\ell)}=\hat{\varphi}^{(\ell)}(h^{(\ell)}_{\text{maxout}}),\;\;\;\textrm{where}\;\;\;\;h^{(\ell)}_{\text{maxout}}=\underset{k\in\\{1,\ldots,m\\}}{\max}\;V^{(\ell)}_{k}h^{(\ell-1)}+b^{(\ell)}_{k}.$
(16)

The original input $x$ corresponds to$h^{(0)}$ . A deep maxout neural-kernel
network can then be formulated as follows [27]:

$\begin{cases}h^{(1)}&=\hat{\varphi}^{(1)}(h^{(1)}_{\text{maxout}}),\;\;\;\textrm{where}\;\;\;\;h^{(1)}_{\text{maxout}}=\underset{k\in\\{1,\ldots,m\\}}{\max}\;V^{(1)}_{k}x+b^{(1)}_{k},\\\
h^{(2)}&=\hat{\varphi}^{(2)}(h^{(2)}_{\text{maxout}}),\;\;\;\textrm{where}\;\;\;\;h^{(2)}_{\text{maxout}}=\underset{k\in\\{1,\ldots,m\\}}{\max}\;V^{(2)}_{k}h^{(1)}+b^{(2)}_{k},\\\
s(x)&=Wh^{(2)}+b.\end{cases}$ (17)

Here $W\in\mathbb{R}^{Q\times d_{2}}$ and $b\in\mathbb{R}^{Q}$ denote the
weight matrix and the bias vector of the fully connected layer before the
softmax layer respectively. The parameters of the described model can be
obtained by solving the following optimization problem [27]:

$\displaystyle\underset{\Theta}{\min}\;J(\Theta)=\gamma\Omega(\theta)+\frac{1}{n}\sum_{i=1}^{n}L(x_{i},y_{i}),$
(18)

where $\Theta$, $\Omega(\Theta)$ and $L(\cdot,\cdot)$ are the trainable model
parameters, regularization term and the cross-entropy loss. As stated
previously, the emphasis given to the regularization term is controlled by the
regularization parameter $\gamma$. The architecture of the described maxout
neural-kernel network is shown in Fig. 6.

Figure 6:  Deep maxout neural-kernel networks. (Image adapted from [27]).

The score variable $s_{\text{test}}$ of the test instance $x_{\text{test}}$,
can be obtained by substituting $x_{\text{test}}$ in (17). Therefore, the
class label for the test instance $x_{\text{test}}$ is computed as follows
[27]:

$\hat{y}_{\text{test}}=\underset{\ell=1,\ldots,Q}{\arg\\!\max}(s_{\text{test}}).$
(19)

There are several possible strategies that one can choose to train the
proposed deep architecture including block wise or end-to-end learning
schemes. In the former, the parameters of the first block are learned, frozen
and used when learning the parameters of the second block [27]. In the later
scheme, all the model parameters are jointly learned in an end-to-end fashion
and this is the approach that we have adopted here. Among existing
regularization techniques one can for instance deploy dropout, batch
normalization or applying various penalties on each block parameters. Here, an
early stopping criterion which keeps track of the validation loss is applied.

###  3.3 Deep average neural-kernel network

The average pooling layer can be used to fuse multiple representations by
giving equal weight to each of the them. The authors in [27] formulated the
$(\ell)$-th average neural-kernel block as follows:

$\displaystyle h^{(\ell)}=$
$\displaystyle\hat{\varphi}^{(\ell)}(h^{(\ell)}_{\text{average}}),$ (20)
$\displaystyle\textrm{where}\;h^{(\ell)}_{\text{average}}=$
$\displaystyle\text{mean}\\{V^{(\ell)}_{1}h^{(\ell-1)}+b^{(\ell)}_{1},\ldots,V^{(\ell)}_{m}h^{(\ell-1)}+b^{(\ell)}_{m}\\}.$

Here, the input representation $h^{(\ell-1)}$, the weight matrices and the
bias vectors $V^{(\ell)}_{k}$ and $b^{(\ell)}_{k}$ for $k=1,\ldots,m$ are
defined as previously. In particular, the original input data $x$ is denoted
by $h^{(0)}$.

One can stack several average neural-kernel blocks to form a deep average
neural-kernel networks, see Fig. 7. The extension of the (20) can then be made
by formulating the deep average neural-kernel network as follows [27]:

$\begin{cases}h^{(1)}_{\text{average}}=&\text{mean}\\{V^{(1)}_{1}x+b^{(1)}_{1},\ldots,V^{(1)}_{m}x+b^{(1)}_{m}\\},\\\
h^{(1)}=&\hat{\varphi}^{(1)}(h^{(1)}_{\text{average}}),\\\
h^{(2)}_{\text{average}}=&\text{mean}\\{V^{(2)}_{1}h^{(1)}+b^{(2)}_{1},\ldots,V^{(2)}_{m}h^{(1)}+b^{(2)}_{m}\\},\\\
h^{(2)}=&\hat{\varphi}^{(2)}(h^{(2)}_{\text{average}}),\\\
s(x)=&Wh^{(2)}+b.\end{cases}$ (21)

The weight matrix and the bias vector of the fully connected layer before the
softmax layer are also denoted by $W\in\mathbb{R}^{Q\times d_{2}}$ and
$b\in\mathbb{R}^{Q}$ respectively. The hidden layers dimensions are defined as
previously.

Figure 7: Deep average neural-kernel networks. Image adapted from [27]).

###  3.4 Deep convolutional neural-kernel network

The convolutional neural-kernel network utilizes a pointwise convolutional
pooling layer which aims at projecting multiple representations onto a new
space [27]. A pointwise convolution is a $(1\times 1)$-convolutional layer
that applies the filters along the depth dimension to increase or reduce the
dimensionality of the given input (see [55]). Following the lines of [27], in
order to keep the number of hidden units among the maxout, average and
convolutional neural-kernel networks comparable, here only one filter in the
$(1\times 1)$-convolutional layer is used. However, it should be noted that in
practice one could potentially deploy more number of filters. Assume that $m$
concatenated linear transformations of the input $h^{(\ell-1)}$ is denoted by
$P^{(\ell)}=\bigg{[}V^{(\ell)}_{1}h^{(\ell-1)}+b^{(\ell)}_{1},\ldots,V^{(\ell)}_{m}h^{(\ell-1)}+b^{(\ell)}_{m}\bigg{]}_{d_{\ell}\times
m}$. It is shown in [27] that the result of applying the $(1\times
1)$-convolution operations with one filter on $P^{(\ell)}$ is equivalent to
calculating the following matrix multiplication:

$P^{(\ell)}D^{(\ell)}1_{m}.$ (22)

Here $D^{(\ell)}$ is a diagonal matrix whose diagonal elements are the filter
parameter $d^{(\ell)}$ corresponding to the $\ell$-th convolutional layer.
$h^{(0)}$ is defined as previously and a $1_{m}$ denotes a vector of all ones
of size $m$.

Figure 8: Deep convolutional neural-kernel networks. (Image adapted from
[27]).

Equation (22) reveals the difference between convolutional kernel blocks and
the average kernel block layers. The convolutional kernel block can be reduced
to the average kernel block, if we set the filter parameter $d^{(\ell)}$ to
$\frac{1}{m}$. Therefore, average kernel block can be considered as a special
case of the convolutional kernel block. The $(\ell)$-th convolutional neural-
kernel block can be formulated as follows [27]:

$h^{(\ell)}=\hat{\varphi}^{(\ell)}(h^{(\ell)}_{\text{conv}}),\;\;\;\;\textrm{where}\;\;\;\;h^{(\ell)}_{\text{conv}}=P^{(\ell)}D^{(\ell)}1_{m}.$
(23)

One can further Stack the convolutional kernel blocks as shown in Fig. 8 to
form a deep convolutional kernel network formulated as follows [27]:

$\displaystyle\begin{cases}h^{(1)}&=\hat{\varphi}^{(1)}(h^{(1)}_{\text{Conv}}),\;\;\;\;\textrm{where}\;\;\;\;h^{(1)}_{\text{Conv}}=P^{(1)}D^{(\ell)}1_{m},\\\
h^{(2)}&=\hat{\varphi}^{(2)}(h^{(2)}_{\text{Conv}}),\;\;\;\;\textrm{where}\;\;\;\;h^{(2)}_{\text{Conv}}=P^{(2)}D^{(\ell)}1_{m},\\\
s(x)&=Wh^{(2)}+b.\end{cases}$ (24)

$W\in\mathbb{R}^{Q\times d_{2}}$ and $b\in\mathbb{R}^{Q}$ denote the weight
matrix and the bias vector of the fully connected layer before the softmax
layer respectively. The hidden layers dimensions are defined as previously.

##  4 Experimental results and discussion

We have conducted several experiments on real-life datasets taken from UCI
machine learning repository [56] and image benchmark datasets. The
descriptions of the used UCI datasets are summarized in Table 1. We follow the
same setup as in [27] and therefore the given UCI dataset is randomly
partitioned to $80\%$ training and $20\%$ test sets respectively. The hyper-
parameters of the discussed deep neural-kernel networks are the dimension of
the explicit feature maps and the variance of the normal distribution [27].
Following the lines of [27], the dimensions of the intermediate layers are
sought in the range of $[50,600]$ using the random search scheme. In all the
experiments the regularization parameter is set to $\gamma=0.0001$. The
employed architectures composed of two neural-kernel blocks with four linear
transformations within each block (see Fig. 6).

Table 2 reports the obtained results of the introduced models in section 3. It
can observed that deep maxout neural-kernel network architecture outperforms
average neural-kernel model and is comparable to the convolutional neural-
kernel model. In particular one can notice a significant test accuracy
improvement for the Titanic, Covertype and Motor datasets. The training
computation time and test accuracy of the discussed deep hybrid, average,
maxout and convolutional neural-kernel networks are shown in 9 and 10
respectively. Fig. 15 and Fig. 11 illustrate the accuracy, training and
validation loss of the deep neural-kernel models discussed in section 3 for
the Covertype and Motor datasets respectively.

Here, an early stopping criterion is used to avoid overfitting. In particular,
the validation loss has been monitored and after twenty consecutive epochs of
no improvement the training process is stopped. It should be mentioned that
the changing patterns of the validation loss is not the same for all the
models and therefore as can be seen from Fig. 15 the training process of some
models stop earlier than the others. In general we observed that the
training/validation losses of the introduced deep maxout, average and
convolutional neural-kernel networks are lower than those of deep hybrid
model. Therefore, the deep maxout, average and convolutional neural-kernel
models are expected to show a better generalization performance on the test in
case there is no distributional mismatch between test and training datasets.

Fig. 12 shows the learned weights of the first block of the deep maxout
neural-kernel networks applied on the Motor dataset. The the learned weights
of the second block for the same dataset are also depicted in Fig. 13. It can
be noted that the magnitude and structure of the weight matrices corresponding
to the first and second block differs. In particular, the weights of the first
block seem to have more sparse preserved structure where only in few positions
one can observe the picks in the magnitude. This is due to the fact that still
the nonlinear transformation is not applied. One can notice changes in the
structure of the weights corresponding to the second block compared to the
first block. This can be explained by the imposed nonlinearity. Inspecting the
the magnitude can also potentially reveal the amount of emphasis the network
is giving to each transformation. Fig. 14 illustrates the The t-SNE
visualization [57] of the hidden layer projections and the score variables
corresponding to the employed deep maxout neural-kernel architecture for the
Motor dataset. Thanks to this visualization technique, one can observe the
changes in the data representations as the data flows through the stacked
hierarchical layers. Ideally, one may expect that the learnt representations
in the deeper layer form a more separable clusters corresponding to the
existing classes [3].

Table 1:  Dataset statistics Dataset | # Instances | # Attributes | # Classes
---|---|---|---
Sonar | 208 | 60 | 2
Monk2 | 432 | 6 | 2
Balance | 625 | 4 | 3
Australian | 690 | 14 | 2
CNAE-9 | 1080 | 856 | 9
Digit-MultiF1 | 2000 | 240 | 10
Digit-MultiF2 | 2000 | 216 | 10
Titanic | 2201 | 3 | 2
Magic | 19,020 | 10 | 2
Motor | 58,509 | 49 | 11
Covertype | 581,012 | 54 | 3
SUSY | 5,000,000 | 18 | 2
Table 2:  The average test accuracy of deep hybrid, maxout, average and convolutional neural-kernel architectures on several benchmark datasets. |  | Deep neural-kernel networks [27]
---|---|---
Dataset | deep hybrid [3] | Average | Maxout | CNN
Sonar | $0.77\pm 0.04$ | $0.77\pm 0.02$ | $\underline{0.78\pm 0.01}$ | $\underline{0.78\pm 0.01}$
Monk2 | $\underline{1.00\pm 0.00}$ | $\underline{1.00\pm 0.00}$ | $\underline{1.00\pm 0.00}$ | $\underline{1.00\pm 0.00}$
Balance | $0.97\pm 0.02$ | $0.98\pm 0.01$ | $\underline{0.99\pm 0.01}$ | $\underline{0.99\pm 0.01}$
Australian | $0.87\pm 0.01$ | $0.88\pm 0.01$ | $\underline{0.90\pm 0.01}$ | $0.88\pm 0.02$
CNAE-9 | $0.94\pm 0.02$ | $0.94\pm 0.01$ | $\underline{0.95\pm 0.01}$ | $0.94\pm 0.01$
Digit-MultiF1 | $\underline{0.98\pm 0.01}$ | $\underline{0.98\pm 0.01}$ | $\underline{0.98\pm 0.02}$ | $\underline{0.98\pm 0.01}$
Digit-MultiF2 | $0.97\pm 0.02$ | $\underline{0.99\pm 0.01}$ | $\underline{0.99\pm 0.02}$ | $\underline{0.99\pm 0.01}$
Titanic | $0.78\pm 0.02$ | $0.81\pm 0.01$ | $\underline{0.84\pm 0.01}$ | $0.83\pm 0.01$
Magic | $0.86\pm 0.01$ | $0.86\pm 0.01$ | $0.87\pm 0.02$ | $\underline{0.88\pm 0.01}$
Motor | $0.96\pm 0.01$ | $0.98\pm 0.01$ | $\underline{0.99\pm 0.01}$ | $\underline{0.99\pm 0.01}$
Covertype | $0.88\pm 0.02$ | $0.94\pm 0.01$ | $\underline{0.95\pm 0.01}$ | $0.94\pm 0.01$
SUSY | $0.81\pm 0.01$ | $0.81\pm 0.02$ | $\underline{0.82\pm 0.01}$ | $0.81\pm 0.01$

Figure 9: The training computation time of deep hybrid, maxout, average and
convolutional neural-kernel networks.
Figure 10: The test accuracy of deep hybrid, maxout, average and convolutional
neural-kernel networks.

|
---|---
|

Figure 11:  The Motor dataset. (a,b,c,d) The training/validation loss and the
accuracy of the deep neural-kernel networks discussed in section 3.

|
---|---
|

Figure 12:  The Motor dataset. (a,b,c,d) The learned weights of the first
block of the maxout neural-kernel networks. (See Fig. 6)

|
---|---
|

Figure 13:  The Motor dataset. (a,b,c,d) The learned weights of the second
block of the maxout neural-kernel networks. (See Fig. 6)

|  |
---|---|---
|  |
|  |
|  |
|  |

Figure 14:  The Motor dataset. The t-SNE projections of: (a) the test data.
(b,c,d,e) the linear transformations of the first block. (f) the maxout layer
of the first block. (g) the explicit feature mapping of the first block.
(h,i,j,k) the linear transformations of the second block. (l) the maxout layer
of the second block. (m) the explicit feature mapping of the second block. (n)
the score variables $S(x)$.

The performance of the discussed deep maxout and convolutional neural-kernel
networks are also evaluated on four real-life benchmark datasets CIFAR-10
[58], CIFAR-100 [58], SVHN [59] and MNIST [1]. The architectures used for
these datasets follow a combination of convolutional neural networks (CNN) for
feature learning and the proposed models for the classification task. The
employed CNN networks consists of three convolutional layers and then
downsampling the feature maps by a factor of two using spatial pooling layer.
The results of the CNN model is fed as input to the deep neural-kernel
networks. The parameters of the two models are jointly learned in an end-to-
end fashion. In this experiment the neural-kernel networks is composed of one
block with three linear transformations. The number of filters of CNN is
gradually increased i.e. 32, 64 and 128. The data statistics of the used
images can be found in Table 3. Next the pre-processing steps applied to each
of the image dataset will be explained. The MNIST images are scaled to [0,1]
before they are fed into the networks. The global contrast normalization as
well as ZCA whitening [54] are applied for preprocessing the CIFAR-10 images.
The weight decay and the size of the local receptive field are tuned using the
validation set. The implemented settings for CIFAR-100 follow those applied on
CIFAR-10 dataset. The task of SVHN dataset is to classify the digit in the
centre of each image. Following the lines of Goodfellow et al. [54] here a
local contrast normalization is applied in order to preprocess the data.

Table 3:  Dataset statistics for Images Dataset | # Image-size | # Training | # Test | # Classes
---|---|---|---|---
MNIST | $28\times 28$ | 60,000 | 10,000 | 10
CIFAR-10 | $32\times 32$ | 50,000 | 10,000 | 10
CIFAR-100 | $32\times 32$ | 50,000 | 10,000 | 100
SVHN | $32\times 32$ | 73,257 | 26,032 | 10

Table 4 summarizes a comparison between the introduced models and those in
[60, 61, 54, 60, 61]. It can be observed from Table 4 that the performance of
the deep neural-kernel models is either comparable or better than those of SVM
and convolutional neural networks. It should be noted that in our experiments
we have not fully explored other regularization techniques such as dropout,
hidden unit sparsity, weight constraints or an exhaustive model selection
search. Next we analyse the influence of the number of neural-kernel blocks
and linear transformations on the performance of the deep maxout neural-kernel
networks. In particular, we vary the number of blocks in the range of 1 to 4
and the number of linear transformation in the range of 2 to 4. Fig. 16 and
Fig. 17 show the effect of the number of blocks and linear transformations on
the accuracy, training and validation loss for the Magic dataset. In general,
we can observe that the larger the number of blocks, the higher the complexity
of the model is and the faster the training loss converges. However, it should
be noted that increase in the complexity of the model results in increasing
the risk of overfitting. Therefore, in practice, one can seek to find the
optimal combinations of these two parameters using for instance a validation
dataset.

|
---|---
|

Figure 15:  The Covertype dataset. (a,b,c,d) The training/validation loss and
accuracy of the deep neural-kernel networks discussed in section 3.
Figure 16: The test accuracy of the deep maxout neural-kernel networks
corresponding to different pairs of the number of blocks and linear
transformations for the Magic datasets.

|
---|---
|
|

Figure 17:  Magic dataset: the training/validation loss of the deep maxout neural-kernel networks when different number of blocks and linear transformations are used. (a,b) two linear transformations. (c,d) three linear transformations. (e,f) four linear transformations. Table 4:  Test set error rates of various networks for MNIST, CIFAR-10, CIFAR-100 and SVHN datasets | Test error (%) |
---|---|---
Method | MNIST | CIFAR-10 | CIFAR-100 | SVHN (%) |
CNN + Maxout Kernel Blocks | $0.48\%$ | $11.52\%$ | $38.77\%$ | $2.41\%$ |
CNN + Conv Kernel Blocks | $0.51\%$ | $11.61\%$ | $39.21\%$ | $2.56\%$ |
Conv. maxout + Dropout [54] | $0.45\%$ | $11.68\%$ | $38.57\%$ | $2.47\%$ |
Stochastic Pooling [60] | $0.47\%$ | $15.13\%$ | $42.51\%$ | $2.80\%$ |
2-Layer CNN + 2-Layer NN [60] | $0.53\%$ | N.A | N.A | N.A |
DLSVM [61] | $0.87\%$ | $11.90\%$ | N.A | N.A |

##  5 Conclusions

In this chapter we have discussed neural-kernel machines a framework that
bridges the artificial neural network and kernel based models. In particular,
deep hybrid, average, maxout and convolutional neural-kernel architectures
have been explained in details. The exiting connections between artificial
neural networks and kernel machines with explicit and implicit feature mapping
are highlighted. In this context, different ways of combining multi-scale
representations of the data are explored and in particular we showed that the
introduced average neural-kernel model is a special case of the neural-kernel
model with pointwise convolutional pooling layer. Finally, the validity and
applicability of the proposed models on several real-life benchmark datasets
have been examined.

## References

  * [1] Y. Bengio, A. Courville, P. Vincent, Representation learning: A review and new perspectives, IEEE transactions on pattern analysis and machine intelligence 35 (8) (2013) 1798–1828.
  * [2] Y. Bengio, Learning deep architectures for AI, Foundations and trends® in Machine Learning 2 (1) (2009) 1–127.
  * [3] S. Mehrkanoon, J. A. K. Suykens, Deep hybrid neural-kernel networks using random Fourier features, Neurocomputing 298 (2018) 46–54.
  * [4] R. Salakhutdinov, G. E. Hinton, Deep Boltzmann Machines., in: AISTATS, Vol. 1, 2009, p. 3.
  * [5] G. Hinton, A practical guide to training restricted boltzmann machines, Momentum 9 (1) (2010) 926.
  * [6] V. Nair, G. E. Hinton, Rectified linear units improve restricted boltzmann machines, in: Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807–814.
  * [7] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-A. Manzagol, Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, Journal of Machine Learning Research 11 (Dec) (2010) 3371–3408.
  * [8] G. E. Hinton, R. R. Salakhutdinov, Reducing the dimensionality of data with neural networks, science 313 (5786) (2006) 504–507.
  * [9] S. Lawrence, C. L. Giles, A. C. Tsoi, A. D. Back, Face recognition: A convolutional neural-network approach, IEEE transactions on neural networks 8 (1) (1997) 98–113.
  * [10] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, 2012, pp. 1097–1105.
  * [11] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (8) (1997) 1735–1780.
  * [12] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7553) (2015) 436–444.
  * [13] V. Vapnik, Statistical learning theory, Wiley, 1998.
  * [14] J. Shawe-Taylor, N. Cristianini, Kernel methods for pattern analysis, Cambridge university press, 2004.
  * [15] C. M. Bishop, N. M. Nasrabadi, Pattern recognition and machine learning, Vol. 1, Springer New York, 2006.
  * [16] Y. Cho, L. K. Saul, Kernel methods for deep learning, in: Advances in neural information processing systems, 2009, pp. 342–350.
  * [17] A. Damianou, N. Lawrence, Deep gaussian processes, in: Artificial Intelligence and Statistics, 2013, pp. 207–215.
  * [18] K. Cutajar, E. V. Bonilla, P. Michiardi, M. Filippone, Random feature expansions for deep gaussian processes, in: International Conference on Machine Learning, 2017, pp. 884–893.
  * [19] J. Mairal, P. Koniusz, Z. Harchaoui, C. Schmid, Convolutional kernel networks, in: Advances in Neural Information Processing Systems, 2014, pp. 2627–2635.
  * [20] Ö. Aslan, X. Zhang, D. Schuurmans, Convex deep learning via normalized kernels, in: Advances in Neural Information Processing Systems, 2014, pp. 3275–3283.
  * [21] I. Steinwart, P. Thomann, N. Schmid, Learning with hierarchical gaussian kernels, arXiv preprint arXiv:1612.00824.
  * [22] U. Heinemann, R. Livni, E. Eban, G. Elidan, A. Globerson, Improper deep kernels, in: Artificial Intelligence and Statistics, 2016, pp. 1159–1167.
  * [23] C. Jose, P. Goyal, P. Aggrwal, M. Varma, Local deep kernel learning for efficient non-linear svm prediction, in: International conference on machine learning, 2013, pp. 486–494.
  * [24] G. Montavon, M. L. Braun, K.-R. Muller, Kernel analysis of deep networks, Journal of Machine Learning Research 12 (Sep) (2011) 2563–2581.
  * [25] L. Belanche, M. Costa-jussa, Bridging deep and kernel methods, in: Proceedings of the 25th European Symposium on Artificial Neural Networks, Computational Intelligence and machine Learning (ESANN), 2017, pp. 1-10., 2017, pp. 1–10.
  * [26] S. Mehrkanoon, A. Zell, J. A. K. Suykens, Scalable hybrid deep neural kernel networks, in: in: Proceedings of the 25th European Symposium on Artificial Neural Networks, Computational Intelligence and machine Learning (ESANN), 2017, pp. 17-22., 2017, pp. 17–22.
  * [27] S. Mehrkanoon, Deep neural-kernel blocks, Neural Networks 116 (2019) 46–55.
  * [28] S. Mehrkanoon, Cross-domain neural-kernel networks, Pattern Recognition Letters 125 (2019) 474–480.
  * [29] J. A. K. Suykens, T. Van Gestel, J. De Brabanter, B. De Moor, J. Vandewalle, Least squares support vector machines, Singapore: World Scientific Pub. Co., 2002\.
  * [30] S. Mehrkanoon, C. Alzate, R. Mall, R. Langone, J. A. K. Suykens, Multiclass semisupervised learning based upon kernel spectral clustering, IEEE Transactions on Neural Networks and Learning Systems 26 (4) (2015) 720–733.
  * [31] S. Mehrkanoon, J. A. K. Suykens, Regularized semipaired kernel CCA for domain adaptation, IEEE transactions on neural networks and learning systems 29 (7) (2018) 3199–3213.
  * [32] S. Mehrkanoon, T. Falck, J. A. Suykens, Approximate solutions to ordinary differential equations using least squares support vector machines, IEEE transactions on neural networks and learning systems 23 (9) (2012) 1356–1367.
  * [33] S. Mehrkanoon, J. A. Suykens, LS-SVM approximate solution to linear time varying descriptor systems, Automatica 48 (10) (2012) 2502–2511.
  * [34] S. Mehrkanoon, J. A. Suykens, Learning solutions to partial differential equations using LS-SVM, Neurocomputing 159 (2015) 105–116.
  * [35] S. Mehrkanoon, S. Mehrkanoon, J. A. K. Suykens, Parameter estimation of delay differential equations: an integration-free LS-SVM approach, Communications in Nonlinear Science and Numerical Simulation 19 (4) (2014) 830–841.
  * [36] S. Mehrkanoon, T. Falck, J. Suykens, Parameter estimation for time varying dynamical systems using least squares support vector machines, in: Proc. of the 16th IFAC Symposium on System Identification (SYSID 2012), Vol. 16, 2012, pp. 1300–1305.
  * [37] N. Cristianini, J. Shawe-Taylor, et al., An introduction to support vector machines and other kernel-based learning methods, Cambridge university press, 2000\.
  * [38] C. E. Rasmussen, Gaussian processes in machine learning, in: Summer School on Machine Learning, Springer, 2003, pp. 63–71.
  * [39] G. H. Golub, C. F. Van Loan, Matrix computations, Johns Hopkins University Press, 2012.
  * [40] C. Williams, M. Seeger, Using the Nyström method to speed up kernel machines, in: Advances in Neural Information Processing Systems 13, 2001\.
  * [41] A. Rahimi, B. Recht, Random features for large-scale kernel machines, in: Advances in neural information processing systems, 2007, pp. 1177–1184.
  * [42] Y.-J. Lee, O. L. Mangasarian, Rsvm: Reduced support vector machines, in: Proceedings of the 2001 SIAM International Conference on Data Mining, SIAM, 2001, pp. 1–17.
  * [43] S. Mehrkanoon, J. A. K. Suykens, Large scale semi-supervised learning using KSC based model, in: Proceedings of International Joint Conference onNeural Networks (IJCNN), IEEE, 2014, pp. 4152–4159.
  * [44] A. J. Smola, B. Schölkopf, Sparse greedy matrix approximation for machine learning, 17th ICML, Stanford, 2000 (2000) 911–918.
  * [45] F. R. Bach, M. I. Jordan, Predictive low-rank decomposition for kernel methods, in: Proceedings of the 22nd international conference on Machine learning, ACM, 2005, pp. 33–40.
  * [46] C. Williams, M. Seeger, Using the Nyström method to speed up kernel machines, in: Proceedings of the 14th Annual Conference on Neural Information Processing Systems, no. EPFL-CONF-161322, 2001, pp. 682–688.
  * [47] T. Yang, Y.-F. Li, M. Mahdavi, R. Jin, Z.-H. Zhou, Nyström method vs random fourier features: A theoretical and empirical comparison, in: Advances in neural information processing systems, 2012, pp. 476–484.
  * [48] S. Mehrkanoon, J. A. K. Suykens, Scalable semi-supervised kernel spectral learning using random Fourier features, in: IEEE Symposium Series on Computational Intelligence (SSCI), IEEE, 2016, pp. 1–8.
  * [49] A. Sinha, J. C. Duchi, Learning kernels with random features, in: Advances in Neural Information Processing Systems, 2016, pp. 1298–1306.
  * [50] R. Agrawal, T. Campbell, J. H. Huggins, T. Broderick, Data-dependent compression of random features for large-scale kernel approximation, in: Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, 2019.
  * [51] G. Pandey, A. Dukkipati, Learning by stretching deep networks, in: International Conference on Machine Learning, 2014, pp. 1719–1727.
  * [52] B. Sriperumbudur, Z. Szabó, Optimal rates for random fourier features, in: Advances in Neural Information Processing Systems, 2015, pp. 1144–1152.
  * [53] X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural networks, in: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011, pp. 315–323.
  * [54] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, Y. Bengio, Maxout networks, arXiv preprint arXiv:1302.4389.
  * [55] M. Lin, Q. Chen, S. Yan, Network in network, arXiv preprint arXiv:1312.4400.
  * [56] A. Asuncion, D. J. Newman, UCI machine learning repository (2007).
  * [57] L. Van der Maaten, G. Hinton, Visualizing data using t-sne, Journal of Machine Learning Research 9 (Nov) (2008) 2579–2605.
  * [58] A. Krizhevsky, G. Hinton, Learning multiple layers of features from tiny images, Tech. rep., Master’s thesis, Department of Computer Science, University of Toronto (2009).
  * [59] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y. Ng, Reading digits in natural images with unsupervised feature learning, in: NIPS workshop on deep learning and unsupervised feature learning, Vol. 2011, 2011, p. 5.
  * [60] M. D. Zeiler, R. Fergus, Stochastic pooling for regularization of deep convolutional neural networks, arXiv preprint arXiv:1301.3557.
  * [61] Y. Tang, Deep learning using linear support vector machines, arXiv preprint arXiv:1306.0239.