# Error Loss Networks

Badong Chen, Yunfei Zheng, and Pengju Ren Badong Chen (corresponding author),
Yunfei Zheng, and Pengju Ren are with the Institute of Artificial Intelligence
and Robotics, Xi’an Jiaotong University, Xi’an 710049, China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>pengjuren@gmail.com).This
work was supported by the National Natural Science Foundation of China under
Grant Numbers (U21A20485, 61976175).

###### Abstract

A novel model called error loss network (ELN) is proposed to build an error
loss function for supervised learning. The ELN is in structure similar to a
radial basis function (RBF) neural network, but its input is an error sample
and output is a loss corresponding to that error sample. That means the
nonlinear input-output mapper of ELN creates an error loss function. The
proposed ELN provides a unified model for a large class of error loss
functions, which includes some information theoretic learning (ITL) loss
functions as special cases. The activation function, weight parameters and
network size of the ELN can be predetermined or learned from the error
samples. On this basis, we propose a new machine learning paradigm where the
learning process is divided into two stages: first, learning a loss function
using an ELN; second, using the learned loss function to continue to perform
the learning. Experimental results are presented to demonstrate the desirable
performance of the new method.

###### Index Terms:

Supervised learning, error loss, error loss networks, radial basis functions.

##  I Introduction

Machine learning aims to build a model based on training samples to predict
the output of new samples. For supervised learning (see Fig. 1), each training
sample consists of a pair of data called respectively the input (typically a
vector) and the desired output (also called the supervisory signal). Given
training samples
$\left\\{{\left({{\mathbf{x}_{1}},{d_{1}}}\right),\cdots,\left({{\mathbf{x}_{N}},{d_{N}}}\right)}\right\\}$
with ${\mathbf{x}_{i}}$ being the input vector and ${d_{i}}$ the desired
output, a supervised learning algorithm usually seeks a function (i.e. the
input-output mapper of the learning machine) $f:\mathcal{X}\to\mathcal{Y}$
where $\mathcal{X}$ denotes the input space and $\mathcal{Y}$ stands for the
output space, such that an empirical loss (risk) is minimized, that is

${f^{*}}=\mathop{\arg\min}\limits_{f\in\mathcal{F}}\frac{1}{N}\sum\limits_{i=1}^{N}{l\left({{d_{i}},{y_{i}}}\right)},$
(1)

where $\mathcal{F}$ is some space of possible functions (usually called the
hypothesis space), ${y_{i}}=f\left({{\mathbf{x}_{i}}}\right)$ is the model
(function) output, and $l:\mathcal{Y}\times\mathcal{Y}\to\mathbb{R}$ is a
certain loss function that measures how well a function fits the training
samples (In this work, the loss function is not limited to be nonnegative). In
many cases, the loss function depends on the error sample
${e_{i}}={d_{i}}-{y_{i}}$, namely the difference between the desired and the
model output. In such conditions, we call it an error loss, denoted by
$l\left({{e_{i}}}\right)$. Some regularization terms are often incorporated
into the loss function to prevent overfitting. However, in this study our
focus is mainly on the error loss term.

Figure 1: Schematic of supervised learning.

How to choose or design a proper error loss function is a key problem in
supervised learning. The squared error loss (i.e. the mean square error
(MSE)), in which , $l\left(e\right)={e^{2}}$ is one of the most widely used
loss functions especially for regression problem because of its mathematical
convenience and computational simplicity. However, MSE cannot deal with non-
Gaussian noise well, and particularly is sensitive to heavy-tailed outliers.
Many alternatives such as mean absolute error (MAE) [1], mean p-power error
(MPE) [2, 3, 4], Huber’s loss [5, 6] and logarithmic loss [7, 8, 9], can thus
be used to improve the robustness to outliers. Over the past decade, some
quantities (such as entropy) related to information theory have been
successfully used as robust loss functions in machine learning, and such
learning methods are called information theoretic learning (ITL) [10]. The
minimum error entropy (MEE) [11, 12] and maximum correntropy criterion (MCC)
[13] are two typical learning criteria in ITL. By Parzen window approach, the
ITL loss functions can be estimated directly from the samples, and with a
Gaussian kernel these loss functions can be viewed as a similarity measure in
kernel space [10]. Compared with other robust error loss functions, such as
MAE and Huber’s loss, the ITL loss functions often show much better robustness
to complex non-Gaussian noises e.g., heavy-tailed, multimode distributed,
discrete-valued, etc.). Theoretical analysis of the robustness of the ITL can
be found in [14, 15, 16].

In recent years, several important variants or extended versions of the ITL
learning criteria (loss functions) have been proposed to further enhance the
robustness or improve the computational efficiency. Examples are quantized
minimum error entropy (QMEE) [17], generalized maximum correntropy criterion
(GMCC) [18], maximum mixture correntropy criterion (MMCC) [19, 20], kernel
risk-sensitive loss (KRSL) [21], kernel mean p-power error (KMPE) [22],
maximum correntropy criterion with variable center (MCC-VC) [23], maximum
multi-kernel correntropy criterion (MMKCC) [24], and so on. These ITL loss
functions have been successfully applied to robust regression, adaptive
filtering, principal component analysis (PCA), point set registration, Granger
causality analysis, etc. [25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37].
How to integrate these loss functions into a unified model is of great
significance, but also a great challenge.

In the present paper, we propose a novel model called error loss network (ELN)
to build an error loss function for supervised learning. The ELN is in
structure similar to a radial basis function (RBF) neural network, but its
input is an error sample and output is a loss corresponding to that error
sample. That means the nonlinear input-output mapper of ELN creates an error
loss function. The proposed ELN provides a unified model for a large class of
error loss functions, including the previously mentioned ITL loss functions as
special cases. The activation function, weight parameters and network size of
the ELN can be predetermined or learned from the error samples. On this basis,
we propose a new machine learning paradigm where the learning process is
divided into two stages: first, learning a loss function using an ELN; second,
using the learned loss function to continue to perform the learning. Moreover,
we propose a probability density function (PDF) matching approach to train the
ELN and a fixed-point iterative algorithm to train the learning machine with a
linear-in-parameter (LIP) model.

The rest of the paper is organized as follows. In section II, we give the
basic concept of ELN and discuss its relationship with ITL. In section III, we
propose an overall supervised learning strategy and discuss how to train the
ELN and the learning machine. Section IV presents the experimental results and
finally, section V gives the concluding remarks.

##  II Error Loss Networks

###  II-A Basic Concept

The error loss is generally a nonlinear function of the error sample (the
difference between the desired and the model output), which has very important
influence on the performance of supervised learning. In most applications,
such functions are predetermined by the algorithm designers based on
experience or some prior knowledge. For example, one often chooses
$l\left(e\right)={e^{2}}$ for ordinary regression or
$l\left(e\right)=\left|e\right|$ for robust regression. In this work, we
propose to build an error loss function in a manner similar to an RBF neural
network, which can be trained from the data (not predetermined). Specifically,
we propose an RBF like model called ELN, with error sample as the input and
corresponding output as the loss (see Fig. 2). The nonlinear input-output
mapper of ELN creates an error loss function $l(.)$.

Figure 2: Schematic of error loss network.

Let $\left\\{{{\phi_{1}},\cdots,{\phi_{M}}}\right\\}$ be $M$ radial basis
functions (whose values depend only on the distances between the inputs and
some fixed points) and
$\boldsymbol{\theta}={\left[{{\theta_{1}},\cdots,{\theta_{M}}}\right]^{T}}$ be
the output weight vector. The output (error loss) of the ELN is a linear
combination of the radial basis functions, given by

$l\left({{e_{i}}}\right)=\sum\limits_{j=1}^{M}{{\theta_{j}}{\phi_{j}}\left({{e_{i}}}\right)}.$
(2)

Given $N$ error samples $\left\\{{{e_{1}},\cdots,{e_{N}}}\right\\}$, the
empirical loss based on the ELN can thus be computed by

$L=\frac{1}{N}\sum\limits_{i=1}^{N}{l\left({{e_{i}}}\right)}=\frac{1}{N}\sum\limits_{i=1}^{N}{\sum\limits_{j=1}^{M}{{\theta_{j}}{\phi_{j}}\left({{e_{i}}}\right)}}.$
(3)

The ELN model can be extended to other neural networks (e.g. Multilayer
Perceptron), but this paper is concerned only with the RBF type network. It is
worth noting that if the radial basis functions (e.g., Gaussian function)
satisfy: i) $\forall
e\in\mathbb{R},{\phi_{j}}\left(e\right)\leq{b_{j}},{b_{j}}>0$; ii)
$\mathop{\lim}\limits_{\left|e\right|\to\infty}\frac{\partial}{{\partial
e}}{\phi_{j}}\left(e\right)=0$, we have

$\left\\{\begin{array}[]{l}\left|{l(e)}\right|\leq\sum\limits_{j=1}^{M}{\left|{{\theta_{j}}}\right|}{b_{j}}\\\
\mathop{\lim}\limits_{\left|e\right|\to\infty}\frac{\partial}{{\partial
e}}l\left(e\right)=0.\end{array}\right.$ (4)

In this case, the error loss function $l(e)$ is always bounded and its
derivative will approach zero as $\left|e\right|\to\infty$. That means the
influence of an error with very large value on the loss function is very
limited, because the loss at large error is bounded and the gradient of the
loss function at large error is also very small. Such loss function will be
robust to outliers that usually cause large errors. Therefore, with an ELN one
can easily obtain an error loss function robust to outliers by choosing the
radial basis functions satisfying the previous two conditions. This is a great
advantage of the proposed ELN.

###  II-B Relation to ITL

The ELN has close relationship with the celebrated ITL, initiated in the late
90s. Basically, ITL uses descriptors from information theory (e.g., entropy)
estimated directly from the data to create the loss functions for machine
learning. One of the most popular ITL learning criteria is the MEE, which
adopts the error’s entropy as the loss for supervised learning (to say, the
learning machine is trained such that the error entropy is minimized). With
Renyi’s quadratic entropy, the empirical loss of MEE can be computed by [10,
11, 12]

$\displaystyle L$
$\displaystyle=-\int\nolimits_{-\infty}^{+\infty}{{{\left({\hat{p}(e)}\right)}^{2}}d}e$
$\displaystyle=-\int\nolimits_{-\infty}^{+\infty}{{{\left({\frac{1}{N}\sum\limits_{i=1}^{N}{{G_{\sigma}}\left({e-{e_{i}}}\right)}}\right)}^{2}}de}$
$\displaystyle=-\frac{{1}}{N^{2}}\sum\limits_{i=1}^{N}{\sum\limits_{j=1}^{N}{{G_{\sqrt{2}\sigma}}\left({{e_{i}}-{e_{j}}}\right)}},$
(5)

where
$\hat{p}(e){\rm{=}}\frac{1}{N}\sum_{i=1}^{N}{{G_{\sigma}}\left({e-{e_{i}}}\right)}$
is the estimated PDF (by Parzen window approach [38]) of the error based on
the $N$ samples $\left\\{{{e_{1}},\cdots,{e_{N}}}\right\\}$, and
${G_{\sigma}}(e)$ is a Gaussian kernel controled by kernel width $\sigma$ and
can be expressed by

${G_{\sigma}}(e)=\frac{1}{\sqrt{2\pi}\sigma}\exp\left({-\frac{{{e^{2}}}}{{2{\sigma^{2}}}}}\right).$
(6)

The above empirical loss can be rewritten as

$\displaystyle L$
$\displaystyle=\frac{1}{N}\sum\limits_{i=1}^{N}{\left({\sum\limits_{j=1}^{N}{\frac{{-1}}{{N}}{G_{\sqrt{2}\sigma}}\left({{e_{i}}-{e_{j}}}\right)}}\right)}$
$\displaystyle=\frac{1}{N}\sum\limits_{i=1}^{N}{\left({\sum\limits_{j=1}^{N}{{\theta_{j}}{\phi_{j}}\left({{e_{i}}}\right)}}\right)},$
(7)

where ${\theta_{j}}=-\frac{{1}}{{N}}$,
${\phi_{j}}\left({{e_{i}}}\right)={G_{\sqrt{2}\sigma}}\left({{e_{i}}-{e_{j}}}\right)$.
Thus, the error loss function $l(e)$ of MEE can be created by an ELN model
with $N$ hidden nodes
${\phi_{j}}\left(e\right)={G_{\sqrt{2}\sigma}}\left({e-{e_{j}}}\right),j=1,\cdots,N$,
and output weight vector
$\boldsymbol{\theta}={\left[{-\frac{{1}}{{N}},\cdots,-\frac{{1}}{{N}}}\right]^{T}}$.

The network size of the ELN for MEE is equal to the sample number, which is
very large for large scale data sets. To reduce the computational complexity
of MEE, the quantized MEE (QMEE) was proposed in [17]. Similar to the MEE, the
QMEE loss can also be viewed as an ELN, but with network size $M\leq N$ in
general. The basic idea behind QMEE is actually to merge several hidden nodes
into one if their centers are very close.

TABLE I: Error loss networks of some ITL loss functions

Loss | $M$ | ${\phi_{i}}(i=1,\cdots,M)$ | ${\theta_{i}}(i=1,\cdots,M)$
---|---|---|---
MCC [13] | $M=1$ | ${\phi_{1}}(e)={G_{\sigma}}(e)$ | ${\theta_{1}}=-1$
GMCC [18] | $M=1$ | ${\phi_{1}}(e)={G_{\alpha,\beta}}(e)$ | ${\theta_{1}}=-1$
KRSL [21] | $M=1$ | ${\phi_{1}}(e)=\exp\left({\lambda\left({1-{G_{\sigma}}(e)}\right)}\right)$ | ${\theta_{1}}=\frac{1}{\lambda}$
KMPE [22] | $M=1$ | ${\phi_{1}}(e)={\left({1-{G_{\sigma}}(e)}\right)^{{p\mathord{\left/{\vphantom{p2}}\right.\kern-1.2pt}2}}}$ | ${\theta_{1}}=1$
MCC-VC [23] | $M=1$ | ${\phi_{1}}(e)={G_{\sigma}}(e-c)$ | ${\theta_{1}}=-1$
MMCC (Type 1) [19] | $M=2$ | $({\phi_{1}}(e),{\phi_{2}}(e))=({G_{{\sigma_{1}}}}(e),{G_{{\sigma_{2}}}}(e))$ | $({\theta_{1}},{\theta_{2}})=(-\alpha,\alpha-1)$
MMCC (Type 2) [20] | $M=2$ | $({\phi_{1}}(e),{\phi_{2}}(e))=({G_{{\sigma}}}(e),{L_{{\sigma}}}(e))$ | $({\theta_{1}},{\theta_{2}})=(-\alpha,\alpha-1)$
MEE [11] | $M=N$ | ${\phi_{i}}(e)={G_{\sigma}}(e-{e_{i}})$ | ${\theta_{i}}=\frac{-1}{N}$
QMEE [17] | $1\leq M\leq N$ | ${\phi_{i}}(e)={G_{\sigma}}(e-{c_{i}})$ | ${\theta_{i}}=\frac{{-{M_{i}}}}{{N}}$
RMEE [39] | $M=3$ | $({\phi_{1}}(e),{\phi_{2}}(e),{\phi_{3}}(e))\\!=\\!({G_{\sigma}}(e\\!-\\!0),{G_{\sigma}}(e\\!+\\!1),{G_{\sigma}}(e\\!-\\!1))$ | $({\theta_{1}},{\theta_{2}},{\theta_{3}})\\!=\\!(\frac{{-{M_{1}}}}{{N}},\frac{{-{M_{2}}}}{{N}},\frac{{-{M_{3}}}}{{N}})$
MMKCC [24] | $M\geq 2$ | ${\phi_{i}}(e)={G_{{\sigma_{i}}}}(e-{c_{i}})$ | ${\theta_{i}}=-\lambda_{i}$

  * 1

The symbol $L_{{\sigma}}$ denotes an Laplacian kernel controlled by the kernel
size $\sigma$.

Another popular learning criterion in ITL is the MCC, which is computationally
much simpler than the MEE criterion (but the performance of MEE is usually
better in the case of complex noise). The empirical loss under MCC is [13]

$L=-\frac{1}{N}\sum\limits_{i=1}^{N}{{G_{\sigma}}\left({{e_{i}}}\right)}.$ (8)

There is a minus sign in the above formula because the loss is minimized. One
can see that the error loss function of MCC is $l(e)=-{G_{\sigma}}(e)$
(sometimes $l(e)=1-{G_{\sigma}}(e)$ is used to ensure the non-negativity of
the loss function). Thus, the MCC loss function can be viewed as a special ELN
with only one hidden node ${\phi_{1}}(e)={G_{\sigma}}(e)$ and corresponding
output weight ${\theta_{1}}=-1$.

Recently, in order to further improve the learning performance, the MCC loss
function has been extended to non-Gaussian kernel functions (e.g., GMCC [18],
KRSL [21], KMPE [22]) or multi-kernel functions (e.g., MMCC [19, 20], MMKCC
[24]). In fact, all these loss functions can be integrated into a unified
model, the ELN. The network size, radial basis functions and output weights of
the ELNs for several ITL loss functions are summarized in Table I.

##  III Supervised Learning with ELNs

###  III-A Overall Learning Strategy

When applying the ELN to supervised learning, before training the learning
machine one must have an ELN available. To this end, one can predetermine an
ELN (e.g., assign an existing ITL loss function) or use the training data to
train an ELN. In this study, we mainly consider the second case, i.e.,
training an ELN to obtain a data-driven loss function for supervised learning.
In this case, the whole learning process consists of two sub-learning
processes, i.e., training the ELN and training the learning machine (see Fig.
3 for the general schematic).

Figure 3: Supervised learning with ELN.

The two sub-learning processes in Fig. 3 are interdependent, that is, training
the ELN depends on the error samples from the trained learning machine;
training the learning machine depends on the loss function from the trained
ELN. How to combine the two sub-learning processes to form a complete learning
process is a key problem. To solve this problem, we first assume that the
initial error samples over the training data set have been obtained with a
predetermined loss function, and then perform the following two sub-learning
processes: first, using the available error samples to train an ELN and obtain
an updated error loss function; second, using the updated loss function to
continually train the learning machine. The two processes can form a cycle,
whose schematic is shown in Fig. 4.

Figure 4: Schematic of the two-stage learning with ELN.

###  III-B Training the ELN

The proposed ELN model consists of two key parts. The first is the weight
vector
$\boldsymbol{\theta}={\left[{{\theta_{1}},\cdots,{\theta_{M}}}\right]^{T}}$
and the second is $M$ radial basis functions
$\left\\{{{\phi_{1}},\cdots,{\phi_{M}}}\right\\}$. In this subsection, we
discuss how to use the error samples to train the ELN model.

####  III-B1 Determine the Weight Vector $\boldsymbol{\theta}$

Suppose we have obtained $N$ error samples
$\left\\{{{e_{1}},\cdots,{e_{N}}}\right\\}$ from the first stage learning. To
simplify the discussion, we assume that $M$ hidden nodes (radial basis
functions) have already been determined (we will discuss this issue later),
and we only need to learn the output weight vector
$\boldsymbol{\theta}={\left[{{\theta}_{1},\cdots,{\theta_{M}}}\right]^{T}}$.
In the following, an example of using the popular PDF matching based method is
provided. Specifically, we propose to learn the weight vector
$\boldsymbol{\theta}$ such that the input-output mapper of the ELN (i.e., the
error loss function $l(e)$) is as close as possible to the error’s negative
PDF $-p(e)$. To further explain why we solve $\boldsymbol{\theta}$ in this
way, here we give some explanation. Fig. 5 illustrates an error’s PDF (solid)
and negative PDF (dotted). If the error loss function is close to the negative
PDF $-p(e)$, it will assign small valued loss to the error with high
probability density and large valued loss to the error with low probability
density. This is in fact very reasonable because the errors at high
probability density regions are usually normal errors (we always assume that
the normal error is the majority), while the errors at low probability density
regions are usually abnormal errors (e.g., errors caused by outliers).

Figure 5:  Error’s PDF and negative PDF.

Based on these discussions, we can minimize the following objective function
to solve the weight vector $\boldsymbol{\theta}$:

$\begin{array}[]{l}J=\int\limits_{-\infty}^{+\infty}{{{\left({l(e)+p(e)}\right)}^{2}}de}\\\
=\int\limits_{-\infty}^{+\infty}{{l^{2}}(e)de}{\rm{+2}}E\left[{l(e)}\right]{\rm{+}}\int\limits_{-\infty}^{+\infty}{{p^{2}}(e)de}\\\
=\int\limits_{-\infty}^{+\infty}{{{\left({\sum\limits_{j=1}^{M}{{\theta_{j}}{\phi_{j}}\left(e\right)}}\right)}^{2}}de}{\rm{+2}}E\left[{\sum\limits_{j=1}^{M}{{\theta_{j}}{\phi_{j}}\left(e\right)}}\right]{\rm{+}}\int\limits_{-\infty}^{+\infty}{{p^{2}}(e)de},\end{array}$

(9)

where $E\left[.\right]$ is the expectation operator. Since the term
$\int\limits_{-\infty}^{+\infty}{{p^{2}}(e)de}$ is independent of
$\boldsymbol{\theta}$, we have

$\begin{array}[]{l}{\boldsymbol{\theta}^{*}}=\mathop{\arg\min}\limits_{\boldsymbol{\theta}\in{\mathbb{R}^{M}}}\left\\{{\int\limits_{-\infty}^{+\infty}{{{\left({\sum\limits_{j=1}^{M}{{\theta_{j}}{\phi_{j}}\left(e\right)}}\right)}^{2}}de}{\rm{+2}}E\left[{\sum\limits_{j=1}^{M}{{\theta_{j}}{\phi_{j}}\left(e\right)}}\right]}\right\\}\\\
{\rm{}}\ \ \
=\mathop{\arg\min}\limits_{\boldsymbol{\theta}\in{\mathbb{R}^{M}}}\left\\{{{\boldsymbol{\theta}^{T}}\textbf{K}\boldsymbol{\theta}{\rm{+2}}{\boldsymbol{\theta}^{T}}\boldsymbol{\xi}}\right\\},\end{array}$

(10)

where $\mathbf{K}$ is an $M\times M$ matrix with
${\textbf{K}_{ij}}=\int_{-\infty}^{+\infty}{{\phi_{i}}\left(e\right){\phi_{j}}\left(e\right)de}$,
and
$\boldsymbol{\xi}={\left[{E\left[{{\phi_{\rm{1}}}\left(e\right)}\right],\cdots,E\left[{{\phi_{M}}\left(e\right)}\right]}\right]^{T}}$.
In practice, the vector $\boldsymbol{\xi}$ can be estimated by
$\hat{\boldsymbol{\xi}}={\left[{\frac{1}{N}\sum_{i=1}^{N}{{\phi_{1}}\left({{e_{i}}}\right)},\cdots,\frac{1}{N}\sum_{i=1}^{N}{{\phi_{M}}\left({{e_{i}}}\right)}}\right]^{T}}$.
Particularly, if the $M$ radial basis functions are assumed to be Gaussian
kernel functions with their centers being $c_{1},c_{2},...,c_{M}$ and kernel
widths being $\sigma_{1},\sigma_{2},...,\sigma_{M}$, respectively, the vector
$\boldsymbol{\xi}$ can be further calculated by
$\hat{\boldsymbol{\xi}}={\left[{\frac{1}{N}\sum_{i=1}^{N}{G_{\sigma_{1}}(e_{i}-c_{1})},\cdots,\frac{1}{N}\sum_{i=1}^{N}{G_{\sigma_{M}}(e_{i}-c_{M})}}\right]^{T}}$
and ${\textbf{K}_{ij}}$ has the form of
${\textbf{K}_{ij}}=G_{\sqrt{{\sigma_{i}}^{2}+{\sigma_{j}}^{2}}}(c_{i}-c_{j})$.
The calculation of ${\textbf{K}_{ij}}$ can refer to Appendix A. Thus, one can
obtain the following solution:

$\boldsymbol{\theta}^{*}=-{(\textbf{K}+\gamma_{{}_{1}}\textbf{I})^{-{\rm{1}}}}\hat{\boldsymbol{\xi}},$
(11)

where $\gamma_{{}_{1}}\geq 0$ is a regularization parameter to avoid the
numerical problem in matrix inversion.

Remark 1: The idea of using PDF to design a loss function has already appeared
in some existing research works, such as the design of MEE criterion and the
design of MMKCC. All of these research works showed that using PDF to design a
loss function is an excellent candidate. However, such method, in general,
involves more calculation cost, since it needs to train the loss itself. In
addition to the PDF matching method discussed in detail in this section, one
can also train an ELN with other productive methods, such as the quantized
method or some fast heuristic search algorithms [17, 40, 41].

####  III-B2 Determine the $M$ Radial Basis Functions

There are many types of radial basis functions that can be selected to
construct the ELN model, such as the Gaussian kernel, the Laplacian kernel,
and some extended Gaussian kernels [18, 21, 22, 42]. In the following, we will
focus on the most commonly used Gaussian kernel; however one should note that
${{\phi_{i}},i\\!=\\!1,2,\\!\cdots\\!,M}$ in the ELN model can be selected
differently for different nodes to get a better performance.

Following the idea of [43] and [44], when the Gaussian kernel is adopted to
construct an RBF network, the centers of $M$ radial basis functions can be
directly selected from training samples. Since the proposed ELN shares a
similar structure to the traditional RBF neural network, such idea is, of
course, suitable to ELN. Therefore, ${\phi_{i}}(e)$ can be expresssed by

${\phi_{i}}(e)\\!=\\!\frac{1}{\sqrt{2\pi}\sigma}\exp\left({\frac{{{-(e-c_{i})^{2}}}}{{2{{\sigma_{i}}^{2}}}}}\right),i\\!=\\!1,2,\\!\cdots\\!,M,$
(12)

where $c_{i}$ is the center of the $i$-th Gaussian kernel, and $\sigma_{i}$ is
the related kernel width. In practice, the selection of
$\left\\{{{c_{1}},\cdots,{c_{M}}}\right\\}$ does not have to be limited to a
fixed manner. For instance, when the number of error samples obtained in the
first stage learning is very limited, all the error samples can be selected to
construct the ELN model, and when the number of available error samples is
very large, the random sampling technology [45], the k-means clustering
technology [46], and the probability density rank-based quantization
technology [47] can be good candidates. In addition, following the idea of
randomized learning machines [48, 49, 50], it would be promising to generate
$\left\\{{{c_{1}},\cdots,{c_{M}}}\right\\}$ in a completely random way that is
independent of the error samples.

Once $\left\\{{{c_{1}},\cdots,{c_{M}}}\right\\}$ are determined, the related
$M$ kernel widths $\left\\{{{\sigma_{1}},\cdots,{\sigma_{M}}}\right\\}$ should
also be properly selected. A simple but useful way to select these kernel
widths is to set all kernel widths to the same value, i. e.,
${{\sigma_{1}}={\sigma_{2}}=\cdots={\sigma_{M}}={\sigma}}$, and then searching
$\sigma$ in a predefined candidate set via cross-validation or other methods.
However, since it is difficult to list the potential values of the kernel
width exhaustively, the final selected $\sigma$ may not be the desired one. To
alleviate this issue, we propose to set kernel widths in the manner of

$\sigma_{i}={\rm max}(\sigma+n_{i},\epsilon),i\\!=\\!1,2,\\!\cdots\\!,M,$ (13)

where $\sigma$ is a reference value for all kernel widths, $n_{i}$ is a small
perturbation term which is assumed to be drawn from Gaussian distribution with
zero-mean and variance $\epsilon$, and “${\rm max}$” denotes the maximum value
operator.

###  III-C Training the Learning Machine

Now we get into the second stage of the learning, i.e., training the learning
machine using the obtained error loss function
$l(e)=\sum\limits_{j=1}^{M}{\theta_{j}^{*}{\phi_{j}}\left(e\right)}$. For a
general case, the input-output mapper of the learning machine can be denoted
by ${y_{i}}=f\left({{\mathbf{x}_{i}},\boldsymbol{\beta}}\right)$, where
$\boldsymbol{\beta}={\left[{{\beta_{1}},\cdots,{\beta_{K}}}\right]^{T}}\in{\mathbb{R}^{K}}$
is a $K$-dimensional weight vector to be learned. In this case, the training
of the learning machine can be formulated as the following optimization:

$\displaystyle{\boldsymbol{\beta}^{\rm{*}}}$
$\displaystyle=\mathop{\arg\min}\limits_{\boldsymbol{\beta}\in{\mathbb{R}^{K}}}L(\boldsymbol{\beta})$
$\displaystyle=\mathop{\arg\min}\limits_{\boldsymbol{\beta}\in{\mathbb{R}^{K}}}\left\\{{\frac{1}{N}\sum\limits_{i=1}^{N}{\sum\limits_{j=1}^{M}{\theta_{j}^{*}{\phi_{j}}\left({{e_{i}}}\right)}}+\frac{\gamma_{{}_{2}}}{2}{{\left\|\boldsymbol{\beta}\right\|}^{2}}}\right\\},$
(14)

where ${e_{i}}=d_{i}-f\left({{\mathbf{x}_{i}},\boldsymbol{\beta}}\right)$,
$L(\boldsymbol{\beta})=\frac{1}{N}\sum\nolimits_{i=1}^{N}{\sum\nolimits_{j=1}^{M}{\theta_{j}^{*}{\phi_{j}}\left({{e_{i}}}\right)}}+\frac{\gamma_{{}_{2}}}{2}{\left\|\boldsymbol{\beta}\right\|^{2}}$,
and $\gamma_{{}_{2}}\geq 0$ is a regularization parameter. The above
optimization problem is in general nonconvex and there is no closed-form
solution. One can, however, use a gradient based method to search the
solution. Below we show that if the mapper
${y_{i}}=f\left({{\mathbf{x}_{i}},\boldsymbol{\beta}}\right)$ is an LIP model
(such as the Gaussian kernel model [43], functional link neural network [48],
extreme leaning machine [49], broad learning system [50], etc.), an efficient
fixed-point iterative algorithm is available to solve the solution.

Consider an LIP model whose output is computed by

$\displaystyle{y_{i}}$ $\displaystyle={\mathbf{h}_{i}}\boldsymbol{\beta}$
$\displaystyle=\left[{{\varphi_{1}}({\mathbf{x}_{i}}),{\varphi_{2}}({\mathbf{x}_{i}}),\cdots,{\varphi_{K}}({\mathbf{x}_{i}})}\right]{\left[{{\beta_{1}},{\beta_{2}},\cdots,{\beta_{K}}}\right]^{T}},$
(15)

where
${\mathbf{h}_{i}}=\left[{{\varphi_{1}}({\mathbf{x}_{i}}),{\varphi_{2}}({\mathbf{x}_{i}}),\cdots,{\varphi_{K}}({\mathbf{x}_{i}})}\right]\in{\mathbb{R}^{K}}$
is the nonlinearly mapped input vector (a row vector), with ${\varphi_{k}}(.)$
being the $k$-th nonlinear mapping function ($k=1,2,\cdots,K$). Let
${{\partial L(\boldsymbol{\beta})}\mathord{\left/{\vphantom{{\partial
L(\omega)}{\partial\omega}}}\right.\kern-1.2pt}{\partial\boldsymbol{\beta}}}=0$
and ${\phi_{j}}\left(e_{i}\right)=G_{\sigma_{j}}\left(e_{i}-c_{j}\right)$, we
have

$\begin{array}[]{l}\frac{1}{N}\sum\limits_{i\\!=\\!1}^{N}\sum\limits_{j\\!=\\!1}^{M}\frac{\theta_{j}^{*}}{\sigma_{j}^{2}}G_{\sigma_{j}}\left(e_{i}\\!-\\!c_{j}\right)\left(e_{i}\\!-\\!c_{j}\right)\mathbf{h}_{i}^{T}\\!+\\!\gamma_{{}_{2}}\boldsymbol{\beta}\\!=\\!0\\\
\\!\Leftrightarrow\\!\sum\limits_{i\\!=\\!1}^{N}\sum\limits_{j\\!=\\!1}^{M}\frac{\theta_{j}^{*}}{\sigma_{j}^{2}}G_{\sigma_{j}}\left(e_{i}\\!-\\!c_{j}\right)\left(d_{i}\\!-\\!{\mathbf{h}_{i}}\boldsymbol{\beta}\\!-\\!c_{j}\right)\mathbf{h}_{i}^{T}\\!+\\!\gamma_{{}_{2}}^{{}^{\prime}}\boldsymbol{\beta}\\!=\\!0\\\
\\!\Leftrightarrow\\!\sum\limits_{i\\!=\\!1}^{N}\psi(e_{i})d_{i}\mathbf{h}_{i}^{T}\\!-\\!\sum\limits_{i\\!=\\!1}^{N}\xi(e_{i})\mathbf{h}_{i}^{T}\\!=\\!\sum\limits_{i\\!=\\!1}^{N}\psi(e_{i})\mathbf{h}_{i}^{T}\mathbf{h}_{i}\boldsymbol{\beta}\\!-\\!\gamma_{{}_{2}}^{{}^{\prime}}\boldsymbol{\beta},\end{array}$

(16)

where $\gamma_{{}_{2}}^{{}^{\prime}}=N\gamma_{{}_{2}}$,
$\psi(e_{i})=\sum\limits_{j\\!=\\!1}^{M}\frac{\theta_{j}^{*}}{\sigma_{j}^{2}}G_{\sigma_{j}}\left(e_{i}-c_{j}\right)$,
and
$\xi(e_{i})=\sum\limits_{j\\!=\\!1}^{M}\frac{c_{j}\theta_{j}^{*}}{\sigma_{j}^{2}}G_{\sigma_{j}}\left(e_{i}-c_{j}\right)$.
From (16), one can obtain

$\boldsymbol{\beta}=\textbf{R}(\boldsymbol{\beta})^{-1}\textbf{p}(\boldsymbol{\beta}),$
(17)

with

$\displaystyle\textbf{R}(\boldsymbol{\beta})$
$\displaystyle=\sum\limits_{i=1}^{N}\psi(e_{i})\mathbf{h}_{i}^{T}\mathbf{h}_{i}-\gamma_{2}^{\prime}\mathbf{I}$
$\displaystyle=\mathbf{H}^{T}\boldsymbol{\Lambda}\mathbf{H}-\gamma_{2}^{\prime}\mathbf{I},$
(18)

and

$\displaystyle\textbf{p}(\boldsymbol{\beta})$
$\displaystyle=\sum\limits_{i=1}^{N}\psi(e_{i})d_{i}\mathbf{h}_{i}^{T}-\sum\limits_{i=1}^{N}\xi(e_{i})\mathbf{h}_{i}^{T}$
$\displaystyle=\mathbf{H}^{T}\boldsymbol{\Lambda}\mathbf{d}-\mathbf{H}^{T}\boldsymbol{\vartheta}.$
(19)

where
$\textbf{H}=[{\mathbf{h}_{1}}^{T},{\mathbf{h}_{2}}^{T},\cdots,{\mathbf{h}_{N}}^{T}]^{T}$,
$\textbf{d}=[d_{1},d_{2},\cdots,d_{N}]^{T}$,
$\boldsymbol{\vartheta}=[\xi(e_{1}),\xi(e_{2}),\cdots,\xi(e_{N})]^{T}$, and
$\boldsymbol{\Lambda}$ is a diagonal matrix with entries
$\boldsymbol{\Lambda}_{i,i}=\psi(e_{i}),i=1,2,\cdots,N$.

It should be noted that both $\textbf{R}(\boldsymbol{\beta})$ and
$\textbf{p}(\boldsymbol{\beta})$ in (17) are the functions of
$\boldsymbol{\beta}$, and hence it is actually a fixed-point equation that can
be described by

$\boldsymbol{\beta}=f_{FP}(\boldsymbol{\beta}),$ (20)

with

$\displaystyle
f_{FP}(\boldsymbol{\beta})=\textbf{R}(\boldsymbol{\beta})^{-1}\textbf{p}(\boldsymbol{\beta}).$
(21)

Such fixed-point equation can be solved by the popular fixed-point iterative
method [51, 52, 53] and the details are shown in Algorithm 1.

Input: training set $\\{\textbf{x}_{i},y_{i}\\}_{i=1}^{N}$.

Output: weight vector $\boldsymbol{\beta}$.

1 1\. Parameters setting: number of radial basis functions :

$M$; reference value for all kernel widths: $\sigma$; variance of perturbation
term: $\epsilon$; regularization parameters: $\gamma_{1}$,
$\gamma_{2}^{\prime}$; maximum number of iterations: $T$; tolerance: $\tau$.

2\. Initialization: set $\boldsymbol{\beta}(0)\\!=\\!\textbf{0}$ and the $M$
kernel widths following (13); meanwhile, construct ${\mathbf{h}_{i}}$
following a

2 preset LIP model.

3\. for $t=1,...,T$ do

4\. Compute the outputs:
${y_{i}}\\!=\\!{\mathbf{h}_{i}}\boldsymbol{\beta}(t\\!-\\!1),i\\!=\\!1,2,\\!\cdots\\!,N$.

5\. Compute the errors:
${e_{i}}\\!=\\!{d_{i}}-{y_{i}},i\\!=\\!1,2,\\!\cdots\\!,N$.

6\. Determine the $M$ centers $\left\\{{{c_{1}},\cdots,{c_{M}}}\right\\}$
using the

random sampling technology or other methods.

7\. Compute K:
${\textbf{K}_{ij}}=G_{\sqrt{{\sigma_{i}}^{2}+{\sigma_{j}}^{2}}}(c_{i}-c_{j})$.

8\. Compute $\hat{\boldsymbol{\xi}}$:
$\hat{\boldsymbol{\xi}}=[\hat{\boldsymbol{\xi}}_{1},\cdots,\hat{\boldsymbol{\xi}}_{m},\cdots,\hat{\boldsymbol{\xi}}_{M}]$,
where

${\boldsymbol{\xi}}_{m}=\frac{1}{N}\sum_{i\\!=\\!1}^{N}{G_{\sigma_{m}}(e_{i}\\!-\\!c_{m})}$.

9\. Compute $\boldsymbol{\theta}^{*}$:
$\boldsymbol{\theta}^{*}=-{(\textbf{K}+\gamma_{{}_{1}}\textbf{I})^{-{\rm{1}}}}\hat{\boldsymbol{\xi}}$.

10\. Compute $\boldsymbol{\Lambda}$:
$\boldsymbol{\Lambda}\\!=\\!\text{diag}[\psi(e_{1}),\\!\cdots\\!,\psi(e_{i}),\\!\cdots\\!,\psi(e_{N})]$,

where
$\psi(e_{i})=\sum\nolimits_{j\\!=\\!1}^{M}\frac{\theta_{j}^{*}}{\sigma_{j}^{2}}G_{\sigma_{j}}\left(e_{i}-c_{j}\right)$.

11\. Compute $\boldsymbol{\vartheta}$:
$\boldsymbol{\vartheta}=[\xi(e_{1}),\cdots,\xi(e_{i}),\cdots,\xi(e_{N})]^{T}$,

where
$\xi(e_{i})=\sum\nolimits_{j\\!=\\!1}^{M}\frac{c_{j}\theta_{j}^{*}}{\sigma_{j}^{2}}G_{\sigma_{j}}\left(e_{i}-c_{j}\right)$.

12\. Compute H:
$\textbf{H}=[{\mathbf{h}_{1}}^{T},{\mathbf{h}_{2}}^{T},\cdots,{\mathbf{h}_{N}}^{T}]^{T}$.

13\. Update $\boldsymbol{\beta}$:
$\boldsymbol{\beta}(t)\\!=\\!\big{(}\mathbf{H}^{T}\boldsymbol{\Lambda}\mathbf{H}\\!-\\!\gamma_{2}^{\prime}\mathbf{I}\big{)}^{\\!-\\!1}\big{(}\mathbf{H}^{T}\boldsymbol{\Lambda}\mathbf{d}\\!-\\!\mathbf{H}^{T}\boldsymbol{\vartheta}\big{)}$.

14\. Until
$\|\boldsymbol{\beta}(t)\\!-\\!\boldsymbol{\beta}(t-1)\|^{2}/\|\boldsymbol{\beta}(t-1)\|^{2}\\!<\\!\tau$.

15\. end for

Algorithm 1 Fixed-point Iterative Algorithm with ELN

Remark 2: Algorithm $1$ is mainly designed for the case of multiple-input-
single-output. To extend it to multiple-input-multiple-output cases, one can
directly refer to the method proposed in [54].

##  IV Experimental Results

In this section, we present experimental results to demonstrate the
performance of the proposed method. Except mentioned otherwise, all results
are obtained using MATLAB (R2016b) on a machine equipped with Intel Xeon
E3-1505M V6 CPU and 16-GB RAM.

###  IV-A Performance Evaluation on Regression Tasks

####  IV-A1 Linear Regression

To test the performance of the proposed method in the scenario of linear
regression, the weight vector of a linear system is assumed to be
$\boldsymbol{\beta}^{*}=[2,1]^{T}$, and the input vectors
$\textbf{x}_{i},i=1,2,\cdots,N$ of the system are assumed to be uniformly
distributed over $[-2,2]\times[-2,2]$. Hence, the system outputs can be
expressed by

$y_{i}=\textbf{x}_{i}^{T}{\boldsymbol{\beta}^{*}},i=1,2,\cdots,N.$ (22)

An additional interference sequence is then added to the system outputs to
model the effect of noise and outliers, i. e.,

$d_{i}=y_{i}+v_{i},i=1,2,\cdots,N,$ (23)

where $d_{i}$ is the noisy observation of the $i$-th output of the system. Our
goal is to find the weight vector of the system based on $N$ pairs of observed
samples $\\{(\textbf{x}_{i},d_{N}),\cdots,(\textbf{x}_{i},d_{N})\\}$. In the
experiments, $N$ is set to $500$, and the interference sequence
$v_{i},i=1,2,\cdots,N$ is modelled as

$v_{i}=(1-\eta_{i})A_{i}+\eta_{i}B_{i},i=1,2,\cdots,N,$ (24)

where $\eta_{i}$ is a binary variable with probability mass $Pr(\eta_{i}=1)=p$
and $Pr(\eta_{i}=0)=1-p$; $A_{i}$ and $B_{i}$ are two processes to model the
inner noise and outliers, respectively. Specifically, we set $p=0.1$, and set
$B_{i}$ to be a Gaussian process with zero-mean and variance $100$. For the
distribution of $A_{i}$, four cases are considered: 1) symmetric Gaussian
mixture density: $1/2\mathcal{N}(-5,0.1)+1/2\mathcal{N}(5,0.1)$, where
$\mathcal{N}(\mu,\delta)$ denotes a Gaussian density function with mean $\mu$
and variance $\delta$; 2) asymmetric Gaussian mixture density:
$1/3\mathcal{N}(-3,0.1)+2/3\mathcal{N}(5,0.1)$; 3) single Gaussian density:
$\mathcal{N}(0,0.1)$; 4) uniform distribution over $[0,1]$. To measure the
performance of the proposed method, the root mean square deviation (RMSD) is
defined by

${\rm
RMSD}=\sqrt{\frac{1}{2}\|\hat{\boldsymbol{\beta}}-\boldsymbol{\beta}^{*}\|^{2}},$
(25)

where $\hat{\boldsymbol{\beta}}$ is the estimate of $\boldsymbol{\beta}^{*}$.

TABLE II: Performance Comparison of Different Methods on Linear Regression Averaged Over $100$ Independent Runs (The best results are marked in bold, the second best results are underlined, and the symbol N/A means that the corresponding value is less than 0.0001) Index | Noise Type | MSE | MCC | MMCC | MCC-VC | GMCC | KRSL | KMPE | QMEE | ELN1 | ELN2
---|---|---|---|---|---|---|---|---|---|---|---
RMSD | Case 1 | 0.1686 | 0.1662 | 0.1659 | 0.0319 | 0.0922 | 0.0899 | 0.1010 | 0.0148 | 0.0116 | 0.0139
| Case 2 | 0.1667 | 0.1619 | 0.1612 | 0.0247 | 0.0817 | 0.0850 | 0.0903 | 0.0145 | 0.0125 | 0.0145
| Case 3 | 0.1133 | 0.0131 | 0.0131 | 0.0131 | 0.0131 | 0.0130 | 0.0138 | 0.0133 | 0.0164 | 0.0125
| Case 4 | 0.1143 | 0.0193 | 0.0187 | 0.0112 | 0.0163 | 0.0147 | 0.0163 | 0.0115 | 0.0119 | 0.0121
Time | Case 1 | N/A | 0.0006 | 0.0008 | 0.0063 | 0.0169 | 0.0134 | 0.0103 | 0.0318 | 0.0570 | 0.2094
| Case 2 | N/A | 0.0006 | 0.0006 | 0.0067 | 0.0282 | 0.0216 | 0.0128 | 0.0653 | 0.0407 | 0.2282
| Case 3 | N/A | 0.0008 | 0.0012 | 0.0008 | 0.0025 | 0.0044 | 0.0043 | 0.0346 | 0.0587 | 0.1653
| Case 4 | N/A | 0.0012 | 0.0012 | 0.0026 | 0.0061 | 0.0054 | 0.0047 | 0.0346 | 0.0310 | 0.3158

First, the performance comparison among ELN, MSE, MCC, MMCC, MCC-VC, GMCC,
KRSL, KMPE, and QMEE is carried out, and the results are shown in Table II.
Herein, both the random sampling technology [45] and the probability density
rank-based quantization technology [47] are considered to generate the centers
of the ELN model. For the convenience of distinction, they are marked as ELN1
and ELN2, respectively. To guarantee a fair comparison, the parameters for
each method are chosen by performing a grid search and measuring performance
by an additional stratified ten-fold cross validation on the training
data111We choose the one group parameters that can achieve minimum average
value of the sum of squares of validation errors.. In detail, the parameter
search ranges of different methods are as follows.

  * •

For MSE, the regularization parameter is searched from
$\\{10^{-5},10^{-4},\cdots,10^{4},10^{5}\\}$.

  * •

For MCC, its regularization parameter is searched in the same range of MSE,
and its kernel width is searched from
$\\{0.1,0.3,0.5,0.7,1,3,5,7,10,15,30,60,100\\}$.

  * •

For MMCC, the regularization parameter is searched in the range of
$\\{10^{-5},10^{-4},\cdots,10^{4},10^{5}\\}$, and the parameters $\sigma_{1}$,
$\sigma_{2}$ and $\lambda$ are chosen from $\\{0.1,0.3,0.5,0.7,1,3,5,7\\}$,
$\\{1,3,5,7,10,15,30,100\\}$, and $\\{0,0.1,0.3,0.5,0.7,0.9,1.0\\}$,
respectively.

  * •

For MCC-VC, its center of Gaussian kernel is searched from
$\\{-5,-3,-1,-0.5,0,0.5,1,3,5\\}$, while the search ranges of the remaining
parameters are kept the same of MCC.

  * •

For GMCC, the regularization parameter is chosen from
$\\{10^{-5},10^{-4},\cdots,10^{4},10^{5}\\}$, the parameter $\alpha$ is
searched from $\\{1,2,3,4,5\\}$, and the parameter $\lambda$ is searched from
$\\{0.0005,0.001,0.005,0.01,0.05,0.1,0.5,1,5\\}$.

  * •

For KRSL, the regularization parameter is chosen from
$\\{10^{-5},10^{-4},\cdots,10^{4},10^{5}\\}$, the parameter $\lambda$ is
searched from $\\{0.1,0.3,0.5,0.7,1,3,5,7,10\\}$, and the kernel width is
searched from $\\{0.1,0.3,0.5,0.7,1,3,5,7,10,15,30,60,100\\}$.

  * •

For KMPE, the regularization parameter and the kernel width are searched in
the same ranges of KRSL, and the parameter $p$ is searched from
$\\{0.1,0.5,1.0,1.5,2.0,2.5,3.0,4.0,5.0\\}$.

  * •

For QMEE, all the parameter search ranges of MCC are shared with it. In
addition, the parameter for performing quantization operation in it is set to
$0.5$.

  * •

For ELN1 and ELN2, the regularization parameter $\gamma_{2}^{\prime}$ is
searched from $\\{10^{-5},10^{-4},\cdots,10^{4},10^{5}\\}$, the reference
value $\sigma$ for all kernel widths is searched from
$\\{0.1,0.3,0.5,0.7,1,3,5,7,10,15,30,60,100\\}$, the number of radial basis
functions is fixed at $M=50$, the regularization parameter $\gamma_{1}$ is set
to $10^{-3}$, and the variance of perturbation term is set as $\epsilon=0$.

In addition, for all iteration-based methods, the maximum number of iteration
is set as $T=50$, and the tolerance for early stopping is set to
$\tau=10^{-7}$. Based on the data shown in Table II, the following results can
be obtained.

  * •

Although MCC, MMCC, MCC-VC, GMCC, KRSL, KMPE, QMEE, ELN1, and ELN2 need more
training time in comparison with MSE criterion, they can outperform the latter
in terms of RMSD, significantly. This demonstrates the effectiveness of these
ITL criteria and the ELN model in robust learning.

  * •

The RMSDs of MMCC, MMCC, MCC-VC, GMCC, KRSL, KMPE, QMEE, ELN1, and ELN2 are
smaller than that of MCC on the most of cases. This demonstrates that the
criterion designed based on a single Gaussian kernel with zero-mean and fixed
variance can be improved by changing its mean, variance, kernel type, or by
integrating the advantages of different kernels.

  * •

The QMEE, ELN1, and ELN2 can, in general, obtain a better performance in terms
of RMSD in comparison with other robust criteria. The reason behind this may
be that more kernels are involved in these three criteria, such that they can
characterize complex noise distributions better [17].

  * •

The ELN1 and ELN2 have the ability to obtain a comparative and even smaller
RMSD than QMEE. A possible reason for this is that QMEE is built with a simple
online-vector quantization (OVQ) technology [17]. Although this technology
also chooses the centers of kernels from error samples, it however can not
keep the distribution information of the original samples well [47]. Moreover,
in the process of performing quantization operator, the contribution of a
discarded error sample is directly measured by its closest center, which
ignores the difference between two adjacent error samples. In contrast, the
proposed ELN model is built based on the idea of RBF networks, whose centers
do not depend on any specific manner. Once an appropriate sampling technology,
such as the random sampling technology adopted in ELN1 or the probability
density rank-based quantization technology in ELN2, is adopted, it can capture
more distribution information hidden in the original error samples. Meanwhile,
the combination coefficient in the ELN model are determined by the idea of PDF
matching, which can, to some extent, allocate the coefficients of different
kernels in a more reasonable manner.

Then, the influence of the free parameters on the learning performance of ELN
is investigated. In particular, when the radial basis function is assumed to
be the Gaussian kernel, the extra parameters in comparison with MMSE criterion
to control the learning performance of an ELN include the regularization
parameter $\gamma_{1}$, the number of radial basis functions $M$, the
reference value $\sigma$ for all kernel widths, and the variance of
perturbation term $\epsilon$. In the following experiments, we investigate the
influence of these parameters on the learning performance of ELN in three
steps.

(a) Case 1

(b) Case 2

(c) Case 3

(d) Case 4

Figure 6: Influence of the parameters $M$ and $\gamma_{1}$ on the learning
performance of ELN under different noise environments.

First, on the premise of setting the values of the other parameters to be the
same as the previous ones222For all cases, $\epsilon=0$ is a default choice.
In addition, the values of $\sigma$ for Case 1, Case2, Case 3, and Case 4 are
set to $1$, $0.7$, $3$, and $0.7$, respectively; the values of
$\gamma_{2}^{\prime}$ for Case 1, Case2, Case 3, and Case 4 are set to
$10^{-1}$, $10^{-1}$, $10^{-2}$, and $10^{0}$, respectively., Fig. 6 shows the
variation of RMSD versus $M$ and $\gamma_{1}$ under different noise
environments. It can be seen from Fig. 6 that, although the choices of $M$ and
$\gamma_{1}$ can affect the learning performance of ELN, there is always a
large flat area that ELN can obtain a small RMSD close to the optimal value.
Moreover, once the value of $\gamma_{1}$ is appropriately chosen and the value
of $M$ exceeds a certain value (for example, $10^{-2}\leq\gamma_{1}\leq
10^{2}$ and $M\geq 50$), the change of $M$ does not affect the final learning
performance of ELN, evidently. These results suggest that $M$ and $\gamma_{1}$
are two parameters which can be easy to be tuned, and hence we do not need to
pay much attention to the selection of them in practice. Then, by setting the
values of the other parameters to be the same of the previous ones, Fig. 7
shows the variation of RMSD versus $\sigma$ under different noise
environments. As can be seen from Fig. 7, the change of $\sigma$ can affect
the learning performance of ELN, significantly. In particular, when the value
of $\sigma$ is set near $2^{0}$, the RMSD of ELN can approach its performance
shown in Table II. However, when the value of $\sigma$ is set to be very large
or very small, the RMSD of ELN can increase dramatically. Therefore, $\sigma$
is a parameter that should be carefully chosen in practice. Other than the
grid search method adopted in this paper, other productive technologies, like
the whale optimization algorithm (WOA) [41] method, can also be a good
candidate.

Figure 7: Influence of the parameter $\sigma$ on the learning performance of
ELN under different noise environments.

Finally, the influence of $\epsilon$ on the learning performance of ELN is
investigated, and the related experimental results are shown in Table III. It
can be seen from Table III that, instead of always setting $\epsilon=0$,
properly adjusting the value of $\epsilon$ can further enhance the learning
performance of ELN. However, it should be noted that the RMSD of ELN will
increase dramatically when the value of $\epsilon$ is set to be greater than
$10^{0}$. Thus, in the following experiments, only smaller values of
$\epsilon$ will be considered as candidate set.

TABLE III: Influence of the parameter $\epsilon$ on the learning performance of ELN under different noise environments $\epsilon$ | RMSD
---|---
Case 1 | Case 2 | Case 3 | Case 4
$0$ | 0.0116 | 0.0125 | 0.0164 | 0.0119
$10^{-7}$ | 0.0117 | 0.0132 | 0.0172 | 0.0110
$10^{-5}$ | 0.0113 | 0.0127 | 0.0169 | 0.0121
$10^{-3}$ | 0.0125 | 0.0141 | 0.0152 | 0.0128
$10^{-1}$ | 0.3911 | 2.3767 | 0.0126 | 1.7032
$10^{0}$ | 0.0114 | 0.0133 | 0.1234 | 0.0120
$10^{1}$ | 0.4792 | 0.3405 | 0.0531 | 0.9983
$10^{3}$ | 1.5811 | 1.5811 | 1.5811 | 1.5811
$10^{5}$ | 1.5811 | 1.5811 | 1.5811 | 1.5811

####  IV-A2 Nonliner Regression

In this subsection, ten real regression benchmark data sets, taken from UCI
machine learning repository [55],
LIACC333https://www.dcc.fc.up.pt/~ltorgo/Regression/DataSets.html and
LIBSVM444https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/, are further
adopted to test the performance of the proposed ELN in nonlinear regression
examples. The descriptions of these data sets are shown in Table IV. In the
experiment, the training and testing samples for each data set are randomly
selected and the data have been normalized into the range of $[0,1]$. In
addition, an additional noise sequence has been added to the outputs of
training samples. The details about the noise model can refer to (24). In
particular, we set $p=0.1$, and set the density functions of $A_{i}$ and
$B_{i}$ to be $1/2\mathcal{N}(-0.8,0.01)+1/2\mathcal{N}(0.8,0.01)$ and
$\mathcal{N}(0,10)$, respectively.

TABLE IV: Specification of Regression Benchmark Data Sets Data Set | Attributes | Training | Testing
---|---|---|---
Concrete Slump Test | 10 | 52 | 52
Machine CPU | 7 | 105 | 104
Residential Building | 106 | 186 | 186
Auto MPG | 8 | 199 | 199
Real Estate Valuation | 7 | 207 | 207
Concrete Compressive Strength | 9 | 515 | 515
Space GA | 7 | 1554 | 1553
Computer Activity | 21 | 4096 | 4096
Pole | 48 | 7500 | 7500
California Housing | 9 | 10320 | 10320

To perform the nonlinear mapping from inputs to outputs of these data sets,
the random vector functional link neural network (RVFLNN) [48] is adopted as
an LIP model for training. Following the idea of RVFLN, the ${\textbf{h}_{i}}$
shown in (III-C) can be constructed by

${\textbf{h}_{i}}=\left[{\textbf{x}_{i},|{h_{1}}({\textbf{x}_{i}}),{h_{2}}({\textbf{x}_{i}}),\cdots,{h_{K}}({\textbf{x}_{i}})}\right],$
(26)

with

${h_{k}}({\textbf{x}_{i}})=g(\textbf{x}_{i}\textbf{w}_{k}+b_{k}),k=1,2,\dots,K,$
(27)

where $K$ is the number of hidden nodes of the network, $\textbf{w}_{k}$ is
the randomly generated weights which connect the $k$ hidden node to the input
layer, $b_{k}$ is the randomly generated bias of the $k$ hidden node, and $g$
is the activation function. In the following experiments, $K$ is set to $200$
for all compared methods and data sets, $\\{\textbf{w}_{k}\\}_{k=1}^{K}$ are
assumed to be drawn from uniform distribution over $[-1,1]$,
$\\{b_{k}\\}_{k=1}^{K}$ are assumed to be drawn from uniform distribution over
$[0,1]$, and $g$ is set as the “sigmoid” function whose expression is

$g(x)=\frac{1}{\text{exp}(-x)+1}.$ (28)

TABLE V: Test RMSE of Different Methods on Regression Benchmark Data Sets Averaged Over $20$ Independent Runs Data Set | MSE | MCC | MMCC | MCC-VC | GMCC | KRSL | KMPE | QMEE | ELN
---|---|---|---|---|---|---|---|---|---
Concrete Slump Test | 0.2997 | 0.2541 | 0.2654 | 0.2471 | 0.2381 | 0.6785 | 0.2946 | 0.2098 | 0.2161
Machine CPU | 0.1524 | 0.1347 | 0.1359 | 0.1330 | 0.1148 | 0.1360 | 0.2493 | 0.0878 | 0.0761
Residential Building | 0.1611 | 0.1525 | 0.1532 | 0.1525 | 0.1524 | 0.1515 | 0.1511 | 0.1110 | 0.0917
Auto MPG | 0.1938 | 0.1957 | 0.1984 | 0.2120 | 0.1420 | 0.1232 | 0.1589 | 0.1341 | 0.0934
Real Estate Valuation | 0.1197 | 0.1175 | 0.1216 | 0.1269 | 0.0995 | 0.0899 | 0.0884 | 0.1022 | 0.0983
Concrete Compressive Strength | 0.2304 | 0.2236 | 0.1686 | 0.2236 | 0.1678 | 0.1786 | 0.1840 | 0.1603 | 0.1443
Space GA | 0.0592 | 0.0513 | 0.0558 | 0.0513 | 0.0460 | 0.0456 | 0.0570 | 0.0417 | 0.0411
Computer Activity | 0.1267 | 0.1188 | 0.1125 | 0.1188 | 0.0758 | 0.0756 | 0.0754 | 0.0456 | 0.0471
Pole | 0.2610 | 0.2546 | 0.2536 | 0.2546 | 0.2372 | 0.2391 | 0.2359 | 0.2235 | 0.2220
California Housing | 0.1497 | 0.1482 | 0.1470 | 0.1482 | 0.1361 | 0.1350 | 0.1349 | 0.1304 | 0.1294

In order to make a fair comparison, the parameters for each method are still
chosen by the grid search method. In detail, the parameter search ranges of
MSE, MCC, MMCC, MCC-VC, GMCC, KRSL, and KMPE are directly set to the same of
those in Section IV-A1\. For QMEE, except that the parameter for performing
quantization operation has been changed to $0.2$, other parameters of it are
searched in the same ranges described in Section IV-A1\. For ELN, the random
sampling technology is chosen to generate the centers of radial basis
functions, the search range of $\gamma_{1}$ has been extended to
$\\{10^{-5},10^{-3},10^{-1}\\}$, the variance of perturbation term is fixed at
$\epsilon=10^{-4}$, while the number of radial basis functions is chosen in
the manner of

$M=\left\\{\begin{array}[]{ll}N,&\ \ N<50\\\ 50,&\ \ 50\leq N<3000\\\ 300,&\ \
N\geq 3000,\\\ \end{array}\right.$ (29)

where $N$ is the number of training samples. In addition, since the optimal
weight vector $\boldsymbol{\beta}^{*}$ is not available in this part, RMSD has
been replaced with the root mean square Error (RMSE) for performance
evaluation, i. e.,

${\rm
RMSE}=\sqrt{\frac{1}{N_{te}}\sum\limits_{i=1}^{N_{te}}\left(d_{i}-\hat{d}_{i}\right)^{2}},$
(30)

where $d_{i}$ and $\hat{d}_{i}$ are the real output and its estimate of the
$i$-th testing sample555After training ends, a bias with its value being equal
to the mean of training errors has been added to the outputs of the model when
QMEE and ELN have been adopted for training. Such operator is common to
supervised learning methods under the MEE criterion, and the details can refer
to [10]. Since QMEE and ELN are closely related to MEE, the outputs of the
models trained with them are both added with an additional bias.,
respectively, and $N_{te}$ is the number of testing samples.

Table V shows the test RMSE of different methods on the ten regression
benchmark data sets. It can be seen from Table V that, although ELN does not
always obtain the minimal RMSE on the ten data sets, it can, at least,
outperform other compared methods on the most of data sets in terms of RMSE.
This suggests that the RVFLNN developed with ELN can be a good candidate to
general regression problems.

###  IV-B Performance Evaluation on Classification Tasks

The good performance of the proposed ELN has been demonstrated in some
regression tasks. In this subsection, its performance will be evaluated on
some classification benchmark data sets. In order to test the robustness of
different methods, the learning with noisy labels is considered in the
following. In particular, the noisy labels are manually generated by a noise
transition matrix $Q$, where $Q_{i,j}=Pr(\hat{t}=j\mid t=i)$ gives that noisy
$\hat{t}$ is flipped from clean $t$. For a classification problem involving
$C$ categories, a popular structure of $Q$ [56] can be formulated by

${Q}\\!=\\!\left[{\begin{array}[]{*{20}{c}}{1-\varepsilon}&\varepsilon&0&\cdots&0&\cdots&0\\\
0&\ddots&\ddots&\ddots&\vdots&\ddots&\vdots\\\
\vdots&\ddots&{1-\varepsilon}&\varepsilon&0&\cdots&0\\\
0&\cdots&0&\ddots&\varepsilon&\ddots&\vdots\\\
\vdots&\ddots&\vdots&\ddots&{1-\varepsilon}&\ddots&0\\\
0&\cdots&0&\cdots&0&\ddots&\varepsilon\\\
\varepsilon&0&\cdots&0&\cdots&0&{1-\varepsilon}\end{array}}\right]\in{\mathbb{R}^{C\times
C}}$

(31)

where $\varepsilon$ is the noise rate which controls the ratio of noisy
labels. Without loss of generality, the accuracy (ACC) defined as

${\rm{ACC}}=\frac{1}{{N}}\sum\limits_{i=1}^{{N}}{{{\delta\left({y_{i},\hat{y}_{i}}\right)}}},$
(32)

is adopted to measure the performance of different methods. Herein, $y_{i}$
and $\hat{y}_{i}$ are respectively the real label and the corresponding
prediction of the $i$th sample, and $\delta(x,y)$ is an indicator function
with its value being $1$ in the case of $x=y$ and $0$ otherwise.

####  IV-B1 Small Scale Classification Data Sets

First, ten small scale classification data sets are adopted for performance
evaluation. The details of these data sets can be found in Table VI. In the
experiment, the training and testing samples for each data set are randomly
selected and all attributes of data have been normalized into the range of
$[-1,1]$. Meanwhile, the labels of data have been transformed to one-hot
vectors. To test the robustness of different methods, $30\%$ of training data
are contaminated by label noise described in (31).

TABLE VI: Specification of Small Scale Classification Data Sets Data Set | Class | Attributes | Training | Testing
---|---|---|---|---
Iris | 3 | 4 | 75 | 75
Sonar | 2 | 60 | 104 | 104
Smooth Subspace | 3 | 15 | 150 | 150
PowerCons | 2 | 144 | 180 | 180
Chinatown | 2 | 24 | 183 | 182
Dermatolog | 6 | 34 | 183 | 183
Balance Scale | 3 | 4 | 313 | 312
Breast Cancer | 2 | 10 | 342 | 341
Australian | 2 | 14 | 345 | 345
Vehicle | 4 | 18 | 423 | 423
TABLE VII: Test ACC of Different Methods on Small Scale Classification Data Sets Averaged Over $20$ Independent Runs (in %) Data Set | MSE | MCC | MMCC | MCC-VC | GMCC | KRSL | KMPE | QMEE | ELN
---|---|---|---|---|---|---|---|---|---
Iris | 72.13 | 74.13 | 82.87 | 74.13 | 73.07 | 74.13 | 77.47 | 72.47 | 86.07
Sonar | 62.31 | 66.20 | 66.06 | 66.20 | 63.89 | 61.92 | 65.63 | 65.34 | 69.52
Smooth Subspace | 71.17 | 80.63 | 80.87 | 80.63 | 67.87 | 80.93 | 79.80 | 76.00 | 80.83
PowerCons | 81.08 | 81.67 | 82.47 | 81.67 | 92.89 | 82.69 | 87.83 | 78.17 | 94.53
Chinatown | 94.18 | 94.12 | 92.36 | 94.12 | 94.40 | 94.23 | 95.52 | 92.64 | 95.03
Dermatolog | 84.64 | 88.63 | 90.44 | 88.63 | 82.98 | 84.64 | 89.67 | 87.35 | 93.52
Balance Scale | 85.71 | 84.54 | 85.08 | 84.54 | 86.07 | 84.58 | 84.66 | 37.84 | 88.06
Breast Cancer | 93.74 | 94.33 | 94.22 | 94.33 | 93.94 | 94.69 | 93.70 | 93.21 | 94.74
Australian | 83.91 | 84.04 | 85.94 | 84.04 | 82.58 | 85.23 | 84.01 | 74.41 | 88.28
Vehicle | 69.09 | 69.14 | 69.33 | 69.18 | 69.76 | 71.69 | 69.18 | 69.98 | 71.57

To establish the connection between the attributes and the labels of these
data sets, the RBF neural network is adopted as an LIP model for training.
Following the idea of RBF, the ${\textbf{h}_{i}}$ shown in (III-C) can be
constructed by

${\textbf{h}_{i}}=\left[{h_{1}}({\textbf{x}_{i}}),{h_{2}}({\textbf{x}_{i}}),\cdots,{h_{N}}({\textbf{x}_{i}})\right],$
(33)

with

${h_{k}}({\textbf{x}_{i}})=\kappa_{\sigma}(\textbf{x}_{i},\textbf{x}_{k}),k=1,2,\dots,N,$
(34)

where $N$ is the number of training samples, $\kappa_{\sigma}$ is a kernel
function controlled by the kernel size $\sigma$. In the following experiments,
the Gaussian kernel will be a default choice for the LIP model, and the kernel
size $\sigma$ is chosen with MSE criterion from the range of
$\\{2^{-5},2^{-3},2^{-1},2^{0},2^{1},2^{3},2^{5}\\}$.

Instead of always training the ELN with PDF matching method, WOA has been
adopted in this part. In addition, we construct the ELN with two extended
Gaussian kernels, and hence it can be expressed as

$L\\!=\\!\frac{1}{N}\sum\limits_{i\\!=\\!1}^{N}\big{(}\theta_{1}{\phi_{1}}+(1-\theta_{1}){\phi_{2}}\big{)},$
(35)

where $\theta_{1}$ is the combination coefficient, and $\phi_{1}$ and
$\phi_{2}$ are constructed by

$\left\\{\begin{array}[]{l}\phi_{1}=-\text{exp}(-\lambda_{G}\mid
e_{i}\mid^{\alpha_{{}_{G}}})\\\
\phi_{2}=\frac{1}{\lambda_{K}}\text{exp}\left({\lambda_{K}\left({1-{G_{\sigma_{K}}}(e_{i})}\right)}\right).\end{array}\right.$
(36)

To search the best parameters of ELN with WOA, the fitness function is set as
the average error recognition rate of five-fold cross validation on the
training set. In addition, the search ranges for $\lambda_{G}$,
$\alpha_{{}_{G}}$, $\lambda_{K}$, $\sigma_{K}$, and $\theta_{1}$ are set as
$[0.01,2]$, $[1,3]$, $[0.1,5]$, $[0.1,3]$, and $[0,1]$, respectively.

Table VII shows the test ACC of different methods on the ten classification
benchmark data sets666We perform the classification process following [54]..
It can be seen from Table VII that, except the data sets “Smooth Subspace”,
“Chinatown” and “Vehicle“, the RBF network under ELN can always obtain a
higher test ACC in comparison with others. Hence, the RBF network developed
with ELN can be a good candidate to the classification problems of small scale
data sets.

####  IV-B2 Large Scale Classification Data Sets

Then, two large scale data sets, including Mixed National Institute of
Standards and Technology (MNIST) [57] and Fashion-MNIST [58], are further
adopted to test the performance of the proposed method. Both of them include
10 classes, 60 000 training samples, and 10 000 testing samples. Each sample
is represented by an image of 28 $\times$ 28 pixels. The main difference
between them is that MNIST is a handwritten digits data set, but Fashion-MNIST
involves different categories of clothes. In addition, the recognition of
Fashion-MNIST is deemed to be more challenging in comparison with the
recognition of MNIST.

To perform the classification tasks on the two large scale data sets, the
label of each image is transformed to be an one-hot vector, and a simple yet
effective convolutional neural network model called LeNet5 [57] is chosen for
training. Meanwhile, the ELN loss is chosen as the combination of three
Gaussian kernels, taking the form of

$L\\!=\\!\frac{1}{N}\sum\limits_{i\\!=\\!1}^{N}\big{(}\theta_{1}{\phi_{1}}+\theta_{2}{\phi_{2}}+(1-\theta_{1}-\theta_{2}){\phi_{3}}\big{)},$
(37)

where $\theta_{1}$ and $\theta_{2}$ are combination coefficients; $\phi_{1}$,
$\phi_{2}$ and $\phi_{2}$ are set as

$\displaystyle\left\\{\begin{array}[]{l}\phi_{1}=-\exp({\frac{{{-\parallel\mathbf{e}_{i}-\mathbf{c}_{1}\parallel^{2}}}}{{2{{\sigma}^{2}}}}})\\\
\phi_{2}=-\exp({\frac{{{-\parallel\mathbf{e}_{i}-\mathbf{c}_{2}\parallel^{2}}}}{{2{{\sigma}^{2}}}}})\\\
\phi_{3}=-\exp({\frac{{{-\parallel\mathbf{e}_{i}-\mathbf{c}_{3}\parallel^{2}}}}{{2{{\sigma}^{2}}}}}),\end{array}\right.$
(41)

with $\mathbf{c}_{1}$, $\mathbf{c}_{2}$ and $\mathbf{c}_{3}$ denoting the
centers of three Gaussian kernels in vector form, and $\sigma$ denotes the
kernel size. In the following experiments, the three centers are simply set to
$\mathbf{c}_{1}=[0,\cdots,0]\in\mathbb{R}^{1\times 10}$,
$\mathbf{c}_{2}=[2,\cdots,2]\in\mathbb{R}^{1\times 10}$, and
$\mathbf{c}_{3}=[-2,\cdots,-2]\in\mathbb{R}^{1\times 10}$. The kernel size
$\sigma$ is chosen from the candidate set of $\\{2^{-1},2^{3}\\}$, and
$\theta_{1}$ and $\theta_{2}$ are chosen from the candidate set of
$\\{0,0.2,0.6\\}$. Once the training is completed, the predicted label of a
given test image is determined by the index position of maximum value of the
related network outputs. Moreover, it should be noted that the network
parameters of LeNet5 are trained using an SGD optimizer with the assistance of
PyTorch 1.10.1. The learning rate of SGD is set to 0.05 at the initial stage,
and is changed to 0.005 and 0.0005 after 60 and 120 epochs, respectively. The
momentum factor is set to 0.9, and the weight decay factor is set to 0.0001.
Meanwhile, the batch size and the number of epochs are set to 128 and 200,
respectively.

TABLE VIII: Test ACC of different methods on Large Scale Data Sets (in %) Data Set | CE Loss | Weighted CE Loss | ELN
---|---|---|---
MNIST ($\varepsilon\\!=\\!0\%$) | 99.06 | 99.17 | 99.29
MNIST ($\varepsilon\\!=\\!40\%$) | 86.15 | 96.07 | 96.45
Fashion-MNIST ($\varepsilon\\!=\\!0\%$) | 91.07 | 90.80 | 90.79
Fashion-MNIST ($\varepsilon\\!=\\!40\%$) | 77.90 | 80.30 | 85.87

Table VIII shows the test ACC of LeNet5 trained with ELN, in which
$\varepsilon\\!=\\!0\%$ and $\varepsilon\\!=\\!40\%$ denote the ratios of
noisy labels in training set. For comparison, the test ACCs of LeNet5 trained
with cross entropy (CE) loss and a weighted CE loss (designed following [59])
are also provided. It can be seen from Table VIII that, when
$\varepsilon\\!=\\!0\%$, test ACCS of LeNet5 trained with three losses are
very close. However, if $40\%$ of training data are contaminated by label
noise, the test ACC of LeNet5 trained with CE loss may degenerate, seriously.
On the contrary, the networks trained with weighted CE loss and ELN can still
obtain relatively higher test ACC. It is worth noting that, although weighted
CE loss shows better robustness in comparison with CE loss, it tends to obtain
a smaller test ACC compared to ELN when $40\%$ of noisy labels are introduced.
The experimental results suggest that ELN is feasible and effective to train
deep neural networks when dealing with the classification problems of large
scale data sets.

##  V Conclusion

This paper proposed a novel loss function model called ELN, whose input is an
error sample and output is a loss function corresponding to that error sample.
It can be verified that ELN provides a unified framework for a large class of
error loss functions, including most of ITL loss functions as special cases.
Based on the fact that the activation function, weight parameters and network
size of ELN can be predetermined or learned from the error samples, we further
proposed a new machine learning paradigm where the learning process is divided
into two stages: first, learning a loss function using an ELN; second, using
the learned loss function to continue to perform the learning. Experimental
results on different regression and classification tasks showed that the
method based on ELN not only has better performance to deal with non-Gaussian
noises than the traditional MSE criterion, but also is very competitive in
comparison with some existing robust loss functions. In the future, more
productive technologies to train the ELN model deserve to be further explored.

##  Appendix A Calculation of K in (11)

Each element of K is formulated as

$\begin{array}[]{l}{{\bf{K}}_{ij}}\\!=\\!\int{{g_{{\sigma_{i}}}}({c_{i}})}\;{g_{{\sigma_{j}}}}({c_{j}})de\\\
\;\;\;\;\;\\!=\\!\int{\frac{1}{{\sqrt{2\pi}{\sigma_{i}}}}\exp(-\frac{{{{(\varepsilon-{c_{i}})}^{2}}}}{{2{\sigma_{i}}^{2}}})\frac{1}{{\sqrt{2\pi}{\sigma_{j}}}}\exp(-\frac{{{{(\varepsilon-{c_{j}})}^{2}}}}{{2{\sigma_{j}}^{2}}})\;d\varepsilon}\\\
\;\;\;\;\;\\!=\\!\frac{1}{{2\pi{\sigma_{i}}{\sigma_{j}}}}\int{\exp\left({-\frac{{{{(\varepsilon-{c_{i}})}^{2}}}}{{2{\sigma_{i}}^{2}}}-\frac{{{{(\varepsilon-{c_{j}})}^{2}}}}{{2{\sigma_{j}}^{2}}}}\right)\;d\varepsilon}\\\
\;\;\;\;\;\\!=\\!r\exp\left({\frac{{{b^{2}}}}{{8a{\sigma_{i}}^{2}{\sigma_{j}}^{2}}}}\right)\int{\exp\left({-\frac{{a{{\left({\varepsilon-\frac{b}{{2a}}}\right)}^{2}}}}{{2{\sigma_{i}}^{2}{\sigma_{j}}^{2}}}}\right)\;d\varepsilon},\end{array}$
(42)

where $r$, $a$, and $b$, are, respectively, calculated by

$\left\\{\begin{array}[]{l}r=\frac{1}{{2\pi{\sigma_{i}}{\sigma_{j}}}}\exp\left({-\frac{{({\sigma_{j}}^{2}{c_{i}}^{2}+{\sigma_{i}}^{2}{c_{j}}^{2})}}{{2{\sigma_{i}}^{2}{\sigma_{j}}^{2}}}}\right)\\\
a={\sigma_{j}}^{2}+{\sigma_{i}}^{2}\\\
b=2({c_{i}}{\sigma_{j}}^{2}+{c_{j}}{\sigma_{i}}^{2}),\\\ \end{array}\right.$
(43)

Let $x=\varepsilon-\frac{b}{{2a}}$ and
${\sigma^{2}}={\sigma_{i}}^{2}{\sigma_{j}}^{2}/a$. Then, (42) can be rewritten
as

$\begin{array}[]{l}\;{{\bf{K}}_{ij}}=r\exp\left({\frac{{{b^{2}}}}{{8a{\sigma_{i}}^{2}{\sigma_{j}}^{2}}}}\right)\int{\exp\left({-\frac{{{x^{2}}}}{{2({\sigma_{i}}^{2}{\sigma_{j}}^{2}/a)}}}\right)\;dx}\\\
\;\;\;\;\;\;=r\exp\left({\frac{{{b^{2}}}}{{8a{\sigma_{i}}^{2}{\sigma_{j}}^{2}}}}\right)\int{\exp\left({-\frac{{{x^{2}}}}{{2{\sigma^{2}}}}}\right)\;dx}\\\
\;\;\;\;\;\;=r\exp\left({\frac{{{b^{2}}}}{{8a{\sigma_{i}}^{2}{\sigma_{j}}^{2}}}}\right)\sqrt{2\pi}\sigma\\\
\;\;\;\;\;\;=\frac{1}{{\sqrt{2\pi({\sigma_{j}}^{2}+{\sigma_{i}}^{2})}}}\exp\left({\frac{{-\left({-2{c_{i}}{c_{j}}+{c_{i}}^{2}+{c_{j}}^{2}}\right)}}{{2({\sigma_{j}}^{2}+{\sigma_{i}}^{2})}}}\right)\\\
\;\;\;\;\;\;={G_{\sqrt{{\sigma_{j}}^{2}+{\sigma_{i}}^{2}}}}\left({{c_{i}}-{c_{j}}}\right).\end{array}$
(44)

## References

  * [1] E. J. Coyle and J. H. Lin, “Stack filters and the mean absolute error criterion,” _IEEE Trans. Acoust. Speech, Signal Process._ , vol. 36, no. 8, pp. 1244–1254, Aug. 1988.
  * [2] S. C. Pei and C. C. Tseng, “Least mean p-power error criterion for adaptive FIR filter,” _IEEE J. Sel. Areas Commun._ , vol. 12, no. 9, pp. 1540–1547, Dec. 1994.
  * [3] J. Yang, F. Ye, H.-J. Rong, and B. Chen, “Recursive least mean p-power extreme learning machine,” _Neural Netw._ , vol. 91, pp. 22–33, Mar. 2017.
  * [4] F. Wen, “Diffusion least-mean p-power algorithms for distributed estimation in alpha-stable noise environments,” _Electron. Lett._ , vol. 49, no. 21, pp. 1355–1356, Oct. 2013.
  * [5] C. Chen, C. Yan, N. Zhao, B. Guo, and G. Liu, “A robust algorithm of support vector regression with a trimmed Huber loss function in the primal,” _Soft Comput._ , vol. 21, no. 18, pp. 5235–5243, Jun. 2017.
  * [6] A. Esmaeili and F. Marvasti, “A novel approach to quantized matrix completion using Huber loss measure,” _IEEE Signal Process. Lett._ , vol. 26, no. 2, pp. 337–341, Feb. 2019.
  * [7] M. O. Sayin, N. D. Vanli, and S. S. Kozat, “A novel family of adaptive filtering algorithms based on the logarithmic cost,” _IEEE Trans. Signal Process._ , vol. 62, no. 17, pp. 4411–4424, Jun. 2014.
  * [8] T. A. Courtade and T. Weissman, “Multiterminal source coding under logarithmic loss,” _IEEE Trans. Inf. Theory_ , vol. 60, no. 1, pp. 740–761, Nov. 2014.
  * [9] C. Xu, D. Tao, and C. Xu, “Multi-view intact space learning,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 37, no. 12, pp. 2531–2544, 2015.
  * [10] J. C. Príncipe, _Information Theoretic Learning: Renyi’s Entropy and Kernel Perspectives_. New York, NY, USA: Springer, 2010.
  * [11] D. Erdogmus and J. C. Principe, “An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems,” _IEEE Trans. Signal Process._ , vol. 50, no. 7, pp. 1780–1786, Jul. 2002.
  * [12] B. Chen, Y. Zhu, and J. Hu, “Mean-square convergence analysis of ADALINE training with minimum error entropy criterion,” _IEEE Trans. Neural Netw._ , vol. 21, no. 7, pp. 1168–1179, Jun. 2010.
  * [13] W. Liu, P. P. Pokharel, and J. C. Príncipe, “Correntropy: properties and applications in non-Gaussian signal processing,” _IEEE Trans. Signal Process._ , vol. 55, no. 11, pp. 5286–5298, Nov. 2007.
  * [14] B. Chen, L. Xing, B. Xu, H. Zhao, and J. C. Pr¨ªncipe, “Insights into the robustness of minimum error entropy estimation,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 29, no. 3, pp. 731–737, Dec. 2018.
  * [15] B. Chen, L. Xing, H. Zhao, S. Du, and J. C. Príncipe, “Effects of outliers on the maximum correntropy estimation: A robustness analysis,” _IEEE Trans. Syst. Man Cybern. -Syst._ , vol. 51, no. 6, pp. 4007–4012, Aug. 2021.
  * [16] L. Bako, “Robustness analysis of a maximum correntropy framework for linear regression,” _Automatica_ , vol. 87, pp. 218–225, Jun. 2018.
  * [17] B. Chen, L. Xing, N. Zheng, and J. C. Príncipe, “Quantized minimum error entropy criterion,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 30, no. 5, pp. 1370–1380, Sep. 2019.
  * [18] B. Chen, L. Xing, H. Zhao, N. Zheng, and J. C. Príncipe, “Generalized correntropy for robust adaptive filtering,” _IEEE Trans. Signal Process._ , vol. 64, no. 13, pp. 3376–3387, Jul. 2016.
  * [19] B. Chen, X. Wang, N. Lu, S. Wang, J. Cao, and J. Qin, “Mixture correntropy for robust learning,” _Pattern Recognit._ , vol. 79, pp. 318–327, Jul. 2018\.
  * [20] Y. Wang, L. Yang, and Q. Ren, “A robust classification framework with mixture correntropy,” _Inf. Sci._ , vol. 491, pp. 306–318, Jul. 2019.
  * [21] B. Chen, L. Xing, B. Xu, H. Zhao, N. Zheng, and J. C. Principe, “Kernel risk-sensitive loss: Definition, properties and application to robust adaptive filtering,” _IEEE Trans. Signal Process._ , vol. 65, no. 11, pp. 2888–2901, Feb. 2017.
  * [22] B. Chen, L. Xing, X. Wang, J. Qin, and N. Zheng, “Robust learning with kernel mean $p$-power error loss,” _IEEE Trans. Cybern._ , vol. 48, no. 7, pp. 2101–2113, Jul. 2018.
  * [23] B. Chen, X. Wang, Y. Li, and J. C. Principe, “Maximum correntropy criterion with variable center,” _IEEE Signal Process. Lett._ , vol. 26, no. 8, pp. 1212–1216, Jun. 2019.
  * [24] B. Chen, Y. Xie, X. Wang, Z. Yuan, P. Ren, and J. Qin, “Multikernel correntropy for robust learning,” _IEEE Trans. Cybern._ , doi: 10.1109/TCYB.2021.3110732.
  * [25] Y. Feng, X. Huang, L. Shi, Y. Yang, and J. A. Suykens, “Learning with the maximum correntropy criterion induced losses for regression,” _J. Mach. Learn. Res._ , vol. 16, no. 30, pp. 993–1034, May 2015.
  * [26] S. Zhao, B. Chen, and J. C. Príncipe, “Kernel adaptive filtering with maximum correntropy criterion,” in _The 2011 International Joint Conference on Neural Networks_ , 2011, pp. 2012–2017.
  * [27] Y. Wang, W. Zheng, S. Sun, and L. Li, “Robust information filter based on maximum correntropy criterion,” _J. Guid. Control Dyn._ , vol. 39, no. 5, pp. 1126–1131, Jan. 2016.
  * [28] X. Liu, B. Chen, B. Xu, Z. Wu, and P. Honeine, “Maximum correntropy unscented filter,” _Int. J. Syst. Sci._ , vol. 48, no. 8, pp. 1607–1615, Jan. 2017\.
  * [29] Z. Wu, J. Shi, X. Zhang, W. Ma, and B. Chen, “Kernel recursive maximum correntropy,” _Signal Process._ , vol. 117, pp. 11–16, Dec. 2015.
  * [30] Z. Wu, S. Peng, B. Chen, and H. Zhao, “Robust Hammerstein adaptive filtering under maximum correntropy criterion,” _Entropy_ , vol. 17, no. 10, pp. 7149–7166, Oct. 2015.
  * [31] W. Ma, H. Qu, G. Gui, L. Xu, J. Zhao, and B. Chen, “Maximum correntropy criterion based sparse adaptive filtering algorithms for robust channel estimation under non-Gaussian environments,” _J. Franklin Inst._ , vol. 352, no. 7, pp. 2708 – 2727, Jul. 2015.
  * [32] B. Chen, X. Liu, H. Zhao, and J. C. Príncipe, “Maximum correntropy Kalman filter,” _Automatica_ , vol. 76, pp. 70 – 77, Feb. 2017\.
  * [33] R. He, B. Hu, W. Zheng, and X. Kong, “Robust principal component analysis based on maximum correntropy criterion,” _IEEE Trans. Image Process._ , vol. 20, no. 6, pp. 1485–1494, Jun. 2011.
  * [34] G. Xu, S. Du, and J. Xue, “Precise 2D point set registration using iterative closest algorithm and correntropy,” in _2016 International Joint Conference on Neural Networks (IJCNN)_ , 2016, pp. 4627–4631.
  * [35] B. Chen, Z. Yuan, N. Zheng, and J. C. Príncipe, “Kernel minimum error entropy algorithm,” _Neurocomputing_ , vol. 121, pp. 160 – 169, Dec. 2013\.
  * [36] B. Chen, R. Ma, S. Yu, S. Du, and J. Qin, “Granger causality analysis based on quantized minimum error entropy criterion,” _IEEE Signal Process. Lett._ , vol. 26, no. 2, pp. 347–351, Jun. 2019.
  * [37] I. Park and J. C. Principe, “Correntropy based Granger causality,” in _2008 IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2008, pp. 3605–3608.
  * [38] E. Parzen, “On estimation of a probability density function and mode,” _Ann. Math. Statist._ , vol. 33, no. 3, pp. 1065–1076, 09 1962.
  * [39] Y. Li, B. Chen, N. Yoshimura, and Y. Koike, “Restricted minimum error entropy criterion for robust classification,” _IEEE Trans. Neural Netw. Learn. Syst._ , doi: 10.1109/TNNLS.2021.3082571.
  * [40] Y. del Valle, G. K. Venayagamoorthy, S. Mohagheghi, J. C. Hernandez, and R. G. Harley, “Particle swarm optimization: Basic concepts, variants and applications in power systems,” _IEEE Trans. Evol. Comput._ , vol. 12, no. 2, pp. 171–195, Mar. 2008.
  * [41] S. Mirjalili and A. Lewis, “The whale optimization algorithm,” _Adv. Eng. Softw._ , vol. 95, pp. 51–67, May 2016.
  * [42] Y. Zheng, S. Wang, Y. Feng, W. Zhang, and Q. Yang, “Convex combination of quantized kernel least mean square algorithm,” in _2015 Sixth International Conference on Intelligent Control and Information Processing (ICICIP)_ , 2015, pp. 186–190.
  * [43] D. Broomhead and D. Lowe, “Multivariable functional interpolation and adaptive networks,” _Complex Systems_ , vol. 2, pp. 3191–3215, 1988.
  * [44] D. Lowe, “Adaptive radial basis function nonlinearities, and the problem of generalisation,” in _First IEE International Conference on Artificial Neural Networks_ , 1989, pp. 171–175.
  * [45] W. Deng, Y. S. Ong, and Q. Zheng, “A fast reduced kernel extreme learning machine,” _Neural. Netw._ , vol. 76, pp. 29–38, Apr. 2016.
  * [46] M. Singhal and S. Shukla, “Centroid selection in kernel extreme learning machine using k-means,” in _5th International Conference on Signal Processing and Integrated Networks (SPIN)_ , 2018, pp. 708–711.
  * [47] Z. Qin, B. Chen, Y. Gu, N. Zheng, and J. C. Principe, “Probability density rank-based quantization for convex universal learning machines,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 31, no. 8, pp. 3100–3113, Sep. 2020.
  * [48] Y. H. Pao and Y. Takefuji, “Functional-link net computing: theory, system architecture, and functionalities,” _Computer_ , vol. 25, no. 5, pp. 76–79, May. 1992.
  * [49] G. Huang, H. Zhou, X. Ding, and R. Zhang, “Extreme learning machine for regression and multiclass classification,” _IEEE Trans. Syst. Man Cybern. Part B_ , vol. 42, no. 2, pp. 513–529, Apr. 2012.
  * [50] C. L. P. Chen and Z. Liu, “Broad learning system: An effective and efficient incremental learning system without the need for deep architecture,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 29, no. 1, pp. 10–24, Jul. 2018.
  * [51] R. P. Agarwal, M. Meehan, and D. O’Regan, _Fixed Point Theory and Applications_. Cambridge, U.K.: Cambridge University Press, 2001.
  * [52] B. Chen, J. Wang, H. Zhao, N. Zheng, and J. C. Principe, “Convergence of a fixed-point algorithm under maximum correntropy criterion,” _IEEE Signal Process. Lett._ , vol. 22, no. 10, pp. 1723–1727, May 2015.
  * [53] A. R. Heravi and G. A. Hodtani, “A new robust fixed-point algorithm and its convergence analysis,” _J. Fixed Point Theory Appl._ , vol. 19, no. 4, pp. 3191–3215, month=, 2017.
  * [54] Z. Xi, J. Yang, Y. Zheng, H. Wu, and B. Chen, “Random fourier features based extended kernel recursive least squares with application to fMRI decoding,” in _2018 Chinese Automation Congress (CAC)_ , 2018, pp. 1390–1395.
  * [55] D. Dua and C. Graff, “UCI machine learning repository,” 2017. [Online]. Available: http://archive.ics.uci.edu/ml
  * [56] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” in _32nd Conference on Neural Information Processing Systems (NeurIPS 2018)_ , 2018, pp. 8535–8545.
  * [57] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, Nov. 1998.
  * [58] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1708.07747
  * [59] T. Liu and D. Tao, “Classification with noisy labels by importance reweighting,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 38, no. 3, pp. 447–461, Jul. 2016.