# Neural Network Ensembles: Theory, Training, and the Importance of Explicit
Diversity

Wenjing Li<EMAIL_ADDRESS>
Department of Mathematical Sciences
Worcester Polytechnic Institute
Worcester, MA 01609, USA Randy C. Paffenroth<EMAIL_ADDRESS>
Department of Mathematical Sciences, Computer Science and Data Science
Worcester Polytechnic Institute
Worcester, MA 01609, USA David Berthiaume<EMAIL_ADDRESS>
Department of Mathematical Sciences
Worcester Polytechnic Institute
Worcester, MA 01609, USA

###### Abstract

Ensemble learning is a process by which multiple base learners are
strategically generated and combined into one composite learner. There are two
features that are essential to an ensemble’s performance, the individual
accuracies of the component learners and the overall diversity in the
ensemble. The right balance of learner accuracy and ensemble diversity can
improve the performance of machine learning tasks on benchmark and real-world
data sets, and recent theoretical and practical work has demonstrated the
subtle trade-off between accuracy and diversity in an ensemble. In this paper,
we extend the extant literature by providing a deeper theoretical
understanding for assessing and improving the optimality of any given
ensemble, including random forests and deep neural network ensembles. We also
propose a training algorithm for neural network ensembles and demonstrate that
our approach provides improved performance when compared to both state-of-the-
art individual learners and ensembles of state-of-the-art learners trained
using standard loss functions. Our key insight is that it is better to
explicitly encourage diversity in an ensemble, rather than merely allowing
diversity to occur by happenstance, and that rigorous theoretical bounds on
the trade-off between diversity and learner accuracy allow one to know when an
optimal arrangement has been achieved.

Keywords: Ensemble Learning, Deep Learning, Neural Networks, Random Forests,
Ensemble Theory

##  1 Introduction

Ensemble learning is a process by which multiple base learners are generated
and combined into one composite learner (Dietterich, 2000). Bagging (Nowlan
and Hinton, 1991), boosting (Freund and Schapire, 1997) (including well-known
Gradient Boosting (Friedman, 2002) and Adaboost (Freund and Schapire, 1997)),
and stacking (Kuncheva, 2004) are classic examples of ensemble algorithms.
Such techniques have been applied to a variety of machine learning domains
including text mining (Williams and Gong, 2014), recommender systems
(Amatriain et al., 2011), and many others. One popular and frequently used
class of ensemble algorithms are random forests (Breiman, 2001) which combine
decision trees, as base classifiers, with bagging, to provide diversity among
the base classifiers. More modern examples of ensembles include ensemble
adversarial training (Tramèr et al., 2017) (an ensemble technique proposed to
defend adversarial attacks), and ensemble deep auto-encoders (Shao et al.,
2018) (an ensemble method for fault diagnosis of rolling bearings).

A neural network ensemble (Li et al., 2018) is an ensemble that combines the
individual outputs of several neural networks, and has numerous machine
learning applications including image classification (Giacinto and Roli,
2001), face recognition (Huang et al., 2000), and intrusion detection (Amini
et al., 2014), to name but a few. Recent work by Choi and Lee (2018) combines
Long Short-Term Memory (LSTM) network ensembles with adaptive weighting to
improve time series forecasting, and Nguyen and Pernkopf (2018) apply nearest
neighbor filters to convolutional neural network ensembles to perform acoustic
scene classifications.

###  1.1 Related Work

Over the years researchers have been investigating ways to build good
ensembles that outperform their component learners (Liu and Yao, 1999;
Dietterich et al., 2002; Zhang and Ma, 2012; Sagi and Rokach, 2018). Two
features of ensembles which have been shown to be key contributing factors to
their performance are the individual accuracies of component learners and the
diversity of the learners in the ensemble (Kuncheva, 2004). In particular,
there is no single widely accepted metric for measuring the diversity in a
given ensemble, but intuitively one can think of a diverse ensemble as one
where the failures of one classifier can be compensated for by the successes
of the others (Sharkey and Sharkey, 1997), that is, to have a diverse
ensemble, the component learners need to avoid making coincident errors.

There also has been a vast amount of research and development on creating
diversity in ensembles (Kuncheva and Whitaker, 2003). In particular, Melville
and Mooney (2005) proposed a DECORATE algorithm that creates diversity in
ensembles by adding randomly generated artificial training examples to the
original training data (the details of this algorithm will be introduced
later). We implemented their algorithm DECORATE on several non-image data
sets, and the results are saved for making future comparisons with our method.

The theorems and numerical results in this paper are inspired by the recent
paper 111From December 2019 (Li and Paffenroth, 2019) where the authors of
that paper demonstrated that there is a subtle trade-off between the
individual accuracies and diversity of an ensemble, and that studying this
trade-off can lead to both theoretical and practical improvements to machine
learning algorithms. In this paper we extend both the theory and results in Li
and Paffenroth (2019) and derive a set of algorithms, armed with rigorous
theoretical bounds, that improve on the performance of state-of-the-art deep
neural networks for both binary and multi-label classification problems.

In many ways, the key results of this paper are quite easy to state. We merely
observe that many approaches to constructing machine learning ensembles only
achieve diversity in their base learners by happenstance. For example, classic
random forests (Breiman, 2001) achieve learner diversity merely by randomly
performing bootstrap sampling of their data and randomly choosing subsets of
their predictors for their constituent learners. Gradient boosting (Friedman,
2002), and similar methods (Freund and Schapire, 1997), improve matters by
taking the errors of the current set of learners into account when
constructing a new learner. _Our proposed algorithm goes even further by
explicitly encouraging diversity between the set of learners in the ensemble
as part of the training process_ , and we use this approach to train ensembles
of neural networks where the individual learners already have state-of-the-art
performance. These ensembles then have classification error rates that are
_2-6 times lower_ than the error rates of the state-of-the-art learners from
which they are constructed.

###  1.2 Our Contribution

Herein we develop a new and deeper understanding of the balance between
individual accuracies and ensemble diversity, building on Li and Paffenroth
(2019), by providing a new proof of one important theorem based upon the
Cauchy-Schwarz inequality (Steele, 2004). This new proof provides valuable
insights into connections between our work, coding theory (Irvine and Harle,
2001), and Welch bounds (Datta et al., 2012). In addition, we demonstrate how
this theoretical work provides methods to assess and improve the optimality of
any given classification ensemble (including random forests and ensembles of
deep neural networks) by relating the simple majority vote accuracy of
ensembles to the learner-learner correlations (which measure ensemble
diversity) and truth-learner correlations (which measure individual learner
accuracies). Furthermore, we propose an algorithm to train ensembles of deep
neural networks and demonstrate the effectiveness of our approach with a
variety of experiments on standard benchmark data sets. In particular, using
our methods one is able to start from state-of-the-art neural networks and
produce ensembles that improve their accuracy beyond their current
capabilities. Moreover, our training algorithm is quite efficient in that we
only need to train the pretrained base neural networks, as part of the
ensemble, by a small number of additional epochs to achieve good performance.

In summary, in this paper, we make the following novel contributions:

  * •

We provide a more concise and illuminating proof of a theorem from the
literature, and extend its applicability from binary classification to multi-
label classification.

  * •

We develop a new theorem that provides a closed-form formula of calculating
the simple majority vote accuracy of ensembles based on the statistical
correlations between pairs of learners and correlations between the learners
and the ground truth.

  * •

This new theoretical work offers a clear understanding of how individual
accuracies of component learners and diversity in an ensemble affect the
performance of an ensemble, and thus inspires a metric to assess the
optimality of given ensembles.

  * •

We propose a training algorithm for deep neural network ensembles that
explicitly encourages ensemble diversity by the way of maximizing our proposed
metric.

  * •

Our algorithm is demonstrated to be more effective than related algorithms by
Melville and Mooney (2005) in creating diversity in neural network ensembles
and maintaining the overall accuracy in the meantime.

  * •

Our algorithm is shown to be general and efficient in improving state-of-the-
art neural networks by experiments on both non-image and image classification
problems with binary and multi-label data.

##  2 Accuracy-Diversity Trade-off in Ensembles from the Perspective of
Statistical Correlations

There is a large extant literature (Brown et al., 2005; Brown, 2004; Brown and
Kuncheva, 2010; Dai et al., 2017; Asafuddoula et al., 2017; Hsu, 2017) that
has studied the relationship between the individual accuracies of component
learners and diversity in ensembles. One recent work by Li and Paffenroth
(2019) showed the balance between these two features from the perspective of
Pearson statistical correlations (Benesty et al., 2009). In particular, the
term “averaged learner-learner correlations” denoted by $r_{LL}^{(ave)}$ was
defined as follows as a measure for the overall diversity in an ensemble (Li
and Paffenroth, 2019):

$r_{LL}^{(ave)}=\frac{1}{N(N-1)/2}\sum_{i=1}^{N}\sum_{j>i}^{N}r_{L_{i},L_{j}},$

where $N$ is the ensemble size (i.e. the number of learners in the ensemble),
and $r_{L_{i},L_{j}}$ is the pairwise Pearson correlation coefficient between
the predicted labels of learners $L_{i}$ and $L_{j}$. Similarly the term
“averaged truth-learner correlations” noted by $r_{TL}^{(ave)}$ was defined as
follows as a measure for the overall accuracy of learners in an ensemble (Li
and Paffenroth, 2019):

$r_{TL}^{(ave)}=\frac{1}{N}\sum_{i=1}^{N}r_{T,L_{i}},$

where $N$ is again the ensemble size, and $r_{T,L_{i}}$ is the Pearson
correlation coefficient between the labels of ground truth $T$ and the
predicted labels of learner $L_{i}$.

To motivate our approach we show in Figure 1 a schematic of the correlation
matrix between the ground truth $T$ and three learners $L_{1}$, $L_{2}$ and
$L_{3}$. In this schematic one can see that $r_{TL}^{(ave)}$ and
$r_{LL}^{(ave)}$ are calculated by taking the average of the blue and orange
elements, respectively.

Figure 1: Correlation matrix for the ground truth $T$ and three learners
$L_{1}$, $L_{2}$ and $L_{3}$. The average of the blue elements is the averaged
truth-learner correlations $r_{TL}^{(ave)}$, and the average of the orange
elements is the averaged learner-learner correlations $r_{LL}^{(ave)}$.

As discussed earlier, the component learners in a diverse ensemble should not
be too similar. Therefore small, or even negative, averaged learner-learner
correlations are preferred. Hence a lower value of $r_{LL}^{(ave)}$ indicates
a higher level of diversity in the ensemble. Similariy, since $r_{T,L_{i}}$
measures the correlation between a learner and the truth (which will be
discussed in detail in Section 3), the higher the value of $r_{TL}^{(ave)}$
is, the higher the overall accuracy of the learners in the ensemble is.

Accordingly, one may be tempted to consider the left correlation matrix in
Figure 2 as optimal for the three-learner cases, as its $r_{LL}^{(ave)}$ has
reached the lowest possible value $-1$ and its $r_{TL}^{(ave)}$ has reached
the highest possible value $1$. However, _no such correlation matrix can
exist_ , as it is not non-negative definite and thus is not a legitimate
correlation matrix. However, the right correlation matrix in Figure 2 is valid
with $r_{TL}^{(ave)}=0.3$ and $r_{LL}^{(ave)}=-0.2$ since all of its
eigenvalues are greater than or equal to $0$.

Figure 2: Two examples of a correlation matrix for the ground truth $T$ and
three learners $L_{1},L_{2}$ and $L_{3}$. The left one is not a valid
correlation matrix, as it is not non-negative definite, while the right one is
valid. These two examples show the fact that not all values between $-1$ and
$1$ are possible for the averaged truth-learner correlations and the averaged
learner-learner correlations.

Generalizing these ideas by studying the eigenstructure of correlation
matrices leads to the two theorems below, originally proven in Li and
Paffenroth (2019), which provide sharp bounds on what values of
$r_{TL}^{(ave)}$ and $r_{LL}^{(ave)}$ are possible for ensembles of different
sizes.

###### Theorem 1

(Originally from Li and Paffenroth (2019) where it appears as Theorem 1) For
an ensemble with $N$ learners we have that

$-\frac{1}{N-1}\leqslant r_{LL}^{(ave)}\leqslant 1.$ (1)

###### Theorem 2

(Originally from Li and Paffenroth (2019) where it appears as Theorem 2) For
an ensemble with $N$ learners we have that

$-\sqrt{\frac{(N-1)\cdot r_{LL}^{(ave)}+1}{N}}\leqslant
r_{TL}^{(ave)}\leqslant\sqrt{\frac{(N-1)\cdot r_{LL}^{(ave)}+1}{N}}.$ (2)

Theorem 1 shows that there is a limitation to the extent of possible negative
learner-learner correlations in an ensemble. In particular, as the ensemble
size $N$ increases, the lower bound $-\frac{1}{N-1}$ in $\eqref{theorem1}$
approaches $0$ from the negative side, meaning that larger ensembles cannot
have large negative values of $r_{LL}^{(ave)}$ as compared to smaller
ensembles. Along these same lines, Theorem 2 provides the tight bounds for
$r_{TL}^{(ave)}$ as functions of $r_{LL}^{(ave)}$.

The original proof in Li and Paffenroth (2019) to Theorem 2 followed the idea
of Sylvester’s criterion (Gilbert, 1991). Herein we propose a new and shorter
proof (in Appendix A.) by applying the Cauchy-Schwarz inequality. This new,
more geometric proof, provides connections between these results and classic
ideas in coding theory (Irvine and Harle, 2001) such as Welch bounds (Datta et
al., 2012) and equiangular tight frames (Sustik et al., 2007).

Perhaps more importantly for the work discussed here, one can combine the two
theorems to visualize the relationship between $r_{LL}^{(ave)}$ and
$r_{TL}^{(ave)}$. Figure 3 shows an example of this relationship for the cases
of an ensemble of size $5$, and ensembles of different sizes will have the
similar pattern. In particular, any ensemble will fall within the region
bounded by the theoretical upper bound curve (colored in red) and the lower
bound curve (colored in blue). An ensemble that is expected to show acceptable
performance should at least fall above the black dotted line, where the
averaged truth-learner correlations is $0$ (meaning that the component
learners are making random guesses on the labels of data instances), as the
Condorcet’s jury theorem (Boland, 1989) states that the individual
probabilities being correct must be greater than $1/2$ to ensure the success
of the group decision under the rule of simple majority vote.

The red curve in Figure 3 can be thought of an optimal boundary, similar to a
Pareto boundary (Sen, 1993). In particular, any ensemble that is strictly
below the red line can have either its individual accuracy or diversity
improved at no cost to the other quantity. For example, the two points on
Figure 3 are examples of ensembles that are close to optimal, where the
component learners in the ensemble at point A are very accurate but not
diverse, while the learner in the ensemble at point B are very diverse but not
that accurate.

Figure 3: A theoretical plot (averaged truth-learner correlations versus
averaged learner-learner correlations) for ensembles of size 5. Ensembles of
different sizes have the similar pattern. Any ensemble will fall within the
region bounded by the red and blue curve, and an acceptable ensemble should
fall at least above the black dotted line. Ensembles that fall near and on the
red curve are optimal, and points A and B are two examples of ensembles that
are close to optimal.

##  3 Theory for Assessing and Improving the Performance of Ensembles

In Section 2, we discussed the balance between individual accuracies and
diversity of learners in ensembles. In this section, we will demonstrate how
exactly the two features affect the performance of ensembles for
classification problems. In particular, the simple majority vote accuracy of
ensembles will be used as the metric for assessing their performance, and we
aim to develop a closed-form formula of calculating simple majority vote
accuracy based on the averaged truth-learner correlations $r_{TL}^{(ave)}$ and
the averaged learner-learner correlations $r_{LL}^{(ave)}$.

For now, we only derive the theory for simple majority vote accuracy for
homogenous ensembles (which will be defined in Section 3.2). However, the
lessons learned from the theory for homogeneous ensembles inspire us on
developing approaches that are still effective for general ensembles, though
without the same theoretical guarantees. In Section 4, we will demonstrate how
such ideas lead to our training algorithm for general neural network
ensembles.

###  3.1 Relationship between the Accuracy of a Learner and its Correlation
with the Truth

Before investigating the theory for assessing the performance of ensembles, it
is worthwhile to derive the mathematical formulation of the individual
accuracy of a learner in terms of its correlation with the truth, as such
formulation is critical to the formulation of simple majority vote accuracy of
ensembles (which we will detail in Section 3.2).

For simplicity, consider a data set with two classes (the labels are indicated
by $0$’s and $1$’s). The mathematical relationship for binary classification
problems between the accuracy of a learner $L$ and its correlation with the
true class labels $T$ can be characterized by the following theorem (Proof in
Appendix B.). Note that extending the theorem to multi-label classification is
one important direction for future work.

###### Theorem 3

$r_{T,L}=\frac{\beta p-\alpha(2\beta
p-p+1-\alpha)}{\sqrt{\alpha(1-\alpha)(2\beta p-p+1-\alpha)(-2\beta
p+p+\alpha))}},$ (3)

where

  * •

the learner $L$ and the truth $T$ are both vectors of $0$’s and $1$’s,

  * •

$r_{T,L}$ is the Pearson correlation coefficient between $L$ and $T$,

  * •

$p$ is the accuracy of learner $L$ (i.e. the proportion of data points
correctly classified by learner $L$),

  * •

$\alpha$ is the proportion of $1$’s in the truth class labels $T$ (note that
$0<\alpha<1$ 222Note that $\alpha$ cannot be 0 or 1, as correlation is not
defined for constants (when true class labels are all $0$’s or all $1$’s).,
and for balanced data sets, $\alpha=\frac{1}{2}$),

  * •

$\beta$ is the ratio of accuracy in class $1$ to accuracy in both classes
(i.e. $\beta=P(T=1\cap L=1)/(P(T=1\cap L=1)+P(T=0\cap L=0))=P(T=1\cap L=1)/p$,
and $P(.)$ represents probability).

Proof: In the Appendix.

The relationship between $p$ and $r_{T,L}$ appears to be complicated in (3).
However one can solve $p$ from (3) by writing out the Taylor series expansion
(Dienes, 1957) of $p$ at $\beta=\frac{1}{2}$ as shown in (4):

$\displaystyle p=$ $\displaystyle
2\alpha(1-\alpha)(1+r_{T,L})-4(\alpha-2\alpha^{2}+2\alpha^{3}$
$\displaystyle+2\alpha r_{T,L}-6\alpha^{2}r_{T,L}+4\alpha^{3}r_{T,L}+\alpha
r_{T,L}^{2}-3\alpha^{2}r_{T,L}^{2}$
$\displaystyle+2\alpha^{3}r_{T,L}^{2})(\beta-\frac{1}{2})+O((\beta-\frac{1}{2})^{2}),$
(4)

where $O((\beta-\frac{1}{2})^{2})$ represents all the higher-order (including
the second-order) terms of $(\beta-\frac{1}{2})$. (4) can be further
simplified as

$\displaystyle p$
$\displaystyle=2\alpha(1-\alpha)(1+r_{T,L})\indent(\text{Under the assumption
of }\beta=1/2)$ (5)
$\displaystyle=2\alpha(1-\alpha)+2\alpha(1-\alpha)r_{T,L}.$

Based on (5), we can see that as long as the learner $L$ performs equally well
on both classes (i.e., $\beta=1/2$) then the accuracy of the learner, $p$, is
a linear function of its correlation with the truth $r_{T,L}$. Notice that the
intercept and slope in (5) share the same function, $2\alpha(1-\alpha)$. Since
$0<\alpha<1$, the slope has $2\alpha(1-\alpha)>0$. Therefore, for any learner
that is expected to perform equally well on both classes, the individual
accuracy $p$ is positively linearly correlated with its correlation with the
truth. Hence, the higher the value of correlation $r_{T,L}$ is, the more
accurate the learner is.

For a balanced binary data set (i.e. $\alpha=1/2$), (5) can be further reduced
to

$p=\frac{1}{2}+\frac{1}{2}\cdot r_{T,L}.\indent(\text{Under assumptions of
}\beta=1/2,\alpha=1/2)$ (6)

Figure 4 visualizes the relationship between accuracy of a learner (the
y-axis) and its correlation with the truth (the x-axis) for $2,000$ randomly
generated pairs of true class labels $T$ and learner $L$, when no assumptions
about $\alpha$ and $\beta$ are made. For each of the $2,000$ pairs, the
correlation $r_{T,L}$ and accuracy $p$ are calculated, and marked as a blue
dot in Figure 4. All the blue dots show a roughly linear trend (colored in
red), where the accuracy is increasing when the correlation increases. There
exists some fluctuation of the blue dots around the linear trend, and such
fluctuation is due to the higher order terms brought by parameters $\alpha$
and $\beta$ to (3).

Figure 4: Accuracy of a Learner versus its Correlation with the Truth. Each of
the 2000 blue points is generated by a pair of randomly generated binary
classifier and true class labels of size 100 with two classes. All the blue
points show a roughly linear trend as indicated by the red straight line.

In conclusion, the fact that a learner’s accuracy is linearly positively
correlated with its correlation with the truth is rigorously true if the
learner is assumed to perform equally well on predicting both class labels,
and is empirically observed even when the assumption is not met. Hence it is
reasonable for us to take the truth-learner correlation $r_{T,L}$ as a measure
for a learner’s individual accuracy. Extending this idea to ensembles, it is
also reasonable to take the averaged truth-learner correlations
$r_{TL}^{(ave)}$ as a measure for the overall accuracy level of learners in an
ensemble.

###  3.2 Simple Majority Vote Accuracy of Homogenous Ensembles

In this section, we will derive the closed-form formula of calculating simple
majority vote accuracy of ensembles. In particular, we focus on the concept of
“homogenous ensemble” which is an extension to ensemble learning of the jury
design mentioned in Kaniovski and Zaigraev (2011), where the authors defined
“homogeneous jury” as a jury in which each vote has an equal probability of
being correct, and each pair of votes correlates with the same correlation
coefficient. Similarly, we define the “homogenous ensemble” as an ensemble
whose component learners correlate with the true class labels with the same
correlation coefficient, and all pairwise learner-learner correlations are
equal.

Consider a homogenous jury under the simple majority vote decision rule, a
closed-form formula for calculating the probability that the jury being
correct (denoted by $M_{n}(p,c)$) is derived in Kaniovski and Zaigraev (2011)
as follows

$M_{n}(p,c)=\sum_{i=\frac{n+1}{2}}^{n}C_{n}^{i}p^{i}(1-p)^{n-i}+0.5c(n-1)(0.5-p)\frac{p^{\frac{n+1}{2}-1}(1-p)^{\frac{n+1}{2}-1}}{B(\frac{n+1}{2},\,\frac{n+1}{2})},$
(7)

where $n$ is the number of jurors in the jury, $p$ is the probability of each
juror being correct, $c$ is the correlation coefficient of each pair of votes,
and $B(.\ ,.)$ is Euler’s Beta function (Chaudhry et al., 1997).

Thinking each juror in the homogenous jury as a learner in a homogenous
ensemble, inspired by $\eqref{eq-reference}$, and replacing $p$ in $\eqref{eq-
reference}$ with $2\alpha(1-\alpha)(1+r_{T,L})$ as derived in
$\eqref{linear}$, we are able to derive the formula for calculating the simple
majority vote accuracy of ensembles (denoted by $M_{N}(r_{T,L},r_{L,L})$) as
follows

$\displaystyle\hskip 21.68121ptM_{N}(r_{T,L},r_{L,L})$ (8)
$\displaystyle=\sum_{i=\frac{N+1}{2}}^{N}C_{N}^{i}(2\alpha(1-\alpha)(1+r_{T,L}))^{i}(1-2\alpha(1-\alpha)(1+r_{T,L}))^{N-i}+0.5\cdot
r_{L,L}(N-1)$
$\displaystyle\cdot(0.5-2\alpha(1-\alpha)(1+r_{T,L}))\cdot\frac{(2\alpha(1-\alpha)(1+r_{T,L}))^{\frac{N-1}{2}}\cdot(1-2\alpha(1-\alpha)(1+r_{T,L}))^{\frac{N-1}{2}}}{B(\frac{N+1}{2},\,\frac{N+1}{2})},$
$\displaystyle(\text{Under the assumption of }\beta=1/2)$

where $N$ is the ensemble size, $r_{T,L}$ is the truth-learner correlation,
$r_{L,L}$ is the Pearson correlation coefficient between each pair of
learners, and $\alpha$ is the proportion of $1$’s in the truth class labels
$T$ of $0$’s and $1$’s.

In a given homogenous ensemble with binary classifiers that are expected to
perform equally well on predicting both classes (i.e. $\beta=1/2$), $\alpha$
can be easily calculated based on the given data set. In particular, $r_{T,L}$
and $r_{L,L}$ in $\eqref{eq-majority-vote}$ are computable from outputs of the
base learners, hence one is able to estimate the simple majority vote accuracy
of the ensemble through the formula given by $\eqref{eq-majority-vote}$. Even
when the assumption $\beta=1/2$ is not met, $\eqref{eq-majority-vote}$ still
offers a reference for evaluating the simple majority vote accuracy of
homogenous ensembles.

For balanced binary data sets (i.e. $\alpha=1/2$), $\eqref{eq-majority-vote}$
can be further simplified as

$\displaystyle M_{N}(r_{T,L},r_{L,L})$
$\displaystyle=\sum_{i=\frac{N+1}{2}}^{N}C_{N}^{i}(0.5(1+r_{T,L}))^{i}(1-0.5(1+r_{T,L}))^{N-i}$
(9)
$\displaystyle-0.25\cdot\frac{r_{T,L}r_{L,L}(N-1)}{B(\frac{N+1}{2},\,\frac{N+1}{2})}(0.5(1+r_{T,L}))^{\frac{N-1}{2}}\cdot(1-0.5(1+r_{T,L}))^{\frac{N-1}{2}}.$
$\displaystyle(\text{Under the assumptions of }\alpha=1/2,\ \beta=1/2)$

For fixed values of $r_{L,L}$ in a homogenous ensemble of size $N$,
$\eqref{eq-majority-vote-simplified}$ is a function of the single variable
$r_{T,L}$, therefore we can visualize the simple majority vote accuracy of the
ensembles when $r_{T,L}$ is varying in $[0,1]$. Figure 5 is one such example
where ensemble size is 5, and the learner-learner correlation $r_{L,L}$ is set
at six different levels (as shown in the legend box). Homogenous ensembles of
different sizes have the similar pattern.

We can see that from Figure 5, for fixed level of truth-learner correlation
$r_{T,L}$, homogenous ensembles with the lowest possible value of $r_{L,L}$
(indicated by the blue curve) show the highest simple majority accuracy.
Moreover, the lower the value of $r_{L,L}$ (that is, the more diverse the
ensembles), the higher the majority vote accuracy, which emphasizes the point
that one should make the homogenous ensemble more diverse in order to achieve
better simple majority vote accuracy, especially in cases that there is not
much improvement that might be possible for the individual accuracies of the
component learners. On the other hand, for a fixed level of $r_{L,L}$, the
value of simple majority vote accuracy increases as the truth-learner
correlation increases, meaning that in order to improve the simple majority
vote accuracy of homogenous ensembles, one should try to make the component
learners more accurate, in cases where the ensemble cannot be made more
diverse.

Figure 5: Simple Majority Vote Accuracy for Homogenous Ensembles of Size 5.
Homogenous ensembles of different sizes have the similar pattern. The learner-
learner correlation is fixed at six different levels as shown in the legend
box, while the truth-learner correlation is varying in $[0,1]$.

In summary, one can assess the performance of any given ensemble by evaluating
its simple majority vote accuracy. Although the formula $\eqref{eq-majority-
vote}$ for calculating simple majority vote accuracy is only valid for
homogenous ensembles with certain assumptions, the idea is still useful when
we develop approaches for general ensembles in Section 4. More importantly,
inspired by the insights obtained from Figure 5, one may improve the
performance of any given ensemble by either making the component learners more
accurate, or making them more diverse.

##  4 Creating Diversity in Ensembles

In this section, we will apply the theoretical analysis on ensembles from our
earlier sections to practical classification problems. In particular, Theorem
1 and Theorem 2 are general in that they apply to any ensembles whatsoever,
and Theorem 3 suggests that $r_{TL}^{(ave)}$ and ensemble accuracy are highly
dependent. Accordingly, in this section we study the importance of and ways to
create diversity in random forests and deep neural network ensembles from the
perspective of statistical correlations ($r_{TL}^{(ave)}$ and
$r_{LL}^{(ave)}$).

Melville and Mooney (2005) proposed a DECORATE algorithm that creates
diversity in ensembles by adding randomly generated artificial training
examples to original training data, however, their algorithm is
computationally expensive especially for large data sets. We thereafter
propose a novel training algorithm following the idea of our theorems
introduced earlier, and demonstrate the effectiveness and advantages of our
approach in the case of deep neural network ensembles by experimental
evaluations on both non-image and image classification problems.

###  4.1 Creating Diversity in Random Forests

Based on the analysis in earlier sections, we see that diversity plays an
important role in the success of ensemble methods. This leads naturally to the
question about how to introduce more diversity into ensembles. Herein, we
start the analysis and experiments with random forests, as they represent a
well-known ensemble method that uses decision trees as component classifiers
and is easy to train.

One possible way to introduce diversity into random forests is varying the
training parameters of the component decisions trees. Table 1 displays the
three types of random forests we generated for the purpose of comparisons. In
particular, an original random forest with $5$ trees is trained with default
parameters, and is used as our baseline model. The second type is “feature
random forest”, where each of the 5 component decision trees is trained with a
random subset of features. And the third type is “depth random forest”, where
the maximum depth of each tree is specified as a given value. One can imagine
generating more types of random forests by varying the other training
parameters.

Table 1: Three Types of Random Forests Original RF |  Original random forest model with $5$ trees using bootstrap sampling of the data for diversity. This is our baseline random forest model for comparison.
---|---
Feature RF |  Random forest with $5$ trees, but each tree is trained on a random subset of $m$ features. For the $5$ “feature random forests” displayed in Figure 6, $m=1,3,5,7,20$, respectively.
Depth RF |  Random forest with $5$ trees with a maximum depth of $d$. For the $5$ “depth random forests” displayed in Figure 6, $d=3,5,7,9,11$, respectively.

We apply the three types of random forests on the train set of the widely
studied CIFAR-10 image data (Krizhevsky and Hinton, 2009) (note: the training
set consists of $50,000$ $32\times 32$ color images in $10$ classes including
“car”, “plane”, “ship”, etc., with 5,000 images per class) to classify the
images, and visualize their calculated metrics ($r_{TL}^{(ave)}$,
$r_{LL}^{(ave)}$ and the simple majority vote accuracy) on Figure 6, where $1$
“original rf”, $5$ “feature rf” and $5$ “depth rf” are shown. The color bar is
the simple majority vote accuracy, with the more yellow the points, the higher
the accuracy of the forests.

Figure 6: Three Types of Random Forests Applied on CIFAR-10 train set (check
Table 1 for details). There is one original random forest (square shaped),
five “feature rf” (triangle shaped), and five “depth rf” (round shaped). The
color bar is the simple majority vote accuracy, with the more yellow the
points, the higher the accuracy of the forests.

We can see from Figure 6, the “original rf” has the highest accuracy among all
the random forests models, as it is the one closest to the upper bound curve.
For “feature rf” and “depth rf”, the closer they are to the upper bound curve,
the higher their simple majority accuracy. “feature rf” and “depth rf” are
created for the purpose of introducing diversity and outperforming the
‘’‘original rf”, but as we can see, the diversity provided by bootstrapping
the data is of higher quality than the other types of diversity.

Based on the analysis on random forests, we can see that for ensembles
consisting of traditional classification models, even though we can tune their
training parameters, it is not guaranteed that such tuning will necessarily
improve the performance of the ensembles. However, the theorems we provide
here, and our measure of diversity, illuminates the underlying causes of the
superior performance of some of the ensembles.

###  4.2 Creating diversity in ensembles using artificial data

To introduce diversity intentionally to ensembles, Melville and Mooney (2005)
proposed an algorithm DECORATE (Diverse Ensemble Creation by Oppositional
Relabeling of Artificial Training Examples), that creates diverse ensembles by
adding randomly generated artificial training examples to the original
training data. The ensemble is initialized with one existing “strong”
classifier, and then at each iteration a new classifier is trained on the
combination of the original training set and a set of artificially generated
data (the size of the artificial data is $r$ times the original training size,
where $r\in[0,1]$) (Melville and Mooney, 2005). For numeric attributes, the
new artificial data is randomly generated from a Gaussian distribution of
which the mean and standard deviation are computed from the original training
set; for nominal attributes, the distribution of the artificial data is
obtained by applying Laplace smoothing on the probabilities of occurrences of
all the distinct values (Melville and Mooney, 2005). The labels of the
artificial training examples are then determined in a way such that the
probability of the selection is inversely proportional to the predictions of
the current ensemble (Melville and Mooney, 2005). To maintain training
accuracy, at each iteration a new classifier will be rejected if adding it to
the current ensemble will decrease its accuracy, and iterations can be
repeated until an ensemble of desired size or the pre-fixed maximum number of
iterations has been reached (Melville and Mooney, 2005).

The idea of the DECORATE algorithm is straightforward while attempting to
increase diversity in ensembles, by forcing the new classifiers to differ from
the current ensemble. However, a few points with the algorithm may raise
doubts on the effectiveness of this method. First of all, their assumptions on
the distribution (Gaussian for numeric attributes) of new generated artificial
data need further careful discussions. Secondly, at each iteration, a new
artificial training set needs to be generated and the probabilities need to be
inverted, which is time-consuming and computationally expensive, especially
when the original training size is large. Last but not least, whether its
success on training data can be generalized to testing data is questionable,
as the ensemble diversity is created based on the combination of the
artificial and original training data, not purely on the original training
data.

In Section 4.3, we develop a different way of creating diversity in neural
network ensembles, which shows better performance than the DECORATE algorithm.

###  4.3 Creating diversity in Neural Network Ensembles

In this section, we will continue working on relating our theoretical results
about accuracy and diversity to real-world ensembles. In particular, we
propose a training algorithm for deep neural network ensembles, apply the
algorithm to a variety of standard benchmark data sets, and demonstrate the
effectiveness of the algorithm on both non-image and image classification
problems.

####  4.3.1 Methodology

One advantage of neural network ensembles over other traditional ensembles
(e.g. random forests) is that we can explicitly control the trade-off between
base learner accuracy and diversity in the loss function we use to train the
networks through the process of backpropagation (Hecht-Nielsen, 1992). The
loss function we proposed, inspired by Li and Paffenroth (2019), is one that
takes both the accuracy and diversity of neural network ensembles into
consideration.

$Loss=-(r_{TL}^{(ave)}-\lambda\cdot r_{LL}^{(ave)}),$ (10)

where $r_{TL}^{(ave)}$ and $r_{LL}^{(ave)}$ is taken as a measure for overall
accuracy and diversity of ensembles, respectively.

Given the above loss function, we propose a training algorithm for neural
network ensembles that works for both binary and multi-label classifications.
The details of our proposed training algorithm are as follows

Input: $X\in R^{n\times q}$, $Y\in R^{n\times m}$, where $n$ is the number of
instances, $q$ is the number of
features, $m$ ($m>1$) is the number of classes, $Y$ is the true class labels
after one-hot
encoding. For epoch in range(num(epochs)): optimizer.zero_grad( ) $O$=[ [ ]
for $j$ in range (ensemble_size) ] for $j$ in range(ensemble_size):
$O[j]$=torch.softmax(nets$[j]$($X$), dim=$1$) $r_{TL}=0,r_{LL}=0$ for $k$ in
range($m$): for $j$ in range(ensemble_size): $r_{TL}+=Corr(Y[:,k],O[j][:,k])$
for $i$ in range(ensemble_size): if $i<j$: $r_{LL}+=Corr(O[i][:,k],O[j][:,k])$
loss=$-(r_{TL}-\lambda\cdot r_{LL})$ loss.backward( ) optimizer.step( )

In particular, the major steps of the training algorithm for a data set with
$m$ classes are:
Step 1. Obtain the one-hot encoded (Harris and Harris, 2010) true class labels
$Y$.
Step 2. Obtain all the corresponding soft outputs $O$ produced from each of
the neural networks in the ensemble of size $N$, where each column of $O$
represents the probability for instances to fall into that category, and each
row sum of $O$ equals the total probability $1$.
Step 3. Compute all column-wise Pearson correlations between the network
outputs $O$ and the true class labels $Y$, for each of the network in the
ensemble. Take the overall sum of these column-wise correlations as a measure
of truth-learner correlations $r_{TL}$ in the neural network ensemble.
Step 4. Similarly compute all column-wise Pearson correlations between the
outputs $O$, for each pair of networks in the ensemble. Take the overall sum
of these column-wise correlations as a measure of learner-learner correlations
$r_{LL}$ in the neural network ensemble.
Step 5. The neural network ensemble now can be trained with the loss function

$Loss=-(r_{TL}-\lambda\cdot r_{LL}),$ (11)

where $r_{TL}$ (calculated by step 3) is a measure for the learner accuracy in
the neural network ensemble (with the higher $r_{TL}$ is, the higher the
learner accuracy is), and $r_{LL}$ (calculated by step 4) is a measure for the
diversity (with the lower $r_{LL}$ is, the more diverse the ensemble is).

Notice that in the loss function $\eqref{new_loss}$, the outer-most minus sign
is introduced, as we aim to minimize the training loss by maximizing $r_{TL}$
and in the meantime minimizing $r_{LL}$. Backpropagation can easily be
performed with this loss function, as all the parts in $\eqref{new_loss}$ are
smooth and differentiable. More importantly, the parameter $\lambda$
($\lambda>0$) controls the diversity level imposed on the neural network
ensemble, with the larger $\lambda$ is, the more diverse the networks in the
ensemble will be.

_The ability to explicitly control the level of diversity is a significant
advantage of our training algorithm, as opposed to other current published
work regarding neural network ensembles that merely allow diversity to occur
by happenstance. Another advantage of our training algorithm is that it is
generally applicable to neural network ensembles of any size and any network
architecture, as all parts in the loss function $\eqref{new_loss}$ are
computable with no restrictions on neural network architecture._

####  4.3.2 Experiments on Non-Image Data

In order to show the effectiveness of our proposed training algorithm on non-
image data classification problems, perhaps more importantly, to show the
advantage of our algorithm over the DECORATE algorithm (Melville and Mooney,
2005) introduced in section 4.2, herein we run experiments on three UCI non-
image data sets (Dua and Graff, 2017) used by Melville and Mooney (2005):
Breast Cancer Wisconsin (Original) Data Set, Iris Data Set, and Image
Segmentation Data Set. The detailed information of the three data sets is
summarized in Table 2.

We compared the performance of the neural network ensemble trained with our
algorithm (with $\lambda=0.9$), the neural network ensemble trained with
DECORATE, and the original J48-based DECORATE ensemble used in Melville and
Mooney (2005) 333The Python code for the experiments in Section 4.3.2 can be
found on https://bitbucket.org/wli5/neural-network-ensembles-theory-training-
and-the-importance-of/src/master/.. Note that in Melville and Mooney (2005)
the DECORATE algorithm has not been applied to neural networks, therefore
herein we apply it to neural network ensembles, which will be compared to the
neural network ensembles trained with our algorithm. To have a fair
comparison, the three ensemble methods share the same ensemble size $15$, and
for DECORATE methods, the amount of artificial data generated is set to be
equal to the size of the original training set. The prediction performance of
each of the three ensemble methods are evaluated using $10$-fold cross
validation (Hastie et al., 2005), that is, each data set in Table 2 is
randomly split into $10$ equal-size segments of which $9$ segments are used
for training and the other one is used for testing. The prediction error rate
is then averaged over the $10$ trials.

Table 2: Summary of Non-Image Data Sets Name | Instances | Classes | Attributes
---|---|---|---
Breast-w | 699 | 2 | 9
Iris | 150 | 3 | 4
Segment | 2310 | 7 | 19

The $10$-fold cross validation error rates of the three ensemble methods on
the three data sets are tabulated in Table 3. We can see that out of the three
non-image datasets, for both Breast-w and Iris data, the neural network
ensemble trained with our algorithm outperforms the DECORATE ensembles, and
the error rate on Breast-w even gets reduced by half when using our algorithm
compared to the original DECORATE ensemble. For Segment data, our method still
shows competitive performance, and gives a lower error rate than the DECORATE
neural network ensemble.

Table 3: $10$-fold Cross Validation Error Rate % (DECORATE Versus Our Algorithm) | Breast-w | Iris | Segment
---|---|---|---
DECORATE in Melville and Mooney (2005) | 3.69 | 5.33 | 1.97
DECORATE on Neural Network Ensemble | 2.04 | 10.67 | 3.90
Our Algorithm on Neural Network Ensemble | 1.74 | 4.00 | 3.33

We can take a closer look at the training process of the neural network
ensemble trained with DECORATE and our algorithm, respectively, by visualizing
their computed averaged truth-learner correlations and averaged learner-
learner correlations on the same graph. Figure 7 shows such an example when
the two neural network ensembles are trained on the Breast-w data set. Note
that following the analysis in previous sections, our ensemble exhibits an
overall higher diversity and higher accuracy than the DECORATE ensemble, and
since our ensemble is closer to the optimal boundary, it is guaranteed by
theory that our ensemble should show a better prediction performance, which is
exactly the case as we look at the 10-fold cross validation error rates on the
Breast-w data.

Figure 7: Training on Breast-w Data. Our neural network ensemble (in orange)
shows an overall higher diversity and higher accuracy than DECORATE neural
network ensemble (in blue).

In summary, compared to the DECORATE algorithm, our algorithm does a better
job on creating diversity in neural network ensembles and keeping the
ensembles accurate in the meantime. In addition, the DECORATE algorithm is
less efficient and much more expensive computationally, especially when the
ensemble size is large as it needs to generate artificial training samples at
each iteration.

####  4.3.3 Experiments on Image Data

All our experimental studies included in this section are conducted using the
library PyTorch 1.1.0 [22] in Python 3.6.9 [23]. We run the Python scripts on
compute nodes with $1$ NVIDIA K20 GPU with $6$ Intel CPU 2.10 GHZ cores and
$32$ GB memory each. 444The Python code for the experiments in section 4.3.3
can be found on https://bitbucket.org/wli5/neural-network-ensembles-theory-
training-and-the-importance-of/src/master/.

Herein our focus is on image classification of the benchmark image data sets
CIFAR-10 and CIFAR-100 (Krizhevsky and Hinton, 2009), where we perform
training on the train sets and testing on the test sets as provided by the
PyTorch library. However, the theory and training algorithm provided in this
paper are also applicable to many other machine learning tasks. To show the
general effectiveness of our training algorithm on neural network ensembles,
we conduct three major comparison studies (results are displayed in Table 4,
5, 6), and compare the prediction performance of the neural network ensembles
trained with our proposed approach and other state-of-the-art individual
networks as well as ensembles of state-of-the-art networks.

Table 4 shows the comparison study on a binary subset (“plane” and “car”) of
the CIFAR-10 image data. In particular, five single pretrained networks
(ShuffleNet-v2 (Ma et al., 2018), GoogLeNet (Szegedy et al., 2015),
MobileNet-v2 (Sandler et al., 2018), ResNet-18 (He et al., 2016) and
DenseNet-121 (Huang et al., 2017) included in “torchvision.models”) originally
trained on ImageNet (Deng et al., 2009) are individually re-trained using the
standard cross entropy loss function. The prediction error rate of the five
single nets are listed in Table 4. Based on the five single nets, two
ensembles are built up: one (displayed as “Model II” in Table 4) is trained
with standard ensemble-version of cross entropy loss (that is, the cross
entropy loss of the ensemble is calculated between the averaged outputs from
the component nets and the true class labels (Li and Paffenroth, 2019)), the
other one (displayed as “Our Model”) is trained by our proposed algorithm with
various specified $\lambda$ values. We can see that the neural network
ensemble trained by our approach outperforms both those single networks and
the ensemble trained with standard approach, in fact, our ensemble with
$\lambda=0.3$ shows the lowest prediction error rate of $1.25\%$, which is
over 2 times lower than the best single net MobileNet-v2, and 6 times lower
than the standard ensemble. _This comparison study successfully demonstrates
the effectiveness of our training algorithm on binary image classification
problems._

Table 4: Pretrained Networks on CIFAR-10 (Binary Subset: “Plane” & “Car”) with
Prediction Error Rate (%)

Model |  Description | Prediction Error Rate (%)
---|---|---
Model I |  Five single re-trained neural networks trained with cross entropy loss until convergence. This is our baseline non-ensemble model for comparison. |  | Network | Error Rate
---|---
ShuffleNet-v2 | 2.80
GoogLeNet | 3.30
MobileNet-v2 | 2.70
ResNet-18 | 5.25
DenseNet-121 | 9.05
Model II |  An ensemble (trained with the ensemble-version of cross entropy loss for $10$ epochs) of the five re-trained nets in Model I . | 7.55
Our Model |  An ensemble (trained with our algorithm for $10$ epochs) of the five re-trained nets in Model I. This model demonstrates the practical effectiveness of our approach on binary classification. |  | $\lambda$ | Error Rate
---|---
0.1 | 1.35
0.3 | 1.25
0.5 | 1.70
0.7 | 1.90
0.9 | 1.80

  * •

Note: This table shows the prediction error rates for experiments on a binary
subset (“plane” and “car”) of CIFAR-10. The neural network ensemble trained by
our approach outperforms both those single networks and the ensemble trained
with standard approach, demonstrating the effectiveness of our approach on
binary classification.

Table 5 shows the comparison study on the entire CIFAR-10 with all $10$
classes. Similarly as the study shown in Table 4, three single pretrained
networks (ShuffleNet-v2, GoogLeNet and MobileNet-v2) are individually re-
trained using the standard cross entropy loss, and two ensembles of size three
are built up: one (displayed as “Model II” in Table 5) is trained with
standard ensemble-version of cross entropy loss, the other one (displayed as
“Our Model”) is trained by our proposed algorithm with various specified
$\lambda$ values. In this case, our ensemble with $\lambda=0.1$ shows the
lowest prediction error rate as $5.76\%$, which is much smaller than the best
single net GoogLeNet and the standard ensemble. _This comparison study
demonstrates the effectiveness of our training algorithm on multi-label image
classification problems._

Table 5: Pretrained Networks on CIFAR-10 with Prediction Error Rate (%)

Model |  Description | Prediction Error Rate (%)
---|---|---
Model I |  Three single re-trained neural networks trained with cross entropy loss using $40$ epochs. This is our baseline non-ensemble model for comparison. |  | Network | Error Rate
---|---
ShuffleNet-v2 | 8.50
GoogLeNet | 6.43
MobileNet-v2 | 7.11
Model II |  An ensemble (trained with the ensemble-version of cross entropy loss function using $40$ epochs) of the three re-trained nets in Model I. | 7.78
Our Model |  An ensemble (trained with our algorithm using 40 epochs) of the three re-trained nets in Model I. This model demonstrates the practical effectiveness of our approach on multi-label classification. |  | $\lambda$ | Error Rate
---|---
0.1 | 5.76
0.2 | 5.91
0.3 | 6.50

  * •

Note: This table shows the prediction error rates for experiments on the
entire CIFAR-10 with all 10 classes. The neural network ensemble trained by
our approach outperforms both those single networks and the ensemble trained
with standard approach, demonstrating the effectiveness of our approach on
multi-label classification.

A third comparison study is performed to show that with our theory and
proposed algorithm, the prediction performance of any given neural network
ensembles can be improved even further. Herein, we will take experiments on a
recent state-of-art neural network EfficientNet (Tan and Le, 2019) as such an
example. Table 6 shows the prediction error rates of experiments on CIFAR-10
and CIFAR-100 data sets. The baseline model for comparison is a single
EfficientNet-b0 net trained with standard cross entropy loss function, while
our ensemble model consisting of three copies of the trained EfficientNet-b0
net is trained with our algorithm. We can see that for both data sets, the
neural network ensembles trained with our algorithm provide lower prediction
error rates than the baseline neural network (which is already very accurate),
meaning that _our algorithm is able to make good neural networks even better
by explicitly imposing diversity and learner accuracy on the loss function._

Table 6: EfficientNet on CIFAR-10 and CIFAR-100 with Prediction Error Rate (%)

Model |  Description |  CIFAR-10 |  CIFAR-100
---|---|---|---
Baseline |  A single EfficientNet-b0 model trained with standard cross entropy loss by $40$ epochs. |  5.44% |  17.28%
Our Model |  An ensemble of three copies of re-trained EfficientNet-b0 model trained with our algorithm by $40$ epochs. |  4.65% ($\lambda=0.1$) |  14.13% ($\lambda=0.1$)

  * •

Note: This table shows the prediction error rate for experiments on CIFAR-10
and CIFAR-100. For both data sets, the neural network ensembles trained with
our algorithm provide lower prediction error rates than the baseline neural
network (which is already very accurate), meaning that our algorithm is able
to make good neural networks even better.

Therefore, based on the three comparison studies shown in Table 4, 5, 6, _our
training algorithm of neural network ensembles consistently achieves the
lowest error rate on a variety of problems and improves the prediction
performance of both state-of-the-art individual networks as well as ensembles
of state-of-the-art networks._

##  5 CONCLUSIONS

In this paper, we develop theory that offers a rigorous understanding of how
individual accuracies of component learners and diversity affect the
performance of an ensemble. These theoretical considerations inspires
methodologies for assessing and improving the optimality of given ensembles.
Just as importantly, we propose a training algorithm for deep neural network
ensembles that explicitly encourages ensemble diversity, and this algorithm is
shown to be generally effective in improving start-of-the-art neural networks
by experiments on standard benchmark data sets. This training algorithm is
also quite efficient in that one only needs to train the ensembles using a
small number of epochs to achieve good performance, assuming that the
individual learners are already trained outside of the ensemble.

Based upon the results in this paper, there are several interesting directions
for future work. First, there appear to be interesting connections between the
proof of Theorem 2 and domains such as coding theory, and it would be quite
useful to understand these connections better. Second, while our numerical
results suggest a strong, and general, connection between $r_{TL}^{(ave)}$ and
ensemble accuracy, our Theorem 3 only address the case of homogeneous
ensembles. Generalizing this result would likely lead to a deep understanding
of the performance of ensembles. Third, and perhaps most importantly, our
numerical experiments beg for application to many more possible machine
learning tasks. There are a wide range of possibilities for taking already
state-of-the-art methods and improving them even further by putting them into
ensembles.

## Appendix A.

Proof to Theorem 2:
Consider the label outputs of the $N$ learners in an ensemble, standardize the
outputs so that each output can be considered as a random variable $L_{i}$
with unit variance, i.e. $Var(L_{i})=1,i=1,2,\cdots,N$. Similarly standardize
the labels of the ground truth $T$ such that $Var(T)=1$.

Define the sum of the outputs of the $N$ learners as: $S=\sum_{i=1}^{N}L_{i}$,
then we have

$\displaystyle Cov(S,T)$
$\displaystyle=Cov(\sum_{i=1}^{N}L_{i},T)=\sum_{i=1}^{N}Cov(L_{i},T)$ (12)
$\displaystyle=\sum_{i=1}^{N}Corr(L_{i},T)=N\cdot r_{TL}^{(ave)},$

where $Cov(.)$ is the covariance, $Corr(.)$ is the correlation. Note that
$Cov(L_{i},T)=Corr(L_{i},T)$ in $\eqref{eq1}$ because $L_{i}$’s and $T$ are
all unit vectors.

Based on one statistical property of variance, we have

$Var(\sum_{i=1}^{N}{L_{i}})=\sum_{i=1}^{N}Var(L_{i})+\sum_{i=1}^{N}\sum_{j\neq
i}^{N}Cov(L_{i},L_{j})$ (13)

Since $Var(L_{i})=1$, $i=1,2,\cdots,N$, the covariance between two random
variables equals the correlation, i.e.
$Cov(L_{i},L_{j})=Corr(L_{i},L_{j})=r_{L_{i},L_{j}}$. We also have
$r_{L_{i},L_{j}}=r_{L_{j},L_{i}}$, therefore we can rewrite $\eqref{eq2}$ as

$\displaystyle Var(\sum_{i=1}^{N}{L_{i}})$
$\displaystyle=N+\sum_{i=1}^{N}\sum_{j\neq i}^{N}r_{L_{i},L_{j}}$ (14)
$\displaystyle=N+2\sum_{i=1}^{N}\sum_{j>i}^{N}r_{L_{i},L_{j}}$
$\displaystyle=N+2\cdot\frac{N(N-1)}{2}r_{LL}^{(ave)}$

Finally based on the Cauchy-Schwarz inequality, we get:

$\displaystyle(Cov(S,T))^{2}$ $\displaystyle\leqslant Var(S)\cdot Var(T)$ (15)
$\displaystyle=Var(S)\cdot 1$ $\displaystyle=Var(\sum_{i=1}^{N}L_{i})$
$\displaystyle=N+N(N-1)r_{LL}^{(ave)}$

Combining $\eqref{eq1}$ and $\eqref{eq4}$,

$(N\cdot r_{TL}^{(ave)})^{2}\leqslant N+N(N-1)\cdot r_{LL}^{(ave)}.$ (16)

Therefore, after simplification, we get

$-\sqrt{\frac{(N-1)\cdot r_{LL}^{(ave)}+1}{N}}\leqslant
r_{TL}^{(ave)}\leqslant\sqrt{\frac{(N-1)\cdot r_{LL}^{(ave)}+1}{N}}$ (17)

## Appendix B.

Proof to Theorem 3:
In this proof, $E(.)$ represents expectation, $Var(.)$ represents variance.
Based on basic statistics, the expectation of $T$ is calculated by

$E(T)=P(T=1)=\alpha,$ (18)

the expectation of $L$ is calculated by

$\displaystyle E(L)$ $\displaystyle=P(L=1)$ $\displaystyle=$ $\displaystyle
P(T=1\cap L=1)+P(T=0\cap L=1)$ $\displaystyle=$ $\displaystyle P(T=1\cap
L=1)+(P(T=0)-P(T=0\cap L=0))$ $\displaystyle=$ $\displaystyle\beta
p+(1-\alpha-(p-\beta p))=2\beta p-p+1-\alpha,$ (19)

and the expectation of the product $TL$ is calculated by

$E(TL)=P(TL=1)=P(T=1\cap L=1)=\beta p.$ (20)

While the variance of $T$ is given by

$\displaystyle Var(T)$ $\displaystyle=E(T^{2})-E(T)^{2}=P(T^{2}=1)-P(T=1)^{2}$
$\displaystyle=P(T=1)-P(T=1)^{2}$
$\displaystyle=P(T=1)(1-P(T=1))=\alpha(1-\alpha),$ (21)

and the variance of $L$ is given by

$\displaystyle Var(L)$ $\displaystyle=E(L^{2})-E(L)^{2}=P(L^{2}=1)-P(L=1)^{2}$
$\displaystyle=P(L=1)-P(L=1)^{2}$ $\displaystyle=P(L=1)(1-P(L=1))$
$\displaystyle=(2\beta p-p+1-\alpha)(-2\beta p+p+\alpha).$ (22)

Therefore, based on the definition of Pearson correlation coefficient,

$\displaystyle r_{T,L}$
$\displaystyle=\frac{E(TL)-E(T)E(L)}{\sqrt{Var(T)Var(L)}}$ (23)
$\displaystyle=\frac{\beta p-\alpha(2\beta
p-p+1-\alpha)}{\sqrt{\alpha(1-\alpha)(2\beta p-p+1-\alpha)(-2\beta
p+p+\alpha))}}.$

## References

  * Amatriain et al. (2011) Xavier Amatriain, Alejandro Jaimes, Nuria Oliver, and Josep M Pujol.  Data mining methods for recommender systems.  In _Recommender systems handbook_ , pages 39–71. Springer, 2011.
  * Amini et al. (2014) Mohammad Amini, Jalal Rezaeenour, and Esmaeil Hadavandi.  Effective intrusion detection with a neural network ensemble using fuzzy clustering and stacking combination method.  _Journal of Computing and Security_ , 1(4):293–305, 2014.
  * Asafuddoula et al. (2017) Md Asafuddoula, Brijesh Verma, and Mengjie Zhang.  An incremental ensemble classifier learning by means of a rule-based accuracy and diversity comparison.  In _2017 International Joint Conference on Neural Networks (IJCNN)_ , pages 1924–1931. IEEE, 2017.
  * Benesty et al. (2009) Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen.  Pearson correlation coefficient.  In _Noise reduction in speech processing_ , pages 1–4. Springer, 2009\.
  * Boland (1989) Philip J Boland.  Majority systems and the condorcet jury theorem.  _Journal of the Royal Statistical Society: Series D (The Statistician)_ , 38(3):181–189, 1989.
  * Breiman (2001) Leo Breiman.  Random forests.  _Machine learning_ , 45(1):5–32, 2001.
  * Brown (2004) Gavin Brown.  _Diversity in neural network ensembles_.  PhD thesis, Citeseer, 2004.
  * Brown and Kuncheva (2010) Gavin Brown and Ludmila I Kuncheva.  “good” and “bad” diversity in majority vote ensembles.  In _International workshop on multiple classifier systems_ , pages 124–133. Springer, 2010.
  * Brown et al. (2005) Gavin Brown, Jeremy L Wyatt, and Peter Tiňo.  Managing diversity in regression ensembles.  _Journal of Machine Learning Research_ , 6(Sep):1621–1650, 2005.
  * Chaudhry et al. (1997) M Aslam Chaudhry, Asghar Qadir, M Rafique, and SM Zubair.  Extension of euler’s beta function.  _Journal of Computational and Applied Mathematics_ , 78(1):19–32, 1997.
  * Choi and Lee (2018) Jae Young Choi and Bumshik Lee.  Combining lstm network ensemble via adaptive weighting for improved time series forecasting.  _Mathematical Problems in Engineering_ , 2018, 2018.
  * Dai et al. (2017) Qun Dai, Rui Ye, and Zhuan Liu.  Considering diversity and accuracy simultaneously for ensemble pruning.  _Applied Soft Computing_ , 58:75–91, 2017.
  * Datta et al. (2012) Somantika Datta, Stephen Howard, and Douglas Cochran.  Geometry of the welch bounds.  _Linear algebra and its applications_ , 437(10):2455–2470, 2012.
  * Deng et al. (2009) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In _CVPR09_ , 2009.
  * Dienes (1957) Paul Dienes.  _The Taylor series: an introduction to the theory of functions of a complex variable_.  Dover New York, 1957.
  * Dietterich (2000) Thomas G Dietterich.  Ensemble methods in machine learning.  In _International workshop on multiple classifier systems_ , pages 1–15. Springer, 2000.
  * Dietterich et al. (2002) Thomas G Dietterich et al.  Ensemble learning.  _The handbook of brain theory and neural networks_ , 2:110–125, 2002.
  * Dua and Graff (2017) Dheeru Dua and Casey Graff.  UCI machine learning repository, 2017.  URL http://archive.ics.uci.edu/ml.
  * Freund and Schapire (1997) Yoav Freund and Robert E Schapire.  A decision-theoretic generalization of on-line learning and an application to boosting.  _Journal of computer and system sciences_ , 55(1):119–139, 1997.
  * Friedman (2002) Jerome H Friedman.  Stochastic gradient boosting.  _Computational statistics & data analysis_, 38(4):367–378, 2002.
  * Giacinto and Roli (2001) Giorgio Giacinto and Fabio Roli.  Design of effective neural network ensembles for image classification purposes.  _Image and Vision Computing_ , 19(9-10):699–707, 2001.
  * Gilbert (1991) George T Gilbert.  Positive definite matrices and sylvester’s criterion.  _The American Mathematical Monthly_ , 98(1):44–46, 1991.
  * Harris and Harris (2010) David Harris and Sarah Harris.  _Digital design and computer architecture_.  Morgan Kaufmann, 2010.
  * Hastie et al. (2005) Trevor Hastie, Robert Tibshirani, Jerome Friedman, and James Franklin.  The elements of statistical learning: data mining, inference and prediction.  _The Mathematical Intelligencer_ , 27(2):83–85, 2005.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hecht-Nielsen (1992) Robert Hecht-Nielsen.  Theory of the backpropagation neural network.  In _Neural networks for perception_ , pages 65–93. Elsevier, 1992\.
  * Hsu (2017) Kuo-Wei Hsu.  A theoretical analysis of why hybrid ensembles work.  _Computational intelligence and neuroscience_ , 2017, 2017.
  * Huang et al. (2000) Fu Jie Huang, Zhihua Zhou, Hong-Jiang Zhang, and Tsuhan Chen.  Pose invariant face recognition.  In _Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580)_ , pages 245–250. IEEE, 2000.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * Irvine and Harle (2001) James Irvine and David Harle.  _Data communications and networks: an engineering approach_.  John Wiley & Sons, 2001.
  * Kaniovski and Zaigraev (2011) Serguei Kaniovski and Alexander Zaigraev.  Optimal jury design for homogeneous juries with correlated votes.  _Theory and decision_ , 71(4):439–459, 2011.
  * Krizhevsky and Hinton (2009) Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  Technical report, Citeseer, 2009.
  * Kuncheva (2004) Ludmila I Kuncheva.  _Combining pattern classifiers: methods and algorithms_.  John Wiley & Sons, 2004.
  * Kuncheva and Whitaker (2003) Ludmila I Kuncheva and Christopher J Whitaker.  Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy.  _Machine learning_ , 51(2):181–207, 2003.
  * Li et al. (2018) Hui Li, Xuesong Wang, and Shifei Ding.  Research and development of neural network ensembles: a survey.  _Artificial Intelligence Review_ , 49(4):455–479, 2018.
  * Li and Paffenroth (2019) Wenjing Li and Randy Paffenroth.  Optimal ensembles for deep learning classification: Theory and practice.  In _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_ , pages 1658–1665. IEEE, 2019.
  * Liu and Yao (1999) Yong Liu and Xin Yao.  Ensemble learning via negative correlation.  _Neural networks_ , 12(10):1399–1404, 1999.
  * Ma et al. (2018) Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 116–131, 2018.
  * Melville and Mooney (2005) Prem Melville and Raymond J Mooney.  Creating diversity in ensembles using artificial data.  _Information Fusion_ , 6(1):99–111, 2005.
  * Nguyen and Pernkopf (2018) Truc Nguyen and Franz Pernkopf.  Acoustic scene classification using a convolutional neural network ensemble and nearest neighbor filters.  In _Workshop on Detection and Classification of Acoustic Scenes and Events_ , 2018.
  * Nowlan and Hinton (1991) Steven J Nowlan and Geoffrey E Hinton.  Evaluation of adaptive mixtures of competing experts.  In _Advances in neural information processing systems_ , pages 774–780, 1991.
  * Sagi and Rokach (2018) Omer Sagi and Lior Rokach.  Ensemble learning: A survey.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 8(4):e1249, 2018.
  * Sandler et al. (2018) Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4510–4520, 2018.
  * Sen (1993) Amartya Sen.  Markets and freedoms: achievements and limitations of the market mechanism in promoting individual freedoms.  _Oxford Economic Papers_ , pages 519–541, 1993.
  * Shao et al. (2018) Haidong Shao, Hongkai Jiang, Ying Lin, and Xingqiu Li.  A novel method for intelligent fault diagnosis of rolling bearings using ensemble deep auto-encoders.  _Mechanical Systems and Signal Processing_ , 102:278–297, 2018.
  * Sharkey and Sharkey (1997) Amanda JC Sharkey and Noel E Sharkey.  Combining diverse neural nets.  _The Knowledge Engineering Review_ , 12(3):231–247, 1997.
  * Steele (2004) J. Michael Steele.  _The Cauchy-Schwarz Master Class: An Introduction to the Art of Mathematical Inequalities_.  Cambridge University Press, 2004.  doi: 10.1017/CBO9780511817106.
  * Sustik et al. (2007) Mátyás A Sustik, Joel A Tropp, Inderjit S Dhillon, and Robert W Heath Jr.  On the existence of equiangular tight frames.  _Linear Algebra and its applications_ , 426(2-3):619–635, 2007.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1–9, 2015.
  * Tan and Le (2019) Mingxing Tan and Quoc V Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  _arXiv preprint arXiv:1905.11946_ , 2019.
  * Tramèr et al. (2017) Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel.  Ensemble adversarial training: Attacks and defenses.  _arXiv preprint arXiv:1705.07204_ , 2017.
  * Williams and Gong (2014) Trefor P Williams and Jie Gong.  Predicting construction cost overruns using text mining, numerical data and ensemble classifiers.  _Automation in Construction_ , 43:23–29, 2014.
  * Zhang and Ma (2012) Cha Zhang and Yunqian Ma.  _Ensemble machine learning: methods and applications_.  Springer, 2012.