# Effective training-time stacking for ensembling of deep neural networks

1st Polina Proskura  Skoltech
Russia
<EMAIL_ADDRESS>2nd Alexey Zaytsev  Skoltech
Russia
<EMAIL_ADDRESS>
###### Abstract

Ensembling is a popular and effective method for improving machine learning
(ML) models. It proves its value not only in classical ML but also for deep
learning. Ensembles enhance the quality and trustworthiness of ML solutions,
and allow uncertainty estimation. However, they come at a price: training
ensembles of deep learning models eat a huge amount of computational
resources.

A snapshot ensembling collects models in the ensemble along a single training
path. As it runs training only one time, the computational time is similar to
the training of one model.

However, the quality of models along the training path is different:
typically, later models are better if no overfitting occurs. So, the models
are of varying utility.

Our method improves snapshot ensembling by selecting and weighting ensemble
members along the training path. It relies on training-time likelihoods
without looking at validation sample errors that standard stacking methods do.
Experimental evidence for Fashion MNIST, CIFAR-10, and CIFAR-100 datasets
demonstrates the superior quality of the proposed weighted ensembles c.t.
vanilla ensembling of deep learning models.

###### Index Terms:

Neural network, Ensembles, Deep Learning, Image classification

##  I Introduction

The process of neural network training is cumbersome. The state-of-the-art
models occupy significant volume and train for days or even weeks [1, 2]. The
problem is even worse for ensembles of deep learning models trained in an
independent way [3], as we need to run training several times.

More time-effective approaches to the training of deep learning model
ensembles exist [4, 5]. The snapshot ensembling methods collect ensemble
members along the training path. As we need only one training, we save
resources with little drop in the quality of the final ensemble [6]. One can
show, that these separate models are diverse enough to reduce the bias of the
estimator, as they come from different local optima of a loss function.

The goal of the article is to propose an approach that works at the upper
weighting level on a training time of single model and outperforms every
single model. By looking at training time likelihood we were able to identify
weights of separate ensemble members that lead to a better ensemble.

We evaluate our ensembling approaches for images classification problems [5]
and investigate our ensembles with respect to the quality, diversity, and
effectiveness. We show, that our stacking procedure produces better models,
than the vanilla snapshot ensembling, while making them more efficient.

Our work has the following sections:

  * •

We review the state-of-the-art in Section II.

  * •

We propose our stacking procedure in Section III.

  * •

We finish with the results of computational experiments in Section IV.

##  II State of the art

###  II-A Ensembling in Machine learning

For classical machine learning (ML) algorithms, ensembling is a very effective
method for model improvement, getting rid of overfitting and decreasing
uncertainty. Basic approaches average model predictions reducing ensemble
model bias [7, 8], while gradient boosting approaches try to reduce both bias
and variance [9, 10, 11, 12, 13] by adding basic models that correct errors of
the current ensemble. For Random forest the other type of ensembling is used –
bagging [3].

###  II-B Stacking

Stacking is one of the ways of ensembling [14]. The main idea of the stacking
is that the combiner algorithm is used on top of regular algorithms to make a
final prediction using the predictions of the trained algorithms as a new
input [15].

The classical approach to the stacking in neural network ensembling is to
train individually several neural networks and then use major voting
principles for the calculation of the final result [16].

###  II-C Ensembles of neural networks

A typical neural network has billions of parameters. So, if we optimize the
loss function for a similar dataset and architecture with approaches based on
stochastic gradient descent, we end up in different local optima due to the
complexity of the loss function surface [17, 18, 19]. So, the total quality of
the ensemble will be better than the quality of individual models [6].

This solution increased the quality but requires more time to train an
ensemble [1, 2]. In [20], the authors explore dependencies of quality on
parameters of the ensemble, stating that ensembles can provide better accuracy
in both quality and time efficiency, than a single model after careful
selection of architecture of the model.

###  II-D Effective ensembles of neural networks

The authors [21] suggest to effectively ensemble models along the training
path: cyclical learning rate [22]. This method varies the learning rate during
the training process [23]. The experiments confirmed the effectiveness of such
methods [22, 24]. Another paper on the application of this idea to ensembling
of neural networks is [4].

The training time is equal to the training time of one model, which means the
significant economy of computer resources. The authors of the articles suggest
the schedule for the learning rate during training.

###  II-E Conclusions

Ensembling methods help to achieve better results compared to a single model
for both classic machine learning and deep learning ensembles. For deep
learning, efficient ensembling happens via snapshot ensembling: we collect
models in the ensemble during a single training run with a cyclical learning
rate.

However, the snapshot ensembling heuristic lacks a procedure to aggregate
basic models into the ensemble, and the published results that these basic
models vary in quality and diversity.

[4, 25] straightforwardly averages individual models into ensemble. Bayes and
stacking perspective at this type of ensembles suggest that there should exist
a way to improve an ensemble.

##  III Methods

###  III-A Background

We consider a fixed architecture of a neural network. The architecture defines
a set of functions $\mathcal{F}=\\{f(\mathbf{x})\\}$ that comprises the space
of the basic models for an ensemble. An example of the input $\mathbf{x}$ is
an image with several channels, an example of the output $f(\mathbf{x})$ is a
probability vector with the size $k$ for the classification problem with $k$
classes from a set $\\{1,\ldots,k\\}$. The vector of the neural network
parameters $\boldsymbol{\theta}$ defines a function
$f_{\boldsymbol{\theta}}(\mathbf{x})\in\mathcal{F}$. We will use
$\boldsymbol{\theta}_{i}$ to define a model
$f_{i}=f_{\boldsymbol{\theta}_{i}}$.

Typically, to produce a model one estimates parameters $\boldsymbol{\theta}$
via minimizing an empirical loss function for a sample
$D=\\{(\mathbf{x}_{j},y_{j})\\}_{j=1}^{m}$. The goal is to build an ensemble
of $N$ models $\\{\boldsymbol{\theta}_{1},\ldots,\boldsymbol{\theta}_{N}\\}$,
and the rule for aggregating the predictions of separate models for gaining
the ensemble prediction $\hat{f}(\mathbf{x})$.

####  III-A1 Ensembling of independent models.

The basic independent algorithm consists of two steps:

  1. 1.

We train each model from an ensemble using a sample $D$ to get
$f_{\boldsymbol{\theta}_{i}}(\mathbf{x})$. As the training of each model
starts from a different initialization of $\boldsymbol{\theta}$, the resulting
models are also different.

  2. 2.

The prediction of the ensemble is the mean of all models
$\hat{f}(\mathbf{x})=\frac{1}{N}\sum_{k=1}^{N}f_{\boldsymbol{\theta}_{k}}(\mathbf{x})$.

If the cost of training one model is $T$, then the cost of the straightforward
training the ensemble is $NT$.

####  III-A2 Snapshot ensembling.

More computationally effective approach is _Snapshot_ ensembling [4]. Suppose,
that we have $T$ epochs for a training of a model. Then the learning rate on
$t$-th epoch will be $\alpha(t)$. We select the learning rate scheduler:
$\alpha(t)$ cyclically changes from $\alpha_{min}$ to
$\alpha_{max}>\alpha_{min}$.

At the $\alpha_{min}$ we reach the local minimum of the loss function and
further by increasing the learning rate we leave the current local minimum and
reach the next one [4]. As a $k$-th model we will choose
$\boldsymbol{\theta}_{k}$, which is the estimation of the parameters on the
$k$-th minimum of the learning rate, so that $\alpha(t)=\alpha_{min}$ at time
$k$. The ensemble prediction is the mean of the base models prediction:
$\hat{f}(\mathbf{x})=\frac{1}{N}\sum_{k=1}^{N}f_{\boldsymbol{\theta}_{k}}(\mathbf{x})$.
The training cost, in this case, is $T$ and is similar to the training cost
for training of a single model.

####  III-A3 SWA approach

The main idea of SWA (Stochastic Weight Averaging) [25] is that the ensembling
happens not in the model space, but in the model parameters space. The method
uses the combination of the model weights at the different stages of the
training. So, the following two properties are achieved:

  * •

If we combine weights, totally there is one model. It speeds up inference.

  * •

The training process has the same estimated complexity but leads to finding a
better local minimum with smoother slopes.

###  III-B Proposed stacking approach

We propose a procedure for the stacking based on effective Snapshot ensembling
[4]. We provide a more precise selection of the iteration for finding the
$\boldsymbol{\theta}_{k}$ in the neighborhood of the $k$-th minimum of
learning rate. We also argue, that as earlier models are worse than later
ones, we should reduce weights for them. In particular, we consider weighting
predictions in an ensemble, such that:

$\hat{f}(\mathbf{x})=\frac{1}{N}\sum_{k=1}^{N}w_{k}f_{\boldsymbol{\theta}_{k}}(\mathbf{x}),$

the weights choosing procedure $w_{k}$ will follow.

####  III-B1 Model collection

For a typical number of epochs $350$ with number of iterations $100$ per epoch
we have more than $3000$ models during training. We can’t use them all due to
computational restrictions and shouldn’t, as neighbor models are similar to
each other and add little value if we take both of them instead of just one.
Moreover, for cyclic learning rates models obtained at high learning rates
typically are of worse quality, so if included in the ensemble, they will
decrease the ensemble quality.

####  III-B2 Models in the minimum of the cyclical learning rate

The basic approach [4] collects models at the minima of the cyclical learning
rate, as they correspond to a neighborhood of a local optima of the loss
function.

####  III-B3 Models in the average value of learning rate

To increase the diversity of an ensemble, we can also add to the ensemble
models picked up at points with $\alpha=\frac{\alpha_{max}+\alpha_{min}}{2}$.
As during stacking, we weight each model, we can assign small weights for such
model in case they decrease the overall ensemble accuracy.

####  III-B4 Choice of the models from the window

Figure 1: Choice of the model from the window near the minimum of learning
rate

During the collection of models into the ensemble, we look at models at a
local optimum of the learning rates. However, we expect that this option can
be not optimal. To validate this assumption of snapshot ensemble, we propose
the following change to the weight selection procedure illustrated by Figure
1. The steps are the following:

  * •

The local minimum of the learning rate occurs at $\alpha(t)$. We check, if
models further on the path is better. A hyperparameter $s$ controls, that we
take models in the range $(\alpha(t-s),\ldots,\alpha(t+s))$.

  * •

For all $2s+1$ models around the local minimum we calculate the loss function.
In particular, we use the loss function at the validation set and get
$\\{q_{t-s},\ldots,q_{t+s}\\}$.

  * •

We select the model with the best $q_{i}$, $i\in\\{{t-s},\ldots,{t+s}\\}$. We
add this model to the ensemble.

####  III-B5 Weighting

Snapshot ensembling has several disadvantages compared to classical
ensembling. One of them is the possible closeness of models in an ensemble, as
we collect them during a single training. Contrary, in the classical ensembles
due to randomization, different models end in different local minimums that
are far from each other, so the different models make errors, which is not
true for snapshot ensembles.

As a solution to this challenge, we propose to use weighting. It helps to
filter the models, which haven’t reached the local minimum by giving them
small weights, and to balance the effect of close models to each other in the
ensemble.

####  III-B6 Weighting based on the loss function value

The main assumption of this approach is the following: the more errors the
model makes, the less weight it should have.

Then the weighting is the following:

  * •

Let $l_{i}$ – value of the loss function for the $i$-th model in ensemble.

  * •

$w_{i}$ is the weight of the model can be calculated according the following
formula:

$w_{i}=s(l_{i})$

  * •

The function $s(\cdot)$ should reflect the following properties:

    * –

Be monotonous: the bigger the value of the loss function, the smaller the
weight.

    * –

Should be fast enough for calculations.

We use the following simple function $s(l)=\frac{1}{l}$ to validate our
approach.

####  III-B7 Weighting based on the likelihood on train and validation sets

The other automated approach has more freedom in the methodology – finding the
coefficients based on likelihood values for some samples.

Likelihood $L(\boldsymbol{\theta})$ is the function which for the current
parameters of the model $\boldsymbol{\theta}$, returns the probability that
the sample $D$ comes from the distribution parametrized by
$\boldsymbol{\theta}$:

$L(\boldsymbol{\theta})=P(D|\boldsymbol{\theta}).$

This approach can be described the following way:

  * •

Let $L_{1},...L_{N}$ be the likelihood for each from $N$ models in the
ensemble calculated in the sample.

  * •

Then the weights of the model will be $w_{1},...,w_{N}$:

$w_{i}=s(L_{i})$

For the experiments was chosen the function $s(l)=l$. The likelihood can be
calculated on the training or validation set.

####  III-B8 Finding temperature for more optimal weighting of the models in
ensemble

This approach is the modification of the previous approach. The difference is
that $s_{\tau}(\cdot)$ is another function, which has the temperature
parameter $\tau$, responsible for weight calibration, to make the approach
more flexible. It has the following form
$s_{\tau}(l)=\exp{\left(\frac{l}{\tau}\right)}$.

Specific choices of temperature parameter $\tau$ can lead to a desired
asymptotic behavior. With a big enough temperature, the weights will be almost
the same and the prediction will be by simple averaging. With a small enough
temperature the effect will be the opposite: the best model will have a big
weight and others relatively small. The selection of a particular temperature
allows a correct trade-off between these two extreme options.

###  III-C Implementation details

Most of the experiments were held using the big and complex neural network
ResNet18 [26]. Neural network ResNet18 is based on convolutions with skip
connections.

As the loss function, we use the negative logarithm of the likelihood:

$Loss=-\log(P(D|\theta)),$

where $D$ is a sample of the pairs of input and targets.

##  IV Experiments

###  IV-A Datasets

We use three datasets for benchmarking: CIFAR-10 [27], CIFAR-100 [27], Fashion
MNIST [28]. All these datasets correspond to multiclass classification
problems.

###  IV-B Models in the minimum of the learning rate

We start with checking the importance of temperature hyperparameters claimed
to be important in previous articles. We average models via likelihood values
at the training sample.

Figure 2 provides results for different temperatures and a different number of
models using weighting based on training likelihood values. Figure 3 uses the
same procedure, but for the validation likelihood values used for the model
weighting. Value $1000$ on the temperature scale represents the results for
the weighting with the equal weights.

Figure 2: Results for CIFAR-10 for the averaging based on the likelihood on
the training set. Figure 3: Results for CIFAR-10 for the averaging based on
the likelihood on the validation set.

We see, that the best result is $90.5\%$ accuracy for a temperature near $1$
and a the maximum number of models, as expected for the training likelihood
values and $90.4\%$ accuracy for ensemble based on validation likelihood
values. The results are similar for the weighting of ensembles using training,
and test time errors are similar, so we can stack models to an ensemble using
training data only.

###  IV-C Models at the fixed distance from minimum

In the following experiment we changed the iteration used to collect model: we
use a base model not at the local optimum of the learning rate, but rather we
wait fixed number of iterations.

Results for the temperature $\tau=1.0$ are in Figure 4.

Figure 4: The dependency of the quality from the number of steps from the
minimum learning rate, temperature = 1.0 for CIFAR-10

The best result is 89.5% occurs for $10$ steps difference from optimum.

###  IV-D Main results

We conclude our experiments by considering several options for snapshot
ensembling model selection:

  * •

min – models in the minimum learning rate, _vanilla snapshot ensemble_.

  * •

mid – models in the minimum and the middle of the learning rate.

  * •

$10$ steps – model, which is in 10 steps from the minimum learning rate.

For model weighting, we consider two options:

  * •

eq – ensemble with equal weights for all basic models.

  * •

stack — ensemble with weights obtained from training likelihood values

We also consider three baselines: a _single_ model performance, Stochastic
weight averaging, and Ensemble that is an ensemble of independent models that
takes prohibitive time for training, but can serve as a number to aim at for
our effective ensembles based on snapshot ensembling.

Table I presents a comparison of the proposed options for snapshot ensembling
and various baselines. We see that the training time stacking in all cases
improve the final results and is the most reasonable option to use. Moreover,
using more models collected at the local maximum of the learning rate, we
further improve the ensemble, as they provide more diversity to the model.

TABLE I: Accuracy in percents for different combinations of the models with the best results for each ensembling type for CIFAR-10 Model | Type | Number | $\tau$ | Accuracy
---|---|---|---|---
|  | of models |  |
Single | – | 1 | – | $89.49$
Ensembles | individual | 5 | – | 91.9
| min, eq | 24 | – | $89.9$
| min, stack | 24 | 0.9 | $90.5$
| min, eq, val | 24 | – | $90.25$
Snapshot | min, stack, val | 24 | 0.5 | $90.50$
ensemble | 10 steps, eq | 22 | – | $89.0$
| 10 steps, stack | 22 | 1.0 | $89.55$
| middle, eq | 15 | – | $89.9$
| middle, stack | 15 | 1.0 | 90.8
Stochastic | min, eq | 24 | – | $89.17$
Weight Averaging | min, stack | 24 | 1.0 | $89.23$

##  V Acknowledgments

Research was supported by Russian Science Foundation, grant 21-11-00373.

##  VI Conclusion

Cyclical learning rate helps to improve the quality of the model and more
precisely detect the loss function minimums. Snapshot ensembling with equal
weights distribution gains the quality improvement compared to the single
model but is worse than weighted ensembling and classical approach.

The choice of the model for the snapshot ensembling in the minimum of the
learning rate is logical and gains the most promising results. Calculating the
weights of the models in the ensemble according to the likelihood of every
model on training and validation sample gains the best results among
considered approaches and increases the quality of snapshot ensemble.

## References

  * [1] R. Livni, S. Shalev-Shwartz, and O. Shamir, “On the computational efficiency of training neural networks,” CoRR, vol. abs/1410.1141, 2014.
  * [2] G. X. Yu, Y. Gao, P. Golikov, and G. Pekhimenko, “Computational performance predictions for deep neural network training: A runtime-based approach,” CoRR, vol. abs/2102.00527, 2021.
  * [3] L. Breiman, “Bagging predictors,” Machine learning, vol. 24, no. 2, pp. 123–140, 1996.
  * [4] G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger, “Snapshot ensembles: Train 1, get m for free,” 2017.
  * [5] T. Garipov, P. Izmailov, D. Podoprikhin, D. Vetrov, and A. G. Wilson, “Loss surfaces, mode connectivity, and fast ensembling of dnns,” 2018.
  * [6] N. Chirkova, E. Lobacheva, and D. Vetrov, “Deep ensembles on a fixed memory budget: One wide network or several thinner ones?,” arXiv preprint arXiv:2005.07292, 2020.
  * [7] T. K. Ho, “Random decision forests,” in Proceedings of 3rd international conference on document analysis and recognition, vol. 1, pp. 278–282, IEEE, 1995\.
  * [8] L. Breiman, “Random forests,” Machine learning, vol. 45, no. 1, pp. 5–32, 2001.
  * [9] J. H. Friedman, “Greedy function approximation: a gradient boosting machine,” Annals of statistics, pp. 1189–1232, 2001.
  * [10] L. Mason, J. Baxter, P. L. Bartlett, and M. R. Frean, “Boosting algorithms as gradient descent,” in Advances in neural information processing systems, pp. 512–518, 2000.
  * [11] Y. Freund and R. Schapire, “A short introduction to boosting,” Journal of Japanese Society for Artificial Intelligence, vol. 14, 1999.
  * [12] B. Kégl, “The return of adaboost. mh: multi-class hamming trees,” arXiv preprint arXiv:1312.6086, 2013.
  * [13] G. Hughes, “On the mean accuracy of statistical pattern recognizers,” IEEE transactions on information theory, vol. 14, no. 1, pp. 55–63, 1968.
  * [14] S. Džeroski and B. Ženko, “Is combining classifiers with stacking better than selecting the best one?,” Machine Learning, vol. 54, pp. 255–273, 03 2004.
  * [15] Z. Ma, P. Wang, Z. Gao, R. Wang, and K. Khalighi, “Ensemble of machine learning algorithms using the stacked generalization approach to estimate the warfarin dose,” PLOS ONE, vol. 13, pp. 1–12, 10 2018.
  * [16] M. A. Ganaie, M. Hu, M. Tanveer, and P. N. Suganthan, “Ensemble deep learning: A review,” CoRR, vol. abs/2104.02395, 2021.
  * [17] M. A. Ganaie, M. Hu, M. Tanveer, and P. N. Suganthan, “Ensemble deep learning: A review,” CoRR, vol. abs/2104.02395, 2021.
  * [18] N. Sendi, N. Abchiche-Mimouni, and F. Zehraoui, “A new transparent ensemble method based on deep learning,” Procedia Computer Science, vol. 159, pp. 271–280, 2019.  Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 23rd International Conference KES2019.
  * [19] L. Hansen and P. Salamon, “Neural network ensembles,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 10, pp. 993–1001, 1990.
  * [20] E. Lobacheva, N. Chirkova, M. Kodryan, and D. Vetrov, “On power laws in deep ensembles,” 2020.
  * [21] L. N. Smith, “Cyclical learning rates for training neural networks,” 2017.
  * [22] M. Mori and M. Nakano, “Efficient cyclic learning rate schedules and their evaluations for neural network ensemble,” in 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1–6, 2018.
  * [23] J. Li and X. Yang, “A cyclical learning rate method in deep learning training,” in 2020 International Conference on Computer, Information and Telecommunication Systems (CITS), pp. 1–5, 2020.
  * [24] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” Proc. of COMPSTAT, 01 2010.
  * [25] P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” arXiv preprint arXiv:1803.05407, 2018.
  * [26] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015.
  * [27] A. Krizhevsky, “Learning multiple layers of features from tiny images,” University of Toronto, 05 2012.
  * [28] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” CoRR, vol. abs/1708.07747, 2017\.

##  Appendix A Baseline results

The table below II demonstrates the baseline results. We see that Snapshot
ensembles are better than individual models. However, if we select a too large
number of models, we end up with suboptimal ensembles.

Model | Amount of models | Accuracy, $\%$
---|---|---
Baseline | 1 | $89.49$
Cyclical learning rate | 1 | $89.62$
Ensemble | 3 | $91.3$
| 4 | $91.8$
| 5 | $91.9$
Snapshot ensemble | 3 | $89.95$
| 4 | $89.94$
| 5 | $89.88$
TABLE II: Comparison of baseline models for CIFAR-10 data. Snapshot ensembles
improve over baselines for a range of models in the ensemble