# Slope and generalization properties of neural networks

Anton Johansson1* &Niklas Engsner1 &Claes Strannegård1 &Petter Mostad1

###### Abstract

Neural networks are very successful tools in for example advanced
classification. From a statistical point of view, fitting a neural network may
be seen as a kind of regression, where we seek a function from the input space
to a space of classification probabilities that follows the "general" shape of
the data, but avoids overfitting by avoiding memorization of individual data
points. In statistics, this can be done by controlling the geometric
complexity of the regression function. We propose to do something similar when
fitting neural networks by controlling the slope of the network.

After defining the slope and discussing some of its theoretical properties, we
go on to show empirically in examples, using ReLU networks, that the
distribution of the slope of a well-trained neural network classifier is
generally independent of the width of the layers in a fully connected network,
and that the mean of the distribution only has a weak dependence on the model
architecture in general. The slope is of similar size throughout the relevant
volume, and varies smoothly. It also behaves as predicted in rescaling
examples. We discuss possible applications of the slope concept, such as using
it as a part of the loss function or stopping criterion during network
training, or ranking data sets in terms of their complexity.

##  1 Introduction

00footnotetext: 1Chalmers University of Technology, Gothenburg,
Sweden.00footnotetext: *Correspondence to: Anton Johansson
<EMAIL_ADDRESS>
Consider the objective of classifying items, for example images, which may be
represented as points in ${\mathbb{R}}^{n_{0}}$, into $n_{c}$ classes. One
approach is to find a map $f\in{\cal A}$ where ${\cal A}$ is the set of all
continuous maps ${\mathbb{R}}^{n_{0}}\to{\mathbb{R}}^{n_{c}}$ and interpret
$\operatorname{softmax}_{i}(f(x))$ as the probability that $x$ is in class
$i$. We aim for a map that fits observed data in terms of the corresponding
loss function while simultaneously avoiding overfitting to these data.

A general way to avoid overfitting is to control the "geometric complexity" of
$f$, interpreting geometric complexity in a similar way as in many
corresponding methods from classical statistics. A common way to limit the
complexity is to define a subset ${\cal B}\subset{\cal A}$, consisting for
example of all those maps expressable with a neural network with a given
architecture, and to use a particular stochastic algorithm to generate a
suitable $f$ that balances loss minimization and avoidance of overfitting.

As the properties we are trying to balance, the loss and the overfitting, both
depend on $f$ and nothing else, we argue that control of geometric complexity
should be measured in terms of properties defined directly on $f$, and not in
terms of any particular neural network representation of $f$. Calling such
properties geometric properties, we focus in this paper on what we call the
slope, essentially the largest speed with which $f$ moves its input at a
point.

Using the hypothesis that current algorithms for training neural network
classifiers are indeed successful when they control the geometric complexity
of $f$ in a meaningful way, we use these algorithms to generate successful
classifiers $f_{1},\dots,f_{k}$ for a given dataset. Under our hypothesis,
these functions should then have similar geometric properties, and in
particular similar slope properties. These slope properties should depend only
on the problem at hand, i.e., the dataset, and not on the particular neural
network architecture used, except as a consequence of the approximations
involved.

In this paper, we first define and study some basic properties of the slope,
before studying what happens with the slope during learning for ReLU networks.
We then go on to check the hypothesis above by varying the network
architecture, data sets and the distance between the points in order to see
how these parameters affect the slope. Finally, we discuss applications of the
slope, for example to directly target a particular slope during the training
of a neural network, or to use the slope for regularization.

###  1.1 Related work and our contribution

Understanding how and why neural network methods work as well as they do is
clearly a vast area of research, and one that has been attacked in a number of
ways. Some examples are information theoretic approaches
DBLP:conf/itw/TishbyZ15 , DBLP:journals/npl/LiL21 , DBLP:conf/isit/Huang0ZW19
, classical statistical learning theory approaches DBLP:conf/iclr/ZhangBHRV17
, DBLP:journals/jmlr/BartlettHLM19  and others DBLP:conf/icml/ZhouF18 . A
geometrical perspective has been considered in different contexts, e.g, by
relating the manifold structure of the data distribution to generalization
properties, see DBLP:journals/corr/abs-1805-10451 ,
DBLP:journals/corr/abs-1909-11500 , or by understanding the inductive bias of
deep neural networks by studying how the distance to the decision boundary
varies as the data representation is changed, see
DBLP:journals/corr/abs-2104-14372 .

A paper taking a somewhat similar view as ours is yoshida2017spectral . In
this paper Yoshida et al impose an upper bound on the spectral norm of the
local affine transformation of ReLU networks by enforcing the spectral norm of
each individual weight matrix in the network to be small. This provides a
regularization method that is related to the slope as we define it, but only
indirectly. In yoshida2017spectral  they then go on to show that their
regularization method has attractive properties.

Contributions:

  * •

We define the concept of slope to capture the geometric complexity of
regression maps.

  * •

We provide theoretical insights into properties associated with the slope,
indicating how it can be used to capture aspects of the evolution and
structure of the underlying geometry.

  * •

Additionally, we validate our theory with empirical results for ReLU networks
and show that the distribution of the slope for well-trained models is close
to invariant to the width of the hidden layers in fully connected network, and
that the mean of the slope distribution only has a weak dependence on the
model architecture for both fully connected and convolutional networks.

##  2 Notation and definitions

###  2.1 Preliminaries

A neural network $f:\mathbb{R}^{n_{0}}\rightarrow\mathbb{R}^{n_{c}}$ will for
us consist of

  * •

a sequence of positive integers $n_{0},n_{1},...,n_{n}=n_{c}$, where
$n_{1},\dots,n_{n}$ denote the width of the hidden layers,

  * •

for $i=1,...,n$, an $(n_{i}\times n_{i-1})$-dimensional matrix $W_{i}$ and a
vector $b_{i}$ of length $n_{i}$, and

  * •

a continuous activation function $g:{\mathbb{R}}\rightarrow{\mathbb{R}}$
applied separately to each dimension.

We define $f^{0}(x)=x$ and for $i=1,\dots,n-1$ a continuous map
$f^{i}:{\mathbb{R}}^{n_{0}}\to{\mathbb{R}}^{n_{i}}$ by setting

$f^{i}(x)=g(W_{i}f^{i-1}(x)+b_{i})$

while we set $f(x)=f^{n}(x)=W_{n}f^{n-1}(x)+b_{n}$. To use the network for
classification, we apply the $\operatorname{softmax}$ function to $f(x)$ to
produce an output which can be interpreted as a probability distribution on
the set of $n_{c}$ classes.

We are mainly concerned with neural networks with activation functions given
by the Rectified Linear Units (ReLU) DBLP:conf/icml/NairH10 , referred to as
ReLU networks. Then

$g(x)=\max(0,x).$

For ReLU networks we can additionally define the concept of an activation
region, the largest open connected sets $R\subset\mathbb{R}^{n_{0}}$ where $f$
can be represented as an affine transformation $f(x)=W_{R}x+b_{R},\forall x\in
R$. These regions correspond to binary patterns indicating which neurons that
are activated when passing an input through the network
DBLP:conf/nips/HaninR19 . While these regions possess many interesting
properties, for our purposes we will mainly use that for $x$ in an activation
region $R$, the Jacobian $J_{f}(x)=W_{R}$ is constant and will thus be denoted
by $J_{f}(R)$.

###  2.2 Slope

The central geometric property we will study in this paper is the slope. All
proofs are relegated to the Appendix.

######  Definition 1.

Given a continuous function $f:{\mathbb{R}}^{n_{s}}\to{\mathbb{R}}^{n_{c}}$
and some $p$ with $1\leq p\leq\infty$, we define its slope (or p-slope) at
$x\in{\mathbb{R}}^{n_{s}}$ as

$\operatorname{Slope}_{f}(x)=\sup_{v\in B^{*}}\left(\lim_{t\downarrow
0}\frac{||f(x+tv)-f(x)||_{p}}{t}\right)$

where $||\cdot||_{p}$ denotes the
p-norm111$||x||_{p}=\left(\sum_{i}|x_{i}|^{p}\right)^{1/p}$, the limit is
taken over positive $t$, and

$B^{*}=\\{v\in{\mathbb{R}}^{n_{s}}:||v||_{p}=1\\}.$

The slope is undefined unless the limit exists for all $v\in B^{*}$.

######  Proposition 1.

If the Jacobian $J_{f}(x)$ exists at $x$, then

$\operatorname{Slope}_{f}(x)=\max_{v\in
B^{*}}||J_{f}(x)v||_{p}=||J_{f}(x)||_{p}.$

Here $||J_{f}(x)||_{p}$ denotes the the matrix p-norm of the Jacobian. Note
that when $p=2$, this is the maximum singular value of $J_{f}(x)$, also called
the spectral norm of $J_{f}(x)$. When $p=1$ it is the maximum over the columns
of $J_{f}(x)$ of the sum of the absolute values of the entries in the column.
When $p=\infty$ it is the maximum over the rows of $J_{f}(x)$ of the sum of
the absolute values of the entries in the row.

######  Proposition 2.

If $f$ is represented by a neural network where the activation function $g$ is
continuously differentiable, then the Jacobian is a continuous function. If
the Jacobian is a continuous function, then the slope is a continuous
function.

######  Proposition 3.

If the Jacobian is a continuous function and if
$\operatorname{Slope}_{f}(x)\leq K$ for all $x\in{\mathbb{R}}^{n_{0}}$ then
for all pairs of points $x,y\in{\mathbb{R}}^{n_{0}}$,

$||f(x)-f(y)||_{p}\leq K||x-y||_{p}$ (1)

If the output space has only one dimension, we see from Proposition 1 that
whenever the gradient $\triangledown f(x)$ exists at a point $x$ we have
$\operatorname{Slope}_{f}(x)=||\triangledown f(x)||_{p}$. If the function $f$
is a type of regression function adapting to data, we would expect the slope
to vary quite a bit, from zero at local extremes to larger values in between
such points.

Consider instead the case where $f(x)$ is multidimensional and the Jacobian
exists. Then we get from Proposition 1 that

$\operatorname{Slope}_{f}(x)=\max_{v\in B^{*}}||J_{f}(x)v||_{p}=\max_{v\in
B^{*}}||\triangledown(f\cdot v)(x)||_{p}=\max_{v\in
B^{*}}\operatorname{Slope}_{f\cdot v}(x).$

In other words, we can understand the slope as follows: Take the output of
$f$, project it along some direction $v$ and take the p-norm of the gradient
at $x$. Then maximize over all possible directions $v$.

If $f$ is used together with a softmax function as a classifier, we would
expect that, at all points $x$, some output coordinates are increasing while
others are decreasing. In other words, there will always be directions in the
output space where the slope in that direction is nonzero. Thus the slope as
we define it is unlikely to be zero anywhere, and is not so much connected to
local extremes as it is to the speed at which the output changes.

A consequence is that it is meaningful to study the average slope $f$. More
specifically,

######  Definition 2.

We define the slope of a network as the expectation of the slope when $x$ has
the distribution of the input data.

Note that the distribution of the input data is unknown. However, we can
estimate the quantity above by using the training data points which are a
sample from the distribution. In our results, we will see that the variation
of the slope across input points $x$ is often remarkably small, making the
concept defined above a useful one.

###  2.3 Slopes of ReLU networks

Our examples are all ReLU networks. For these, the Jacobian does not exist
everywhere, but the slope still exists. All points $x$ inside an activation
region $R$ have the same Jacobian $J_{f}(R)$, so we may define

$\operatorname{Slope}_{f}(R)=\operatorname{Slope}_{f}(x)=||J_{f}(R)||_{p}.$

######  Proposition 4.

If $f$ is represented by a ReLU network then $\operatorname{Slope}_{f}(x)$
exists for all $x\in{\mathbb{R}}^{n_{0}}$ and

$\operatorname{Slope}_{f}(x)\leq\max_{R\,:\,x\in\overline{R}}\,\,\operatorname{Slope}_{f}(R)$

where $\overline{R}$ denotes the closure of $R$.

For ReLU networks it is easy to find the Jacobian. In fact,

$J_{f}(x)=W_{n}Z_{f^{n-1}(x)}W_{n-1}\cdots Z_{f^{2}(x)}W_{2}Z_{f^{1}(x)}W_{1}$
(2)

where $Z_{f^{i}(x)}$ is a diagonal matrix having 0’s and 1’s along its
diagonal, depending on the value of $f^{i}(x)$. If $Z_{f^{i}(x)}=0$ for some
$i$ then $J_{f}(x)=0$. Let us assume below that this is not the case; we then
get $||Z_{f^{i}(x)}||_{p}=1$. In a similar way as in yoshida2017spectral  we
can take the p-norm of Equation 2 to obtain

$\displaystyle||J_{f}(x)||_{p}$ $\displaystyle\leq$
$\displaystyle||W_{n}||_{p}\cdot||Z_{f^{n-1}}(x)||_{p}\cdots||W_{2}||_{p}\cdot||Z_{f^{1}(x)}||_{p}\cdot||W_{1}||_{p}$
$\displaystyle=$
$\displaystyle||W_{n}||_{p}\cdots||W_{2}||_{p}\cdot||W_{1}||_{p}$

Further222This follows as the spectral norm is equal to the largest singular
value of $W_{i}$, while the Frobenius norm is equal to the square root of the
sum of the squares of the singular values of $W_{i}$., we have
$||W_{i}||_{2}\leq||W_{i}||_{F}$ where $||W_{i}||_{F}$ denotes the Frobenius
norm of $W_{i}$, i.e., the square root of the sum of the squares of the
entries of $W_{i}$. This shows that limiting the size of the entries of the
$W_{i}$ matrices implies limiting the $||W_{i}||_{2}$ values. In turn, we have
shown above that limiting $||W_{i}||_{p}$ for any $1\leq p\leq\infty$ implies
limiting $||J_{f}(x)||_{p}$, i.e., the slope.

However, the reverse is not the case. In fact, our conjecture is that
controlling the slope $||J_{f}(x)||_{p}$ is a much more fine-tuned and precise
way of controlling the geometry of $f$ than standard regularization.

######  Proposition 5.

Proposition 1 holds also when $f$ is a ReLU network.

Propositions 1 and 5 indicate how the slope directly connects classification
probabilities in the output space with distances in the input space. If we
somehow increase the distances in the input space with with a factor $c$, we
might expect the slopes of similarly well-trained classifiers to decrease with
the same factor $c$.

In fact, we will investigate this effect in the case of image resolutions.
Assume the resolution of the images in an image classification dataset is
changed using some algorithm. For example, images in standard datasets with
$28\times 28$ resolution might be rescaled to a $56\times 56$ resolution,
multiplying the total number of dimensions by 4. The exact change in the
Euclidean distances between images will depend on the rescaling algorithm
used, but as a rough estimate we may assume that the change is the same as the
change of distances between independent points with a standard normal
distribution when the dimension is multiplied by 4. Using Lemma 2 in the
Appendix we get that Euclidean distances between such points are doubled. We
will compare this with empirical observations in Section 4.3.

##  3 Slopes and learning

Let us start with some theory:

######  Proposition 6.

Assume we have a ReLU network $f(x)=(f_{1}(x),\dots,f_{n}(x))$ followed by a
softmax classifier into $n$ categories. If the network classifies a data point
$x$ correctly, the term in the loss corresponding to $x$ will decrease if $f$
is replaced by $cf$ where $c>1$ is a constant.

######  Proposition 7.

For any ReLU network there exists at least one vector $v$ in the parameter
space such that the gradient in the direction of $v$ corresponds to
multiplying the network map $f$ with a constant $c$.

Finally, notice that for any network map $f$ and $c>0$ we have
$\operatorname{Slope}_{cf}(x)=c\operatorname{Slope}_{f}(x)$.

Now, assume we are learning the parameters of a particular network, and have
reached a "good model". Then, generally, most points will be correctly
classified. It is then reasonable to expect that changes to the parameters
along a vector like those described in the previous proposition will on
average lead to a decline in the loss. Of course, there will often be many
vectors along which the loss declines. However, if the training is continued
for an unlimited number of epochs, the directions described in the previous
theorem may become dominant. Thus, the training will lead to larger and larger
slopes.

It is a well-studied issue with the types of neural networks we are studying
here that training tends to lead to larger and larger parameters. Two
important methods to control this effect are regularization and batch
normalization. Regularization may attempt to limit the growth of the values in
the $W_{i}$ and $b_{i}$ parameters. Batch normalization re-centers and re-
scales data values between layers. We saw in the previous section how
regularization also controls the 2-slope. A similar argument can be made
regarding batch normalization. However, we conjecture that controlling the
geometric complexity of $f$ using measures defined in terms of $f$ (such as
slope) should yield better and more precise results than using measures that
depend on the particular neural network representation of $f$, such as
standard regularization and batch normalization.

###  3.1 Finding a well trained model

Naïve attempts to build a classifier $f$ may try to predict all points in the
training data as well as possible, i.e., one may focus only on minimizing the
loss as much as possible. This will lead to functions $f$ whose complexity
tend to increase without bound as the amount of data increases. As discussed
in the previous section this is connected to an ever-increasing slope for $f$.
It is also a recipe for overfitting.

In this paper we take the Bayesian viewpoint that the information content in
the training data is not big enough to build a model that classifies perfectly
on the training set and optimally on validation and test sets. Instead, one
should aim for functions $f$ that weigh loss minimization against $f$ being
"reasonable", in some sense, as a classifier. Successful classification
algorithms avoid overfitting in a multitude of ways: By restricting the set of
allowed functions $f$, by regularization that prioritizes "reasonable" $f$, by
using network architectures that lead to gradients pointing toward
"reasonable" $f$, by various stochastic mechanisms introducing noise, and by
stopping the training process based on carefully chosen criteria.

In this paper, we select, for each of a set of datasets and models that have
been seen to produce fitted models with good classification accuracy on test
sets. For each dataset, we train these models a multitude of times, producing
a sequence of classification functions $f_{1},\dots,f_{k}$. Following the
language of the paragraph above, these functions will have been produced by
limiting the "unreasonableness" of the classification function in slightly
different ways. However, our hypothesis is that these ways are sufficiently
similar, and related to the specific geometric property we are studying, that
we can also detect that the functions $f_{1},\dots,f_{k}$ have similar slope
properties.

This empirical investigation is performed by investigating the slope (measured
with $||\cdot||_{2}$ for convenience) of networks trained on MNIST, KMNIST and
FashionMNIST. Additionally, in order to include a non-image data set, we
investigate the slope of networks trained on the Forest Cover data set. Due to
time computational constraints, we do not work with the full Forest Cover data
set but instead work with a random subset of 10000 data points, which are
further split into 8000 training and 2000 validation points.

For each of these models we run Stochastic Gradient Descent (SGD) with a
momentum of 0.8, batch size of 64 and learning rate of 0.001 for 150 epochs
and the optimal model during training is chosen as the one obtained at the
epoch where the validation loss was the lowest. This setup and hyperparameters
are chosen so that the training proceeds long enough to give an accurate
picture of the evolution of the slope, while ensuring that all models can be
trained to yield accurate classifications. Unless it is otherwise mentioned,
in all subsequent experiments we summarize the slope of $f$ into a single
number by computing the average slope over 750 training data points chosen at
random. All error-bars are obtained as the standard deviation of the slope
over 5 separate runs.

##  4 Results

###  4.1 Consistently increasing slopes

A first observation is that the slope is monotonously increasing during the
larger extent of the training period. This can be seen in Figure 1 where the
evolution of the slope during training is shown for a variety of fully
connected and convolutional networks. Each curve represents the evolution of
the slope for one unique model (the exact info of the considered models can be
found in Appendix A.2). The convolutional networks are trained on MNIST,
FashionMNIST, KMNIST while the fully connected network is additionally trained
on the Forest Cover data set.

This continuous increase of the geometric complexity indicates that similar
gradient directions to that of Proposition 7 control the majority of the
training evolution. It can also be seen that convolutional networks generally
seem to reach higher slopes earlier than fully connected networks. An
explanation may be that as the convolutional networks are adapted to the image
analysis problem at hand, the training goes faster, i.e., takes fewer epochs,
than for fully connected networks.

The behaviour of an increasing slope is also visible in a different form in
Figure 2 where the slope distribution at initialization and for the optimal
models obtained for the Forest Cover data set can be seen. From these results
it is clear that the slope of a well performing model is generally higher than
that at initialization. Here it can also be seen that the distribution of the
slope for the optimal model is roughly invariant to the width of the layers in
the network, seen by the alignment of the estimated distributions as the layer
width changes. While the distribution of the slope changes as more hidden
layers are added, the mean of the distributions is relatively invariant to the
model architecture.

Figure 1: Increasing slope during training. Each curve represents the
evolution of the slope when training one unique network for 150 epochs on
either MNIST, FashionMNIST, KMNIST or the Forest Cover data set. The orange
lines detail the evolution for convolutional networks while the blue detail it
for fully connected networks. The convolutional networks are only trained on
MNIST, FashionMNIST and KMNIST. Figure 2: Violinplot of how the slope changes
from initialization to the optimal model for the Forest Cover data set. Each
estimated distribution is created by measuring the slope on 750 random
training points. Similar plots are obtained for the other considered data sets
and convolutional networks.

###  4.2 Slopes are consistent across different SGD simulations and different
network architectures

In order to ensure that the slope contains information of the underlying
geometry and that different runs of SGD produce functions $f$ with consistent
slopes, we perform several repeated runs with different random seeds and
measure the average slope and variance for the optimal model.

The effect on the distributions for the optimal models when using different
random seeds can be seen in Figure 3. It can be seen that the random seed has
a minor effect on the shape of the estimated distribution for KMNIST, while
for MNIST there are some minor discrepancies in the alignment of the
distributions. This minor discrepancy can be expected given the stochastic
nature of the SGD algorithm and that piecewise linear functions can locally
change slope quickly without having a major effect on the overall behaviour of
the function. While not shown, the effect of the random seed on the slope
distribution for the optimal models trained on FashionMNIST and Forest Cover
are similar to that of the effect on KMNIST.

Figure 3: The estimated slope distributions for KMNIST (top) and MNIST
(bottom) for two different random seeds are shown to measure the stability of
the distribution of the slope to variations of the seed used for SGD.

The results when only considering the effect of the seed on the mean of the
distributions can be seen in Figure 4 where the network structure is varied
and the mean slope for the optimal model is recorded. These results show the
stability of the slope in spite of random fluctuations in the learning
algorithm, but they also indicate that for some data sets there might be a
small range of slopes where the model will perform well, and that this range
is almost independent of the network structure.

There are however intriguing differences in slopes between fully connected and
convolutional networks. The slope for the FashionMNIST data set seems to be of
larger magnitude for the convolutional models while for MNIST and opposite
effect can be observed.

Generally, as we observed in Figure 1, training of convolutional networks uses
fewer iterations to reach functions $f$ with higher slopes. Depending on the
specifics of the dataset, this may mean that the "well fitted model" as
defined in our computation is reached at an $f$ with a higher or lower slope
compared to the fully connected case. Further investigation of this effect is
needed.

Figure 4: Experiment to measure the stability of the slope mean to variations
in the structure for fully connected (top) and convolutional networks
(bottom). Each curve represents how the slope mean varies as the model
architecture is changed.

###  4.3 Slopes decrease with increasing image resolution

In order to understand the relation between the slope and distance between
input points, we devise an experiment where we use bi-linear interpolation to
increase the resolution of images in KMNIST, MNIST and FashionMNIST and
investigate how the slope varies for the optimal classifiers. This setup moves
input points further away from each other while it can be simultaneously
argued that complexity of the classification task is preserved. The results of
the experiment when increasing the image resolution from 28x28 to 84x84 can be
seen below in Figure 5.

While the decrease is relatively linear for all three data sets, it can be
seen that the decrease in slope deviates from the ideal hypothesized factor of
2 from the argument following Proposition 5. This deviation is likely to stem
from that the assumed normality required for Lemma 2 does not fully capture
how the true distances between input points vary, but instead only provides a
rough approximation.

Figure 5: Relation between the slope and the distance between input points.
The image resolution is increased through bi-linear interpolation.

###  4.4 Local variation in slope

While the above experiments mainly consider the global properties associated
with the slope, it can also be of interest to see how the slope varies
locally. This is explored below in Figure 6 where we choose 250 randomly
chosen training points in FashionMNIST and sample points on concentric spheres
with increasing radii and measure the relative difference between the slope
for the sampled points on the spheres and the slope of the training point at
the center of the sphere. The variation of the slope on each radii is
summarized by sampling 500 points on each sphere and computing the relative
difference to the slope at the center of the sphere. For ease of displaying
the results, this is only performed for a fully connected network with 3
hidden layers, each of width 200 and the procedure is performed for the
optimal network parameters, but similar results do hold for other network
architectures. In the figure it can be seen that the relative slope difference
is small and increasing for all radii and training points. The discontinuous
nature of the slope for ReLU networks is not immediately visible but instead
the slope exhibits a smooth and almost continuous change as the distance is
increased.

Figure 6: Experiment to measure the stability of the slope to local
perturbations. Each curve is created by choosing a random training point on
FashionMNIST and sampling points on concentric spheres with increasing radii
around that point. The mean of the absolute slope difference between the
sampled points and the training point is recorded for each radii. On each
circle 250 points are sampled to estimate the difference in slope. Similar
results are obtained for the other data sets.

##  5 How can the slope be used?

We have established that well fitted classification neural networks (i.e.
having a good accuracy and limited overfitting) for a given dataset have in
common similar slope properties. Thus these can be associated with the
geometry of a good classifier for the problem. A number of uses of this can be
imagined.

One possibility is to view slope as a property that should be limited. In
other words, any increase in the slope should be weighed against a decrease in
the loss. From a Bayesian point of view one may imagine using a prior for $f$
computed from its slope. Equivalently, one might use a regularization term in
the optimization computed from the slope. We aim to explore this possibility
in a later paper. The explanation that good classifiers for a particular
problem have similar slopes would be that this slope represents a good
compromise between loss minimization and generalizability.

Another possibility is to directly target a particular slope interval for a
particular classification problem. One might first use large neural networks
to establish a reasonable slope for a good classifier for a problem. Then one
might use this target slope when optimizing other types of networks, for
example smaller networks, or networks with particular properties such as
robustness.

A third possibility is that the network slope of a fitted network can be used
as a measure of the "complexity" of a classification problem. Note that this
measure would generally be different from measures based on the accuracy
obtained using a particular neural network.

##  6 Discussion

We have established the concept of slope for a neural network, together with
some attractive theoretical properties. In examples, we have shown that the
slope often does not vary much throughout the input space, that it varies
smoothly, and that it tends to increase during training. In examples, we have
shown that the distribution of the slope for well trained fully connected
networks is almost invariant to the width of the hidden layers. Some
characteristics of the distribution seems to be dependant on the number of
hidden layers, but the mean of the distribution is fairly constant to these
changes. There additionally seems to be some dependence on whether a fully
connected or convolutional architecture is used, and further study is needed
to determine the cause of this.

Our examples have been using quite simple classification problems, such as
MNIST and FashionMNIST, together with quite small neural networks. Further
study is needed to determine how general our conclusions are when increasing
the size of the problem and the networks.

Additionally, while the theory holds for general values of $p$, all empirical
results only consider the slope for $p=2$. While the results are conjectured
to generalize to other values of $p$, this should be verified in future
studies.

Nonetheless, the general idea that the generalization properties of a neural
network function $f$ is determined by its geometrical properties seems
supported, and the slope seems to be an example of such a geometric property.
This opens up a number of interesting usages, both theoretical in connection
with understanding neural network generalization properties, and practical, in
terms of controlling the slopes of networks.

## References

  * [1] Peter L. Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian.  Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks.  J. Mach. Learn. Res., 20:63:1–63:17, 2019.
  * [2] Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová.  Modelling the influence of data structure on learning in neural networks.  CoRR, abs/1909.11500, 2019.
  * [3] Boris Hanin and David Rolnick.  Deep relu networks have surprisingly few activation patterns.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 359–368, 2019.
  * [4] Shao-Lun Huang, Xiangxiang Xu, Lizhong Zheng, and Gregory W. Wornell.  An information theoretic interpretation to deep neural networks.  In IEEE International Symposium on Information Theory, ISIT 2019, Paris, France, July 7-12, 2019, pages 1984–1988. IEEE, 2019.
  * [5] Na Lei, Zhongxuan Luo, Shing-Tung Yau, and Xianfeng David Gu.  Geometric understanding of deep learning.  CoRR, abs/1805.10451, 2018.
  * [6] Junjie Li and Ding Liu.  Information bottleneck theory on convolutional neural networks.  Neural Process. Lett., 53(2):1385–1400, 2021.
  * [7] Vinod Nair and Geoffrey E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In Johannes Fürnkranz and Thorsten Joachims, editors, Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pages 807–814. Omnipress, 2010.
  * [8] Guillermo Ortiz-Jiménez, Itamar Franco Salazar-Reque, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard.  A neural anisotropic view of underspecification in deep learning.  CoRR, abs/2104.14372, 2021.
  * [9] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [10] Kenneth H. Rosen.  Discrete Mathematics and Its Applications.  McGraw-Hill, Inc., USA, 3rd edition, 1996.
  * [11] Naftali Tishby and Noga Zaslavsky.  Deep learning and the information bottleneck principle.  In 2015 IEEE Information Theory Workshop, ITW 2015, Jerusalem, Israel, April 26 - May 1, 2015, pages 1–5. IEEE, 2015.
  * [12] Yuichi Yoshida and Takeru Miyato.  Spectral norm regularization for improving the generalizability of deep learning.  arXiv preprint arXiv:1705.10941, 2017.
  * [13] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
  * [14] Pan Zhou and Jiashi Feng.  Understanding generalization and optimization performance of deep cnns.  In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 5955–5964. PMLR, 2018\.

##  Appendix A Appendix

###  A.1 Proofs

###### Proof.

(Proof of proposition 1) Assuming that the Jacobian $J_{f}(x)$ exists we have

$\lim_{t\downarrow 0}\frac{||f(x+tv)-f(x)||_{p}}{t}=\lim_{t\downarrow
0}||J_{f}(x)v+o(1)||_{p}=||J_{f}(x)v||_{p}$

where $||v||=1$. The slope is thus reduced to

$\operatorname{Slope}_{f}(x)=\sup_{v\in B^{*}}||J_{f}(x)v||_{p}$

which by definition is the p-norm of the matrix $J_{f}(x)$. [10]. ∎

###### Proof.

(Proof of Proposition 2) The Jacobian can be constructed using the chain rule
and differentiation at every layer. The differentiations are either of linear
functions or of the activation functions, thus yielding continuous results.
The first part follows. The second part follows from the continuity of the
matrix p-norm.

∎

###### Proof.

(Proof of proposition 3) Assume the Jacobian is a continuous function
everywhere and let $x,y\in{\mathbb{R}}^{n_{0}}$. Defining
$g(t)=f(x+t(y-x))-f(x)$ for $t\in[0,1]$ we get

$f(y)-f(x)=\int_{0}^{1}g^{\prime}(t)\,dt=\left(\int_{0}^{1}J_{f}(x+t(y-x))\,dt\right)(y-x).$

Note that the integrals are taken component-wise and that the right-hand side
of the equation is a matrix multiplied with a vector. Applying the p-norm to
this equation and using the definition of the matrix p-norm we get

$||f(y)-f(x)||_{p}\leq\left|\left|\int_{0}^{1}J_{f}(x+t(y-x))\,dt\right|\right|_{p}||y-x||_{p}.$
(3)

The p-norm of a matrix is a convex function, so by Jensen’s inequality we get

$\left|\left|\int_{0}^{1}J_{f}(x+t(y-x))\,dt\right|\right|_{p}\leq\int_{0}^{1}||J_{f}(x+t(y-x))||_{p}\,dt.$
(4)

If the Jacobian exists everywhere then $\operatorname{Slope}_{f}(x+t(y-x))\leq
K$ implies $||J_{f}(x+f(y-x))||_{p}\leq K$. Putting this together with
Equations 3 and 4 we get the desired result.

∎

To prove Proposition 4, we first prove the following Lemma:

######  Lemma 1.

Let $f$ be a ReLU network. For any $x\in{\mathbb{R}}^{n_{0}}$ and $v\in B^{*}$
there exists a region $R_{x,v}$ and an $\epsilon_{x,v}>0$ such that
$x+vt\in\overline{R_{x,v}}$ for all $t\in[0,\epsilon_{x,v}]$.

###### Proof.

Note that the regions $R$ are associated with each component of $f^{i}$ in
every layer being non-zero, i.e, $f_{j}^{i}(x)\neq 0,\forall(i,j)$. If for any
$(i,j)$ we have $f_{j}^{i}(x)=0$ then $x$ does not lie in a region, but
instead on the "edge" of a region. For a given $x$, we thus get a separation
of the neurons into three sets, the active neurons
$\mathcal{A}=\\{(i,j)\in\mathbbm{N}\times\mathbbm{N}:f_{j}^{i}(x)>0\\}$, the
inactive neurons
$\mathcal{I}=\\{(i,j)\in\mathbbm{N}\times\mathbbm{N}:f_{j}^{i}(x)<0\\}$ and
the edge neurons
$\mathcal{E}=\\{(i,j)\in\mathbbm{N}\times\mathbbm{N}:f_{j}^{i}(x)=0\\}$.

If $x$ lies inside a region then the proof is straightforward, so assume that
passing $x$ through the network gives rise to the sets
$\mathcal{A},\mathcal{I},\mathcal{E}$ with $\mathcal{E}\neq\varnothing$.
First, we will show that $x$ lies in the closure of the regions given by
distributing the elements of $\mathcal{E}$ to either $\mathcal{A}$ or
$\mathcal{I}$, i.e, the closure of the regions associated with the active set
$\mathcal{A}_{\mathcal{E}}$ and the inactive set $\mathcal{I}_{\mathcal{E}}$
where we have that
$e\in\mathcal{A}_{\mathcal{E}}\bigcup\mathcal{I}_{\mathcal{E}},\forall
e\in\mathcal{E}$ and
$\mathcal{A}\subset\mathcal{A}_{\mathcal{E}},\mathcal{I}\subset\mathcal{I}_{\mathcal{E}}$.
Due to the piecewise linearity of $f$, we have that locally the set
$\\{x\in\mathbb{R}^{n_{0}}:f^{i}_{j}(x)=0,(i,j)\in\mathcal{E}\\}$ is given by
an intersection of hyperplanes. Each edge neuron has an associated hyperplane
and each hyperplane has an associated normal vector $n_{i}$, which when we
move in that direction will switch that neuron from an edge neuron to an
active or inactive neuron. There thus exists constants $c_{i}\neq
0,i=1,..,|\mathcal{E}|$, a region $R$ and $\epsilon>0$ such that the point
$x+\sum_{i=1}^{|\mathcal{E}|}tc_{i}v_{i}\in R$ for all $t\in(0,\epsilon]$.
Consequently, the point $x$ lies in $\overline{R}$.

If the vector $v$ is not given as a linear combination of the normal vectors
to the planes but instead points in a general direction, then for $\epsilon>0$
we can say that the vector $x+\epsilon v$ yields a new set of active
$\mathcal{A}_{x,v,\epsilon}$, inactive $\mathcal{I}_{x,v,\epsilon}$ and edge
neurons $\mathcal{E}_{x,v,\epsilon}$. Due to the continuity of ReLU networks,
for small $\epsilon>0$ we have that
$\mathcal{A}\subset\mathcal{A}_{x,v,\epsilon}$,
$\mathcal{I}\subset\mathcal{I}_{x,v,\epsilon}$ and consequently
$\mathcal{E}_{x,v,\epsilon}\subset\mathcal{E}$. Thus only some edge neurons
switched to either an active or inactive state. Since by redistributing the
elements of $\mathcal{E}$ and $\mathcal{E}_{x,v,\epsilon}$ to active or
inactive states we can obtain the same sets of active and inactive neurons, we
get that from the argument above that there is a region $R$ such that the
point $x$ and $x+tv$ lie in $\overline{R}$ for all $t\in(0,\epsilon]$, from
which the desired statement follows. ∎

###### Proof.

(Proof of Proposition 4) If $x$ and $y$ are both in a region $R$, then

$f(y)-f(x)=J_{f}(R)(y-x).$

By continuity of $f$ and convexity of $R$ this is also true when $x$ and $y$
are in the closure $\overline{R}$. Using this together with Lemma 1 we get

$\displaystyle\operatorname{Slope}_{f}(x)$ $\displaystyle=$
$\displaystyle\sup_{v\in B^{*}}\left(\lim_{x\downarrow
0}\frac{||f(x+vt)-f(x)||_{p}}{t}\right)$ $\displaystyle=$
$\displaystyle\sup_{v\in B^{*}}\left(\lim_{t\downarrow
0}\frac{||J(R_{x,v})tv||_{p}}{t}\right)$ $\displaystyle=$
$\displaystyle\sup_{v\in B^{*}}||J(R_{x,v})v||_{p}$ $\displaystyle\leq$
$\displaystyle\max_{R:x\in\overline{R}}\,\,\sup_{v\in B^{*}}||J(R)v||_{p}$
$\displaystyle=$
$\displaystyle\max_{R:x\in\overline{R}}\,\,\operatorname{Slope}_{f}(R)$

∎

###### Proof.

(Proof of Proposition 5) If $f$ is piecewise linear then we construct the
function $h:[0,1]\rightarrow\mathbb{R}^{n_{c}}$ as $h(t)=f(x(1-t)+ty)$. Assume
that $h$ is constructed by $m$ different linear sections. Then there exists
$m$ intervals $I_{j}=[t_{j},t_{j+1}]$ with $\bigcup_{j=1}^{m}I_{j}=[0,1]$ such
that $h$ restricted to interval $j$ is a linear function in a region $R_{j}$.
The desired inequality can then be obtained by an application of the triangle
inequality as follows,

$\displaystyle||f(x)-f(y)||_{p}=||h(0)-h(1)||_{p}$
$\displaystyle\leq\sum_{j=1}^{m}||h(t_{j+1})-h(t_{j})||_{p}$ (5)
$\displaystyle=\sum_{j=1}^{m}||J_{f}(R_{j})(y-x)(t_{j+1}-t_{j})||_{p}$ (6)
$\displaystyle\leq\sum_{j=1}^{m}||J_{f}(R_{j})||_{p}||y-x||_{p}(t_{j+1}-t_{j})$
(7) $\displaystyle\leq\sum_{j=1}^{m}K(t_{j+1}-t_{j})||x-y||_{p}$ (8)
$\displaystyle=K||x-y||_{p}.$ (9)

∎

###### Proof.

(Proof of Proposition 6) The relevant term in the loss function is

$-\log\frac{\exp(f_{i}(x))}{\sum_{j}\exp(f_{j}(x))}$

where $i$ is the class $x$ is classified into. If $f$ classifies this point
correctly we have that $f_{i}(x)>f_{j}(x)$ for all $j\neq i$. Thus, for any
$c>1$, $c(f_{j}(x)-f_{i}(x))<f_{j}(x)-f_{i}(x)$. Summing over $j=1,\dots,n$ we
get

$\sum_{j}c(f_{j}(x)-f_{i}(x))<\sum_{j}f_{j}(x)-f_{i}(x)$

which is equivalent to

$-\log\frac{\exp(cf_{i}(x))}{\sum_{j}\exp(cf_{j}(x))}<-\log\frac{\exp(f_{i}(x))}{\sum_{j}\exp(f_{j}(x))}.$

This shows that the loss at $x$ decreases, as claimed. ∎

###### Proof.

(Proof of proposition 7) Choose the vector $v$ such that a step in the
direction of $v$ multiplies the weights $W_{i}$ and bias $b_{i}$ at layer $i$
and the bias at all subsequent layers $j>i$ with a constant $c>0$. For
$i=1,...,n$ this step will create a new sequence of functions $f^{i}_{c}$.
Since no change has been made to parameters in layers $k<i$ we have that
$f^{i}_{c}=f^{i}$. For $k\geq i$ we obtain

$f^{k}_{c}(x)=\begin{cases}\operatorname{ReLU}(cW_{k}f^{k-1}(x)+cb_{k})=cf^{k}(x),~{}~{}\textrm{for
$k=i$}\\\
\operatorname{ReLU}(W_{k}cf^{k-1}(x)+cb_{k})=cf^{k}(x),~{}~{}\textrm{for
$k>i$}\end{cases}$

Thus we see that $f^{n}_{c}=cf^{n}=cf$ as desired. ∎

######  Lemma 2.

When $x,y\sim\operatorname{Normal}_{n}(0,I)$ then, approximately when $n$ is
large, $||x-y||\sim\operatorname{Normal}(\sqrt{2n},1)$.

###### Proof.

We get $(x-y)/\sqrt{2}\sim\operatorname{Normal}(0,I)$ and
$||x-y||^{2}/2\sim\chi^{2}_{n}$, so as a first-order approximation when $n$ is
large,

$\frac{1}{2}||x-y||^{2}\sim\operatorname{Normal}(n,2n)$

and as a further approximation when $n$ is large,

$||x-y||\sim\operatorname{Normal}(\sqrt{2n},1).$

∎

###  A.2 Experimental details

All experiments are carried out in PyTorch [9] and the code can be obtained on
github333github.com/antonFJohansson/slope_and_generalization . Every network
is trained for 150 epochs and 750 random training points are chosen at the
beginning of the training where the slope is subsequently measured at every
epoch. To connect the slope with the generalization properties of the models
we need to obtain a model that has a good generalization capability, i.e, we
want to obtain an "optimal" model. This is done by choosing the optimal model
to be the model at the epoch where the lowest validation loss was obtained
when training the network for 150 epochs. Every network was trained with
Stochastic Gradient Descent with a learning rate of 0.001, momentum of 0.8,
batch size of 64 and every experiment is repeated 5 times.

The considered fully connected model architectures are obtained by varying the
number of hidden layers as 1,2,3 and varying the number of neurons in the
hidden layers (every hidden layer has the same number of neurons) as
25,50,100,200.

For the convolutional model architectures we vary the number of hidden layers
as 3,6 and the number of channels (all hidden layers has the same number of
channels) as 3,6,12,24. After the convolutional layer the feature
representation is flattened and fed through a fully connected layer to the
final output layer. All convolutional layers uses padding such that the
feature representation maintains the same shape throughout the network.

###  A.3 Info regarding the optimal models

The validation accuracies and validation losses for all of the optimal models,
both convolutional and fully connected, can be found in Table 1 \- 4. All
tables contain the mean and standard deviation of 5 runs. While some models
achieve a higher accuracy/lower loss than others, it can be argued that all
models generalize.

| MNIST | KMNIST | FashionMNIST | Forest Cover
---|---|---|---|---
(1, 25) | $0.965\pm 1.13e-03$ | $0.819\pm 3.00e-03$ | $0.869\pm 1.72e-03$ | $0.718\pm 4.60e-03$
(1, 50) | $0.975\pm 8.15e-04$ | $0.856\pm 2.16e-03$ | $0.876\pm 1.62e-03$ | $0.721\pm 2.62e-03$
(1, 100) | $0.979\pm 1.85e-04$ | $0.878\pm 2.19e-03$ | $0.882\pm 1.39e-03$ | $0.723\pm 2.18e-03$
(1, 200) | $0.981\pm 7.93e-04$ | $0.891\pm 7.52e-04$ | $0.887\pm 5.95e-04$ | $0.728\pm 2.08e-03$
(2, 25) | $0.965\pm 8.84e-04$ | $0.822\pm 2.64e-03$ | $0.869\pm 2.02e-03$ | $0.722\pm 1.56e-03$
(2, 50) | $0.973\pm 5.04e-04$ | $0.852\pm 2.07e-03$ | $0.876\pm 5.12e-04$ | $0.727\pm 3.41e-03$
(2, 100) | $0.977\pm 5.84e-04$ | $0.875\pm 2.80e-03$ | $0.878\pm 6.69e-04$ | $0.730\pm 2.12e-03$
(2, 200) | $0.980\pm 8.93e-04$ | $0.886\pm 2.49e-03$ | $0.883\pm 1.39e-03$ | $0.734\pm 2.58e-03$
(3, 25) | $0.963\pm 1.24e-03$ | $0.815\pm 3.84e-03$ | $0.864\pm 4.36e-03$ | $0.724\pm 5.11e-03$
(3, 50) | $0.971\pm 7.88e-04$ | $0.846\pm 4.88e-03$ | $0.872\pm 1.40e-03$ | $0.725\pm 3.50e-03$
(3, 100) | $0.975\pm 6.91e-04$ | $0.863\pm 3.25e-03$ | $0.876\pm 8.08e-04$ | $0.732\pm 4.27e-03$
(3, 200) | $0.978\pm 7.91e-04$ | $0.877\pm 2.88e-03$ | $0.879\pm 1.67e-03$ | $0.739\pm 3.98e-03$
Table 1: Validation accuracies for the optimal fully connected models for each data set. | MNIST | KMNIST | FashionMNIST | Forest Cover
---|---|---|---|---
(1, 25) | $0.0019\pm 4.97e-05$ | $0.0097\pm 9.81e-05$ | $0.0058\pm 7.17e-05$ | $0.0106\pm 3.68e-05$
(1, 50) | $0.0013\pm 4.39e-05$ | $0.0079\pm 8.03e-05$ | $0.0055\pm 3.34e-05$ | $0.0105\pm 6.94e-05$
(1, 100) | $0.0011\pm 3.09e-05$ | $0.0067\pm 7.18e-05$ | $0.0053\pm 2.73e-05$ | $0.0104\pm 4.54e-05$
(1, 200) | $0.0010\pm 1.61e-05$ | $0.0060\pm 4.32e-05$ | $0.0051\pm 2.57e-05$ | $0.0103\pm 3.88e-05$
(2, 25) | $0.0019\pm 4.87e-05$ | $0.0098\pm 2.99e-04$ | $0.0058\pm 6.17e-05$ | $0.0105\pm 9.92e-05$
(2, 50) | $0.0014\pm 2.82e-05$ | $0.0082\pm 2.80e-05$ | $0.0055\pm 3.31e-05$ | $0.0103\pm 7.35e-05$
(2, 100) | $0.0012\pm 5.71e-05$ | $0.0070\pm 1.39e-04$ | $0.0054\pm 2.72e-05$ | $0.0101\pm 6.72e-05$
(2, 200) | $0.0010\pm 3.29e-05$ | $0.0064\pm 1.19e-04$ | $0.0053\pm 5.91e-05$ | $0.0100\pm 3.60e-05$
(3, 25) | $0.0020\pm 4.42e-05$ | $0.0103\pm 2.74e-04$ | $0.0061\pm 1.65e-04$ | $0.0105\pm 7.99e-05$
(3, 50) | $0.0016\pm 4.50e-05$ | $0.0086\pm 2.00e-04$ | $0.0058\pm 1.01e-04$ | $0.0103\pm 6.18e-05$
(3, 100) | $0.0013\pm 2.80e-05$ | $0.0076\pm 2.09e-04$ | $0.0056\pm 4.08e-05$ | $0.0101\pm 6.51e-05$
(3, 200) | $0.0012\pm 3.34e-05$ | $0.0069\pm 4.06e-05$ | $0.0055\pm 2.60e-05$ | $0.0099\pm 4.22e-05$
Table 2: Validation losses for the optimal fully connected models for each data set. | MNIST | KMNIST | FashionMNIST
---|---|---|---
(3, 3) | $0.980\pm 1.20e-03$ | $0.851\pm 5.88e-03$ | $0.889\pm 3.32e-03$
(3, 6) | $0.983\pm 1.24e-03$ | $0.879\pm 6.71e-03$ | $0.900\pm 2.94e-03$
(3, 12) | $0.985\pm 1.13e-03$ | $0.899\pm 3.31e-04$ | $0.903\pm 2.13e-03$
(3, 24) | $0.986\pm 9.83e-04$ | $0.907\pm 1.34e-03$ | $0.908\pm 6.95e-04$
(6, 3) | $0.983\pm 9.60e-04$ | $0.862\pm 8.26e-03$ | $0.893\pm 2.12e-03$
(6, 6) | $0.984\pm 1.84e-03$ | $0.886\pm 3.11e-03$ | $0.898\pm 4.92e-03$
(6, 12) | $0.985\pm 1.56e-03$ | $0.896\pm 6.14e-03$ | $0.900\pm 2.45e-03$
(6, 24) | $0.985\pm 8.50e-04$ | $0.904\pm 4.00e-04$ | $0.900\pm 2.46e-03$
Table 3: Validation accuracies for the optimal convolutional networks for each data set. | MNIST | KMNIST | FashionMNIST
---|---|---|---
(3, 3) | $0.0010\pm 1.05e-04$ | $0.0087\pm 2.20e-04$ | $0.0050\pm 1.44e-04$
(3, 6) | $0.0008\pm 7.39e-05$ | $0.0070\pm 2.33e-04$ | $0.0045\pm 1.56e-04$
(3, 12) | $0.0007\pm 3.35e-05$ | $0.0061\pm 1.16e-04$ | $0.0044\pm 1.06e-04$
(3, 24) | $0.0006\pm 4.04e-05$ | $0.0056\pm 1.77e-04$ | $0.0043\pm 6.18e-05$
(6, 3) | $0.0009\pm 6.79e-05$ | $0.0079\pm 5.92e-04$ | $0.0048\pm 1.94e-04$
(6, 6) | $0.0008\pm 8.20e-05$ | $0.0070\pm 1.61e-04$ | $0.0045\pm 2.35e-04$
(6, 12) | $0.0008\pm 4.61e-05$ | $0.0064\pm 3.00e-04$ | $0.0045\pm 9.90e-05$
(6, 24) | $0.0007\pm 2.98e-05$ | $0.0058\pm 1.85e-04$ | $0.0045\pm 8.23e-05$
Table 4: Validation losses for the optimal convolutional models for each data
set


* Chung . (2021) chung2021beyondChung, Y., Neiswanger, W., Char, I. Schneider, J.  2021\.  Beyond pinball loss: Quantile methods for calibrated uncertainty quantification Beyond pinball loss: Quantile methods for calibrated uncertainty quantification.  Advances in Neural Information Processing Systems3410971–10984.
  * Cobb . (2018) CobbGal2018LossCobb, A.D., Roberts, S.J. Gal, Y.  2018\.  Loss-Calibrated Approximate Inference in Bayesian Neural Networks Loss-Calibrated Approximate Inference in Bayesian Neural Networks.  Theory of deep learning workshop, ICML. Theory of deep learning workshop, ICML.
  * Cui . (2020) cui2020calibratedCui, P., Hu, W. Zhu, J.  2020\.  Calibrated Reliable Regression using Maximum Mean Discrepancy Calibrated reliable regression using maximum mean discrepancy.  Advances in Neural Information Processing Systems33.
  * DeGroot Fienberg (1983) degroot1983DeGroot, M.H. Fienberg, S.E.  1983\.  The Comparison and Evaluation of Forecasters The Comparison and Evaluation of Forecasters.  Journal of the Royal Statistical Society. Series D (The Statistician)321/212–22.
  * Ding . (2021) ding2021localDing, Z., Han, X., Liu, P. Niethammer, M.  2021\.  Local temperature scaling for probability calibration Local temperature scaling for probability calibration.  Proceedings of the IEEE/CVF International Conference on Computer Vision Proceedings of the ieee/cvf international conference on computer vision ( 6889–6899).
  * Fagerland . (2008) fagerland2008multinomialFagerland, M.W., Hosmer, D.W. Bofin, A.M.  2008\.  Multinomial goodness-of-fit tests for logistic regression models Multinomial goodness-of-fit tests for logistic regression models.  Statistics in Medicine27214238-4253.  https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.3202  https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.3202 https://doi.org/10.1002/sim.3202
  * Fasiolo . (2020) fasiolo2020fastFasiolo, M., Wood, S.N., Zaffran, M., Nedellec, R. Goude, Y.  2020\.  Fast calibrated additive quantile regression Fast calibrated additive quantile regression.  Journal of the American Statistical Association1–11.
  * Fawcett Niculescu-Mizil (2007) fawcett2007Fawcett, T. Niculescu-Mizil, A.  2007\.  PAV and the ROC convex hull PAV and the ROC convex hull.  Machine Learning68197–106.
  * P. Flach (2012) flach2012machineFlach, P.  2012\.  Machine learning: the art and science of algorithms that make sense of data Machine learning: the art and science of algorithms that make sense of data.  Cambridge University Press.
  * P. Flach (2014) flach2014classificationFlach, P.  2014\.  Classification in context: Adapting to changes in class and cost distribution Classification in context: Adapting to changes in class and cost distribution.  First international workshop on learning over multiple contexts at European conference on machine learning and principles and practice of knowledge discovery in databases (ECML-PKDD). First international workshop on learning over multiple contexts at European conference on machine learning and principles and practice of knowledge discovery in databases (ECML-PKDD).
  * P.A. Flach Kull (2015) flach2015Flach, P.A. Kull, M.  2015\.  Precision-Recall-Gain Curves: PR Analysis Done Right Precision-Recall-Gain Curves: PR Analysis Done Right.  Advances in Neural Information Processing Systems (NIPS’15) Advances in neural information processing systems (nips’15) ( 838–846).
  * Ghandeharioun . (2019) ghandeharioun2019characterizingGhandeharioun, A., Eoff, B., Jou, B. Picard, R.  2019\.  Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data Bias Characterizing sources of uncertainty to proxy calibration and disambiguate annotator and data bias.  2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) 2019 ieee/cvf international conference on computer vision workshop (ICCVW) ( 4202–4206).
  * Gneiting Raftery (2007) gneiting2007Gneiting, T. Raftery, A.E.  2007\.  Strictly proper scoring rules, prediction, and estimation Strictly proper scoring rules, prediction, and estimation.  Journal of the American Statistical Association102477359–378.
  * Gretton . (2012) gretton2012kernelGretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B. Smola, A.  2012\.  A kernel two-sample test A kernel two-sample test.  The Journal of Machine Learning Research131723–773.
  * Guillory . (2021) guillory2021predictingGuillory, D., Shankar, V., Ebrahimi, S., Darrell, T. Schmidt, L.  2021\.  Predicting with confidence on unseen distributions Predicting with confidence on unseen distributions.  Proceedings of the IEEE/CVF International Conference on Computer Vision Proceedings of the ieee/cvf international conference on computer vision ( 1134–1144).
  * Guo . (2017) guo2017Guo, C., Pleiss, G., Sun, Y. Weinberger, K.Q.  2017\.  On Calibration of Modern Neural Networks On Calibration of Modern Neural Networks.  34th International Conference on Machine Learning 34th international conference on machine learning ( 1321–1330).  Sydney, Australia.
  * Hagedorn . (2005) hagedorn2005Hagedorn, R., Doblas-Reyes, F.J. Palmer, T.N.  2005\.  The rationale behind the success of multi-model ensembles in seasonal forecasting — I. Basic concept The rationale behind the success of multi-model ensembles in seasonal forecasting — I. Basic concept.  Tellus A: Dynamic Meteorology and Oceanography573219–233.
  * Hallenbeck (1920) hallenbeck1920Hallenbeck, C.  1920\.  Forecasting Precipitation in Percentages of Probability. Forecasting Precipitation in Percentages of Probability.  Monthly Weather Review4811645–647.
  * Hinton . (2015) hinton2015distillingHinton, G., Vinyals, O., Dean, J. .  2015\.  Distilling the knowledge in a neural network Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.0253127.
  * Hosmer Lemeshow (2003) Hosmer2003appliedHosmer, D. Lemeshow, S.  2003\.  Applied Logistic Regression Applied logistic regression.  Wiley.
  * Hüllermeier . (2022) hullermeier2022quantifyingHüllermeier, E., Destercke, S. Shaker, M.H.  202201–05 Aug.  Quantification of Credal Uncertainty in Machine Learning: A Critical Analysis and Empirical Comparison Quantification of credal uncertainty in machine learning: A critical analysis and empirical comparison.  J. Cussens K. Zhang (), Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence Proceedings of the thirty-eighth conference on uncertainty in artificial intelligence ( 180, 548–557).  PMLR.  https://proceedings.mlr.press/v180/hullermeier22a.html
  * Hüllermeier Waegeman (2021) hullermeier2021aleatoricHüllermeier, E. Waegeman, W.  2021\.  Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  Machine Learning1103457–506.
  * Jiang . (2021) jiang2021assessingJiang, Y., Nagarajan, V., Baek, C. Kolter, J.Z.  2021\.  Assessing generalization of sgd via disagreement Assessing generalization of sgd via disagreement.  arXiv preprint arXiv:2106.13799.
  * Johansson . (2018) johansson2018Johansson, U., Löfström, T., Sundell, H., Linusson, H., Gidenstam, A. Boström, H.  2018\.  Venn predictors for well-calibrated probability estimation trees Venn predictors for well-calibrated probability estimation trees.  A. Gammerman, V. Vovk, Z. Luo, E. Smirnov R. Peeters (), Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications Proceedings of the seventh workshop on conformal and probabilistic prediction and applications ( 3–14).  PMLR.
  * Koenker Hallock (2001) koenker2001quantileKoenker, R. Hallock, K.F.  2001\.  Quantile regression Quantile regression.  Journal of economic perspectives154143–156.
  * Kristiadi . (2020) kristiadi2020beingKristiadi, A., Hein, M. Hennig, P.  2020\.  Being bayesian, even just a bit, fixes overconfidence in relu networks Being bayesian, even just a bit, fixes overconfidence in relu networks.  International Conference on Machine Learning International conference on machine learning ( 5436–5446).
  * Kuleshov Deshpande (2022) kuleshov2022calibratedKuleshov, V. Deshpande, S.  2022\.  Calibrated and sharp uncertainties in deep learning via density estimation Calibrated and sharp uncertainties in deep learning via density estimation.  International Conference on Machine Learning International conference on machine learning ( 11683–11693).
  * Kuleshov . (2018) kuleshov2018Kuleshov, V., Fenner, N. Ermon, S.  2018\.  Accurate uncertainties for deep learning using calibrated regression Accurate uncertainties for deep learning using calibrated regression.  35th International Conference on Machine Learning, ICML 2018 35th international conference on machine learning, ICML 2018 ( 6, 4369–4377).  International Machine Learning Society (IMLS).
  * Kuleshov Liang (2015) kuleshov2015calibratedKuleshov, V. Liang, P.S.  2015\.  Calibrated structured prediction Calibrated structured prediction.  Advances in Neural Information Processing Systems283474–3482.
  * Kull Flach (2015) kull2015novelKull, M. Flach, P.  2015\.  Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration.  Joint European Conference on Machine Learning and Knowledge Discovery in Databases Joint european conference on machine learning and knowledge discovery in databases ( 68–85).
  * Kull . (2019) kull2019Kull, M., Perello-Nieto, M., Kängsepp, M., Filho, T.S., Song, H. Flach, P.  2019\.  Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration.  Advances in Neural Information Processing Systems (NeurIPS’19) Advances in neural information processing systems (NeurIPS’19) ( 12316–12326).
  * Kull, Silva Filho Flach (2017) pmlr-v54-kull17aKull, M., Silva Filho, T. Flach, P.  201720–22 Apr.  Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers.  A. Singh J. Zhu (), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics Proceedings of the 20th international conference on artificial intelligence and statistics ( 54, 623–631).  Fort Lauderdale, FL, USAPMLR.
  * Kull, Silva Filho Flach (2017) kull2017beyondKull, M., Silva Filho, T.M. Flach, P.  2017\.  Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta calibration Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta calibration.  Electronic Journal of Statistics1125052–5080.
  * Kumar . (2019) kumar2019Kumar, A., Liang, P. Ma, T.  2019\.  Verified Uncertainty Calibration Verified Uncertainty Calibration.  Advances in Neural Information Processing Systems (NeurIPS’19). Advances in neural information processing systems (NeurIPS’19).
  * Kumar . (2018) kumar2018Kumar, A., Sarawagi, S. Jain, U.  201810–15 Jul.  Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings.  J. Dy A. Krause (), 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research 35th international conference on machine learning, volume 80 of proceedings of machine learning research ( 80, 2805–2814).  Stockholmsmässan, Stockholm SwedenPMLR.
  * Kängsepp . (2022) kangsepp2022usefulnessKängsepp, M., Valk, K. Kull, M.  2022\.  On the Usefulness of the Fit-on-the-Test View on Evaluating Calibration of Classifiers. On the usefulness of the fit-on-the-test view on evaluating calibration of classifiers.  arXiv.  https://arxiv.org/abs/2203.08958  10.48550/ARXIV.2203.08958
  * Lachiche Flach (2003) lachiche2003improvingLachiche, N. Flach, P.A.  2003\.  Improving accuracy and cost of two-class and multi-class probabilistic classifiers using ROC curves Improving accuracy and cost of two-class and multi-class probabilistic classifiers using roc curves.  Proceedings of the 20th International Conference on Machine Learning (ICML-03) Proceedings of the 20th international conference on machine learning (ICML-03) ( 416–423).
  * Leathart . (2017) leathart2017Leathart, T., Frank, E., Holmes, G. Pfahringer, B.  2017jul.  Probability Calibration Trees Probability Calibration Trees.  Asian Conference on Machine Learning Asian conference on machine learning ( 145–160).
  * Lichtenstein . (1977) lichtenstein1977calibrationLichtenstein, S., Fischhoff, B. Phillips, L.D.  1977\.  Calibration of probabilities: The state of the art Calibration of probabilities: The state of the art.  Decision making and change in human affairs Decision making and change in human affairs ( 275–324).  Springer.
  * Lichtenstein . (1982) lichtenstein1982Lichtenstein, S., Fischhoff, B. Phillips, L.D.  1982\.  Calibration of probabilities: The state of the art to 1980 Calibration of probabilities: The state of the art to 1980.  Judgment under Uncertainty: Heuristics and Biases Judgment under uncertainty: Heuristics and biases ( 306–334).  Cambridge University Press.
  * J.Z. Liu . (2019) liu2019accurateLiu, J.Z., Paisley, J., Kioumourtzoglou, MA. Coull, B.  2019\.  Accurate uncertainty estimation and decomposition in ensemble learning Accurate uncertainty estimation and decomposition in ensemble learning.  arXiv preprint arXiv:1911.04061.
  * S. Liu . (2018) liu2018generalizedLiu, S., Long, M., Wang, J. Jordan, M.I.  2018\.  Generalized zero-shot learning with deep calibration network Generalized zero-shot learning with deep calibration network.  Advances in Neural Information Processing Systems Advances in neural information processing systems ( 2005–2015).
  * Ma Blaschko (2021) ma2021metaMa, X. Blaschko, M.B.  2021\.  Meta-cal: Well-controlled post-hoc calibration by ranking Meta-cal: Well-controlled post-hoc calibration by ranking.  International Conference on Machine Learning International conference on machine learning ( 7235–7245).
  * Menon Williamson (2018) menon2018lossMenon, A.K. Williamson, R.C.  2018\.  A loss framework for calibrated anomaly detection A loss framework for calibrated anomaly detection.  Proceedings of the 32nd International Conference on Neural Information Processing Systems Proceedings of the 32nd international conference on neural information processing systems ( 1494–1504).
  * Milios . (2018) milios2018Milios, D., Michiardi, P., Rosasco, L. Filippone, M.  2018\.  Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification.  Advances in Neural Information Processing Systems (NIPS’18) Advances in neural information processing systems (NIPS’18) ( 6005–6015).
  * Miller (1962) miller1962statisticalMiller, R.G.  1962\.  Statistical Prediction by Discriminant Analysis Statistical prediction by discriminant analysis.  Statistical Prediction by Discriminant Analysis Statistical prediction by discriminant analysis ( 1–54).  Boston, MAAmerican Meteorological Society.
  * Müller . (2019) muller2019doesMüller, R., Kornblith, S. Hinton, G.  2019\.  When does label smoothing help? When does label smoothing help?  arXiv preprint arXiv:1906.02629.
  * Murphy Winkler (1977) murphy1977reliabilityMurphy, A.H. Winkler, R.L.  1977\.  Reliability of Subjective Probability Forecasts of Precipitation and Temperature Reliability of subjective probability forecasts of precipitation and temperature.  Journal of the Royal Statistical Society. Series C (Applied Statistics)26141–47.
  * M.P. Naeini Cooper (2016) naeini2016Naeini, M.P. Cooper, G.F.  2016\.  Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models.  IEEE 16th International Conference on Data Mining (ICDM) IEEE 16th international conference on data mining (ICDM) ( 360–369).  Institute of Electrical and Electronics Engineers (IEEE).
  * P. Naeini . (2015) naeini2015Naeini, P., Cooper, G.F. Hauskrecht, M.  2015\.  Obtaining Well Calibrated Probabilities Using Bayesian Binning Obtaining Well Calibrated Probabilities Using Bayesian Binning.  29th AAAI Conference on Artificial Intelligence 29th AAAI conference on artificial intelligence ( 29).
  * Niculescu-Mizil Caruana (2005) niculescu-mizil2005Niculescu-Mizil, A. Caruana, R.  2005\.  Predicting good probabilities with supervised learning Predicting good probabilities with supervised learning.  22nd International Conference on Machine Learning (ICML’05) 22nd international conference on machine learning (ICML’05) ( 625–632).  New York, New York, USAACM Press.
  * Ovadia . (2019) ovadia2019canOvadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S.Snoek, J.  2019\.  Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift.  Advances in Neural Information Processing Systems 32 Advances in neural information processing systems 32 ( 32).
  * Patel . (2021) patel2021multiclassPatel, K., Beluch, W.H., Yang, B., Pfeiffer, M. Zhang, D.  2021\.  Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning Multi-class uncertainty calibration via mutual information maximization-based binning.  International Conference on Learning Representations. International conference on learning representations.  https://openreview.net/forum?id=AICNpd8ke-m
  * Perello-Nieto . (2021) pycalibPerello-Nieto, M., Song, H., Silva Filho, T. Kängsepp, M.  202108\.  PyCalib a library for classifier calibration. Pycalib a library for classifier calibration.  Zenodo.  https://doi.org/10.5281/zenodo.5518877  If you use this software, please cite it as below.  10.5281/zenodo.5518877
  * J. Platt (2000) platt2000Platt, J.  2000\.  Probabilities for SV Machines Probabilities for SV Machines.  A.J. Smola, P. Bartlett, B. Schölkopf D. Schuurmans (), Advances in Large-Margin Classifiers Advances in large-margin classifiers ( 61–74).  MIT Press.
  * J.C. Platt (1999) platt99probabilisticoutputsPlatt, J.C.  1999\.  Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  Advances in large margin classifiers Advances in large margin classifiers ( 61–74).  MIT Press.
  * Pleiss . (2017) pleiss2017fairnessPleiss, G., Raghavan, M., Wu, F., Kleinberg, J. Weinberger, K.Q.  2017\.  On fairness and calibration On fairness and calibration.  arXiv preprint arXiv:1709.02012.
  * Roelofs . (2021) roelofs2021mitigatingRoelofs, R., Cain, N., Shlens, J. Mozer, M.C.  2021\.  Mitigating Bias in Calibration Error Estimation Mitigating bias in calibration error estimation.  arXiv preprint arXiv:2012.08668.
  * Romano . (2019) romano2019conformalizedRomano, Y., Patterson, E. Candes, E.  2019\.  Conformalized quantile regression Conformalized quantile regression.  Advances in Neural Information Processing Systems323543–3553.
  * Sahoo . (2021) sahoo3reliableSahoo, R., Zhao, S., Chen, A. Ermon, S.  2021\.  Reliable Decisions with Threshold Calibration Reliable decisions with threshold calibration.  Threshold33–5.
  * Sanders (1963) sanders1963Sanders, F.  1963apr.  On Subjective Probability Forecasting On Subjective Probability Forecasting.  Journal of Applied Meteorology22191–201.
  * Savage (1971) savage1971Savage, L.J.  1971\.  Elicitation of personal probabilities and expectations Elicitation of personal probabilities and expectations.  Journal of the American Statistical Association66336783–801.
  * Senge . (2014) senge2014reliableSenge, R., Bösner, S., Dembczyński, K., Haasenritter, J., Hirsch, O., Donner-Banzhoff, N. Hüllermeier, E.  2014\.  Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty.  Information Sciences25516–29.
  * Shaker Hüllermeier (2021) shaker2021ensembleShaker, M.H. Hüllermeier, E.  2021\.  Ensemble-based Uncertainty Quantification: Bayesian versus Credal Inference. Ensemble-based uncertainty quantification: Bayesian versus credal inference.  arXiv.  https://arxiv.org/abs/2107.10384  10.48550/ARXIV.2107.10384
  * Song . (2019) pmlr-v97-song19aSong, H., Diethe, T., Kull, M. Flach, P.  201909–15 Jun.  Distribution calibration for regression Distribution calibration for regression.  K. Chaudhuri R. Salakhutdinov (), 36th International Conference on Machine Learning 36th international conference on machine learning ( 97, 5897–5906).  Long Beach, California, USAPMLR.
  * Thulasidasan . (2019) thulasidasan2019mixupThulasidasan, S., Chennupati, G., Bilmes, J., Bhattacharya, T. Michalak, S.  2019\.  On mixup training: Improved calibration and predictive uncertainty for deep neural networks On mixup training: Improved calibration and predictive uncertainty for deep neural networks.  arXiv preprint arXiv:1905.11001.
  * Tran . (2019) tran2019calibratingTran, GL., Bonilla, E.V., Cunningham, J., Michiardi, P. Filippone, M.  2019\.  Calibrating deep convolutional gaussian processes Calibrating deep convolutional gaussian processes.  The 22nd International Conference on Artificial Intelligence and Statistics The 22nd international conference on artificial intelligence and statistics ( 1554–1563).
  * Vaicenavicius . (2019) vaicenavicius2019evaluatingVaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J. Schön, T.  2019\.  Evaluating model calibration in classification Evaluating model calibration in classification.  The 22nd International Conference on Artificial Intelligence and Statistics The 22nd international conference on artificial intelligence and statistics ( 3459–3467).
  * Vovk . (2005) vovk2005algorithmicVovk, V., Gammerman, A. Shafer, G.  2005\.  Algorithmic learning in a random world Algorithmic learning in a random world.  Springer Science & Business Media.
  * Vovk Petej (2012) vovk2012vennVovk, V. Petej, I.  2012\.  Venn-abers predictors Venn-abers predictors.  arXiv preprint arXiv:1211.0025.
  * Vovk . (2003) vovk2003selfVovk, V., Shafer, G. Nouretdinov, I.  2003\.  Self-calibrating probability forecasting Self-calibrating probability forecasting.  Advances in neural information processing systems16.
  * Wang . (2019) wang2019calibratingWang, Y., Li, L. Dang, C.  2019\.  Calibrating classification probabilities with shape-restricted polynomial regression Calibrating classification probabilities with shape-restricted polynomial regression.  IEEE transactions on pattern analysis and machine intelligence4181813–1827.
  * Wen . (2020) wen2020batchensembleWen, Y., Tran, D. Ba, J.  2020\.  Batchensemble: an alternative approach to efficient ensemble and lifelong learning Batchensemble: an alternative approach to efficient ensemble and lifelong learning.  arXiv preprint arXiv:2002.06715.
  * Wenger . (2020) wenger2020Wenger, J., Kjellström, H. Triebel, R.  2020\.  Non-parametric calibration for classification Non-parametric calibration for classification.  International Conference on Artificial Intelligence and Statistics International conference on artificial intelligence and statistics ( 178–190).
  * Widmann . (2019) widmann2019calibrationWidmann, D., Lindsten, F. Zachariah, D.  2019\.  Calibration tests in multi-class classification: A unifying framework Calibration tests in multi-class classification: A unifying framework.  Advances in Neural Information Processing Systems Advances in neural information processing systems ( 12257–12267).
  * Widmann . (2022) widmann2022calibrationWidmann, D., Lindsten, F. Zachariah, D.  2022\.  Calibration tests beyond classification. Calibration tests beyond classification.  arXiv.  https://arxiv.org/abs/2210.13355  10.48550/ARXIV.2210.13355
  * Winkler (1969) winkler1969scoringWinkler, R.L.  1969\.  Scoring Rules and the Evaluation of Probability Assessors Scoring rules and the evaluation of probability assessors.  Journal of the American Statistical Association643271073–1078.
  * Zadrozny Elkan (2001) zadrozny2001obtainingZadrozny, B. Elkan, C.  2001\.  Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers.  ICML ICML ( 1, 609–616).
  * Zadrozny Elkan (2002) zadrozny2002Zadrozny, B. Elkan, C.  2002\.  Transforming Classifier Scores into Accurate Multiclass Probability Estimates Transforming Classifier Scores into Accurate Multiclass Probability Estimates.  8th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD ’02 8th ACM SIGKDD international conference on knowledge discovery and data mining - KDD ’02 ( 694–699).  New York, New York, USAACM Press.
  * Zhang . (2020) zhang2020mixZhang, J., Kailkhura, B. Han, T.YJ.  2020\.  Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning.  International Conference on Machine Learning International conference on machine learning ( 11117–11128).