# Decorrelating neurons using persistence

Rubén Ballester
Departament de Matemàtiques i Informàtica
Universitat de Barcelona
Gran Via de les Corts Catalanes, 585, 08007 Barcelona, Catalonia, Spain
<EMAIL_ADDRESS>
&Carles Casacuberta
Departament de Matemàtiques i Informàtica
Universitat de Barcelona
Gran Via de les Corts Catalanes, 585, 08007 Barcelona, Catalonia, Spain
<EMAIL_ADDRESS>
&Sergio Escalera
Departament de Matemàtiques i Informàtica
Universitat de Barcelona
Gran Via de les Corts Catalanes, 585, 08007 Barcelona, Catalonia, Spain
<EMAIL_ADDRESS>

###### Abstract

We propose a novel way to improve the generalisation capacity of deep learning
models by reducing high correlations between neurons. For this, we present two
regularisation terms computed from the weights of a minimum spanning tree of
the clique whose vertices are the neurons of a given network (or a sample of
those), where weights on edges are correlation dissimilarities. We provide an
extensive set of experiments to validate the effectiveness of our terms,
showing that they outperform popular ones. Also, we demonstrate that naive
minimisation of all correlations between neurons obtains lower accuracies than
our regularisation terms, suggesting that redundancies play a significant role
in artificial neural networks, as evidenced by some studies in neuroscience
for real networks. We include a proof of differentiability of our
regularisers, thus developing the first effective topological persistence-
based regularisation terms that consider the whole set of neurons and that can
be applied to a feedforward architecture in any deep learning task such as
classification, data generation, or regression.

##  1 Introduction

Neural networks have proven to be powerful models to solve complex tasks.
Usual neural networks show a high capacity to generalise properly beyond the
training dataset used to fit their parameters [47]. Although there is no
general explanation of why this happens yet, abundant literature is available
to tackle this problem [16, 19, 28, 29, 31]. Moreover, many regularisation
methods have been proposed to improve generalisation capacity from both
theoretical and practical perspectives. According to experimental results,
explicit regularisation may improve generalisation performance [47].

Evidence from neuroscience indicates that correlation among neurons is a
significant factor in the brain’s ability to encode and process information
[12, 33]. This was pointed out in [30], where it was observed that the
generalisation error of a deep network is monotonic with respect to the
correlation between weight matrices of neurons or filters, suggesting that
decreasing this correlation can be beneficial to improve the generalisation
capacity of a network. Furthermore, in [11], overfitting of neural networks
was reduced by decorrelating their neuron activations. Overall, these studies
suggest that a reduced correlation between neuron activations could improve
the robustness of a network.

However, neuroscience also suggests that redundancy appears naturally in brain
circuits and is useful to perform certain computations [20, 37]. For this
reason, aggressively minimising correlations between all activations or
weights may be detrimental for the performance of a neural network.

In this work, we propose a way to minimise only the most relevant high
correlations between neurons. For each batch of data during training, we
compute regularisation terms based on edge weights of a minimum spanning tree
of the clique generated by the most relevant neurons for the batch, based on
an importance measure inspired by the activation criterion for neural network
pruning presented in [38]. Edge weights in the clique are pairwise
correlations between activation vectors of neurons. In order to prove that our
regularisation terms are almost everywhere differentiable, we use the
differential calculus framework for persistent homology developed in [36].

###  1.1 Contributions

The main contributions of our work can be described as follows:

  1. 1.

We propose two novel regularisation terms that minimise only some of the
highest correlations of the most relevant neurons in a specific training
batch. This approach allows for some redundancy in the neural network, unlike
other articles such as [11, 30], which propose to minimise all pairwise
correlations. Each regularisation term employs a distinct method, and each of
them outperforms the other in specific networks, thereby complementing each
other.

  2. 2.

We use differentiable persistence descriptors to ensure differentiability of
our regularisation terms, thus developing, to the best of our knowledge, the
first topological regularisation terms that depend on the whole set of hidden
internal representations of the neurons of a neural network.

  3. 3.

We provide an extensive set of experiments to validate the effectiveness of
our topological regularisation terms. We also compare our regularisers with
several popular regularisation terms and find that our regularisers achieve a
better performance in the experiments.

The article is structured as follows. In Section 2, we analyse the current
approaches to regularisation of neural networks using correlations and
topological data analysis. In Section 3, we describe our topological
regularisation terms and prove that they are differentiable almost everywhere.
In Section 4, we describe the experiments performed to validate our
regularisation terms and discuss their results. In Section 5, we discuss
limitations of our approach and possible future work. Section 6 contains our
conclusions, and basic facts about differentiability of persistence
descriptors are detailed in the Appendix.

##  2 Related work

Many works in deep learning study how to apply explicit regularisation to
improve the generalisation capacity of neural network models. Popular
approaches include dropout [44], in which neurons of neural networks are
dropped randomly during training, and the classical $l_{1}$ and $l_{2}$
regularisation terms [46], that control the size of weights of a neural
network. However, many regularisation approaches have been developed in the
literature. Among them, some have used correlations between weights and
activations of neurons to regularise neural networks. In particular, [30] uses
weight correlations for convolutional and fully-connected layers to improve
deep network learning outcomes, and [11] proposes to minimise a loss function
computed from a covariance matrix of neuron activations on a batch. Both
methods, however, have limitations:

  1. 1.

In [30], the definition of weight correlation is hardcoded only for fully-
connected and convolutional layers and thus does not apply to some relevant
modern architectures.

  2. 2.

In [11], correlations between neurons are computed for a set of neurons
defined by the user. This leaves the practitioner with the need to choose the
neurons to decorrelate, which is not an easy task for large neural networks.

  3. 3.

Both articles [11, 30] reduce all correlations at the same time, without
taking into consideration that redundancy between neurons can be important.
Also, the regularisation terms proposed in these articles use only layerwise
correlations, without taking into account interactions between neurons of
different layers.

We propose two regularisation terms that, first, can be used in any
feedforward architecture and are not restricted to correlations in the same
layer, and secondly, they only decorrelate the neurons with highest
correlations, thus allowing the neural network to be flexible enough to keep
an amount of redundancy that might be useful for the task.

During the last years, the popularity of topological methods in machine
learning has rapidly increased. An overall survey of these methods can be
found in [21]. Particularly interesting for this paper is the use of
topological priors for regularising neural networks. In [6, 36], frameworks
for differential calculus on persistence barcodes were defined, allowing to
optimise point cloud shapes and thus to construct topological regularisation
terms. Thanks to the possibility of differentiating persistence diagrams,
several neural network layers have been created with the objective of
leveraging topological properties in the process of learning [5, 18, 22, 25,
32, 40]. In [8], the first regularisation term for neural networks based on
the topology of the decision region was proposed. From there, specific
topological regularisation terms have been discussed for image segmentation
[3, 10, 26, 27], autoencoder latent space [23], and classification using
decision boundaries [9].

Among the current approaches on regularising neural networks, the most similar
to our method are the ones suggested in [2, 24]. In [24], zero-dimensional
persistent homology is used to optimise the mass concentration of the internal
representations in the last hidden layer assuming that the mini-batches used
during training are equally distributed among all classes. In [2], an upper
bound of the generalisation gap of a neural network $\mathcal{N}$ was found
theoretically in terms of the persistent homology dimension of the continuous
set of weights generated during the training of $\mathcal{N}$. The higher the
dimension, the higher the upper bound on the generalisation gap, and thus it
was proposed to regularise neural networks by adding a topological term
consisting of an approximation of the persistent homology dimension of the set
of weights generated during training.

Although the existing topological regularisation approaches perform
satisfactorily when used appropiately, most of them are restricted to specific
tasks or have limitations that can be complemented with our approach, that is
fundamentally different from previous methods, for the following reasons:

  1. 1.

Topological regularisation terms based on decision boundaries or the evolution
of the whole set of weights do not consider the whole structure of a network,
that provides substantial information about the network’s generalisation [13,
14, 31, 41].

  2. 2.

Topological regularisation terms designed for specific tasks, such as
segmentation, or specific models, such as autoencoders, are too restrictive in
general scenarios.

  3. 3.

In [24] it is assumed that the batches during training have equally
distributed samples among the different labels and can be applied only to
classification tasks. For unbalanced datasets or non-classification tasks,
such assumptions need not be satisfied.

Our topological regularisation terms based on correlations of network neurons
are agnostic to the problem (regression, classification, generative models,
etc.) and compatible with most model types (CNNs, transformer-based, MLPs,
etc.), and they work _on the whole structure of the network_ by means of
neuron activations.

##  3 Methodology

The presence of high correlations between neurons in a neural network may
imply the existence of redundant features learned from the data. This becomes
particularly significant when two neurons, denoted as $\nu_{1}$ and $\nu_{2}$,
exhibit a Pearson correlation coefficient equal to $\pm 1$. In such cases,
there exist real numbers $a\neq 0$ and $b$ such that $\nu_{1}=a\nu_{2}+b$ with
probability one. Consequently, one of the neurons can be eliminated, resulting
in a smaller neural network that represents the same function as the original
neural network almost everywhere. This observation suggests that excessively
high correlations between neurons restrict the network’s capacity to fully
utilise its expressivity for learning the problem, highlighting the need to
prevent such behaviour.

Figure 1: Sequence of steps needed to compute the proposed regularisation
terms $\mathcal{T}_{1}$ and $\mathcal{T}_{2}$. In the first step, the neural
network is fed with the training batch and the activations for each neuron are
computed. Then, a set of neurons per hidden layer (maroon nodes) is chosen.
For the importance sampling algorithm described in the methodology, we take a
fixed percentage of $0.5$% of neurons per hidden layer and the whole set of
neurons in the output layer. In the second step, the persistence diagram —that
is, a multiset of weights of the edges of a minimum spanning tree— is computed
(red edges). Finally, in the third step, a suitable real-valued function
$\mathcal{R}$ from the weights of the minimum spanning tree is computed. By
fixing batch examples, the computation of the real value given by
$\mathcal{R}$ using the previous steps is differentiable almost everywhere
with respect to neural network parameters due to the chain rule and Theorem 1.

However, we conjecture that fully avoiding correlated features may be
detrimental for learning tasks. This is because  (1) it imposes hard
restrictions to the weights of neural networks during training, and  (2) there
is evidence from neuroscience suggesting that correlation is beneficial in
brain operation [20, 37].  In this paper, we propose to generate a balanced
amount of correlation between neurons by reducing only some of the highest
ones. To do this, we use dimension zero persistent homology to build two
regularisation terms that work in a complementary way.

For a thorough introduction to persistent homology, we refer the reader to the
book [17] and to the survey [21]. To a finite set $X$ and a symmetric function
$d\colon X\times X\to\mathbb{R}_{\geq 0}$ such that $d(x,x)=0$ for all $x\in
X$ one associates a _persistence diagram_ in every homological dimension
greater than or equal to zero as described e.g. in [17] (details are also
given in the Appendix).

In this work we only use persistent homology in dimension zero. For this
reason, since points in a zero-dimensional persistence diagram are aligned
along the positive $y$ axis, we only focus on their $y$-coordinates. Hence we
associate to each pair $(X,d)$ as above a finite multiset $D(X,d)$ of positive
real numbers, ignoring the point at infinity. Such numbers are the weights of
the edges of a minimum spanning tree of the undirected weigthed graph
$(V,E,w)$ with $V=X$ and $E=\left\\{\\{u,v\\}:u,v\in V\right\\}$, with weights
$w(\\{u,v\\})=d(u,v)$. This graph is a _clique_ (or a _complete graph_) since
every pair of vertices is joined by an edge, and a _minimum spanning tree_
(MST) is a subgraph without cycles containing all the vertices with the
minimum possible total edge weight.

Given a zero-dimensional persistence diagram $D(X,d)$, we use the framework
for differential calculus of [36] to build differentiable regularisation terms
depending on $D(X,d)$. In particular, we are interested in maximizing the
overall values of persistence diagrams for pairs $(X,d)$ where $X$ represents
a set of neurons and $d$ is a dissimilarity function measuring correlations
between them.

Let us denote by $\mathcal{N}\colon\mathcal{X}\to\mathcal{Y}$ a feedforward
neural network, and let
$\mathcal{D}=\left\\{(x_{1},y_{1}),\ldots,(x_{n},y_{n})\right\\}\subseteq\mathcal{X}\times\mathcal{Y}$
be a dataset —in our case, batches of the training dataset. Let
$G_{\mathcal{N}}$ be the clique graph of the network $\mathcal{N}$, whose
vertices are the neurons of $\mathcal{N}$. As we do not know the marginal
distribution of the data on $\mathcal{X}$, we approximate the correlation
between two neurons seen as random variables
$u,v\colon\mathcal{X}\to\mathbb{R}$ using the sample correlation for the
neuron activations in the dataset $\mathcal{D}$ restricted to the inputs.
Recall that the _sample correlation_ of two vectors $x,y\in\mathbb{R}^{n}$ is
defined as

$\text{corr}(x,y)=\frac{\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}\sqrt{\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}},$

where $\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_{i}$ and
$\bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_{i}$. Thus, the sample correlation between
two neurons $u$ and $v$ given $\mathcal{D}$ is
$\text{corr}_{\mathcal{D}}(u,v)\triangleq\text{corr}\left(u(\mathcal{D}),v(\mathcal{D})\right)$
where $u(\mathcal{D})\triangleq(u(x_{1}),\ldots,u(x_{n}))$ and
$v(\mathcal{D})\triangleq(v(x_{1}),\ldots,v(x_{n}))$.

In our case, neurons are considered to be similar when they share a large
correlation in absolute value. Since correlations take values between $-1$ and
$1$, we define the _correlation dissimilarity_ between neurons as a function
$d\colon V(G_{\mathcal{N}})\times V(G_{\mathcal{N}})\to[0,1]$ given by
$d(u,v)=1-\left|\text{corr}_{\mathcal{D}}(u,v)\right|$. Given this
dissimilarity function, we can study correlations between any subset of
neurons $V^{\prime}\subseteq V(G_{\mathcal{N}})$ by means of the persistence
diagram $D(V^{\prime},d)$.

By the cut property of minimum spanning trees, each MST of a clique contains,
for each of its vertices, at least one edge with the minimum weight among its
incident edges. In our case, this is translated into the fact that the diagram
$D(V^{\prime},d)$ contains the value

$1-\max_{u,v\in V^{\prime},\,u\neq
v}\left|\text{corr}_{\mathcal{D}}(u,v)\right|$

among, possibly, other high correlations to form the MST. Therefore, by
maximising the values of $D(V^{\prime},d)$, we are in fact minimising a set of
correlations between neurons in $V^{\prime}$ containing the highest
correlations achieved by neurons in the set.

Current neural networks contain an enormous quantity of neurons and computing
a MST of the weighted clique $(V(G_{\mathcal{N}}),d)$ is not feasible in many
cases, as computing a minimum spanning tree has a complexity of
$\mathcal{O}(e\cdot\alpha(e,v))$ [7, Theorem 1.1], where $\alpha(e,v)$ is the
functional inverse of Ackermann’s function [45] and $e$ and $v$ are the number
of edges and vertices, respectively, with $e=\binom{v}{2}$ because the graph
is a clique.

For this reason, we consider, for each batch
$\mathcal{B}=\\{(x_{1},y_{1}),\ldots,(x_{\left|\mathcal{B}\right|},y_{\left|\mathcal{B}\right|})\\}$
during training, a subset of neurons $V_{\mathcal{B}}\subseteq
V(G_{\mathcal{N}})$ that may have smaller cardinality than
$V(G_{\mathcal{N}})$. In particular, for the cases in which we cannot set
$V_{\mathcal{B}}=V(G_{\mathcal{N}})$, we sample $V_{\mathcal{B}}$ using an
_importance sampling algorithm_ for each batch. We take the top percentage $P$
of most important neurons of each layer given the batch, except for the last
layer, where we take all the neurons, where $P$ is a hyperparameter depending
on the size of the neural network. This is because the last layer showed to
contain relevant information with respect to generalisation in other works
like [4]. For our experiments with big neural networks, we set $P$ to $0.5$%
due to practical hardware limitations.

The _importance_ of a neuron given a batch is set to the average quantity of
absolute activation achieved by the neuron, and it is inspired by the
activation criterion for pruning presented in [38, Section 2.2]. The higher
this value for a neuron is, the more relevance we allot to the neuron. More
precisely, denote
$\bar{v}_{\mathcal{B}}=\left|\mathcal{B}\right|^{-1}\sum_{i=1}^{\left|\mathcal{B}\right|}\left|v(x_{i})\right|$
and let $V(G_{\mathcal{N}})_{l}=\\{v_{1},\ldots,v_{n_{l}}\\}$ be the neurons
of the $l$-layer of $\mathcal{N}$ in any descending order of their
$\bar{v}_{\mathcal{B}}$ values. In our case, we use the order given by the
argsort function of TensorFlow. Let

$V_{\mathcal{B}}=\bigcup_{l}V_{\mathcal{B},l}\quad\text{where}V_{\mathcal{B},l}=\begin{cases}\\{v_{1},\ldots,v_{\lfloor
0.005\,n_{l}\rfloor}\\}\text{ if }l\neq L,\\\\[2.84544pt]
V(G_{\mathcal{N}})_{l}\text{ otherwise},\end{cases}$

where $l$ iterates over all possible layers of $\mathcal{N}$ and $L$ is the
number of the last layer.

Recall that maximising the values of $D(V^{\prime},d)$ is equivalent to
minimising a set of high correlations between neurons in $V^{\prime}$. By
convention, we assume that regularisation terms are minimised by network
training algorithms, so to maximise a function $f(\theta)$ we minimise its
opposite function $-f(\theta)$. We propose two regularisation terms that
maximise persistence diagram values in different ways:

$\hskip
76.82234pt\mathcal{T}_{1}(\theta)\triangleq-\\!\\!\\!\\!\\!\\!\sum_{y\in
D(V_{\mathcal{B}},d)}\\!\\!\\!\\!\\!\\!y,$ (1)
$\\!\\!\\!\\!\\!\mathcal{T}_{2}(\theta)\triangleq-\alpha\bar{D}(V_{\mathcal{B}},d)+\beta\sigma(D(V_{\mathcal{B}},d)),$
(2)

where $\alpha,\beta\in\mathbb{R}_{\geq 0}$ are weight parameters, $\theta$ is
the set of parameters of the neural network being trained, and

$\bar{D}(V_{\mathcal{B}},d)=\frac{1}{\left|D(V_{\mathcal{B}},d)\right|}\sum_{y\in
D(V_{\mathcal{B}},d)}\\!\\!\\!\\!\\!\\!y,\qquad\sigma^{2}(D(V_{\mathcal{B}},d))=\frac{1}{\left|D(V_{\mathcal{B}},d)\right|}\sum_{y\in
D(V_{\mathcal{B}},d)}\\!\\!\\!\\!\\!(y-\bar{D}(V_{\mathcal{B}},d))^{2}$

are the mean and variance of the total persistence of $D(V_{\mathcal{B}},d)$.
The regularisation term $\mathcal{T}_{1}$ given by Equation (1) maximises the
sum of values in the persistence diagram, while the regularisation term
$\mathcal{T}_{2}$ given by Equation (2) is a more involved term that focuses
on how the entries of the persistence diagram are distributed, minimising
their dispersion and maximising their average value. In our case, we pick
$\alpha=\beta=1/2$ since we treat mean and dispersion with the same strength.
Further exploration and optimisation of these hyperparameters is left for
future work.

######  Theorem 1.

Let $c,n\geq 2$ and let $d\colon\mathbb{R}^{n}\times\mathbb{R}^{n}\to[0,1]$ be
given by $d(x,y)=1-\left|\textnormal{corr}(x,y)\right|$ where corr denotes
correlation. There exists an open dense subset
$\mathfrak{D}_{c,n}\subseteq\mathbb{R}^{cn}$ such that the functions

$\mathcal{T}_{1}(x_{1},\ldots,x_{c})\triangleq-\\!\\!\\!\\!\\!\\!\sum_{y\in
D(X,d)}\\!\\!\\!\\!\\!y,\qquad\mathcal{T}_{2}(x_{1},\ldots,x_{c})\triangleq-\alpha\bar{D}(X,d)+\beta\sigma(D(X,d))$

are $\mathcal{C}^{\infty}$ on $\mathfrak{D}_{c,n}$ for all
$\alpha,\beta\in\mathbb{R}_{\geq 0}$, where $X=\\{x_{1},\ldots,x_{c}\\}$ and
$\bar{D}(X,d)$ and $\sigma(D(X,d))$ denote average and standard deviation,
respectively, of the zero-dimensional persistence diagram $D(X,d)$.

A proof of this result is provided in the Appendix.

Using the chain rule, our regularisation terms are well defined as soon as the
neuron activations of the set of neurons
$V_{\mathcal{B}}=\\{\nu_{1},\ldots,\nu_{c}\\}$ in the batch
$\mathcal{B}=\left\\{x_{1},\ldots,x_{n}\right\\}$ form a vector

$\mathbf{v}=(\nu_{1}(x_{1}),\ldots,\nu_{1}(x_{n})),\ldots,(\nu_{c}(x_{1}),\ldots,\nu_{c}(x_{n}))\in\mathbb{R}^{cn}$

such that $\mathbf{v}\in\mathfrak{D}_{c,n}$ and such that the neuron
activations are obtained in a differentiable way. Experimentally, we need not
control when this vector $\mathbf{v}$ is inside $\mathfrak{D}_{c,n}$ thanks to
the fact that $\mathfrak{D}_{c,n}$ is a dense set. However, we note that
ignoring points where non-differentiability may occur in the domain could
introduce errors in some iterations during training, as it may also happen
with ReLU [1].

##  4 Results

In this section, we first describe the experimental setup and the
computational resources that we use to validate the hypothesis stated in the
previous section. This is done in Subsection 4.1. Then, we present and discuss
the results in Subsection 4.2. The code used to perform these experiments is
attached as supplementary material
111https://github.com/rballeba/DecorrelatingNeuronsUsingPersistence.

###  4.1 Experimental setup

We prove the plausibility of the hypotheses that we formulated in Section 3 by
making two blocks of experiments in which we train several neural networks
with different regularisation terms, including our proposed ones, and without
regularisation terms. For each block, we train several models following a
common architecture: multilayer perceptron models for the first block and VGG-
like models for the second one. The networks of the first block are trained in
the MNIST dataset whereas the networks of the second one are trained in
CIFAR-10. In both blocks, we explore the same set of weights for the
regularisation terms. Finally, to compare the accuracies of our proposed
regularisation terms to the other alternatives, we use the Friedman
statistical test with its Nemenyi post-hoc. Details of the experiments,
training procedures, and evaluation methods are provided through this section.

Multilayer perceptron experiments

In the first block of experiments, we examine our regularisation terms in a
simplified problem. We train three different multilayer perceptron
architectures with $1000$ hidden neurons, labelled 0, 1, and 2 using the MNIST
dataset [35]. Networks 0 and 1 share the same fully connected architecture.
However, network 1 is trained using dropout with a $50$% probability of
dropping a hidden neuron at each iteration. Specifically, architectures 0 and
2 have a trapezium shape consisting of a sequence of hidden layers of $450$,
$350$, and $200$ neurons for the first network, and of $300$, $250$, $200$,
$150$, and $100$ for the second one, respectively.

PGDL experiments

In the second block of experiments, the objective is to see if the method
scales properly to more complex datasets and models. We train eight different
neural network architectures from the PGDL dataset introduced in [28] during
NeurIPS 2020 competition track. The PGDL dataset is a collection of tasks
where each task is composed by one dataset and a set of different neural
network architectures trained with the dataset of the task. The eight
different neural network architectures we take belong to the first task, that
is composed of VGG-like neural networks and the CIFAR10 dataset [34]. The
architectures we selected are the ones corresponding to the numbers 20, 21,
22, 23, 148, 149, 150, and 151 from the dataset. Architectures 20, 21, 148,
and 149 are the same as the architectures 22, 23, 150, and 151, but with a
layerwise dropout probability of $0.5$, respectively. The difference between
models 22 and 23 is the width of their convolutions, where architecture 22 has
convolution widths of $256$ and architecture 23 has convolution widths of
$512$. Finally, the architectures 150 and 151 are the same as the
architectures 22 and 23 but with one more dense layer.

Training procedures

In these experiments, we train the different architectures with different
regularisation terms weighted with several values. To train the neural
networks in the different experiments, we replicate approximately the training
performed by the PGDL dataset used in the second block of experiments.

For both blocks of experiments we split the data into training, validation,
and test datasets. For the MNIST dataset, we split the original training
dataset into new training and validation datasets with $80$% and $20$% of the
original data, respectively. Finally, we use the original test dataset as test
dataset. For the CIFAR10 dataset, we split the original training dataset into
new training and validation datasets, where we choose $1000$ examples of each
class randomly for the validation dataset and we place the remaining examples
into the training dataset. Again, we reuse the original test dataset.

For the training procedure, we train for a maximum of $1200$ epochs with early
stopping after $20$ epochs of no improvement, and with a batch size of $256$.
The algorithm used for training is the usual stochastic gradient descent (SGD)
with momentum $0.9$. For the first block of experiments, we use an adaptive
learning rate $\alpha_{i}\triangleq\alpha_{0}\cdot(0.95)^{i/3520}$ where $i$
is the iteration where the learning rate $\alpha_{i}$ is used and
$\alpha_{0}=0.01$. For the second block of experiments, we use a fixed
learning rate of $0.001$, for which we obtained similar accuracies to the ones
given by the original trainings of the PGDL neural networks.

Let $\text{CCE}(\theta)$ denote the categorical cross entropy loss for a fixed
neural network, clear from the context, with a set of parameters $\theta$. For
each of the networks described before we perform several trainings with the
different regularisation terms that we study, weighed by different values. In
particular, each training minimises a loss function

$\mathcal{L}(\theta)=\text{CCE}(\theta)+\omega\mathcal{R}(\theta),$ (3)

where $\omega\in\\{10^{-6},10^{-5},10^{-4},0.001,0.01,0.1,1.5,10,100\\}$
represents one of the possible weight values used in the experiments, and
$\mathcal{R}(\theta)$ represents one of the regularisation terms. We also
train the models without any regularisation term, i.e., with $\omega=0$.

We use the full and sampled version of our regularisers $\mathcal{T}_{1}$ and
$\mathcal{T}_{2}$ for the first and second blocks of experiments,
respectively, due to the small size of the networks of the first block and to
the large size of the networks of the second one. To see if reducing only some
correlations between neurons is better than minimising all of them, we also
study the regularisation term $\mathcal{C}(\theta)$ defined as

$\mathcal{C}(\theta)=\frac{1}{|\mathfrak{C}|}\sum_{(x,y)\in\mathfrak{C}}|\text{corr}_{\mathcal{B}}(x,y)|,$
(4)

where $\mathfrak{C}=\left\\{(x,y)\in V_{\mathcal{B}}\times
V_{\mathcal{B}}:x\neq y\text{ and }\text{corr}_{\mathcal{B}}(x,y)\neq
0\right\\}$, and $V_{\mathcal{B}}$ is defined as for the terms
$\mathcal{T}_{1}$ and $\mathcal{T}_{2}$ in Section 3. Note that for the first
block of experiments we consider all the non-input neurons and for the second
block of experiments we perform the same sample of neurons due to the
computational complexity of computing all the possible pairwise correlations
for each iteration of the training. Finally, we also train the networks with
the classic $l_{1}$ and $l_{2}$ regularisation terms [46].

Evaluation procedure

To evaluate the performances of the regularisation terms compared, we use a
Friedman test with the Nemenyi post-hoc, as proposed in [15], and we report
the test accuracies for each regularisation term and network. To obtain test
accuracies, we choose, for each term and network, the weight that maximises
the validation accuracy after training. Then, we compute the test accuracy
using the selected weight. For the training procedures without regularisation
terms, we compute directly their test accuracies.

####  4.1.1 Resources used and computation

The experiments were computed in a server with 503 GB of RAM, a CPU AMD EPYC
7452 32-Core Processor with a frequency up to 3.35 GHz, and seven GPUs NVIDIA
GeForce RTX 3090 with 24 GiB of memory. The storage consisted of 3 Samsung
SSDs, two of them with 3840 GB of memory and the other one with 960 GB. All
the experiments were executed in parallel using one of the GPUs per
experiment. The computational bottlenecks were related to the computation of
correlation matrices of neurons and persistence diagrams. The first process
was done using TensorFlow (in GPU mode) and the second one was performed using
the library giotto-ph [39].

###  4.2 Results and analysis

Table 1: Nemenyi $p$-value matrix. Each row represents a different regularisation term except for the first row, that represents training without regulariser. Each cell is the $p$-value of the Nemenyi post-hoc test. The null-hypothesis of this test is that there is no difference between the accuracies yielded by the two training approaches. Those $p$-values lower than or equal to $0.05$ are bolded. $\mathcal{T}_{1}$: First topological regularisation term as in (1); $\mathcal{T}_{2}$: Second topological regularisation term, as in (2); $l_{1}$: Lasso regression regularisation term; $l_{2}$: Ridge regression regularisation term; $\mathcal{C}$: Regularisation term minimising all the pairwise correlations in $V_{\mathcal{B}}$, as in (4); $\emptyset$: no regulariser. | $\emptyset$ | $\mathcal{T}_{1}$ | $\mathcal{T}_{2}$ | $l_{1}$ | $l_{2}$ | $\mathcal{C}$
---|---|---|---|---|---|---
$\emptyset$ |  | 0.018 | 0.002 | 0.900 | 0.785 | 0.900
$\mathcal{T}_{1}$ |  |  | 0.900 | 0.036 | 0.380 | 0.159
$\mathcal{T}_{2}$ |  |  |  | 0.006 | 0.122 | 0.036
$l_{1}$ |  |  |  |  | 0.900 | 0.900
$l_{2}$ |  |  |  |  |  | 0.900
Table 2: Test accuracies for the different training procedures and networks. Each row represents a training with a different topological regularisation term except for the first row, that represents a training without a regularisation term. Each column represents a network in the PGDL dataset. Each cell is the test accuracy of the regularisation term with the weight that obtained the best validation accuracy for the term in the network specified by the column. Best accuracies per network are bolded. Rows $\mathcal{T}_{1}$, $\mathcal{T}_{2}$, $l_{1}$, $l_{2}$, $\mathcal{C}$, and $\emptyset$ are described in Table 1. | MNIST and MLP | PGDL and VGG-like
---|---|---
| 0 | 1 | 2 | 20 | 21 | 22 | 23 | 148 | 149 | 150 | 151
$\emptyset$ | 0.929 | 0.501 | 0.636 | 0.681 | 0.680 | 0.685 | 0.682 | 0.672 | 0.677 | 0.675 | 0.680
$\mathcal{T}_{1}$ | 0.928 | 0.547 | 0.883 | 0.687 | 0.705 | 0.675 | 0.700 | 0.688 | 0.704 | 0.678 | 0.698
$\mathcal{T}_{2}$ | 0.923 | 0.540 | 0.879 | 0.691 | 0.701 | 0.688 | 0.706 | 0.689 | 0.698 | 0.688 | 0.695
$l_{1}$ | 0.914 | 0.536 | 0.870 | 0.682 | 0.680 | 0.682 | 0.683 | 0.677 | 0.675 | 0.685 | 0.678
$l_{2}$ | 0.919 | 0.531 | 0.878 | 0.681 | 0.688 | 0.686 | 0.683 | 0.680 | 0.680 | 0.681 | 0.679
$\mathcal{C}$ | 0.923 | 0.530 | 0.881 | 0.679 | 0.687 | 0.680 | 0.686 | 0.678 | 0.69 | 0.683 | 0.674

Table 2 contains test accuracies for each network in the MNIST and PGDL
experiments for the different regularisation terms and the training procedure
without regularisation term. For each regularisation term and network, we
compute the test accuracy for the weight of the regularisation term that
obtained the best validation accuracy for the specific network. Using these
accuracies, the Friedman test with null hypothesis that all the training
algorithms are equivalent [15] gives a $p$-value of $1.04e-05<0.05$, so we
reject the null hypothesis. Therefore, we perform the Nemenyi post-hoc test,
obtaining the $p$-value matrix shown in Table 1.

When comparing test accuracies individually, $\mathcal{T}_{1}$ and
$\mathcal{T}_{2}$ outperform the other training methods for all the networks
except for the model 0. Concretely, $\mathcal{T}_{1}$ and $\mathcal{T}_{2}$
obtain the best test accuracies in five of the networks each, respectively,
out of eleven total networks. Due to this, there is no winner between the two
proposed regularisation terms. This is contrasted by the Nemenyi matrix, where
the $p$-value between both methods has a high value of $0.900$. Also, both of
them are significantly better than training without regularisation term,
according to the Nemenyi $p$-value matrix, obtaining $p$-values of $0.018$ and
$0.002$ for $\mathcal{T}_{1}$ and $\mathcal{T}_{2}$, respectively.

Regarding the differences between minimising all the pairwise correlations in
the set of relevant neurons and minimising only the highest ones, i.e., the
differences between the regularisation terms $\mathcal{T}_{1}$,
$\mathcal{T}_{2}$, and $\mathcal{C}$, we see that both regularisation terms
obtain low $p$-values in the Nemenyi test. In particular, for
$\mathcal{T}_{2}$ and $\mathcal{C}$ we obtain a significant low $p$-value of
$0.036$, that validates the hypothesis that minimising only the highest
correlations is better than minimising all the correlations for a sampled set
of relevant neurons in our experiments.

Concerning the classical regularisation terms, both regularisation terms
$\mathcal{T}_{1}$ and $\mathcal{T}_{2}$ obtain $p$-values lower than or equal
to $0.05$ with respect to $l_{1}$, making our regularisation terms
significantly better than $l_{1}$. As for $l_{2}$, the $p$-values of
$\mathcal{T}_{1}$ and $\mathcal{T}_{2}$ with respect to $l_{2}$ are much lower
than the other $p$-values for $l_{2}$, although not as low as for $l_{1}$.
This, together to the fact that $l_{2}$ does not obtain the best accuracy for
any network, suggests that our regularisation terms are probably better than
$l_{2}$, although more experiments are needed to confirm this claim.

Overall, we see a trend in the results that $\mathcal{T}_{1}$ and
$\mathcal{T}_{2}$ are superior to the other training approaches, which
supports the hypothesis that we stated in the Methodology.

##  5 Limitations and future work

The computational cost of determining persistence diagrams, which is a crucial
component of the pipeline to compute our proposed regularisation terms, can
pose limitations in practical scenarios that require fast training or involve
huge networks. To extend the applicability of our regularisation terms, an
efficient algorithm for computing persistence diagrams must be developed. One
approach is to leverage the computational power of GPUs, as done in the
Ripser++ package [48]. However, Ripser++ does not generate all the necessary
information required for computing gradients in differentiable persistent
homology, thus necessitating further development. Moreover, dimension zero
persistent homology represents a unique case wherein persistence diagrams can
be computed using any algorithm capable of computing minimum spanning trees of
weighted graphs. Therefore, advancements in implementing these algorithms
efficiently in a distributed or GPU-accelerated manner, as discussed in [42]
or [43], could be of considerable interest.

Other crucial bottleneck in our pipeline is the computation of pairwise
correlations. With a large number of neurons, it is impractical to compute
correlations for all neurons at each training step due to constraints in
execution time and memory. We hypothesise that neuron selection greatly
impacts the performance of regularisation terms, and thus, further study is
needed. For large neural networks, a potential solution could involve
computing regularisation terms within different clusters of neurons grouped by
the proximity of their layers, as we hypothesise that distant neurons have low
correlation due to non-linear activations applied from one layer to the other.

The last limitation is related to the theoretical guarantees of our method.
All our process is experimental, and we cannot guarantee that these
regularisers work properly in any kind of learning task, despite neuroscience
heuristic and experimental evidences. Further work to understand how
correlations affect the generalisation capacity of neural networks is needed.

$1$$2$$3$$4$$5$$6$$\mathcal{T}_{2}$$\mathcal{T}_{1}$$l_{2}$$\mathcal{C}$$l_{1}$$\emptyset$
Figure 2: Critical difference diagrams [15] for the Friedman and Nemenyi post-
hoc statistical tests conducted in both blocks of experiments. The position of
each training approach on the diagram corresponds to its average rank based on
the test accuracies of the trained network —see Table 2. Lower ranks indicate
that a regularisation term, or the training without regulariser, outperforms
competitors with higher ranks. Regularisation terms are connected if the
$p$-value obtained from the Nemenyi post-hoc test is greater than $0.05$. See
Table 1 for a description of the training approaches and the $p$-values.

##  6 Conclusions

In this work, we introduced regularisation terms that minimise high
correlations between the most important neurons given a training batch, by
maximising the values of their zero-dimensional persistence diagram computed
with the dissimilarity function
$d(u,v)=1-\left|\text{corr}_{\mathcal{D}}(u,v)\right|$. Our regularisation
terms outperformed classical regularisation terms and significantly improved
the performance compared to minimising all pairwise correlations of important
neurons in the MNIST and CIFAR10 datasets with MLP and VGG-like architectures,
respectively. Additionally, we demonstrated that, when minimising higher
correlations using persistent homology, several loss functions that are used
with the same objective can yield different performances, suggesting that, for
differentiable persistent descriptors, the choice of a loss function is a
crucial step in the process. These findings support the hypothesis that neuron
correlations play a crucial role in the generalisation capacity of neural
networks, consistent with previous studies such as [11, 30].

Our results also show that topological regularisation terms can be used to
improve the performance of neural networks not only by considering the final
representations of the data, but also by looking at the intermediate
representations as well. This, along with other topological methods in deep
learning, highlights the relevance of topological data analysis tools in
understanding the behaviour of neural networks and the importance of shape
features of the data and the models on the generalisation capacity of neural
networks. We hope that our work can encourage further research in this
direction, as well as the development of new topological methods for deep
learning. In summary, our findings provide valuable insights into the role of
topology and neuron correlations in deep learning and their potential for
future advances in the field.

## Acknowledgements

This work was supported by the Ministry of Science and Innovation of Spain
through the research projects PID2022-136436NB-I00 and PID2020-117971GB-C22,
and the Ministry of Universities of Spain through contract FPU21/00968.

## References

  * Bertoin et al. [2021] David Bertoin, Jérôme Bolte, Sébastien Gerchinovitz, and Edouard Pauwels.  Numerical influence of ReLU’(0) on backpropagation.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , volume 34, pages 468–479. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper_files/paper/2021/file/043ab21fc5a1607b381ac3896176dac6-Paper.pdf.
  * Birdal et al. [2021] Tolga Birdal, Aaron Lou, Leonidas J Guibas, and Umut Simsekli.  Intrinsic dimension, persistent homology and generalization in neural networks.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , volume 34, pages 6776–6789. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper_files/paper/2021/file/35a12c43227f217207d4e06ffefe39d3-Paper.pdf.
  * Byrne et al. [2021] Nick Byrne, James R. Clough, Giovanni Montana, and Andrew P. King.  A persistent homology-based topological loss function for multi-class CNN segmentation of cardiac MRI.  In Esther Puyol Anton, Mihaela Pop, Maxime Sermesant, Víctor Campello, Alain Lalande, Karim Lekadir, Avan Suinesiaputra, Oscar Camara, and Alistair Young, editors, _Statistical Atlases and Computational Models of the Heart. M &Ms and EMIDEC Challenges_, pages 3–13, Cham, 2021. Springer International Publishing.  ISBN 978-3-030-68107-4.
  * Carlsson and Gabrielsson [2020] Gunnar Carlsson and Rickard Brüel Gabrielsson.  Topological approaches to deep learning.  In Nils A. Baas, Gunnar E. Carlsson, Gereon Quick, Markus Szymik, and Marius Thaule, editors, _Topological Data Analysis_ , pages 119–146, Cham, 2020. Springer International Publishing.  ISBN 978-3-030-43408-3.
  * Carrière et al. [2020] Mathieu Carrière, Frédéric Chazal, Yuichi Ike, Théo Lacombe, Martin Royer, and Yuhei Umeda.  Perslay: A neural network layer for persistence diagrams and new graph topological signatures.  In Silvia Chiappa and Roberto Calandra, editors, _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , volume 108 of _Proceedings of Machine Learning Research_ , pages 2786–2796. PMLR, 26–28 Aug 2020.  URL https://proceedings.mlr.press/v108/carriere20a.html.
  * Carrière et al. [2021] Mathieu Carrière, Frédéric Chazal, Marc Glisse, Yuichi Ike, Hariprasad Kannan, and Yuhei Umeda.  Optimizing persistent homology based functions.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 1294–1303. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/carriere21a.html.
  * Chazelle [2000] Bernard Chazelle.  A minimum spanning tree algorithm with inverse-Ackermann type complexity.  _J. ACM_ , 47(6):1028–1047, nov 2000.  ISSN 0004-5411.  doi: 10.1145/355541.355562.  URL https://doi.org/10.1145/355541.355562.
  * Chen et al. [2019a] Chao Chen, Xiuyan Ni, Qinxun Bai, and Yusu Wang.  A topological regularizer for classifiers via persistent homology.  In Kamalika Chaudhuri and Masashi Sugiyama, editors, _Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics_ , volume 89 of _Proceedings of Machine Learning Research_ , pages 2573–2582. PMLR, 16–18 Apr 2019a.  URL https://proceedings.mlr.press/v89/chen19g.html.
  * Chen et al. [2019b] Chao Chen, Xiuyan Ni, Qinxun Bai, and Yusu Wang.  A topological regularizer for classifiers via persistent homology.  In Kamalika Chaudhuri and Masashi Sugiyama, editors, _Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics_ , volume 89 of _Proceedings of Machine Learning Research_ , pages 2573–2582. PMLR, 16–18 Apr 2019b.  URL https://proceedings.mlr.press/v89/chen19g.html.
  * Clough et al. [2022] J. R. Clough, N. Byrne, I. Oksuz, V. A. Zimmer, J. A. Schnabel, and A. P. King.  A topological loss function for deep-learning based image segmentation using persistent homology.  _IEEE Transactions on Pattern Analysis; Machine Intelligence_ , 44(12):8766–8778, dec 2022.  ISSN 1939-3539.  doi: 10.1109/TPAMI.2020.3013679.
  * Cogswell et al. [2016] Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra.  Reducing overfitting in deep networks by decorrelating representations.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , 2016.
  * Cohen and Kohn [2011] Marlene R Cohen and Adam Kohn.  Measuring and interpreting neuronal correlations.  _Nature Neuroscience_ , 14(7):811–819, 2011.
  * Corneanu et al. [2020] C. A. Corneanu, S. Escalera, and A. M. Martinez.  Computing the testing error without a testing set.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2674–2682, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society.  doi: 10.1109/CVPR42600.2020.00275.  URL https://doi.ieeecomputersociety.org/10.1109/CVPR42600.2020.00275.
  * Corneanu et al. [2019] Ciprian A. Corneanu, Meysam Madadi, Sergio Escalera, and Aleix M. Martinez.  What does it mean to learn in deep networks? And, how does one detect adversarial attacks?  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4752–4761, 2019.  doi: 10.1109/CVPR.2019.00489.
  * Demšar [2006] Janez Demšar.  Statistical comparisons of classifiers over multiple data sets.  _Journal of Machine Learning Research_ , 7(1):1–30, 2006.  URL http://jmlr.org/papers/v7/demsar06a.html.
  * Dziugaite et al. [2020] Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, Ioannis Mitliagkas, and Daniel M. Roy.  In search of robust measures of generalization.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.  ISBN 9781713829546.
  * Edelsbrunner and Harer [2022] Herbert Edelsbrunner and John L Harer.  _Computational Topology: an Introduction_.  American Mathematical Society, 2022.
  * Gabrielsson et al. [2020] Rickard Brüel Gabrielsson, Bradley J. Nelson, Anjan Dwaraknath, and Primoz Skraba.  A topology layer for machine learning.  In Silvia Chiappa and Roberto Calandra, editors, _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , volume 108 of _Proceedings of Machine Learning Research_ , pages 1553–1563. PMLR, 26–28 Aug 2020.  URL https://proceedings.mlr.press/v108/gabrielsson20a.html.
  * Graf et al. [2022] Florian Graf, Sebastian Zeng, Bastian Rieck, Marc Niethammer, and Roland Kwitt.  On measuring excess capacity in neural networks.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 10164–10178. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/420492060687ca7448398c4c3fa10366-Paper-Conference.pdf.
  * Hennig et al. [2018] Jay A Hennig, Matthew D Golub, Peter J Lund, Patrick T Sadtler, Emily R Oby, Kristin M Quick, Stephen I Ryu, Elizabeth C Tyler-Kabara, Aaron P Batista, Byron M Yu, and Steven M Chase.  Constraints on neural redundancy.  _Elife_ , 7, August 2018.
  * Hensel et al. [2021] Felix Hensel, Michael Moor, and Bastian Rieck.  A survey of topological machine learning methods.  _Frontiers in Artificial Intelligence_ , 4, 2021.  ISSN 2624-8212.  doi: 10.3389/frai.2021.681108.  URL https://www.frontiersin.org/articles/10.3389/frai.2021.681108.
  * Hofer et al. [2017] Christoph Hofer, Roland Kwitt, Marc Niethammer, and Andreas Uhl.  Deep learning with topological signatures.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/file/883e881bb4d22a7add958f2d6b052c9f-Paper.pdf.
  * Hofer et al. [2019a] Christoph Hofer, Roland Kwitt, Marc Niethammer, and Mandar Dixit.  Connectivity-optimized representation learning via persistent homology.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2751–2760. PMLR, 09–15 Jun 2019a.  URL https://proceedings.mlr.press/v97/hofer19a.html.
  * Hofer et al. [2020] Christoph Hofer, Florian Graf, Marc Niethammer, and Roland Kwitt.  Topologically densified distributions.  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4304–4313. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/hofer20a.html.
  * Hofer et al. [2019b] Christoph D. Hofer, Roland Kwitt, and Marc Niethammer.  Learning representations of persistence barcodes.  _Journal of Machine Learning Research_ , 20(126):1–45, 2019b.  URL http://jmlr.org/papers/v20/18-358.html.
  * Hu et al. [2019] Xiaoling Hu, Fuxin Li, Dimitris Samaras, and Chao Chen.  Topology-preserving deep image segmentation.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/file/2d95666e2649fcfc6e3af75e09f5adb9-Paper.pdf.
  * Hu et al. [2021] Xiaoling Hu, Yusu Wang, Li Fuxin, Dimitris Samaras, and Chao Chen.  Topology-aware segmentation using discrete Morse theory.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=LGgdb4TS4Z.
  * Jiang et al. [2020] Yiding Jiang, Pierre Foret, Scott Yak, Daniel M. Roy, Hossein Mobahi, Gintare Karolina Dziugaite, Samy Bengio, Suriya Gunasekar, Isabelle Guyon, and Behnam Neyshabur.  NeurIPS 2020 competition: Predicting generalization in deep learning, 2020.  URL https://arxiv.org/abs/2012.07976.
  * Jiang* et al. [2020] Yiding Jiang*, Behnam Neyshabur*, Hossein Mobahi, Dilip Krishnan, and Samy Bengio.  Fantastic generalization measures and where to find them.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=SJgIPJBFvH.
  * Jin et al. [2020] Gaojie Jin, Xinping Yi, Liang Zhang, Lijun Zhang, Sven Schewe, and Xiaowei Huang.  How does weight correlation affect generalisation ability of deep neural networks?  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 21346–21356. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/f48c04ffab49ff0e5d1176244fdfb65c-Paper.pdf.
  * Kawaguchi et al. [2022] K. Kawaguchi, Y. Bengio, and L. Kaelbling.  Generalization in deep learning.  In _Mathematical Aspects of Deep Learning_ , pages 112–148. Cambridge University Press, dec 2022.  doi: 10.1017/9781009025096.003.  URL https://doi.org/10.1017%2F9781009025096.003.
  * Kim et al. [2020] Kwangho Kim, Jisu Kim, Manzil Zaheer, Joon Sik Kim, Frédéric Chazal, and Larry Wasserman.  PLLay: Efficient topological layer based on persistence landscapes.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.  ISBN 9781713829546.
  * Kohn and Smith [2005] Adam Kohn and Matthew A Smith.  Stimulus dependence of neuronal correlation in primary visual cortex of the macaque.  _Journal of Neuroscience_ , 25(14):3661–3673, 2005.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * LeCun et al. [2010] Yann LeCun, Corinna Cortes, and CJ Burges.  MNIST handwritten digit database.  _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ , 2, 2010.
  * Leygonie et al. [2022] Jacob Leygonie, Steve Oudot, and Ulrike Tillmann.  A framework for differential calculus on persistence barcodes.  _Foundations of Computational Mathematics_ , 22(4):1069–1131, Aug 2022.  ISSN 1615-3383.  doi: 10.1007/s10208-021-09522-y.  URL https://doi.org/10.1007/s10208-021-09522-y.
  * Mizusaki and O’Donnell [2021] Beatriz E. P. Mizusaki and Cian O’Donnell.  Neural circuit function redundancy in brain disorders.  _Current Opinion in Neurobiology_ , 70:74–80, 2021.  ISSN 0959-4388.  doi: https://doi.org/10.1016/j.conb.2021.07.008.  URL https://www.sciencedirect.com/science/article/pii/S0959438821000787.  Computational Neuroscience.
  * Molchanov et al. [2017] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz.  Pruning convolutional neural networks for resource efficient inference.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=SJGCiw5gl.
  * Pérez et al. [2021] Julián Burella Pérez, Sydney Hauke, Umberto Lupo, Matteo Caorsi, and Alberto Dassatti.  Giotto-ph: A Python library for high-performance computation of persistent homology of Vietoris–Rips filtrations, 2021.
  * Reinauer et al. [2022] Raphael Reinauer, Matteo Caorsi, and Nicolas Berkouk.  Persformer: A transformer architecture for topological machine learning, 2022.
  * Rieck et al. [2019] Bastian Rieck, Matteo Togninalli, Christian Bock, Michael Moor, Max Horn, Thomas Gumbsch, and Karsten Borgwardt.  Neural persistence: A complexity measure for deep neural networks using algebraic topology.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=ByxkijC5FQ.
  * Rostrup et al. [2013] Scott Rostrup, Shweta Srivastava, and Kishore Singhal.  Fast and memory-efficient minimum spanning tree on the GPU.  _Int. J. Comput. Sci. Eng._ , 8(1):21–33, feb 2013.  ISSN 1742-7185.  doi: 10.1504/IJCSE.2013.052115.  URL https://doi.org/10.1504/IJCSE.2013.052115.
  * Sanders and Schimek [2023] Peter Sanders and Matthias Schimek.  Engineering massively parallel MST algorithms, 2023.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958, 2014.  URL http://jmlr.org/papers/v15/srivastava14a.html.
  * Tarjan [1975] Robert Endre Tarjan.  Efficiency of a good but not linear set union algorithm.  _J. ACM_ , 22(2):215–225, apr 1975.  ISSN 0004-5411.  doi: 10.1145/321879.321884.  URL https://doi.org/10.1145/321879.321884.
  * Tibshirani [2011] Robert Tibshirani.  Regression shrinkage and selection via the Lasso: a retrospective.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 73(3):273–282, 2011.  doi: https://doi.org/10.1111/j.1467-9868.2011.00771.x.  URL https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-9868.2011.00771.x.
  * Zhang et al. [2021] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning (still) requires rethinking generalization.  _Commun. ACM_ , 64(3):107–115, February 2021\.  ISSN 0001-0782.  doi: 10.1145/3446776.  URL https://doi.org/10.1145/3446776.
  * Zhang et al. [2020] Simon Zhang, Mengbai Xiao, and Hao Wang.  GPU-accelerated computation of Vietoris-Rips persistence barcodes.  In _36th International Symposium on Computational Geometry (SoCG 2020)_. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2020.

##  Appendix A Differentiability of functions on persistence diagrams

In this appendix, we prove Theorem 1 using methods and results from [36]. We
consider finite ordered sets $X\subseteq\mathbb{R}^{n}$ with $c$ elements,
where $n\geq 2$ and $c\geq 2$. Each such set $X$ corresponds to an element
$(p_{1},\dots,p_{c})\in\mathbb{R}^{cn}$, where $p_{i}\in\mathbb{R}^{n}$
denotes the $i$th point of $X$.

Given points $p_{1},\dots,p_{c}$ in $\mathbb{R}^{n}$, where $c\geq 2$, recall
that the _covariance_ between $p_{i}$ and $p_{j}$ is

$\text{cov}(p_{i},p_{j})=\frac{1}{n}\sum_{k=1}^{n}(p_{i,k}-\bar{p}_{i})(p_{j,k}-\bar{p}_{j}),$

where $\bar{p_{i}}=\frac{1}{n}\sum_{k=1}^{n}p_{i,k}$. Since the covariance
function is polynomial on the entries, the set

$\mathcal{D}_{c,n}=\left\\{(p_{1},\ldots,p_{c})\in\mathbb{R}^{cn}:\text{$\text{cov}(p_{i},p_{j})\neq
0$ for all $i,j\in\\{1,\ldots,c\\}$}\right\\}$

is open and dense in $\mathbb{R}^{cn}$.

Suppose given a continuous function
$d\colon\mathbb{R}^{n}\times\mathbb{R}^{n}\to\mathbb{R}_{\geq 0}$ such that
$d(x,y)=d(y,x)$ for all $x$ and $y$ and $d(x,x)=0$ for every $x$ —we call such
a function a _dissimilarity_. The _persistence diagram_ of
$X=\\{p_{1},\dots,p_{c}\\}$ in homological dimension $k$ with respect to the
given dissimilarity $d$ is an unordered set of points
$\\{(b_{i},d_{i})\\}_{i\in I}$ in $\mathbb{R}^{2}$ where $b_{i}$ is the birth
parameter and $d_{i}$ is the death parameter (possibly infinite) of an element
in a full set $I$ of linearly independent generators of $k$th simplicial
homology $H_{k}(V_{t}(X,d))$ of the _Vietoris–Rips filtered simplicial
complex_ $\\{V_{t}(X,d)\\}_{t\geq 0}$. This simplicial complex has an
$m$-simplex at level $t$ for every collection of points
$p_{i_{0}},\dots,p_{i_{m}}$ in $X$ such that $d(p_{i_{r}},p_{i_{s}})\leq t$
for $i_{r},i_{s}\in\\{0,\dots,m)$. By convention, persistence diagrams include
all points in the diagonal $\Delta^{\infty}=\\{(x,x):x\geq 0\\}$ with infinite
multiplicity. Simplicial homology is computed with coefficients in any field.

A persistence diagram in homological dimension $k$ can be viewed as a function
$B_{k}={\rm Dgm}_{k}\circ F$ defined on $\mathbb{R}^{cn}$ taking values in the
set $\mathbf{Diag}$ of families of points with multiplicities in the upper
quadrant of $\mathbb{R}^{2}$ extended with points at infinity:

$\mathbb{R}^{cn}\xrightarrow[]{\hskip 11.38092ptF\hskip
11.38092pt}\mathbb{R}^{K}\xrightarrow[]{\hskip 5.69046pt\text{Dgm}_{k}\hskip
5.69046pt}\mathbf{Diag}.$ (5)

Here we denote by $\mathbb{R}^{K}$ the set of all functions $f\colon
K\to\mathbb{R}$, where $K$ is the collection of nonempty subsets of
$\\{1,\dots,c\\}$, which we view as faces of a $(c-1)$-dimensional simplex.
The function $F$ is defined as

$F(X)(\sigma)=\max_{i,j\in\sigma}\,d(p_{i},p_{j})$

for $\sigma\in K$ and $X=\\{p_{1},\dots,p_{c}\\}$. The function
$\text{Dgm}_{k}$ assigns to each function $f\colon K\to\mathbb{R}$ the
corresponding Vietoris–Rips persistence diagram in homological dimension $k$,
where $f$ is treated as a filtering function on the faces of a
$(c-1)$-dimensional simplex.

Differentiability of functions valued in $\mathbf{Diag}$ is defined in [36] as
follows. For $m,\ell\in\mathbb{Z}_{\geq 0}$, consider the quotient map
$Q_{m,\ell}\colon\mathbb{R}^{2m}\times\mathbb{R}^{\ell}\to\mathbf{Diag}$
sending each point

$\tilde{D}=(x_{1},y_{1},\ldots,x_{m},y_{m},z_{1},\ldots,z_{\ell})\in\mathbb{R}^{2m}\times\mathbb{R}^{\ell}$

to the diagram obtained by forgetting the order of the points:

$Q_{m,p}(\tilde{D})=\left\\{(x_{i},y_{i})\right\\}_{i=1}^{m}\cup\left\\{(z_{j},\infty)\right\\}_{j=1}^{\ell}\cup\Delta^{\infty}.$

Let $\mathcal{M}$ be a smooth manifold and let
$B\colon\mathcal{M}\to\mathbf{Diag}$ be any map. For $x\in\mathcal{M}$ and
$r\in\mathbb{Z}_{\geq 0}\cup\\{\infty\\}$, the map $B$ is said to be _$r$
-differentiable_ at $x$ if there exists an open neighborhood $U$ of $x$ and
there exist integers $m,\ell\in\mathbb{Z}_{\geq 0}$ and a map $\tilde{B}\colon
U\to\mathbb{R}^{2m}\times\mathbb{R}^{\ell}$ of class $\mathcal{C}^{r}$ such
that $B=Q_{m,\ell}\circ\tilde{B}$ on $U$. Similarly, for a smooth manifold
$\mathcal{N}$, a map $V\colon\mathbf{Diag}\to\mathcal{N}$ is said to be _$r$
-differentiable_ at a diagram $D$, where $r\in\mathbb{Z}_{\geq
0}\cup\\{\infty\\}$, if for all $m,\ell\in\mathbb{Z}_{\geq 0}$ and all
$\tilde{D}\in\mathbb{R}^{2m}\times\mathbb{R}^{\ell}$ such that
$Q_{m,\ell}(\tilde{D})=D$ the map $V\circ
Q_{m,\ell}\colon\mathbb{R}^{2m}\times\mathbb{R}^{\ell}\to\mathcal{N}$ is
$\mathcal{C}^{r}$ on an open neighborhood of $\tilde{D}$.

As proved in [36, Proposition 3.14], if a function
$B\colon\mathcal{M}\to\mathbf{Diag}$ is $r$-differentiable at
$x\in\mathcal{M}$ and another function $V\colon\mathbf{Diag}\to\mathcal{N}$ is
$r$-differentiable at $B(x)$, then $V\circ B\colon\mathcal{M}\to\mathcal{N}$
is $\mathcal{C}^{r}$ at $x$ as a map between smooth manifolds.

In what follows, we consider the projections for $i,j\in\\{1,\dots,c\\}$,

$\pi_{i,j}\colon\mathbb{R}^{cn}\to\mathbb{R}^{n}\times\mathbb{R}^{n},\qquad\pi_{i,j}(p_{1},\dots,p_{c})=(p_{i},p_{j}).$

######  Proposition 1.

Let $d\colon\mathbb{R}^{n}\times\mathbb{R}^{n}\to\mathbb{R}$ be a
dissimilarity which is $\mathcal{C}^{r}$ on an open set
$U\subseteq\mathbb{R}^{n}\times\mathbb{R}^{n}$, where $r\geq 0$. Let
$p=(p_{1},\dots,p_{c})\in\mathbb{R}^{cn}$ such that $p\in\pi^{-1}_{i,j}(U)$
for all $i,j\in\\{1,\ldots,c\\}$. Suppose that $d(p_{i},p_{j})\neq
d(p_{k},p_{l})$ when $\\{i,j\\}\neq\\{k,l\\}$, where
$i,j,k,l\in\\{1,\ldots,c\\}$. Then the function
$B_{k}=\textnormal{Dgm}_{k}\circ F$ defined in (5) is $r$-differentiable at
$p$.

###### Proof.

Since $d(p_{i},p_{j})\neq d(p_{k},p_{l})$ for $\\{i,j\\}\neq\\{k,l\\}$, the
values $d(p_{i},p_{j})$ for $i\neq j$ are strictly ordered. As the projections
$\pi_{i,j}$ are $\mathcal{C}^{\infty}$ and $d$ is $\mathcal{C}^{r}$ on $U$,
and $\pi_{i,j}(p)\in U$, we infer that $d\circ\pi_{i,j}$ is $\mathcal{C}^{r}$
in $p$. Since, in particular, $d\circ\pi_{i,j}$ is continuous, there is a
neighbourhood $U^{\prime}$ of $p$ where the order of the values
$d(p^{\prime}_{i},p^{\prime}_{j})$ remains the same. Then $F(p)$ and
$F(p^{\prime})$ induce the same preorder on the set of simplices of $K$ for
every $p^{\prime}\in U\cap\bigcap_{i,j}\pi_{i,j}^{-1}(U)$. Hence, the
hypotheses of [36, Theorem 4.7] hold and therefore $B_{k}$ is
$r$-differentiable at $p$. ∎

In what follows, we denote, for a given dissimilarity $d$,

$\mathfrak{D}_{c,n}=\left\\{(p_{1},\ldots,p_{c})\in\mathcal{D}_{c,n}:\text{$d(p_{i},p_{j})\neq
d(p_{k},p_{l})$ if $\\{i,j\\}\neq\\{k,l\\}$}\right\\}.$

We note that, if the dissimilarity $d(x,y)=1-|{\rm corr}(x,y)|$ is chosen,
then $\mathfrak{D}_{c,n}$ is an open dense subset of $\mathbb{R}^{cn}$, since
$d(p_{i},p_{j})=d(p_{k},p_{l})$ precisely when $|{\rm
corr}(p_{i},p_{j})|=|{\rm corr}(p_{k},p_{l})|$, and the square of correlation
is a rational function.

######  Proposition 2.

Let $c,n\geq 2$, $k\in\mathbb{Z}_{\geq 0}$, and
$d\colon\mathbb{R}^{n}\times\mathbb{R}^{n}\to[0,1]$ be the dissimilarity given
by $d(x,y)=1-|{\rm corr}(x,y)|$. The function $B_{k}={\rm Dgm}_{k}\circ F$
defined in (5) is $\infty$-differentiable on $\mathfrak{D}_{c,n}$.

###### Proof.

Take $p=(p_{1},\ldots,p_{c})\in\mathfrak{D}_{c,n}$. By the definition of
$\mathfrak{D}_{c,n}$, we have that $d(p_{i},p_{j})\neq d(p_{k},p_{l})$ for all
$\\{i,j\\}\neq\\{k,l\\}$. Furthermore, the correlation is well-defined and
$\mathcal{C}^{\infty}$ on $\pi_{i,j}(p)=(p_{i},p_{j})$ for all
$i,j\in\\{1,\ldots,c\\}$, because $\text{cov}(p_{i},p_{j})\neq 0$ for points
in $\mathfrak{D}_{c,n}$. We also have that $|\text{corr}(x,y)|$ is
$\mathcal{C}^{\infty}$ on every $(p_{i},p_{j})$, since the absolute value
function is $\mathcal{C}^{\infty}$ on $\mathbb{R}\smallsetminus\\{0\\}$.
Therefore, $d$ is $\mathcal{C}^{\infty}$ on $\pi_{i,j}(p)=(p_{i},p_{j})$ for
all $i,j\in\\{1,\ldots,c\\}$ and thus the assumptions of Proposition 1 hold,
implying that $B_{k}$ is $\infty$-differentiable on $\mathfrak{D}_{c,n}$. ∎

###### Proof of Theorem 1.

By Proposition 2, the function $B_{0}$ is $\infty$-differentiable on
$\mathfrak{D}_{c,n}$, where $B_{0}(X)$ is the persistence diagram of $X$ in
homological dimension zero. Therefore, we only need to display functions
$\tilde{\mathcal{T}}_{i}\colon\mathbf{Diag}\to\mathbb{R}$ that are
$\infty$-differentiable on $B_{0}(\mathfrak{D}_{c,n})$ such that
$\mathcal{T}_{i}=\tilde{\mathcal{T}}_{i}\circ B_{0}$ for $i\in\\{1,2\\}$.

Here we view zero-dimensional persistence diagrams as consisting of points
$(0,y)$, although we keep denoting them in the general form $(b,d)$. When
computing the average persistence and standard deviation of persistence of the
points in $B_{0}(X)$, the number of points in the diagram is assumed to be
equal to the number of edges of a minimum spanning tree for $(X,d)$, that is,
$|X|-1$. Therefore, the functions $\tilde{\mathcal{T}}_{i}$ can be defined as

$\tilde{\mathcal{T}}_{1}(D)\triangleq\frac{1}{c-1}\sum_{(b,d)\in
D^{*}}\\!\\!(d-b),\qquad\tilde{\mathcal{T}}_{2}(D)\triangleq-\alpha\tilde{\mathcal{T}}_{1}(D)+\beta\sigma(D),$

where

$\sigma^{2}(D)\triangleq\frac{1}{c-1}\sum_{(b,d)\in
D^{*}}((d-b)-\tilde{\mathcal{T}}_{1}(D))^{2}$

with $D^{*}=\left\\{(b,d)\in D:d<\infty\right\\}$. Points in the diagonal
$\Delta^{\infty}$ are sent to zero by $d-b$ and are not taken into
consideration in the sums, and neither are points at infinity.

In order to prove that the functions $\tilde{\mathcal{T}}_{i}$ are
$\infty$-differentiable, take any $m,\ell\in\mathbb{Z}_{\geq 0}$ and
$\tilde{D}\in\mathbb{R}^{2m}\times\mathbb{R}^{\ell}$ such that
$Q_{m,\ell}(\tilde{D})=D$. If we write
$\tilde{D}=(x_{1},y_{1},\ldots,x_{m},y_{m},z_{1},\ldots,z_{\ell})$, then the
functions $\tilde{\mathcal{T}}_{i}\circ Q_{m,\ell}$ are given by

$\tilde{\mathcal{T}}_{1}(Q_{m,\ell}(\tilde{D}))=\frac{1}{c-1}\sum_{i=1}^{m}(y_{i}-x_{i}),\qquad\tilde{\mathcal{T}}_{2}(Q_{m,\ell}(\tilde{D}))=-\alpha\tilde{\mathcal{T}}_{1}(Q_{m,\ell}(\tilde{D}))+\beta\sigma(\tilde{D}),$

where

$\sigma^{2}(\tilde{D})\triangleq\frac{1}{c-1}\sum_{i=1}^{m}[(y_{i}-x_{i})-\tilde{\mathcal{T}}_{1}(Q_{m,\ell}(\tilde{D}))]^{2}.$

The functions $\mathcal{\tilde{T}}_{i}\circ Q_{m,\ell}$ are
$\mathcal{C}^{\infty}$ on all their domain because they are compositions of
$\mathcal{C}^{\infty}$ functions on a neighborhood of $\tilde{D}$. The only
function that is not $\mathcal{C}^{\infty}$ in all its domain is the square
root function, which is not differentiable at zero. However, for points
$p\in\mathfrak{D}_{c,n}$ we have pairwise different distances, and
consequently the persistence diagram $B_{0}(X)$ contains at least two
different points, making $\sigma(\tilde{D})\neq 0$ for a neighbourhood $U$ of
$\tilde{D}$ and thus making $\sigma(\tilde{D})$ a $\mathcal{C}^{\infty}$
function on $U$. Hence, $\tilde{\mathcal{T}}_{i}$ is $\infty$-differentiable.
Therefore, as $B_{0}$ and $\tilde{\mathcal{T}}_{i}$ are
$\infty$-differentiable in $\mathfrak{D}$ and $B_{0}(\mathfrak{D}_{c,n})$
respectively, the functions $\mathcal{T}_{i}$ are $\mathcal{C}^{\infty}$ on
$\mathfrak{D}_{c,n}$. ∎