# Topological understanding of Neural Networks

Tushar Pandey<EMAIL_ADDRESS>Department of Mathematics, Texas A&M
University, College Station, TX 77840

###### Abstract

In this review paper, we look at the internal structure of neural networks
which is usually treated as a black box. The easiest and most comprehensible
thing to do is to look at a binary classification and try to understand the
approach a neural network takes. We review the significance of different
activation functions, types of network architectures associated to them, and
some empirical data. At the end, we conclude with describing some possible
choices of activation functions for different problems and techniques.

##  I Introduction

One of the prominent questions in deep learning is understanding what happens
inside the black box, i.e. the hidden layers. The theme of the paper is to
understand what happens to the data when it goes through different layer.
There are some other approaches taken, one where each data point is looked at
after every layer, usually images. The second one is to understand the
boundary manifold and how that changes in the hidden layer. Even though these
methods are important, we believe it’s more important to look at the
transformation of the entire data set as it goes through the hidden layers,
and see the representation of the data space in the final layer.
We begin the paper by looking at some smooth activation functions, where width
plays an important role. We provide some intuition behind selecting neural
networks with different architectures. We point out some possible errors in
considering such methods and the time complexity that comes with it.
In the third section, we look at a comparison between smooth and non-smooth
activation functions, along with changing width and depth of the network. We
try to answer a widely asked question, what makes ReLU better than other
activation functions in practice? [1], [2], [3] We begin with a simulated
dataset, where the topology is known, in order to understand the changes that
could take place in an actual manifold. Under different conditions, these
changes are measured. Once there is an idea of the change, it is verified on
real world data in high dimensions. Due to computational power constraints,
some parameters of the architecture are not adjusted from the simulated data
to real world data with a large difference in dimensions.
After reviewing these methods, we draw some conclusions from both the
approaches and provide with experiments to extend the results to different
architectures in order to improve the understanding of the black box.
For most of the paper, we will consider the case where the data has two
classes. All the definitions are provided at the end of the paper in Appendix.

##  II Smooth Activation functions and change in topology

In this section, we look at the change in topology when the activation
function is smooth. This section is based on the work of [4]
Consider an architecture where there is no hidden layer. Let’s assume the data
forms two lines as described in fig 1.

Figure 1:  No hidden Layer

Since there is no hidden layer, the output layer is a linear function,
therefore the neural network tries to classify the data by separating it
through a straight line (or a hyperplane in case of higher dimensions). In
this example, it can not properly classify the data set as no straight line
can possibly distinguish the two classes completely. Note that there is no
activation function involved so far.
Now consider the case with one hidden layer. We look at the $tanh$ function
and see the boundary line between the two classes of data in fig 2.

Figure 2:  ”tanh” activation function with one hidden layer. The green line is
the boundary

This separation is seen in the actual data space, by which we mean the
original way the data is represented. However, internally, the network doesn’t
try to change the boundary shape from a line to a curve, rather, it changes
the shape of the data and then fits a linear separating boundary. More
precisely, the ambient space of the data changes after applying the activation
functions and therefore it’s easier for the neural network to construct a
linear boundary. An example of the same is demonstrated in fig 3.

Figure 3: The data space is transformed and the final boundary (in green) is
linear.

For a smooth activation functions, this change in the data space is a
homeomorphism. A natural question that arises:
”Is one hidden layer enough to change the data space (manifold) in order to
separate two classes via a linear boundary for smooth activation functions?”
Answer: No. In fact, depth is not sufficient to decide whether or not two data
sets are separable. The depth of the network corresponds to the number of
transformations. Since the functions are homeomorphisms, they can’t change
some topological properties of the space. One such property of a space is the
homology (or the Betti numbers). Back to neural networks; if one has an
annulus as one class of data points and in the hole, there is a cluster
corresponding to a different class, no homoemorphism in $\mathbb{R}^{2}$ can
distinguish between these classes for a smooth activation neural network.
Since it’s a homeomorphism, $\beta_{1}$ will remain 1, which means, no linear
boundary can separate the two classes completely.

Question: How about changing the width?

Answer: yes! It will work. Width corresponds to an embedding in a higher
dimensional space, where one can lift up the class inside the hole and by a
hyperplane, separate them. This means, if we know the minimum dimension of the
ambient space where the data can be embedded properly, we can use a neural
network to completely classify the dataset.

######  Theorem II.1.

For the data space described above, any neural network with a smooth
activation function, width 1 or 2 is not enough to completely classify the two
data classes regardless of the depth.

###### Proof.

For a smooth activation function, hidden layer corresponds to a homeomorphism,
which means $\beta_{1}=1$ after all the hidden layers. The last layer is a
linear transformation, therefore conserves $\beta_{1}$ as well. In order to
divide the space by a linear boundary, $\beta_{1}$ needs to be 0. Therefore, a
contradiction. ∎

Note that this was true only because of the smoothness of the activation
function. Which means, if the activation function is not smooth (e.g. ReLU),
the Neural network can form a linear boundary to distinguish the two classes.
For a dataset with n dimensional points, they can be embedded in ambient space
of dimension 2n+2, such that a linear (hyperplane) boundary can separate them.
Therefore, if the hidden layer has width $\geq 2n+2$, then the data can be
separated.

######  Theorem II.2.

[4] There is an ambient isotopy between the input and a network layer’s
representation if: a) W isn’t singular, b) we are willing to permute the
neurons in the hidden layer, and c) there is more than 1 hidden unit.

So far, we have a way to find a nice boundary to potentially get $100\%$
accuracy on the training data. This however, does not mean it’s the actual
separation. The author [4] also suggests that based on some empirical results,
changing the last layer from softmax to KNN, the accuracy increases. This
approach concludes that for smooth activation, the width needs to be wide
enough and there is a need of sufficient (but small) depth in order to not
force the network to get stuck at a local minima.

##  III Smooth vs non-smooth activation functions

We follow [5] for most of the section. Once again, the data is divided into
two classes $M=M_{a}\cup M_{b}$. The first conclusion of this section
describes ReLU outperforming smooth activation function, namely tanh, and the
second one is describing the depth instead of the fact that shallow networks
can approximate most functions pretty well. [5] have performed the analysis on
real and simulated data to conclude such observations. The topological work is
done through TDA, based on the idea of persistent homology originally inspired
by [6, 7, 8]
The interesting aspect of this study is that instead of looking at the change
in each data point, the overall change in the data shape is observed, which
provides a better insight. The focus of the study is to look at the change in
Betti numbers and how it is affected in the training.

Figure 4: [5] Change in Betti numbers:
$\beta(red):(1,2,0)\rightarrow(1,2,0)\rightarrow(2,1,0)\rightarrow(2,0,0)\rightarrow(1,0,0)\rightarrow(1,0,0);\\\
\beta(green)::(2,2,0)\rightarrow(2,2,0)\rightarrow(2,1,0)\rightarrow(2,0,0)\rightarrow(2,0,0)\rightarrow(1,0,0).$

In fig 4, the actual network changes the shape of the data like demonstrated.
In order to unlink the components, it has to break the topology, and change
the betti numbers. Generally speaking, for binary classification problems, the
idea is to decrease the betti numbers, s.t.
$\beta_{1}(M_{a})=0=\beta_{1}(M_{b})$ and
$\beta_{2}(M_{a})=0=\beta_{2}(M_{b})$, whereas
$\beta_{0}(M_{a})=1=\beta_{0}(M_{b})$. Note that the last condition is not
very strict. Even if $\beta_{0}(M_{a})>1$ while other $\beta_{j}(M_{a})=0$ for
$j\geq 1$, it’s still a good enough classification. In fact, if $\beta_{0}>1$,
the suggests possibility of another class (or subclass), providing more
insights about the data.
Out of some of the obstructions this hypothesis possesses, one of them is that
the data usually doesn’t come in such nice form. The data will more likely be
in a point cloud form with some noise. But, that’s where persistent homology
comes into picture. Presence of small noise does not change the effective
$\beta_{j}$ for the point cloud data. If the data does come from some sampling
of a manifold structure, then persistent homology recovers the $\beta_{j}$
almost precisely. The authors of the paper have not only looked at the
topological changes for simulated data, but also for real world data including
images.
Some important questions which will be answered here are:
1) Why does ReLU perform better than others empirically?
2) Are these topological changes observed through this method robust?
3) Why do deep neural network work better than shallow ones even after the
approximation theorems?

###  III.1 Topology

We try to look into topological complexity of the manifold and the
generalization gap of the dataset which measures the difference in test and
training accuracy of the model. The Generalization gap is defined as

$\displaystyle GG(X_{train},X_{test},m)=Acc(X_{train},m)-Acc(X_{test},m)$

where m is the model.
For piece wise linear function the upper bound for topological complexity is
given in terms of linear regions. However, the number of linear regions
determined through the training set is not stable under small perturbations
[9]. Instead of looking at the decision boundary at different stages, the data
space transformations are the object of interest here. In order to understand
the transformations, one can look at the change in Betti numbers. Furthermore,
instead of looking at the Betti number of the entire Manifold, it’s sufficient
to look at the Betti number progression of each component. In practice, it’s
usually difficult to compute the homology for a point cloud.

The standard practices in Topological Data Analysis (TDA) is i) Discard
outliers, noise ii) Construct $\epsilon-$Vietoris-Rips complex iii) Simplify
VR Complex without changing the topology. The topological structure can be
altered by i) and ii) based on choices of noise reduction/smoothing and
$\epsilon$ value in VR complex.

###  III.2 Setup

  1. 1.

The problem assumed is a binary classification problem, $M=M_{a}\cup M_{b}$
with an additional assumption, $\inf\\{||x-y||:x\in M_{a},y\in M_{b}\\}>0$.

  2. 2.

For the simulated dataset, we know the manifold, so to generate the point
cloud a large sample is selected uniformly and densely. The neural network
chosen in feed forward, with depth l.

  3. 3.

The network function is $v:s\circ f_{l}\circ\dots\circ
f_{1}:\mathbb{R}\rightarrow[0,1]$.s is the score function,
$s:\mathbb{R}^{n_{l+1}}\rightarrow[0,1]$.

  4. 4.

Let $n_{j}$ denote the width of the layer j. We let $n_{1}=d,n_{l+1}=p$ and
$v_{j}:f_{j}\circ\dots\circ f_{1}$.

  5. 5.

The simulated data is non-realistic with complicated topology in low-
dimension. Real world data is in high dimensions but most likely simpler in
terms of entangled classes. For the simulated data we look at large
$\beta_{0}$, non-zero $\beta_{j}$ for $j\geq 1$ and large Topological
complexity.

  6. 6.

The model is trained to near-zero generalization error.

Spoiler alert: with sufficient depth, the last layer maps $v(T_{a}),v(T_{b})$
on the opposite ends of $[0,1]$. There are some challenges moving from
simulated to real data. The topology is not known so the persistent homology
is harder to compute at each layer.
The work is done on three different dataset which can be seen in the following
figure [5]. The red part of the data is $M_{b}$ and the green part is $M_{a}$.

Figure 5:  [5] a) $\beta(M_{a})=(9,0)$, $\beta(M_{b})=(1,9)$, b)
$\beta(M_{a})=(9,9,0)$, $\beta(M_{b})=(9,9,0)$,
c) $\beta(M_{a})=(9,0,9)$, $\beta(M_{b})=(18,0,9)$

###  III.3 Training

  1. 1.

Different activation functions: tanh, ReLU, leaky ReLU

  2. 2.

Different depth from 4-10

  3. 3.

Different width between 6-50

  4. 4.

Criss entropy categorical loss

  5. 5.

ADAM with 18000 epoch

  6. 6.

$\eta=0.02-0.04$ with exponential decay $\eta^{t/d}$, $d=2500$ and t is the
epoch.

  7. 7.

For bottleneck architecture (narrow width in middle), d = 4000, $\eta=0.5$

  8. 8.

Score = softmax function

  9. 9.

Metric $\delta_{k}$ for $VR_{\epsilon}$ Complex is the graph geodesic distance
on K-nearest neighbours. $\delta_{k}(x_{i},x_{j})=$ minimum no. of edges
between them in knn graph. It preserves connectivity and normalizes distance.

  10. 10.

Two hyperparameters, $k$ and $\epsilon$. The persistant homology is used
through filtered complex w.r.t. $\delta_{k}$ at $\epsilon$. Find $k^{*}$ with
$\epsilon=1$ first by $\beta_{0}(VR_{k^{*}})=\beta_{0}(M)$. Then find
$\epsilon^{*}$ by equating $\beta_{1}$ and $\beta_{2}$.

###  III.4 Results

  1. 1.

Clear decay in $\beta_{0}$ across all possible neural network architectures.
It’s slower in tanh, faster in Leaky ReLU and fastest in ReLU.

  2. 2.

Width:

     * •

Narrow: 6 neurons each layer, changes topology faster

     * •

Bottleneck: one of the middle layer has 3 neurons while other have 15, sudden
change in topology at bottleneck.

     * •

Wider: 50 neurons each, smoother reduction in topological complexity.

  3. 3.

depth:

     * •

For highly entangled classes, more depth required for $100\%$ accuracy. Low
depth makes it difficult to get the accuracy we want

     * •

Faster reduction in Topological complexity in the final layers.

     * •

Sometimes, there’s non-required simplification for deeper neural networks.

  4. 4.

For wide enough layers, $\beta_{1}(M)=0$ and $\beta_{2}(M)=0$ is not required.
Shallow networks sometimes forces this but deeper ones don’t, therefore
preserving more structure.

###  III.5 Graphical Results

Figure 6: [5] Average change in $\beta_{0}$ for different activation functions
for Dataset I. The dark line is the average whereas the shaded regions are
results of multiple simulations. Figure 7: [5] Change in $\beta_{0}$ and
$\beta_{1}$ for Dataset II for different activation functions. Figure 8:  [5]
Change in $\beta_{0}$ and $\beta_{2}$ for Dataset III for different activation
functions. Figure 9:  [5] Topological complexity for varying network depth for
all three datasets.

###  III.6 Real Dataset

These properties were verified for MNIST, HTRU2, UCI Banknotes and UCI
sensorless drive datasets. Since the dimension of these datasets are high, the
persistent homology can not be performed as efficiently and needs to be
performed on every layer. The generalization gap is relaxed, from $2-5\%$. The
width and depth of the networks are fixed.
MNIST: $\mathbb{R}^{784}$, for this dataset, the topological observations are
made on top 50 principal components, HTRU2 Dataset: $\mathbb{R}^{8}$, UCI
Banknotes: $\mathbb{R}^{4}$, UCI drive: $\mathbb{R}^{49}$
The table of results are available in the appendix. The observation as
expected were:

  1. 1.

The topological complexity is reduced overall. The network tries to reduce
$\beta_{0}$ to 1 and other $\beta_{j}$ to 0.

  2. 2.

Smooth activation function reduces the topological complexity slower than non-
smooth ones with ReLU performing the most simplification.

  3. 3.

ReLU adds a folding to the data space but not entirely. This is immediate from
the definition of ReLU or the absolute value function.

  4. 4.

More layers makes it easier to train the model by ”taking it’s time
simplifying the space step by step”.

##  IV Future work

While the possibility of research in this direction is never ending, some
possible options for near term research experiments are:

###  IV.1 Different activation functions in different layers

For training a model on multiple data sets or for training a dataset with
composite architectures, setting the activation function for the initial
layers as a smooth one, and applying ReLU or Leaky ReLU at the end could
better transform the data. The idea behind this is that smooth activation
function preserves the structure in the same dimension, but embedding it in
higher dimension would reduce some complexity with some structures still being
preserved.

###  IV.2 Relationship between time complexity and robustness against noise

There are some results about different linear folding because of ReLU which
are not stable under noise. There seems to be a trade-off between
oversimplification and time. It’ll be interesting to relate the stability of
folding under a network architecture with non-ReLU activation functions in
some layers as well.

###  IV.3 Different neural network architectures

Different architectures for the same dataset, e.g. CNN, simple feed forward
network and ResNet should be compared in terms of changes in topological
complexity.

##  V References

## References

  * Nair and Hinton [2010] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In _ICML 2010_ , pages 807–814, 2010.
  * Maas [2013] Andrew L. Maas.  Rectifier nonlinearities improve neural network acoustic models.  2013\.
  * Glorot et al. [2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Deep sparse rectifier neural networks.  In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics_ , volume 15 of _Proceedings of Machine Learning Research_ , pages 315–323, Fort Lauderdale, FL, USA, 11–13 Apr 2011\. PMLR.  URL https://proceedings.mlr.press/v15/glorot11a.html.
  * Olah [2014] C. Olah.  Neural networks, manifolds, and topology, 2014.  URL http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/.
  * Naitzat et al. [2022] Gregory Naitzat, Andrey Zhitnikov, and Lek-Heng Lim.  Topology of deep neural networks.  _J. Mach. Learn. Res._ , 21(1), jun 2022.  ISSN 1532-4435.
  * Carlsson [2012] G. Carlsson.  The shape of data, 2012.
  * Carlsson [2014] Gunnar Carlsson.  Topological pattern recognition for point cloud data.  _Acta Numerica_ , 23:289–368, 2014.  doi: 10.1017/S0962492914000051.
  * Carlsson and Zomorodian [2007] Gunnar Carlsson and Afra Zomorodian.  The theory of multidimensional persistence.  In _Proceedings of the Twenty-Third Annual Symposium on Computational Geometry_ , SCG ’07, page 184–193, New York, NY, USA, 2007. Association for Computing Machinery.  ISBN 9781595937056.  doi: 10.1145/1247069.1247105.  URL https://doi.org/10.1145/1247069.1247105.
  * Zhang et al. [2018] Liwen Zhang, Gregory Naitzat, and Lek-Heng Lim.  Tropical geometry of deep neural networks.  In Jennifer Dy and Andreas Krause, editors, _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 5824–5832. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/zhang18i.html.
  * Anand et al. [2019] D. Vijay Anand, Zhenyu Meng, Kelin Xia, and Yuguang Mu.  Weighted persistent homology for osmolyte molecular aggregation and hydrogen-bonding network analysis.  _Scientific Reports_ , 10, 2019.

##  Appendix A Definitions

The definitions listed here are more intuitive than formal for the sake of
understanding for a wider audience.

######  Definition A.1.

Homeomorphism: A function $f:M\rightarrow N$ is considered to be a
homeomorphism if f is continuous, f is a bijection (one-one and onto) and
$f^{-1}$ is continuous.

######  Definition A.2.

Connected Component: A space M is called connected if it’s not a disjoint
union of two components. The number of connected component is the cardinality
of the components that are connected.

######  Definition A.3.

Homology: Let M be a manifold (locally homeomorphic to $\mathbb{R}^{n}$ for
some $n\in\mathbb{N}$)
The $0^{th}$ homology, $\mathbb{H}_{0}(M)$ is defined to be the number of
connected components of M.
The first homology, $\mathbb{H}_{1}(M)$ is related to how many distinct
circles (non-trivial) can one embed in the manifold M. Based on the number and
the relationship between them, a group with integer coefficients are
associated to it.
Similarly one can also define $\mathbb{H}_{2}(M)$ by relating how many
spherical holes are present in the manifold.
Betti numbers denote the rank rank of the homology group. e.g.
$\beta_{0}=\mathrm{rank}(H_{0}(M)),\beta_{1}=\mathrm{rank}(H_{1}(M)),\beta_{2}=\mathrm{rank}(H_{2}(M))$

######  Definition A.4.

Persistent Homology: Suppose there are some data points in an ambient space.
At each point, start with a ball of radius $\epsilon$. If two such balls
intersect, add an edge. If there are three balls which intersect pairwise and
together as well, draw triangular face. In this way, based on intersections,
one builds a simplicial complex from data set by slowly increasing the radius
$\epsilon$ of the ball around each data point.
At each radius value, the homology of the data set is computed. The homology
which remains the same for the longest time, or in other words, persists, is
considered the homology of the data set (manifold).

######  Definition A.5.

Topological Complexity: It’s $\sum_{j=1}^{d}\beta_{j}$ for a d-dimension
manifold.

######  Definition A.6.

Simplicial Complex: Set composed of points, edges, faces, tetrahedrons and so
on, such that all faces of these elements are in the complex as well as, if
two elements intersect, they intersect at one face (corresponding to the
object)

######  Definition A.7.

Vietoris-Rips Complex: It’s the simplex formed in the definition of persistent
homology for different $\epsilon$ radius values.

##  Appendix B Examples

Example of Homology:

Figure 10: The images are : a) Disk b) Circle c) Annulus d) Torus
$\mathbb{H}_{0}$ for each of the four parts is $\mathbb{Z}$ because there is
one connected component.
a) $\mathbb{H}_{1}(D^{1})={e}$ which mean trivial. Any small circle inside the
disk can be contracted to a point while being inside the disk at all times.
b) $\mathbb{H}_{1}(S^{1})=\mathbb{Z}$ because there is one circle.
c) $\mathbb{H}_{1}(Annulus)=\mathbb{Z}$. Two types of circles are possible,
one which goes around the hole and another one which is small, completely
inside the blue region. The smaller one can shrink to a point, but the other
one can not be. Therefore, it is $\mathbb{Z}$
d) $\mathbb{H}_{1}(\mathbb{T}^{2})=\mathbb{Z}\times\mathbb{Z}$. There are two
non-trivial distinct circles, the red one and the purple one. They can not be
continuously deformed into one another without leaving the surface of torus.
Figure 11: [10] Persistent homology filtration, persistent barcodes based on
the radius and the homology of the simplicial complex, topological invariants
Figure 12: 3D simplicial complex


# Experimenting with Normalization Layers in Federated Learning on non-IID
scenarios

Bruno Casella, Roberto Esposito, Antonio Sciarappa, Carlo Cavazzoni, Marco
Aldinucci Bruno Casella, Roberto Esposito and Marco Aldinucci are with the
Computer Science Dpt. of the University of Turin, Corso Svizzera 185, Turin,
Italy (emails: [bruno.casella, roberto.esposito, marco.aldinucci]@unito.it).
Antonio Sciarappa and Carlo Cavazzoni are with Leonardo S.p.A., Italy (emails:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
©2022 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

###### Abstract

Training Deep Learning (DL) models require large, high-quality datasets, often
assembled with data from different institutions. Federated Learning (FL) has
been emerging as a method for privacy-preserving pooling of datasets employing
collaborative training from different institutions by iteratively globally
aggregating locally trained models. One critical performance challenge of FL
is operating on datasets not independently and identically distributed (non-
IID) among the federation participants. Even though this fragility cannot be
eliminated, it can be debunked by a suitable optimization of two hyper-
parameters: layer normalization methods and collaboration frequency selection.
In this work, we benchmark five different normalization layers for training
Neural Networks (NNs), two families of non-IID data skew, and two datasets.
Results show that Batch Normalization, widely employed for centralized DL, is
not the best choice for FL, whereas Group and Layer Normalization consistently
outperform Batch Normalization. Similarly, frequent model aggregation
decreases convergence speed and mode quality.

###### Index Terms:

Federated Learning, Federated Averaging, non-IID, Neural Networks,
Normalization Layers, Batch Normalization

##  I Introduction

The constant development of Information and Communication Technologies has
boosted the availability of computational resources and data, leading us to
the Big Data era, where data-driven approaches have become a fundamental
aspect of everyday decisions. Both computational resources and data are
ubiquitous and inherently distributed. All public and private sectors, from
scientific research to companies, take benefit from a vast amount of diverse
data to support the growth of their business and to develop more accurate
Artificial Intelligence (AI) systems.

Data is often spread and segregated in silos across different institutions and
even different business units of the same organization. It is essential to
make data accessible to all the partners to train high-quality models and
exploit the entire data’s value [1]. Many recent open science works have
encouraged data sharing between institutions in order to improve research
possibilities, create collaborations and publish reproducible results. For
example, data sharing across countries has been a crucial information tool
during the CoViD-19 pandemic [2].

However, data is often not shareable due to issues like privacy, security,
ownership, trust and economic reasons. For instance, the European regulation
GDPR [3] places stringent constraints on the possibility of sharing sensitive
data between parties; industrial companies do not share their data because
leveraging it is seen as a competitive advantage. Also, exposing data to other
institutions can raise concerns like lack of ownership and lack of trust.

To address these problems, model-sharing strategies (MSS) have emerged as an
alternative to data sharing. In MSS, the idea is to share AI models between
the involved parties in order to achieve collaboration without sharing raw
data. In these approaches, the AI model can range from simpler Machine
Learning (ML) algorithms like linear regression to more complex models such as
those learned by Deep Learning techniques using Neural Networks (NNs). Recent
years have seen the growth of different model-sharing approaches ranging from
the ”model-to-data remote access” approaches to Federated Learning [4]. In
”model-to-data remote access” approaches, AI models are run remotely directly
on the machines that hold the data and security is enforced by leveraging
secure remote connections and Trusted Execution Enviroments (TEEs) enclaves.
Federated Learning has also emerged as a popular approach. In FL, the involved
parties collaborate by aggregating locally trained models into a globally
shared one. The process is usually iterative and based on NNs (FedAvg) [4],
even if recently methods based on non-NN distributed boosting algorithms have
been proposed [5]. These algorithms allow parties to aggregate any kind of
model without making assumptions about the kind of model being aggregated or
assuming a training procedure based on gradient descent [6].

FL is a distributed ML technique originally proposed by Google in 2016 to deal
with sensitive data of mobile devices [4]. FL is an iterative version of
model-sharing: clients (the data owners) create a federation (hence the name)
together with the server and build a shared model based on the following
steps: _1)_ clients send their metadata, like the number of classes, training
set size, test set size and shape of the input features, to the server, that
initializes a model based on the received metadata characteristics; _2)_ the
server sends the initialized model to all the participants of the federation;
_3)_ after performing one or more steps of gradient descent, clients send the
trained model back to the server; _4)_ server acts as an aggregator performing
a combination (a function like average, sum, maximum, minimum and so on) of
the received models. The aggregated model is now sent to the clients, and
steps _3)_ and _4)_ are repeated until a specified number of rounds are
performed, or a convergence criterion is met. The first proposed FL algorithm
is the FedAvg [4] algorithm, where the aggregation function used to combine
models is the average. In this way, all datasets are kept within the
proprietary organizations, and the only information that gets exchanged is the
model parameters, that in the case of NN, are matrices of floating point
numbers representing the weights and the biases associated with the neurons.

Federated Learning performs well when the data is independently and
identically distributed (IID) among the involved institutions. Unfortunately,
real-world data is often non-IID, and it is well known that this scenario
poses critical issues to FL [1]. In a non-IID setting, the data statistics of
a single client may be unrepresentative of the global statistics and make the
model diverge from the intended solution. Interestingly, Huang at al. show
that if the loss surface of the optimization problem is both smooth and convex
(which is hardly true in a real scenario), then FedAvg will also converge when
the data is non-IID [7].

Recent works have proposed several FL algorithms to cope with non-IIDness
problems, such as FedProx [8], FedNova [9], SCAFFOLD [10], and FedCurv [11],
which has been tested in [12, 13]. Notice that all these algorithms are
modified versions of FedAvg and they preserve the principle underneat FedAvg:
to average the weights in all the layers of the NN. Most of the common NN
architectures employ Batch Normalization (BN) [14], a technique for improving
the training of NNs to make them converge faster and to a more stable
solution. BN works by standardizing the layers’ input for each mini-batch.

In this work, we investigate two aspects of the training FL models which,
differently from cenytralized case, they happen to be hyper-parameters that
can be optimized: the normalization layers and the frequency of models
aggregation (epochs per round). We show that the most popular normalization
layer (BN) does not couple well with FL for non-IID data and that substituting
BN with alternative normalization FL a better model can be produced for both
the non-IID and IID case. We also show that building a global model
aggregating local models at each epoch is not a good strategy, neither for
quality of the model nor for the execution time. We experiment with two
network architectures and five different normalization layers on two public
image datasets: MNIST [15] and CIFAR-10 [16]. Results show that the
performance of the networks is strictly related to the type of normalization
layer adopted.

The main contributions of this work are:

  * •

We provide benchmarks for five different normalization layers: BN, GN, LN, IN,
BRN;

  * •

We provide results of experiments on FedAvg on two non-IID settings
considering a feature distribution skew and a labels distribution skew (in
addition to the IID case). To the best of our knowledge, this is the first
work providing empirical evidence on the behaviour of these normalization
layers in common non-IID cases;

  * •

for the most promising normalization layers, we ran extensive tests to discuss
how performances are affected by the following factors:

    1. 1.

Batch size.

    2. 2.

Number of epochs per round (E).

    3. 3.

Number of clients.

  * •

We show that choosing the right normalization layer and a suitable number of
local gradient descent steps is crucial for obtaining good performances.

This work extends the typical search for optimization of machine learning
parameters to federated learning.

The rest of the paper is organized as follows. In Section II, we introduce and
discuss recent related works. In Section III, the most used normalization
layers are reviewed. In Section IV, the most typical non-IID scenarios are
described. Section V shows and discusses experimental results. Finally,
Section VI concludes the paper.

##  II Related Work

The main challenges in FL are statistical heterogeneity (non-iidness) and
systems heterogeneity (variability of the devices of the federation). In this
work, we address the former. In [1], the most common non-IID data settings
that are quantity skew, labels quantity skew (prior shift), feature
distribution skew (covariate shift), same label but different features and
same features but different labels, are reviewed. To the best of our
knowledge, there are only a few benchmarks for FL dealing with non-IID data.
Li et al. in [12] report the analysis of FedAvg [4], FedNova [9], FedProx [8]
and SCAFFOLD [10] on nine public image datasets, including MNIST [15] and
CIFAR10 [16], split according to three of the previous mentioned non-IID
partition strategies, i.e. quantity skew, labels quantity skew and three
different versions of feature distribution skew: noise-based, synthetic and
real-world feature imbalance. Authors show that none of those algorithms
outperforms others in all the cases and that non-iidness degrades the
performance of FL systems in terms of accuracy, especially in the case of
labels quantity skew. Another recent work [13] reports an empirical assessment
of the behaviour of FedAvg and FedCurv [11] on MNIST, CIFAR10 and MedMNIST
[17]. Datasets are split according to the same non-IID settings of [12].
Authors show that aggregating models at each epoch is not necessarily a good
strategy: performing local training for multiple epochs before the aggregation
phase can significantly improve performance while also reducing communication
costs. FedAvg produced better models in most non-IID settings despite
competing with an algorithm explicitly developed to deal with these scenarios
(FedCurv).

Results in [13] also confirmed literature sentiment: labels quantity skew and
its pathological variant are the most detrimental ones for the algorithms. The
same non-IID partitions have already been tested in [5], which proposes a
novel technique of non-gradient-descent FL on tabular datasets. Our paper
extends [13], deepening the experiments about the number of epochs per round,
a hyper-parameter that, if tuned appropriately, can lead to large performance
gains. Moreover, we aim to investigate which type of normalization layer
better fits FL on non-IID data. Indeed, when data are non-IID, batch
statistics do not represent the global statistics, leading NNs equipped with
BN to poor results. The most common alternatives to BN are: Group
Normalization (GN) [18], Layer Normalization (LN) [19], Instance Normalization
(IN) [20] and Batch Renormalization (BRN) [21]. To the best of our knowledge,
there are no works benchmarking normalization layers for FL on non-IID data. A
previous work [22], proposing a novel form of Transfer Learning through test-
time parameters’ aggregation, shows that a NN with Batch Normalization [14]
does not learn at all, while performance improves only when using Group
Normalization [18]. Andreaux et al. propose a novel FL approach by introducing
local-statistic BN layers [23]. Their method, called SiloBN, consists in only
sharing the learned BN parameters $\gamma$ and $\beta$ across clients, while
BN statistics $\mu$ and $\sigma^{2}$ remain local, allowing the training of a
model robust to the heterogeneity of the different centres. SiloBN showed
better intra-centre generalization capabilities than existing FL methods.
FedBN [24] is a FL algorithm that excludes BN layers from the averaging step,
outperforming both FedAvg and FedProx in a feature distribution skew setting.

##  III Normalization Layers

The majority of the FL algorithms simply apply an aggregation function (like
averaging) to all the components of a NN, including weights and biases of the
normalization layers. Most of the common NN architectures, like residual
networks [25], adopt BN [14] as normalization layer. However, in contexts like
Federated or Transfer Learning, BN may not be the optimal choice, especially
when dealing with non-IID data. In this chapter will be reviewed the main
characteristics of Batch Normalization and several possible alternatives like
Group Normalization (GN) [18], Layer Normalization (LN) [19], Instance
Normalization (IN) [20] and Batch Renormalization (BRN) [21].

###  III-A Batch Normalization

Batch Normalization has seen a recent extensive adoption by neural networks
for their training. The key issue that BN tackles is Internal Covariate Shift
(ICS), which is the change in the distribution of the data (or network
activations), i.e. the input variables of training and test sets. Informally,
at each epoch of training, weights are updated, input data are different, and
the algorithm faces difficulties. This results in a slower and more difficult
training process because lower learning rates and careful parameter
initialization are then required. BN attempts to reduce ICS by normalizing
activations to stabilize the mean and variance of the layer’s inputs. This
accelerates training by allowing the use of higher learning rates and reduces
the impact of the initialization. During training, BN normalizes the output of
the previous layers along the batch size, height and width axes to have zero
mean and unit variance:

$\hat{x}_{i}=\frac{x_{i}-\mu_{m}}{\sqrt{\sigma^{2}_{m}+\epsilon}}$

where $x,\mu_{m}$ and $\sigma^{2}_{m}$ are respectively the input, the mean
and the variance of a minibatch $m$, and $\epsilon$ is arbitrarily constant
greater than zero used for stability in case the denominator is zero. BN also
adds two learnable parameters, $\gamma$ and $\beta$ that are a scaling and a
shifting step, to fix the representation in case the normalization alters what
the layer represents: $y_{i}=\gamma\hat{x_{i}}+\beta$. Normalized activations
will depend on the other samples contained in the minibatch. In the test
phase, BN can not calculate statistics; otherwise, it will learn from the test
dataset, so it uses the moving averages of minibatch means and variances of
the training set. In the case of IID mini-batches, statistical estimations
will be accurate if the batch size is large; otherwise, inaccuracies will be
compounded with depth, reducing the quality of the models. Non-IID data can
have a more detrimental effect on models equipped with BN because batch
statistics do not represent global statistics, leading to even worse results.
Therefore there is a need to investigate alternatives to BN that can work well
with non-IID data and small batch sizes.

###  III-B Group Normalization

Group Normalization is a simple alternative to BN. It divides the channels
into different groups and computes within each group the mean $\mu_{i}$ and
the variance $\sigma_{i}$ along the height and width axes. GN overcomes the
constraint on the batch size because it is completely independent of the other
input features in the batch, and its accuracy is stable in a wide range of
batch size. Indeed, GN has a 10.6$\%$ lower error than BN on ResNet-50 [25]
trained on ImageNet [18]. The number of groups G is a pre-defined
hyperparameter which needs to divide the number of channels C. When G=C, it
means that each group contains one channel, and GN becomes Instance
Normalization, while when G=1, it means that one group contains all the
channels, and GN becomes Layer Normalization. Both Instance and Layer
Normalizations are described below.

###  III-C Instance Normalization

Instance Normalization is another alternative to BN, firstly proposed for
improving NN performances in image generation. It can be seen as a Group
Normalization with G=C or as a BN with a batch size of one, so applying the BN
formula to each input feature individually. Indeed, IN computes the mean
$\mu_{i}$ and the variance $\sigma_{i}$ along the height and width axes. As
stated before, BN suffers from small batch sizes, so we expect that
experiments made with IN will produce worse results than the ones with BN or
GN, which can exploit the dependence across the channels.

###  III-D Layer Normalization

Layer Normalization was first proposed to stabilize hidden state dynamics on
Recurrent Neural Networks (RNNs) [19]. It computes the mean and the variance
along the channel, height and width axes. LN overcomes the constraint on the
batch size because it is completely independent of the other input features in
the batch. LN performs the same computation both at training and inference
times. It can be seen as a GN with G=1, so with only one group controlling all
the channels. As a result, when there are several distributions to be learned
among the group of channels, it can perform worse than GN.

###  III-E Batch Renormalization

Batch Renormalization [21] is an extension of BN that ensures training and
inference models generate the same outputs that depend on individual examples
rather than the entire minibatch. BRN is an augmentation of a network which
contains batch normalization layers with a per-dimension affine transformation
applied to the normalized activations to ensure the match between training and
inference models. Reducing the dependence of activation of each sample with
other samples in the minibatch can result in a performance increase when data
are non-IID.

##  IV Non-IID Data

The most common non-IID data settings are reviewed in [1] that lists five
different partitioning strategies: 1) quantity skew, 2) labels quantity skew
(prior shift), 3) feature distribution skew (covariate shift), 4) same labels
but different features and 5) same features but different labels. In this
paper, we consider the same distributions tested in [13, 12] apart from
quantity skew, which is not treated. Indeed, [13, 12] showed that quantity
skew does not hurt the performance of FL models, probably because it results
in a different quantity of samples per client, but the distribution of samples
is uniform, which is easy to deal with. In this paper, labels quantity skew,
which is the most detrimental to the FL models’ performance, has been
extensively tested in a lot of scenarios to show how it is possible to
overcome its difficulties. The cases adopted (both IID and non-IID) are
briefly described.

  * •

Uniform Distribution (IID): each client of the federation holds the same
amount of data, and the distribution is uniform among parties. This is the
simplest case for FL algorithms because the distribution is IID.

  * •

Labels Quantity Skew: the marginal distributions of labels $P(y_{i})$ vary
across parties, even if $P(x_{i}|y_{i})$ is the same. This especially happens
when dealing with real FL applications where clients of the federation are
distributed among different world regions. Certain data are present only in
some countries, leading to the labels quantity skew. In this work, we adopted
the simplest version of labels quantity skew, where each client holds samples
belonging to only a fixed amount of classes. In our experiments, we used two
as number of classes per client. Other versions of labels quantity skew can be
the Dirichlet labels skew (each client holds samples such that classes are
distributed according to the Dirichlet function) and the Pathological labels
skew (data are firstly sorted by label and then divided in shards).

  * •

Feature Distribution Skew: the marginal distributions $P(x_{i})$ vary across
parties, even if $P(y|x)$ is shared. This can happen in a lot of ML scenarios;
for example, in handwriting recognition, the same words can be written with
different styles, stroke widths, and slants. The covariate shift was obtained
according to the procedure described in [5]: samples are distributed among
clients according to the results of a Principal Component Analysis (PCA)
performed on the data.

##  V Experiments

Our experiments have been realized using OpenFL [26], the new framework for FL
developed by Intel Internet of Things Group and Intel Labs. OpenFL is a Python
3 library for FL that enables organizations to collaboratively train a model
without sharing sensitive information. OpenFL is DL framework-agnostic.
Training of statistical models may be done with any deep learning framework,
such as TensorFlow or PyTorch, via a plugin mechanism. OpenFL is based on a
Director-Envoy workflow which uses long-lived components in a federation to
distribute more experiments in the federation. The Director is the central
node of the federation. It starts an Aggregator for each experiment, sends
data to connected collaborator nodes, and provides updates on the status. The
Envoy runs on Collaborator nodes connected to the Director. When the Director
starts an experiment, the Envoy starts the Collaborator to train the global
model. All the experiments were computed in a distributed environment with ten
collaborators. Each collaborator is run on an Intel Xeon CPU (8 cores per
CPU), 1 Tesla T4 GPU. The code used for experimental evaluation is publicly
available at https://github.com/alpha-unito/Benchmarking-Normalization-Layers-
in-Federated-Learning-for-Image-Classification-Tasks-on-non-IID.

Dataset: We tested FedAvg on MNIST [15] and CIFAR10 [16], that are default
benchmarks in ML literature. The details of the datasets are summarized in
Table I.

TABLE I: Statistics of the datasets.
Dataset | Train samples | Test samples | # labels
---|---|---|---
MNIST | 60.000 | 10.000 | 10
CIFAR10 | 50.000 | 10.000 | 10

Preprocessing: both datasets were not rescaled: MNIST images are 28x28 while
CIFAR10 images are 32x32. As for data augmentation, we performed random
horizontal flips and random crop with a probability of 50%. All the datasets
were normalized according to their mean and standard deviation.

Model: We employed ResNet-18 [25] and EfficientNet-B0 [27] as classification
models, trained by minimizing the cross-entropy loss with mini-batch gradient
descent using the Adam optimizer with learning rate $10^{-3}$. The local batch
size was 128. We used two networks to show that the results are not model-
dependent (See Appendix for the EfficientNet-B0’s results). The scores of
baseline models and federated experiments on the uniform and non-IID settings
(section. V-A, Tables II, IV and V) are the average ($\pm$ standard deviation)
over five runs. For the extensive experiments on batch size, number of local
training steps, and number of clients, we tested only ResNet-18 for only one
run.

Normalization Layers: All the normalization layers described before, i.e.
Batch Norm, Group Norm, Instance Norm, Layer Norm and Batch Renormalization,
have been applied to the classification model in each experiment. For the most
promising normalization layers have been ran additional tests to study the
impact of the batch size, the number of epochs per round and the number of
clients. For BN, we set the momentum, i.e. the importance given to the
previous moving average, to 0.9, according to the SOTA [28] for ResNet-18. For
GN, the number of channels must be divisible by the number of groups, so we
set the number of groups to 32 for ResNet-18 (one of the possible divisors)
and 8 for EfficientNet-B0 (the only possible divisor). All the other
normalization layers have been used with their standard PyTorch configuration.

Top-1 accuracy has been employed as a classification metric to compare the
performance. Results show the best aggregated model’s accuracy. The learning
curve of all the experiments can be studied from Figure 1 to Figure 5. Table
II reports about a non-federated baseline, i.e., the typical AI scenario where
the data are centralized. The remaining tables show the performance of FedAvg
in different data partitioning scenarios and for different values of some
hyperparameters such as batch size, number of epochs per round and number of
clients.

TABLE II: Accuracy in the non-federated setting.
Dataset | BN | GN | IN | LN | BRN
---|---|---|---|---|---
MNIST | $99.22\%\pm 0.07\%$ | $99.29\%\pm 0.06\%$ | $11.36\%\pm 0.00\%$ | $\textbf{99.32\%}\pm\textbf{0.07\%}$ | $99.00\%\pm 0.20\%$
CIFAR10 | $\textbf{82.60\%}\pm\textbf{0.24\%}$ | $81.61\%\pm 0.24\%$ | $9.89\%\pm 0.00\%$ | $82.06\%\pm 0.40\%$ | $81.77\%\pm 0.23\%$

###  V-A Normalization Layers and non-IID data

This subsection presents the results of the three data partitioning scenarios
presented: uniform, labels quantity skew and covariate shift. Table III shows
that normalization levels have a huge impact on the performance of a NN,
ranging from very poor levels to almost reaching the level of accuracy in the
centralized case. ResNet-18-LN performs slightly better than BN and GN while
outperforming IN and BRN in the uniform setting (fig. 1a). In both the labels
quantity skew and the covariate shift scenarios, both GN and LN outperform all
the other normalization layers; however, they require more training steps to
converge, as shown in fig. 1b and fig. 1c. IN does not learn in FL; indeed,
since both MNIST and CIFAR10 have ten classes, ResNet-18-IN’s performance is
like tossing a coin. BRN seems to have a very long learning curve; in fact, it
needs a lot of training rounds to reach convergence. However, its performance
is still far from the best performances of BN, GN and LN. For this reason, the
following subsections will report results only for the most promising
normalization layers: BN, GN and LN.

TABLE III: Accuracy in the uniform setting.
Dataset | BN | GN | IN | LN | BRN
---|---|---|---|---|---
MNIST | $97.12\%\pm 0.24\%$ | $98.26\%\pm 0.19\%$ | $11.28\%\pm 0.04\%$ | $\textbf{98.51\%}\pm\textbf{0.06\%}$ | $86.54\%\pm 4.24\%$
CIFAR10 | $47.12\%\pm 0.82\%$ | $53.99\%\pm 0.23\%$ | $10.04\%\pm 0.07\%$ | $\textbf{59.06\%}\pm\textbf{0.25\%}$ | $32.36\%\pm 1.07\%$
TABLE IV: Accuracy in the labels quantity skew setting.
Dataset | BN | GN | IN | LN | BRN
---|---|---|---|---|---
MNIST | $82.25\%\pm 7.70\%$ | $97.32\%\pm 0.46\%$ | $18.53\%\pm 4.18\%$ | $\textbf{97.68\%}\pm\textbf{0.22\%}$ | $74.66\%\pm 3.20\%$
CIFAR10 | $38.02\%\pm 1.64\%$ | $\textbf{58.91\%}\pm\textbf{3.10\%}$ | $19.00\%\pm 5.47\%$ | $57.98\%\pm 3.88\%$ | $27.34\%\pm 5.56\%$
TABLE V: Accuracy in the covariate shift setting.
Dataset | BN | GN | IN | LN | BRN
---|---|---|---|---|---
MNIST | $96.42\%\pm 0.25\%$ | $\textbf{97.96\%}\pm\textbf{0.19\%}$ | $11.51\%\pm 0.23\%$ | $97.37\pm 1.63\%$ | $91.01\%\pm 1.12\%$
CIFAR10 | $44.58\%\pm 1.96\%$ | $51.04\%\pm 2.83\%$ | $10.08\%\pm 0.23\%$ | $\textbf{55.54}\pm\textbf{2.22\%}$ | $26.20\%\pm 1.48\%$

(a) IID MNIST

(b) Non-IID (labels quantity skew) MNIST

(c) Non-IID (covariate shift) MNIST

(d) IID CIFAR10

(e) Non-IID (labels quantity skew) CIFAR10

(f) Non-IID (covariate shift) CIFAR10

Figure 1: Accuracies of ResNet-18 on the uniform and non-IID cases. BN, GN,
and LN require few rounds to reach convergence in uniform setting, while they
need more training steps to converge in non-IID scenarios. It is clearly shown
that BRN requires a very long learning curve and that IN does not learn in FL.

###  V-B Normalization Layers and batch size

TABLE VI: Accuracy in the labels quantity skew setting as the batch size
varies.
Dataset | Batch size | BN | GN | LN
---|---|---|---|---
MNIST | 8 | $94.77\%$ | $97.85\%$ | $98.32\%$
16 | $81.02\%$ | $97.92\%$ | $98.83\%$
32 | $98.00\%$ | $97.93\%$ | $98.36\%$
64 | $73.89\%$ | $95.91\%$ | $98.13\%$
128 | $87.07\%$ | $96.84\%$ | $97.39\%$
256 | $90.33\%$ | $97.22\%$ | $97.39\%$
512 | $87.86\%$ | $97.97\%$ | $96.99\%$
CIFAR10 | 8 | $40.23\%$ | $59.95\%$ | $66.63\%$
16 | $45.41\%$ | $61.87\%$ | $56.42\%$
32 | $42.26\%$ | $55.61\%$ | $52.70\%$
64 | $43.02\%$ | $59.43\%$ | $55.07\%$
128 | $38.02\%$ | $60.17\%$ | $60.36\%$
256 | $46.69\%$ | $56.18\%$ | $56.92\%$
512 | $35.86\%$ | $51.92\%$ | $59.73\%$
Figure 2: Accuracies on CIFAR10 and different batch sizes. Accuracy degrades
when the batch size becomes too large.

We examined the effect of a range of batch sizes on training NNs with
different normalization layers (Tab. VI and Fig.2). We trained ResNet-18 on
both MNIST and CIFAR10 with a batch size of 8, 16, 32, 64, 128, 256 and 512.

In Tab.VI we can see that GN and LN variants of ResNet-18 consistently
outperform BN (batch sizes 8 and 16). In all three variants, the accuracy
degrades when the batch size becomes too large (in almost all cases, there is
a significant drop in performance when passing batch size 256 to 512). A
possible explanation for this phenomenon is that, as stated in [29], ”the lack
of generalization ability is due to the fact that large-batch methods tend to
converge to sharp minimizers of the training function”. This is especially
true in contexts such as FL, where clients have fewer data than in centralized
scenarios, and therefore increasing the batch size has a greater effect.

###  V-C Normalization layers and number of epochs per round

We considered two types of experiments to study how accuracy is affected by
the number of epochs per round:

  * •

Fix the number of rounds and increase the number of local training steps
(Tab.VII).

  * •

Fix the number of training epochs to 1000 and vary the ratio of epochs to
rounds (Tab. VIII).

It can be noted that models take benefit from more local steps of gradient
descent before doing aggregation. Indeed, accuracy increases as E increases. A
possible explanation is that this happens because clients of the federation
share a similar loss function shape, and going more and more towards the local
minima can be beneficial to reach global optima.

Interestingly, when E=1, BN converges quickly, while GN and LN require more
training steps to converge. However, when E increases to 10 or 100, BN also
requires more rounds to reach convergence, while the learning curves of GN and
LN are unaffected by significant changes.

These results can also be analyzed from a communication point of view: with
the same amount of epochs, less communication achieves better results. For
example, on CIFAR10, ResNet-GN with E=2 and 500 rounds achieves higher
accuracy than ResNet-GN with E=1 and 1000 rounds (Fig.4). This means that
perhaps counter-intuitively, training locally before performing aggregation
can boost the model’s accuracy. This seems to indicate that pursuing local
optimizations can lead to better approximations of the local optima. However,
at a certain point, increasing E and reducing the number of rounds decreases
the performance. This pattern is clearly visible with all the normalization
layers and in both datasets. Table VIII shows that we always need an
appropriate ratio of epochs to round.

(a) BN

(b) GN

(c) LN

Figure 3: Accuracies on CIFAR10 and different epochs per round. Accuracy
increases as BN converges lates as E increases, while GN and LN follow an
inverse pattern.

(a) BN

(b) GN

(c) LN

Figure 4: Accuracies on CIFAR10 fixing the number of epochs to 1000 and
varying the ratio of epochs to rounds. TABLE VII: Accuracy in the labels
quantity skew setting as the number of epochs per round varies.
Dataset | Epochs | BN | GN | LN
---|---|---|---|---
MNIST | 1 | $87.07\%$ | $96.84\%$ | $97.39\%$
10 | $79.82\%$ | $98.83\%$ | $98.95\%$
100 | $92.88\%$ | $97.83\%$ | $99.04\%$
CIFAR10 | 1 | $38.02\%$ | $60.17\%$ | $60.36\%$
10 | $55.53\%$ | $54.89\%$ | $63.68\%$
100 | $54.19\%$ | $73.32\%$ | $68.40\%$
TABLE VIII: Comparison between different epochs per rounds in labels quantity
skew setting.
Dataset | Epochs | Rounds | BN | GN | LN
---|---|---|---|---|---
MNIST | 1 | 1000 | $91.15\%$ | $98.92\%$ | $98.77\%$
2 | 500 | $84.68\%$ | $98.85\%$ | $99.05\%$
5 | 200 | $92.42\%$ | $98.33\%$ | $99.12\%$
10 | 100 | $79.40\%$ | $98.46\%$ | $98.69\%$
20 | 50 | $77.86\%$ | $97.17\%$ | $97.26\%$
50 | 20 | $54.18\%$ | $79.77\%$ | $74.40\%$
100 | 10 | $60.29\%$ | $78.02\%$ | $78.86\%$
CIFAR10 | 1 | 1000 | $43.12\%$ | $56.94\%$ | $57.03\%$
2 | 500 | $50.09\%$ | $65.14\%$ | $62.63\%$
5 | 200 | $47.85\%$ | $63.68\%$ | $64.56\%$
10 | 100 | $51.37\%$ | $51.23\%$ | $61.22\%$
20 | 50 | $48.16\%$ | $54.33\%$ | $65.36\%$
50 | 20 | $38.92\%$ | $56.53\%$ | $52.67\%$
100 | 10 | $33.06\%$ | $48.24\%$ | $50.56\%$

###  V-D Normalization layers and number of clients

TABLE IX: Accuracy in the labels quantity skew setting as the number of
collaborators varies.
Dataset | Clients | BN | GN | LN
---|---|---|---|---
MNIST | 2 | $99.02\%$ | $99.61\%$ | $99.85\%$
4 | $91.85\%$ | $99.47\%$ | $98.36\%$
8 | $88.38\%$ | $96.49\%$ | $97.68\%$
10 | $87.07\%$ | $96.84\%$ | $97.39\%$
CIFAR10 | 2 | $44.57\%$ | $74.15\%$ | $75.63\%$
4 | $55.30\%$ | $50.50\%$ | $67.69\%$
8 | $51.27\%$ | $60.75\%$ | $65.00\%$
10 | $38.02\%$ | $60.17\%$ | $60.36\%$

We tested the scalability of FL by measuring the effect of the number of
clients of the federation, as shown in Fig. 5, and considering two types of
experiments:

  * •

a labels quantity skew split of the dataset across a different number of
clients (namely 2, 4, 8 and 10). Results are reported in Table IX.

  * •

a uniform dataset split across clients, but considering only some parties.
Here the idea is to show how increasing the number of participants, and so the
quantity of data, can be beneficial to the federation. Results are reported in
Table X.

We can observe (Table IX) that the accuracy significantly increases when
decreasing the number of clients. Indeed, when the number of parties is small,
the amount of local data increases, leading to better local models, and also
aggregating fewer models can result in less information loss. Moreover, we can
note the importance of normalization layers in FL: GN and LN variants of
ResNet-18 in the ten clients scenario perform better than BN in a two clients
scenario on CIFAR10, while on MNIST there is only a slightly drop.
Table X shows the results in an IID scenario considering only some shards of
the dataset. In this case, the amount of local data remains the same in each
configuration; however, the federation’s total amount of data varies according
to the number of parties. Increasing the quantity of data in the federation by
increasing the number of clients benefits the aggregated model.

Figure 5: Accuracies on CIFAR10 and different number of clients. Accuracy of
ResNet-18-BN on 2 parties is still lower than ResNet-18-GN/LN on 10 parties.
TABLE X: Accuracy in the i.i.d. setting using only some shards of the dataset.
Dataset | Clients | BN | GN | LN
---|---|---|---|---
MNIST | 2 | $94.64\%$ | $96.26\%$ | $97.38\%$
4 | $96.18\%$ | $97.52\%$ | $98.10\%$
8 | $96.79\%$ | $97.98\%$ | $98.34\%$
10 | $97.12\%$ | $98.26\%$ | $98.51\%$
CIFAR10 | 2 | $41.80\%$ | $45.37\%$ | $51.73\%$
4 | $43.47\%$ | $48.74\%$ | $54.55\%$
8 | $45.81\%$ | $53.18\%$ | $58.36\%$
10 | $47.12\%$ | $53.99\%$ | $59.06\%$

##  VI Conclusions

This work aims to improve the effectiveness of federated learning, focusing on
hyper-parameter optimization, starting from understanding which hyper-
parameters specifically affect the training of a federated model differently
from centralized training. We specifically focused on layer normalization,
which is also a hyper-parameter of centralized training, and frequency of
model aggregation, which is not an issue in centralized training.

We experimented with two network architectures and five normalization layers
on two public image datasets. We tested Batch, Group, Instance, Layer
Normalization and Batch Renormalization in the uniform, label quantity skew
and covariate shift settings. Although BN is the state-of-the-art for
classical ML techniques, in our experiments, GN and LN outperformed the other
normalization layers in all the FL partitioning strategies.

Through extensive experimentation, we analyzed how the batch size, the number
of epochs per round, the number of rounds and the number of clients of the
federation affect the aggregated model performance. These additional tests
have been conducted in the labels quantity skew scenario, which is the most
challenging for FL algorithms, considering the best three normalization
layers: BN, GN and LN.

GN and LN outperform BN in almost all the tests. Results show that regardless
of the batch size, GN and LN consistently outperform BN, although batch size
affects the model’s performance in all cases. Unexpectedly, we observed that
the plot of the quality of the model against the frequency of model
aggregation (epochs per round) consistently exhibits a maximum at a few epochs
per round. For FL, the number of epochs per round exhibits similar behaviour
of batch size for centralized training.

Eventually, we tested the scalability of FL systems. We noted that FL is not
scalable under strong scalability assumption, i.e. increasing the number of
clients while maintaining constant the size of local datasets. However, GN and
LN on ten clients still outperform BN on two clients. The scalability has also
been tested in the IID scenario under the weak scalability assumption, i.e.
increasing the number of clients while maintaining constant the size of the
local dataset per client. In this case, the federation’s data changes with the
number of clients, and the model’s performance increases with the number of
parties.

## Acknowledgements

This work has been supported by the Spoke ”FutureHPC $\&$ BigData” of the ICSC
– Centro Nazionale di Ricerca in ”High Performance Computing, Big Data and
Quantum Computing”, funded by European Union – NextGenerationEU, and by the
European Union within the H2020 RIA “European Processor Initiative—Specific
Grant Agreement 2” G.A. 826647, https://www.european-processor-initiative.eu/.

## References

  * [1] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. A. Bonawitz, Z. Charles, G. Cormode, R. Cummings, R. G. L. D’Oliveira, H. Eichner, S. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, A. Gascón, B. Ghazi, P. B. Gibbons, M. Gruteser, Z. Harchaoui, C. He, L. He, Z. Huo, B. Hutchinson, J. Hsu, M. Jaggi, T. Javidi, G. Joshi, M. Khodak, J. Konečný, A. Korolova, F. Koushanfar, S. Koyejo, T. Lepoint, Y. Liu, P. Mittal, M. Mohri, R. Nock, A. Özgür, R. Pagh, H. Qi, D. Ramage, R. Raskar, M. Raykova, D. Song, W. Song, S. U. Stich, Z. Sun, A. T. Suresh, F. Tramèr, P. Vepakomma, J. Wang, L. Xiong, Z. Xu, Q. Yang, F. X. Yu, H. Yu, and S. Zhao, “Advances and open problems in federated learning,” _Found. Trends Mach. Learn._ , vol. 14, no. 1-2, pp. 1–210, 2021.
  * [2] S. Callaghan, “Data sharing in a time of pandemic,” _Patterns_ , vol. 1, no. 5, p. 100086, 2020.
  * [3] P. Voigt and A. v. d. Bussche, _The EU General Data Protection Regulation (GDPR): A Practical Guide_ , 1st ed. Springer Publishing Company, Incorporated, 2017.
  * [4] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Proc. of the 20th Intl. Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA_ , ser. Proc. of Machine Learning Research, A. Singh and X. J. Zhu, Eds., vol. 54. PMLR, 2017, pp. 1273–1282.
  * [5] M. Polato, R. Esposito, and M. Aldinucci, “Boosting the federation: Cross-silo federated learning without gradient descent,” in _International Joint Conference on Neural Networks, IJCNN 2022, Padua, Italy, July 18-23, 2022_. IEEE, 2022, pp. 1–10.
  * [6] Y. Arfat, G. Mittone, I. Colonnelli, F. D’Ascenzo, R. Esposito, and M. Aldinucci, “Pooling critical datasets with federated learning,” in _Proc. of 31st Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)_. Napoli, Italy: IEEE, 2023.
  * [7] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” _CoRR_ , vol. abs/1907.02189, 2019.
  * [8] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in _Proc. of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020_ , I. S. Dhillon, D. S. Papailiopoulos, and V. Sze, Eds. mlsys.org, 2020.
  * [9] J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” in _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020.
  * [10] S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh, “SCAFFOLD: stochastic controlled averaging for federated learning,” in _Proc. of the 37th Intl. Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , ser. Proc. of Machine Learning Research, vol. 119. PMLR, 2020, pp. 5132–5143.
  * [11] N. Shoham, T. Avidor, A. Keren, N. Israel, D. Benditkis, L. Mor-Yosef, and I. Zeitak, “Overcoming forgetting in federated learning on non-iid data,” _CoRR_ , vol. abs/1910.07796, 2019.
  * [12] Q. Li, Y. Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An experimental study,” in _38th IEEE International Conference on Data Engineering, ICDE 2022, Kuala Lumpur, Malaysia, May 9-12, 2022_. IEEE, 2022, pp. 965–978.
  * [13] B. Casella, R. Esposito, C. Cavazzoni, and M. Aldinucci, “Benchmarking fedavg and fedcurv for image classification tasks,” in _Proc. of th 1st Italian Conference on Big Data and Data Science, ITADATA22, Milan, September 20-21, 2022_ , 2022.
  * [14] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , ser. JMLR Workshop and Conference Proceedings, F. R. Bach and D. M. Blei, Eds., vol. 37. JMLR.org, 2015, pp. 448–456.
  * [15] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proc. IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [16] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” pp. 32–33, 2009.
  * [17] J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni, “Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification,” _CoRR_ , vol. abs/2110.14795, 2021.
  * [18] Y. Wu and K. He, “Group normalization,” in _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII_ , ser. Lecture Notes in Computer Science, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds., vol. 11217. Springer, 2018, pp. 3–19.
  * [19] L. J. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _CoRR_ , vol. abs/1607.06450, 2016.
  * [20] D. Ulyanov, A. Vedaldi, and V. S. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _CoRR_ , vol. abs/1607.08022, 2016\.
  * [21] S. Ioffe, “Batch renormalization: Towards reducing minibatch dependence in batch-normalized models,” in _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 1945–1953.
  * [22] B. Casella, A. B. Chisari, S. Battiato, and M. V. Giuffrida, “Transfer learning via test-time neural networks aggregation,” in _Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISIGRAPP 2022, Volume 5: VISAPP, Online Streaming, February 6-8, 2022_ , G. M. Farinella, P. Radeva, and K. Bouatouch, Eds. SCITEPRESS, 2022, pp. 642–649.
  * [23] M. Andreux, J. O. du Terrail, C. Beguier, and E. W. Tramel, “Siloed federated learning for multi-centric histopathology datasets,” in _Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning - Second MICCAI Workshop, DART 2020, and First MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4-8, 2020, Proceedings_ , ser. Lecture Notes in Computer Science, S. Albarqouni, S. Bakas, K. Kamnitsas, M. J. Cardoso, B. A. Landman, W. Li, F. Milletari, N. Rieke, H. Roth, D. Xu, and Z. Xu, Eds., vol. 12444. Springer, 2020, pp. 129–139.
  * [24] X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou, “Fedbn: Federated learning on non-iid features via local batch normalization,” in _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * [25] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_. IEEE Computer Society, 2016, pp. 770–778.
  * [26] G. A. Reina, A. Gruzdev, P. Foley, O. Perepelkina, M. Sharma, I. Davidyuk, I. Trushkin, M. Radionov, A. Mokrov, D. Agapov, J. Martin, B. Edwards, M. J. Sheller, S. Pati, P. N. Moorthy, H. S. Wang, P. Shah, and S. Bakas, “Openfl: An open-source framework for federated learning,” _CoRR_ , vol. abs/2105.06413, 2021.
  * [27] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 2019, pp. 6105–6114.
  * [28] T. Moreau, M. Massias, A. Gramfort, P. Ablin, P. Bannier, B. Charlier, M. Dagréou, T. D. la Tour, G. Durif, C. F. Dantas, Q. Klopfenstein, J. Larsson, E. Lai, T. Lefort, B. Malézieux, B. Moufad, B. T. Nguyen, A. Rakotomamonjy, Z. Ramzi, J. Salmon, and S. Vaiter, “Benchopt: Reproducible, efficient and collaborative optimization benchmarks,” _CoRR_ , vol. abs/2206.13424, 2022.
  * [29] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,” in _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.

Bruno Casella is currently a PhD student in Modeling and Data Science, at the
University of Turin. He received the B.S. degree in Computer Engineering in
2020, and the M. Sc. in Data Science for management from the University of
Catania. His current research interests include federated learning, transfer
learning and distributed computing.
---

x

Roberto Esposito is an associate professor at the University of Turin. He
received the PhD in Computer Science in 2003 from the University of Turin.
---
Antonio Sciarappa is currently a researcher at the Leonardo Labs of Genova. He
received the PhD in Theoretical Particle Physics in 2015 from the SISSA. His
current research interests include parallel and distributed computing.
---
Carlo Cavazzoni is currently Senior Vice President of Cloud Computing and Head
of the High Performance Computing of Leonardo S.p.A. He received the PhD in
Computational Physics from the SISSA.
---
Marco Aldinucci is a full professor and P.I. of the Parallel Computing
research group at the University of Turin. He received his PhD from the
University of Pisa (2003), and he has been a researcher at the Italian
National Research Council (CNR).
---

## Appendix

Accuracies on the uniform and non-iid data settings using and EfficientNet-B0
[27].

TABLE XI: Accuracy in the uniform setting with EfficientNet-B0
Dataset | BN | GN | IN | LN | BRN
---|---|---|---|---|---
MNIST | $65.09\%$ | $95.07\%$ | $11.26\%$ | $96.12\%$ | $11.02\%$
CIFAR10 | $29.88\%$ | $29.93\%$ | $10.08\%$ | $38.98\%$ | $10.48\%$
TABLE XII: Accuracy in the labels quantity skew setting with EfficientNet-B0
Dataset | BN | GN | IN | LN | BRN
---|---|---|---|---|---
MNIST | $55.83\%$ | $91.91\%$ | $24.13\%$ | $94.35\%$ | $45.69\%$
CIFAR10 | $29.98\%$ | $27.53\%$ | $14.92\%$ | $37.71\%$ | $22.61\%$
TABLE XIII: Accuracy in the covariate shift setting with EfficientNet-B0
Dataset | BN | GN | IN | LN | BRN
---|---|---|---|---|---
MNIST | $65.10\%$ | $94.86\%$ | $11.19\%$ | $95.81\%$ | $61.90\%$
CIFAR10 | $26.78\%$ | $28.01\%$ | $10.16\%$ | $48.10\%$ | $10.01\%$