aainstitutetext: Bogoliubov Laboratory of Theoretical Physics JINR, 141980
Dubna, Russia

# The Gorishny-Isaev vacuum integrations and UV(IR)-regime

I. V. Anikin<EMAIL_ADDRESS>
###### Abstract

We present the further development of the vacuum massless integrations. In
particular, in the Gorishny-Isaev formula, it has been shown that the delta
function representing UV-regime should be treated within the sequential
approach. It allows us to resolve the problem of vacuum integrations related
to the analytical continuation of diagram indices.

Based on the materials presented as a talk at The XXVth International Baldin-
Burov Seminar on High Energy Physics Problems, September, 21st 2023, Dubna

##  1 Introduction

The effective potential computation with multi-loop accuracy in the different
quantum field models plays an very important role for the studies, for
example, of the spontaneous symmetry breaking manifestation. As well-known,
the calculation of effective potential involves ultimately the vacuum
integrations. The presence of massive parameters (or particle masses) in the
theory can significantly complicate the multi-loop calculations. On the other
hand, the massless propagators in the corresponding loops simplify any multi-
loop calculations. Moreover, the massless loops give a possibility to use of
conformal symmetry Anikin:2023wkk ; Anikin:2023ogb . The price to be paid for
the simplification is the necessity of operation with the
$\delta(0)$-singularity after the use of Gorishny-Isaev (GI) vacuum
integrations Gorishnii:1984te .

In the paper, we demonstrate that $\delta(0)$-singularity can be treated as
either the ultraviolet (IV) or infrared (IR) regime. To avoid working with the
singular generated function, we also advocate the use of sequential approach
to such kinds of generated functions. It allows us to resolve a problem with
the corresponding analytical extension of the GI vacuum integration up to the
real indices of diagrams, see Gorishnii:1984te ; Anikin:2020dlh .

Schematically, the process of effective potential computation can be presented
as

(7)

##  2 The massless loop integration in dimensional regularization

We adhere the dimensional regularization where the loop integration measure is
defined for $D=d-2\varepsilon$. Due to the dimensional analysis, the vacuum
massless integration gives

$\displaystyle\int\,\frac{d^{D}k}{[k^{2}]^{n}}=0\quad\text{for}\quad\forall\,\,n.$
(8)

However, if $n=D/2$ the dimensional analysis arguments do not work.
Nevertheless, we can check that the zero is achieved owing to the
cancellation, each other, of ultraviolet and infrared divergencies
Grozin:2005yg . As a rule, for the effective potential calculations, we
concentrate on only the ultraviolet (UV) divergences Anikin:2023wkk ;
Anikin:2023ogb . Hence, focusing on the only UV-divergency we deal with the
non-zero contribution which is an object of our consideration. In the case of
the IR-divergency, our argumentation is not much different because the IR-
regime can be obtained from the UV-regime by the corresponding replacement
$\varepsilon\to-\varepsilon$.

##  3 On the singular generated functions

It is worth to remind that every singular functional is being a limit of the
regular functional, i.e.

$\displaystyle\delta(x)=\lim_{\varepsilon\to
0}\delta_{\varepsilon}(x)\quad\text{\footnotesize in the function space}$
$\displaystyle\Rightarrow\lim_{\varepsilon\to 0}\int
d\mu(x)\delta_{\varepsilon}(x)\quad\text{\footnotesize in the functional
space},$ (9)

where the integration measure $d\mu(x)=dx\,\phi(x)$ involves the restricted
(finite) function $\phi(x)$. In Eqn. (3), the function
$\delta_{\varepsilon}(x)$ can be determined by different ways. For example, we
have

$\displaystyle\delta_{\varepsilon}(x)=\Big{\\{}\frac{1}{\pi}\frac{\varepsilon}{\varepsilon^{2}+x^{2}};\quad\frac{1}{2\sqrt{\pi\varepsilon}}e^{-\frac{x^{2}}{4\varepsilon}};\quad\frac{1}{\pi}\frac{\sin
x/\varepsilon}{x}\Big{\\}}.$ (10)

If the argument of delta function is equal to zero, then we get the following
behaviour:

$\displaystyle\delta_{\varepsilon}(x)\stackrel{{\scriptstyle
x=0}}{{\thicksim}}\Big{\\{}\big{[}\frac{1}{\varepsilon}];\quad\big{[}\frac{1}{\sqrt{\varepsilon}}];\quad\big{[}\frac{1}{\varepsilon}]\Big{\\}},$
(11)

where $\stackrel{{\scriptstyle x=0}}{{\thicksim}}$ means “behaves as” if
$x=0$. One can see that Eqn. (11) gives the different parametrizations of the
$\delta(0)$-singularity. Actually, it is well-known that the
$\delta(0)$-singularity can be presented through the different meromorphic
(and not only) functions Gelfand:1964 .

##  4  $\Delta_{F}(0)$-singularity

As the first example, let us consider the tad-pole diagrams which, by
definition, include the singularity as $\Delta_{F}(0)$. Using the Fourier
transform, the propagator $\Delta_{F}(0)$ can be write as

$\displaystyle\Delta_{F}(0)=\int\frac{(d^{D}k)}{k^{2}}=\Gamma(D/2-1)\int(d^{D}z)\,\,\frac{\delta(z)}{\big{(}z^{2}\big{)}^{D/2-1}},$
(12)

where the causal prescription $+i0$ is omitted in the denominators. If we
assume that $D/2-1=0$, then

$\displaystyle\Delta_{F}(0)=\Gamma(0)\int(d^{D}z)\,\delta(z)\Rightarrow\Gamma(0),$
(13)

where

$\displaystyle\Gamma(0)=\lim_{\epsilon\to 0}\Gamma(\epsilon)=\lim_{\epsilon\to
0}\Big{\\{}\frac{1}{\epsilon}+....\Big{\\}}.$ (14)

The condition given by $D/2-1=0$ has to be applied before the integration over
$(d^{D}k)$ in order to avoid the uncertainty.

On the other hand, the vacuum integration results in the delta function
Gorishnii:1984te . A key moment of Gorishny-Isaev’s method goes as below.
Using the spherical system, $\Delta_{F}(0)$ can be represented as

$\displaystyle\Delta_{F}(0)=\int\frac{(d^{D}k)}{k^{2}}=\frac{1}{2}\int
d\Omega\int_{0}^{\infty}d\beta\,\beta^{D/2-2}.$ (15)

The replacement $\beta=e^{y}$ leads to the following expression

$\displaystyle\Delta_{F}(0)=\frac{1}{2}\int
d\Omega\int_{-\infty}^{\infty}(dy)\,e^{iy\big{[}(-i)(D/2-1)\big{]}}\stackrel{{\scriptstyle
n}}{{=}}\frac{1}{2}\delta\big{(}i[D/2-1]\big{)}=\frac{1}{2\,|i|}\delta\big{(}D/2-1\big{)}.$
(16)

The angular integration with $d\Omega$ gives only the finite numerical
coefficient.

It is worth to notice that the representation given by Eqn. (16) is valid if
and only if the delta function argument is a complex number.

Now, restoring all coefficients, it reads

$\displaystyle\Delta_{F}(0)=-\,2i\,\pi^{1+D/2}\,\delta(1-D/2)\Big{|}_{D=2}=-\,2i\,\pi^{2}\,\delta(0).$
(17)

So, for the case of $D=2$, owing to the corresponding matching, one can
observe that the singularities given by two different functions $\Gamma(0)$
and $\delta(0)$ can be parametrized in the same way, i.e.

$\displaystyle(-i)\,\Delta_{F}(0)=\Gamma(0)=-\,2\,\pi^{2}\,\delta(0).$ (18)

With this, we may conclude that $\delta(0)$-singularity can be treated as the
singularity of $\Gamma(0)$. The same inference has been reached by the
different method, see Anikin:2020dlh . Notice that the physical nature of the
mentioned singularity has been somewhat hidden.

##  5 $\delta(0)$-singularity and the sequential approach

As above-mentioned, based on the dimensional analysis, we may conclude that

$\displaystyle\mathcal{V}_{n}=\int\frac{(d^{D}k)}{[k^{2}]^{n}}=0\quad\text{for}\,\,n\not=D/2.$
(19)

The case of $n=D/2$ (or $n=2$ if $\varepsilon\to 0$) requires the special
consideration because the dimensional analysis argumentation does not now
work. The nullification of $\mathcal{V}_{D/2}$ takes still place but thanks to
different reasons. It turns out, UV- and IR- divergencies are cancelled each
other. Hence, if only the ultraviolet divergencies are under our
consideration, $\mathcal{V}_{D/2}$ is not equal to zero.

To demonstrate it, we dwell on the vacuum integration where the IR-
regularization has been applied. Notice that the mentioned IR-regularization
is needed as the external operation to be focused on the UV-regime only. In
the spherical co-ordinate system, we write the following representation
111$\mu^{2}$ plays a role of IR-regularization.

$\displaystyle\mathcal{V}_{2}=\int_{UV}\frac{(d^{D}k)}{[k^{2}]^{2}}\equiv\frac{\pi^{D/2}}{\Gamma(D/2)}\int_{\mu^{2}}^{\infty}d\beta\beta^{D/2-3}\,\,\,\,\text{with}\,\,\,\beta=|k|^{2},$
(20)

Next, calculating $\beta$-integration, we reach the representation as

$\displaystyle\mathcal{V}_{2}=\frac{\pi^{2-\varepsilon}\mu^{-2\varepsilon}}{\Gamma(2-\varepsilon)}\,\frac{1}{\varepsilon}\Big{|}_{\varepsilon\to
0},$ (21)

where the $\epsilon$-pole corresponds to the UV-divergency only (the IR-
divergency is absent by construction thanks for $\mu^{2}$) Grozin:2005yg .

On the other hand, we are able to calculate the vacuum integration by
Gorishny-Isaev’s method Gorishnii:1984te . In this case, $\mathcal{V}_{n}$
reads

$\displaystyle\mathcal{V}_{n}=\int\frac{(d^{D}k)}{[k^{2}]^{n}}=\frac{2i\,\pi^{1+D/2}}{(-1)^{D/2}\,\Gamma(D/2)}\delta(n-D/2).$
(22)

Supposing $D=4-2\varepsilon$, the only contribution is given by

$\displaystyle\mathcal{V}_{2}=\int\frac{(d^{D}k)}{[k^{2}]^{2}}=\frac{2i\,\pi^{3-\varepsilon}}{\Gamma(2-\varepsilon)}\delta(\varepsilon)\,\not=\,0.$
(23)

Hence, the delta function of argument $\epsilon$ reflects, in fact, the UV-
divergency because the representations of $\mathcal{V}_{2}$ given by Eqns.
(21) and (23) should be equivalent.

In terms of generated functions (functionals), the delta function is a linear
singular functional defined on the space of finite basic functions. We remind
that a singular generated function cannot be generated by the locally-
integrated functions. Also, the delta function can be understood with the help
of the fundamental sequences of regular functionals provided the corresponding
weak limit, see Eqn. (3) Gelfand:1964 ; Antosik:1973 . Besides, one of the
delta function representations is related to the following realization

$\displaystyle\delta(t)=\lim_{\varepsilon\to
0}\delta_{\varepsilon}(t)\equiv\lim_{\varepsilon\to
0}\frac{\Theta(-\varepsilon\leq t\leq 0)}{\varepsilon},$ (24)

where $\Theta(-\varepsilon\leq t\leq 0)$ implies the extended step-function
without any uncertainties. This representation of delta function augments the
set of possible representations, see Eqn. (10).

One can see that the treatment of $\delta(\varepsilon)$ as the linear
(singular) functional on the finite basic function space with
$d\mu(\varepsilon)=d\varepsilon\phi(\varepsilon)$ meets some difficulties
within the dimensional regularization approach. Indeed, for the practical use,
$\varepsilon$ is not a convenient variable for the construction of the finite
function space because we finally need to be focused on the limit as
$\epsilon\to 0$.

Meanwhile, within the sequential approach Antosik:1973 ; Gelfand:1964 , the
delta function might be considered as the usual singular (meromorphic)
function and the $\delta(0)$-singularity can be treated as a pole of the first
order Anikin:2020dlh ,

$\displaystyle\delta(0)=\lim_{\varepsilon\to
0}\delta_{\varepsilon}(0)\equiv\lim_{\varepsilon\to 0}\frac{1}{\varepsilon}.$
(25)

For the demanding mathematician, this representation should be understood
merely as a symbol. That is, $\delta(0)$ denotes alternatively the limit of
$1/\epsilon$.

Generally speaking, the parametrization of $\delta(0)$-singularity can be
realized by many different ways, see Eqns. (11) and (25). However, in our
consideration, the unique parametrization in the form of Eqn. (25) can be
fixed by the fact that Eqns. (21) and (23) are equivalent ones.

Thus, in our study, the $\delta(0)$-singularity is parametrized in the form of
the meromorphic function (see Eqn. (25)) within the sequential approach
Antosik:1973 ; Gelfand:1964 . Compared to Eqn. (16), the representation of
Eqn. (25) together with Eqn. (21) are not limited by the complex argument of
delta function only. This is an obvious preponderance of the sequential
approach.

##  6 Conclusions

To conclude, we have presented the important extension of the Gorishny-Isaev
massless vacuum integrations. In the note, we have demonstrated the
preponderance of sequential approach where the singular generated functions
(distributions) are treated as a fundamental sequences of regular functionals.
Due to this treatment, the singularity as $\delta(0)$ can be parametrized with
the help of the meromorphic function of first order. The uniqueness of
parametrization is shown to be ensured by the fact that the delta function
represents the UV-regime.

## Acknowledgements

We thank M. Hnatic, A. Manashov, S.V. Mikhailov and L. Szymanowski for very
useful discussions.

## References

  * (1) I. V. Anikin, “Conformal Symmetry and Effective Potential: I. Vacuum $V_{z,x}$-operation for the Green functions,” [arXiv:2306.15373 [hep-ph]].
  * (2) I. V. Anikin, “Conformal Symmetry and Effective Potential: II. Evolution,” [arXiv:2306.17018 [hep-ph]].
  * (3) S. G. Gorishnii and A. P. Isaev, Theor. Math. Phys. 62, 232 (1985) [Teor. Mat. Fiz. 62, 345 (1985)].
  * (4) I. V. Anikin, Phys. Part. Nucl. Lett. 18, 290 (2021)
  * (5) A. Grozin, “Lectures on QED and QCD,” [arXiv:hep-ph/0508242 [hep-ph]].
  * (6) I. M. Gelfand and G. E. Shilov, “Generalized Functions Vol 1 Properties And Operations,” Academic Press, 1964, ISBN-0-12-279501-6
  * (7) Antosik P., Mikusinsky Y., Sikorsky R., “Theory of Generalized Functions: A Sequential Approach,” (PWN – Polish Scientific Publisher, 1973)


# Transformer-based Federated Learning for Multi-Label Remote Sensing Image
Classification

###### Abstract

Federated learning (FL) aims to collaboratively learn deep learning model
parameters from decentralized data archives (i.e., clients) without accessing
training data on clients. However, the training data across clients might be
not independent and identically distributed (non-IID), which may result in
difficulty in achieving optimal model convergence. In this work, we
investigate the capability of state-of-the-art transformer architectures
(which are MLP-Mixer, ConvMixer, PoolFormer) to address the challenges related
to non-IID training data across various clients in the context of FL for
multi-label classification (MLC) problems in remote sensing (RS). The
considered transformer architectures are compared among themselves and with
the ResNet-50 architecture in terms of their: 1) robustness to training data
heterogeneity; 2) local training complexity; and 3) aggregation complexity
under different non-IID levels. The experimental results obtained on the
BigEarthNet-S2 benchmark archive demonstrate that the considered architectures
increase the generalization ability with the cost of higher local training and
aggregation complexities. On the basis of our analysis, some guidelines are
derived for a proper selection of transformer architecture in the context of
FL for RS MLC. The code of this work is publicly available at https://git.tu-
berlin.de/rsim/FL-Transformer.

Index Terms—  Federated learning, multi-label image classification,
transformers, remote sensing.

##  1 Introduction

Recent advances in satellite technologies lead to a significant increase in
the amount of remote sensing (RS) data that is distributed across
decentralized image archives (i.e., clients). The development of accurate
multi-label classification (MLC) methods in massive archives of RS images is a
growing research interest in RS. Deep learning (DL) methods have achieved
remarkable success in MLC of RS images. However, employing DL typically
necessitates full access to training data for learning model parameters [1, 2,
3]. In some clients, training data may not be publicly accessible due to
commercial interests, privacy concerns, and legal regulations [4]. To address
this problem, federated learning (FL) that allows to jointly train a DL model
without any access to the data on the clients (i.e., decentralized image
archives) and to find the optimal model parameters of a global model on a
central server can be used.

It is worth noting that the training data in different clients might be not
independent and identically distributed (non-IID). The presence of non-IID
data in FL can reduce the overall MLC performance as it affects the
convergence of the global model [5]. Although there are many studies that aim
to mitigate the effects of non-IID in computer vision [5, 6, 7, 8, 9, 10], it
is seldom studied in RS [11, 12, 13]. As an example, in [12], an FL algorithm
is introduced, aiming to train multiple global models using hierarchical
clustering, when RS images are non-IID among clients. There are also other
studies in RS related to FL but addressing different problems [14, 15, 16]. As
an example, in [14], an FL framework that learns DL model parameters from
decentralized and unshared multi-modal data archives is introduced. In [15],
local differential privacy federated learning algorithm is proposed to
collaboratively train DL models while secured by local differential privacy.
Most of the methods that address training data heterogeneity aim to improve
either standard empirical risk minimization of local training or standard
parameter aggregation of model averaging [11]. In [17], it is shown that
Vision Transformers (ViTs) [18] and Swin Transformers [19] are beneficial to
alleviate the effects of training data heterogeneity in FL models. In [17],
non-IID data partitions are simulated by only considering label distribution
skew in the context of single-label classification problems. In MLC problems,
different clients might have varying combinations of labels associated with
their data, potentially leading to even more skewed label distributions across
clients. In addition to that, there might be other factors that cause non-IID
data partitions. In detail, various clients may have different amounts of
training data (i.e., quantity skew) [20]. Also, the images that belong to the
same class might exhibit different data distributions among different clients
(i.e., concept drift) [21], which may reduce the generalization capability of
DL models.

In this paper, we focus our attention on the state-of-the-art transformer
architectures (MLP-Mixer [22], ConvMixer [23], PoolFormer [24]) and present a
study to analyze and compare them both theoretically and experimentally in the
framework of FL for RS MLC. In particular, we assess their capability to
address training data heterogeneity when quantity skew, label distribution
skew, and concept drift exist across decentralized RS image archives.

##  2 Methodology

###  2.1 Problem Statement

Let $K$ be the total number of clients and $C^{i}$, $1\leq i\leq K$ be the
$i$th client. In detail, $C^{i}$ holds the corresponding training set
$D_{i}=\\{(\boldsymbol{x}_{z}^{i},\boldsymbol{y}_{z}^{i})\\}_{z=1}^{M^{i}}$,
where $M^{i}$ is the number of images, $\boldsymbol{x}_{z}^{i}$ is the $z$th
RS image of the $i$th client and $\boldsymbol{y}_{z}^{i}$ is the corresponding
class label.
$\boldsymbol{y}_{z}^{i}=[{y}_{z,1}^{i},...,{y}_{z,P}^{i}]\in\\{0,1\\}^{P}$ is
a multi-label vector of $\boldsymbol{x}_{z}^{i}$, stating the presence of $P$
unique classes (i.e., ${y}_{z,p}^{i}=1$ indicates that $p$-th class is present
and ${y}_{z,p}^{i}=0$ indicates that $p$-th class is not present in
$\boldsymbol{x}_{z}^{i}$). There is at least one class label associated with
each image. The DL model $\phi^{i}$ is trained using $D_{i}$ for each $C^{i}$.
To this end, we use binary cross-entropy (BCE) loss, which is given for a
single sample as follows:

$\begin{split}\mathcal{L}_{BCE}(\phi^{i}(\boldsymbol{x}_{z}^{i}),\boldsymbol{y}_{z}^{i})=&\\\
-\sum^{P}_{p=1}{y}_{z,p}^{i}\log(&r_{z,p}^{i})+(1-{y}_{z,p}^{i}\log(1-r_{z,p}^{i})\end{split}$
(1)

where $\phi^{i}(\boldsymbol{x}_{z}^{i})=[r_{z,1}^{i},...,r_{z,P}^{i}]$ is the
class probability vector obtained by the DL model. In this paper, we assume
that the data in clients is not shared and FL is used to learn global model
parameters $w$ over the whole training set
$M=\bigcup_{i\in\\{1,2,...,K\\}}M^{i}$ while minimizing the following global
objective function as follows:

$\arg\min_{w}L(w)=\sum_{i=1}^{K}\frac{M^{i}}{\lvert M\rvert}L_{i}(w),$ (2)

where $L_{i}$ is the empirical loss of $C^{i}$. The parameter update of the
algorithm is as follows:

$w=\sum_{i=1}^{K}\alpha_{i}w^{i},$ (3)

where $\alpha_{i}$ is a hyperparameter to adjust the importance of the local
parameters in each client for aggregation, $w^{i}$ is the local model
parameters of $C^{i}$.

In this paper, we aim to study the effectiveness of transformer architectures
in terms of their: 1) robustness to training data heterogeneity; 2) local
training complexity; and 3) aggregation complexity in FL settings for RS MLC
problems under non-IID data. In particular, we focus on three architectures:
1) MLP-Mixer; 2) ConvMixer; and 3) PoolFormer. As FL algorithms, we select
federated averaging (FedAvg) [25] (which does not explicitly address training
data heterogeneity) and model-contrastive federated learning (denoted as MOON)
[5] (which is known for its strong performance in the case of data
heterogeneity among clients).

###  2.2 Selected Federated Learning Algorithms

As one of the first studies, the FedAvg algorithm is introduced in [25] to
train the local models using local data on clients and the central server
aggregates the updated model parameters through iterative model averaging.
There are four steps of iterative model averaging: 1) the server sends the
global model to clients; 2) the clients update the model parameters; 3) the
clients send the model parameters to central server; and 4) the central server
aggregates the model parameters.

In [5], the MOON algorithm is introduced to address the challenges of training
data heterogeneity by adding the proximal term to the local objective function
of each client. The aim of this algorithm is to limit the size of local
updates, leading to more robust performance when the training data is non-IID.
The proximal term is calculated by increasing the similarity between the image
features obtained from the global model and the local model, and reducing the
similarity of features obtained between current and previous local models. The
proximal term of the MOON algorithm is defined as follows:

$l_{con}=-\frac{e^{S(z_{t+1},z)/\tau}}{e^{S(z_{t+1},z)/\tau}+e^{S(z_{t},z)/\tau}}$
(4)

where $S(.,.)$ measures the cosine similarity, $\tau$ is a temperature
parameter. $z_{t+1}$, $z_{t}$, $z$ are the representation vectors from the
current local model $w_{t+1}^{i}$, previous local model $w_{t}^{i}$, global
model $w$, respectively.

###  2.3 Selected Transformer Architectures

####  2.3.1 MLP-Mixer

The MLP-Mixer [22] is introduced as an alternative computer vision model to
CNNs and the Vision Transformers (ViT) [18] to reduce the computational time.
Similar to the ViT, the MLP-Mixer employs the concept of channel mixing and
token mixing to enable communication across different channels as well as
spatial locations. The token mixing and channel mixing operations allow the
local models to learn diverse representations of the input data. This
capability enables global model to adapt to variations in data distributions
across various clients. This can help global model to be more robust to
training data heterogeneity. Unlike many other transformer architectures, it
does not use any self-attention or convolutions but rather only utilizes
multi-layer perceptrons (MLPs) that are applied to patches of an input image.
The absence of convolutional layers and self-attention mechanisms reduce the
computational costs required to train a DL model per client (i.e., local
training complexity), as there is no need to perform convolution operations or
maintain convolutional kernels. Since MLP-Mixer leverages a simple parameter
structure, it results in using fewer trainable model parameters. This reduces
the cost of aggregating model parameters at the central server (i.e.,
aggregation complexity).

####  2.3.2 ConvMixer

The ConvMixer architecture [23] builds upon the MLP-Mixer by including channel
and token mixing mechanisms to process channel and spatial features. The
ConvMixer replaces all MLP operations in MLP-Mixer with convolutional layers.
The image is initially embedded in a patchwise manner via a convolutional
layer. Then, the patch embedding is passed to a depthwise convolutional layer.
The following pointwise convolutional layer performs $1x1$ convolution to
achieve channel mixing. Since ConvMixer does not use any attention mechanism,
it may struggle to focus on important features or relationships within the
data. This can reduce the generalization capability of global model in FL
setups with heterogeneous data distributions. The ConvMixer architecture
typically requires a large number of convolutional filters across its layers
to capture spatial hierarchies effectively. The increased number of filters
may lead to high local training complexity. Furthermore, the architectural
choices regarding to the number of layers and filters can notably increase the
aggregation complexity, given that each filter encompasses a set of trainable
parameters.

####  2.3.3 PoolFormer

The PoolFormer architecture [24] replaces the attention-based token mixer
module by the simple average pooling operation as a token mixer. The average
pooling operation provides a summary representation of the most important
features within each pooling window. As clients obtain smoother local model
parameters, the global model becomes more robust to the limitations of
training data heterogeneity. Since the self-attention layers introduce
quadratic runtime complexity, the absence of self-attention reduces local
training complexity. The average pooling layers do not introduce additional
parameters to be learned. Therefore, the aggregation complexity of PoolFormer
is lower than those of MLP-Mixer and ConvMixer.

##  3 Experimental Results

The experiments were conducted on the BigEarthNet-S2 benchmark archive [26].
In detail, we used a subset of BigEarthNet-S2 that includes 52,906 Sentinel-2
images acquired over Austria, Belgium, Finland, Ireland, Lithuania, Serbia and
Switzerland in summer. Each image is annotated with multi-labels provided by
the CORINE Land Cover Map database. We used the 19 class nomenclature
introduced in [26]. In the experiments, we used the train and test split
proposed in [26]. To assess the architectures on different non-IID levels, we
designed two different data partitions (i.e., scenarios) on the official train
split as follows:

  * •

Decentralization Scenario 1 (DS1): The training images were randomly
distributed to different clients.

  * •

Decentralization Scenario 2 (DS2): The training images were distributed in a
way that each client holds the images associated with only one country.

Since the images are randomly distributed to the clients in DS1, the level of
non-IID in DS1 is lower than that of DS2. We have compared the results of MLP-
Mixer [22], ConvMixer [23] and PoolFormer [24] architectures with those of the
ResNet-50 in the context of FedAvg [25] and MOON [5] algorithms for RS MLC.
The models were trained for 30 communication rounds and three epochs using
Adam optimizer with the learning rate of 0.001, mini-batch size of 128. We
compare the performance of architectures in terms of MLC accuracy (in
$F_{1}$-Score), local training complexity, and aggregation complexity. Table 1
shows the $F_{1}$-scores (%) under both scenarios. One can see from the table
that the results under DS1 are higher than those under DS2. This is due to the
fact that the level of training data heterogeneity of DS1 is lower than that
of DS2. According to the results obtained using FedAvg algorithm, the highest
accuracy was obtained by ResNet-50, which is 2.77% higher than that obtained
by the worst performing architecture (i.e., ConvMixer) under DS1. In detail,
the results obtained with MLP-Mixer and PoolFormer architectures are similar
to each other and higher than those of ConvMixer by almost 1.5%. Unlike the
results obtained under DS1, the lowest accuracy was obtained with ResNet-50
under DS2. This indicates that the transformer architectures more effectively
handle the challenges of training data heterogeneity and increase the accuracy
of the global model. The highest accuracy was achieved with PoolFormer under
DS2, which is 12.53% higher than the results obtained with ResNet-50.
According to the results obtained using MOON algorithm, the $F_{1}$-scores
obtained under DS2 are higher than those obtained with FedAvg due to the
capability of the MOON algorithm to address the limitations of training data
heterogeneity. However, this increase is less prominent with transformer
architectures. As an example, although MOON algorithm outperforms FedAvg
algorithm by 7.51% higher $F_{1}$-score with Resnet-50, MOON algorithm does
not significantly increase the $F_{1}$-score with PoolFormer. This shows that
the transformer architectures already alleviate the effects of training data
heterogeneity. Therefore, it is feasible to achieve high $F_{1}$-score without
employing an FL method that specifically tackles non-IID data.

Table 1: $F_{1}$-scores (%) obtained by FedAvg and MOON algorithms with different architectures under DS1 and DS2. Algorithms | Architectures | DS1 | DS2
---|---|---|---
FedAvg [25] | ResNet-50 | 75.24 | 47.32
MLP-Mixer [22] | 73.87 | 59.79
ConvMixer [23] | 72.47 | 53.66
PoolFormer [24] | 74.09 | 59.85
MOON [5] | ResNet-50 | 72.82 | 54.83
MLP-Mixer [22] | 74.83 | 60.05
ConvMixer [23] | 73.28 | 58.76
PoolFormer [24] | 74.47 | 60.26

We also assess the effectiveness of the architectures in terms of local
training complexity and aggregation complexity. Table 2 shows the
computational times (in seconds) to complete a single communication round of
local training. By analyzing the table, one can see that the computational
time of MOON algorithm is higher than that of FedAvg algorithm for all
selected architectures. This is because of the fact that the MOON algorithm
requires extra forward passes to extract image feature vectors. One can
observe from the table that the computational time required with ResNet-50 is
the lowest. The local training complexity of MLP-Mixer and PoolFormer are
similar and lower than that of ConvMixer. Table 2 shows the number of model
parameters shared by clients. According to the results, PoolFormer is the most
efficient architecture among all the selected architectures in terms of
aggregation complexity. As an example, the number of parameters of a model
that is trained by PoolFormer is less than that of other selected
architectures.

Table 2: Computational times (in seconds) and number of parameters shared by each client with different architectures obtained by FedAvg and MOON algorithms. Algorithms | Architectures |  Computational Time |  Number of Shared Parameters
---|---|---|---
FedAvg [25] | ResNet-50 | 767 | 23.60M
MLP-Mixer [22] | 1092 | 18.34M
ConvMixer [23] | 2432 | 20.62M
PoolFormer [24] | 1231 | 11.24M
MOON [5] | ResNet-50 | 972 | 23.60M
MLP-Mixer [22] | 1323 | 18.34M
ConvMixer [23] | 2683 | 20.62M
PoolFormer [24] | 1447 | 11.24M

##  4 Conclusion

This paper analyzes and compares different transformer-based architectures
(MLP-Mixer, ConvMixer, PoolFormer) under different non-IID levels in the
framework of FL for MLC problems in RS. The selected architectures have been
compared in term of their: 1) robustness to training data heterogeneity; 2)
local training complexity; and 3) aggregation complexity (see Table 3).
Through an experimental comparison of these architectures, we have derived a
guideline for selecting appropriate architectures in the context of FL for RS
MLC problems as follows:

  1. 1.

The architectures achieve similar accuracies when the level of training data
heterogeneity is low. ResNet-50 has the lowest local training complexity.
Therefore, although the aggregation complexity of ResNet-50 is the highest
among transformer architectures, it can be selected if training data
heterogeneity is low.

  2. 2.

The accuracy of ResNet-50 drops significantly when the level of training data
heterogeneity increases. MLP-Mixer and PoolFormer achieve the highest
accuracies among the selected architectures when the level of training data
heterogeneity is high. Since the aggregation complexity of PoolFormer is lower
than that of MLP-Mixer, it can be selected if the training data is highly non-
IID.

  3. 3.

MOON outperforms FedAvg by a large margin when the level of training data
heterogeneity is high with ResNet-50. However, the accuracy obtained with MOON
is very close to that of FedAvg when the transformer architectures are used.
Therefore, the FedAvg algorithm can be used with transformers instead of
complex algorithms that alleviate the effects of training data heterogeneity.

As a final remark, we would like to note that, although MLC has been selected
as the classification task to assess different transfomer-based architectures
for FL in this paper, our study can be extended to the other learning tasks
such as semantic segmentation. As a future work, we plan to conduct research
in this direction.

Table 3: Comparison of the selected architectures. Three marks ”H” (High), ”M” (Medium), or “L” (Low) are given for the considered criteria. Architectures |  |  Accuracy |  Local Training Complexity |  Aggregation Complexity
---|---|---|---|---
ResNet-50 |  | L | L | H
MLP-Mixer [22] |  | H | M | H
ConvMixer [23] |  | M | H | H
PoolFormer [24] |  | H | M | L

##  5 ACKNOWLEDGEMENTS

This work is supported by the European Research Council (ERC) through the
ERC-2017-STG BigEarth Project under Grant 759764.

## References

  * [1] T. Burgert, M. Ravanbakhsh, and B. Demir, “On the effects of different types of label noise in multi-label remote sensing image classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–13, 2022.
  * [2] G. Sumbul and B. Demir, “Generative reasoning integrated label noise robust deep image representation learning,” _IEEE Transactions on Image Processing_ , vol. 32, pp. 4529–4542, 2023.
  * [3] L. Möllenbrok, G. Sumbul, and B. Demir, “Deep active learning for multi-label classification of remote sensing images,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 20, pp. 1–5, 2023.
  * [4] N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar, “Semi-supervised knowledge transfer for deep learning from private training data,” _International Conference on Learning Representations_ , 2017.
  * [5] Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10 713–10 722, 2021.
  * [6] L. Gao, H. Fu, L. Li, Y. Chen, M. Xu, and C.-Z. Xu, “Feddc: Federated learning with non-iid data via local drift decoupling and correction,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10 112–10 121, 2022.
  * [7] M. Mendieta, T. Yang, P. Wang, M. Lee, Z. Ding, and C. Chen, “Local learning matters: Rethinking data heterogeneity in federated learning,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8397–8406, 2022.
  * [8] X. Ma, J. Zhang, S. Guo, and W. Xu, “Layer-wised model aggregation for personalized federated learning,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10 092–10 101, 2022.
  * [9] Q. Li, Z. Wen, Z. Wu, S. Hu, N. Wang, Y. Li, X. Liu, and B. He, “A survey on federated learning systems: Vision, hype and reality for data privacy and protection,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 35, no. 4, pp. 3347–3366, 2023.
  * [10] Z. Qin, L. Yang, Q. Wang, Y. Han, and Q. Hu, “Reliable and interpretable personalized federated learning,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 20 422–20 431, 2023.
  * [11] B. Büyüktaş, G. Sumbul, and B. Demir, “Federated learning across decentralized and unshared archives for remote sensing image classification,” _arXiv preprint arXiv:2311.06141_ , 2023.
  * [12] B. Zhang, X. Zhang, M.-O. Pun, and M. Liu, “Prototype-based clustered federated learning for semantic segmentation of aerial images,” _IEEE International Geoscience and Remote Sensing Symposium_ , pp. 2227–2230, 2022.
  * [13] X. Zhang, B. Zhang, W. Yu, and X. Kang, “Federated deep learning with prototype matching for object extraction from very-high-resolution remote sensing images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 61, pp. 1–16, 2023.
  * [14] B. Büyüktaş, G. Sumbul, and B. Demir, “Learning across decentralized multi-modal remote sensing archives with federated learning,” _IEEE International Geoscience and Remote Sensing Symposium_ , pp. 4966–4969, 2023.
  * [15] Z. Zhang, X. Ma, and J. Ma, “Local differential privacy based membership-privacy-preserving federated learning for deep-learning-driven remote sensing,” _Remote Sensing_ , vol. 15, no. 20, p. 5050, 2023.
  * [16] S. Chen, T. Shu, H. Zhao, J. Wang, S. Ren, and L. Yang, “Free lunch for federated remote sensing target fine-grained classification: A parameter-efficient framework,” _arXiv preprint arXiv:2401.01493_ , 2024.
  * [17] L. Qu, Y. Zhou, P. P. Liang, Y. Xia, F. Wang, E. Adeli, L. Fei-Fei, and D. Rubin, “Rethinking architecture design for tackling data heterogeneity in federated learning,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10 061–10 071, 2022.
  * [18] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [19] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” _IEEE/CVF International Conference on Computer Vision_ , pp. 10 012–10 022, 2021.
  * [20] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _Foundations and Trends® in Machine Learning_ , vol. 14, no. 1–2, pp. 1–210, 2021.
  * [21] X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou, “Fedbn: Federated learning on non-iid features via local batch normalization,” _arXiv preprint arXiv:2102.07623_ , 2021.
  * [22] I. O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit _et al._ , “MLP-Mixer: An all-MLP architecture for vision,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 24 261–24 272, 2021.
  * [23] A. Trockman and J. Z. Kolter, “Patches are all you need?” _arXiv preprint arXiv:2201.09792_ , 2022.
  * [24] W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, J. Feng, and S. Yan, “MetaFormer is actually what you need for vision,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10 819–10 829, 2022.
  * [25] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” _Artificial Intelligence and Statistics_ , pp. 1273–1282, 2017.
  * [26] G. Sumbul, A. D. Wall, T. Kreuziger, F. Marcelino, H. Costa, P. Benevides, M. Caetano, B. Demir, and V. Markl, “BigEarthNet-MM: A large-scale, multimodal, multilabel benchmark archive for remote sensing image classification and retrieval,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 9, no. 3, pp. 174–180, 2021.