# An Evaluation of Non-Contrastive Self-Supervised Learning for Federated
Medical Image Analysis

Soumitri Chattopadhyay*1 Soham Ganguly*1 Sreejit Chaudhury*1 Sayan Nag*2
Samiran Chattopadhyay1
1Jadavpur University 2University of Toronto
{soumitri.chattopadhyay, sohamgangulysmail, sreejitchau,
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Privacy and annotation bottlenecks are two major issues that profoundly affect
the practicality of machine learning-based medical image analysis. Although
significant progress has been made in these areas, these issues are not yet
fully resolved. In this paper, we seek to tackle these concerns head-on and
systematically explore the applicability of non-contrastive self-supervised
learning (SSL) algorithms under federated learning (FL) simulations for
medical image analysis. We conduct thorough experimentation of recently
proposed state-of-the-art non-contrastive frameworks under standard FL setups.
With the SoTA Contrastive Learning algorithm, SimCLR as our comparative
baseline, we benchmark the performances of our 4 chosen non-contrastive
algorithms under non-i.i.d. data conditions and with a varying number of
clients. We present a holistic evaluation of these techniques on 6
standardized medical imaging datasets. We further analyse different trends
inferred from the findings of our research, with the aim to find directions
for further research based on ours. To the best of our knowledge, ours is the
first to perform such a thorough analysis of federated self-supervised
learning for medical imaging. All of our source code will be made public upon
acceptance of the paper.

Figure 1: Overall pipeline of the presented study. The SSL methods have been
described in Figure 2. **footnotetext: denotes equal contribution.

##  1 Introduction

Medical image analysis [1, 20] is a topic of active research in the machine
learning community. It involves an array of different tasks like disease
detection [22], classification [2, 36] and segmentation [4, 6]. Classically,
standard supervised computer vision techniques have been applied to solve
medical imaging problems. Although such methods have achieved commendable
performance across medical vision tasks, there are two unique challenges the
field faces, which are yet to be fully tackled by such methods.

The first major challenge in practical medical imaging is learning over
medical data stored in a distributed manner. In the field of medical imaging,
the entire data is rarely stored in a centralized server; it is distributed
across various servers and devices (say, servers of different hospitals
containing patient data). Data stored at different sites may even have very
different distributions [10]. Furthermore, owing to privacy and legal
concerns, such data can neither be accumulated together into a centralised
server nor be shared with other sources for training deep learning models.
Thus, one requires decentralized training across all such client devices that
preserves data privacy. This has been facilitated by the paradigm of federated
learning (FL) – introduced in the seminal work by [24]. FL involves individual
model training in each client, followed by aggregating the individual model
weights into a global server model, whose copies are then sent to the clients
for inference. This is non-trivial, since the server model would have to fit
different data distributions across the clients.

Secondly, annotating medical imaging datasets is an arduous task, requiring a
significant amount of time and effort from skilled clinicians. In fact, this
is a major reason why only a handful of large-scale labeled medical imaging
datasets are available to the community. This has led to the development of
annotation-efficient paradigms such as semi-supervised [34, 5], weakly-
supervised [30] and most notably, self-supervised learning (SSL) algorithms.
In particular, SSL [14, 7, 3, 23, 40] has shown great potential to learn
robust representations in computer vision [13, 14, 23] and medical imaging [2,
22, 31] tasks, achieving comparable performance to even supervised approaches.

In this study, we aim to tackle the two aforementioned problems in the field
of medical imaging head-on by leveraging SSL to federated setups for medical
imaging. In particular, we conduct a thorough investigation of the
performances of SoTA _non-contrastive_ SSL algorithms under various federated
learning setups. The motivation for our focus on non-contrastive methods [9,
3, 37, 40] is in light of their using lower batch sizes for reducing
computational demands. To evaluate their performance, we utilise the SoTA
contrastive learning algorithm SimCLR [7] as a comparative baseline. On the
federated side, we conduct experiments using three standard FL algorithms,
namely FedAVG [24], FedBN [19] and FedProx [18]. By varying the number of
clients, we put forth a rigorous evaluation of such simulations on datasets of
the standardized medical imaging suite, MedMNIST [32, 33], which can be used
as a ready reference for current and future research, something that has not
been done so far. Moreover, we further discuss the various trends observed in
the evaluation, which could provide intriguing insights and pave the way for
further study. To the best of our knowledge, such a holistic analysis has not
been performed in literature to date, making our contribution significant. We
believe such a study would be very helpful to the medical vision community in
a very practical manner.

Figure 2: Comparison of different SSL methods used in the presented study.

To sum up, the contributions of our work are as follows:

  1. 1.

We present the first thorough investigation of the applicability of non-
contrastive self-supervision algorithms under three different federated
learning setups in literature for medical image analysis.

  2. 2.

We perform a rigorous evaluation of the performance of these algorithms under
non-i.i.d. data splits with variation in the number of clients participating
in federated learning.

  3. 3.

The benchmarking is performed on the MedMNIST suite, which consists of
standardized medical image datasets. Furthermore, we systematically analyse
different trends inferred from the study, paving the way for further research.

Our experimental pipeline is schematically represented in Figure 1. All source
codes will be made publicly available upon acceptance.

##  2 Related Work

Federated Learning: FL [24, 18, 39, 19, 21] aims at training machine learning
models across a number of private devices having their own datasets, with the
underlying assumption that the clients’ data are mutually independent of each
other [24]. In other words, FL involves decentralized training of models in a
distributed manner. For generalization, the individually trained models are
aggregated at a global server, via an aggregation function. Thus, no data is
shared among the clients, only the model parameters being communicated in the
aggregation process [29]. First proposed in [24], the classic FedAVG
algorithm, which performs a simple weighted averaging of model weights across
clients, is still considered a strong baseline framework. Among other popular
FL methods, FedProx [18] introduced a proximal term to the local client losses
so as to mitigate possible weight divergences, FedAVG-Share [39] proposed to
use a small globally shared data subset for robust generalization, and FedBN
[19] used local batch normalizations to capture different data distributions
and alleviate representational shift prior to aggregation. In this paper, we
adopted FedAVG [24], FedBN [19] and FedProx [18] as for conducting the FL
simulations over SSL algorithms.

Self-Supervised Learning: Being a subset of unsupervised learning, SSL aims at
learning representations without the need for explicit annotations by means of
pre-training tasks in which the so-called pseudo-ground-truths are generated
from the data itself. While classical approaches include colorization [38],
inpainting [28] and solving jigsaw puzzles [25], recent SoTA approaches have
focused on contrastive learning [8, 7, 11], which aims at pulling similar data
points (positive samples) closer together, along with pushing apart dissimilar
points (negative samples) in the embedding space. Although popular contrastive
learning frameworks such as SimCLR [7] and MoCo [11] have proven to be
successful at learning robust visual representations, a bottleneck they face
is the requirement of a large number of negative samples in a mini-batch,
which compels the use of large batch sizes. To alleviate this limitation, more
recent SSL works have directed towards the paradigm of non-contrastive
learning [37, 40, 3] – wherein the objective is to solely bring similar
entities closer together in the latent space, thereby mitigating the need for
negative samples altogether. Barlow Twins [37] uses information maximization
to maximize similarity along with reducing redundancy among neurons, inspired
by neuroscience theories. VICReg [3] extended upon the former by introducing a
hinge function for variance regularization. In our study, we adopted some of
the recently proposed non-contrastive SSL variants – SimSiam [9], Barlow Twins
[37] and VICReg [3]. We also considered TiCo [40], which combines the flavours
of both contrastive and non-contrastive learning along with the use of an
implicit feature memory. For evaluation purpose, SimCLR [7] was chosen as the
contrastive baseline method.

##  3 Federated Self-Supervised Learning

As explained, our study investigates self-supervision models under FL
simulations. In this section, we provide background on the SSL and federated
learning algorithms used in this paper.

###  3.1 SSL Methods

Contrastive: We treat contrastive SSL as a baseline and employ the SoTA SimCLR
[7] algorithm, which uses the InfoNCE [8, 26, 15] contrastive loss to push
embeddings of different views of the same image closer and pull apart those of
different images.

Non-contrastive: As our primary focus is towards non-contrastive SSL, we
experiment with several such methods, namely SimSiam [9], Barlow Twins [37],
VICReg [3] and TiCo [40]. It is worth noting that TiCo can be interpreted as
both contrastive and non-contrastive; it has elements of both the SSL
variants.

For a high-level understanding, the SSL methods adopted in this study have
been depicted figuratively in Figure 2. Other details have been provided in
Table 1.

###  3.2 FL Algorithms

We simulate three standard FL algorithms in our study, namely FedAVG [24],
FedBN [19] and FedProx [18].

FedAVG: FedAVG utilizes iterative averaging over client weights to update the
global model [24]. In each communication round, a set of selected clients
$S_{t}$ receive the current global model, locally train for a predetermined E
epochs, and send the local weights back to the central server. The server
updates the global model by averaging the client weights weighted by number of
local samples $n_{k}$ at each client.

$\displaystyle
w_{t+1}\leftarrow\sum_{k\mathcal{E}S_{t}}\frac{n_{k}}{n}w^{k}_{t+1}$ (1)

Here, $w_{t+1}$ represents the updated global model in round $t+1$,
$w^{k}_{t+1}$ are the weights received from the $k^{th}$ client and $n$ is the
total number of samples present in selected clients.

FedBN: FedBN is a modification of the FedAVG algorithm designed to address the
different marginal or conditional feature distributions across clients, a
problem known as feature shift non-i.i.d. [19]. FedBN utilizes local batch
normalization, with updates to global model weights via weighted averaging,
_leaving out_ the weights from the batch normalization layers.

FedProx: Proposed in [18], FedProx is a generalisation of FedAVG with
modifications intended to address heterogeneous data and systems. Like FedAVG,
in each communication round, the server selects a set of clients and sends
them the current global model $w^{t}$. Unlike FedAVG, FedProx allows local
optimization on clients to run for a variable number of epochs by making
clients optimise a regularised loss with a proximal term. Each client
approximately minimizes the a new object $h_{k}$ defined over its original
local objective $F_{k}$ as (Here, $\mu$ is the regularization hyperparameter):

$\displaystyle\min_{w}h_{k}(w;w^{t})=F_{k}+\frac{\mu}{2}\|w-w^{t}\|^{2}$ (2)

##  4 Experimental Protocol

###  4.1 Datasets

We evaluate the federated self-supervised simulations on the MedMNIST data
suite [32, 33], a large-scale collection of standardized real-world biomedical
images resized to MNIST-like dimensions. The datasets used in this work are
Pneumonia, Breast, Retina, Organ-A, Organ-C and Organ-S. It is essential to
mention that {RetinaMNIST, BreastMNIST and PneumoniaMNIST} are “small
datasets” (having less than 6k images in total), while the remaining are
different views (A=axial, C=coronal, S=sagittal) of abdominal CT images, each
having more than 20k images, thereby being “big datasets”. More details about
the datasets can be found in 111https://medmnist.com/.

###  4.2 Configuration

We conducted our experiments in PyTorch [27] accelerated by a 16GB NVIDIA
Tesla V100 GPU. For self-supervised model training, a ResNet-18 [12] encoder
was used with random initialization, which is considered a de facto standard
for SSL works [7, 37]. For the projector head, a $128$-dimensional fully
connected (FC) output layer was considered for SimCLR, while a
$512$-dimensional FC layer was used as the expander for the non-contrastive
learning algorithms. Following standard works, we set the number of clients to
$5,10$ and $20$ for our investigations. For creating class imbalance in our
clients to simulate the non-i.i.d. condition of FL, we utilised Dirichlet
distribution [24, 19] to generate data splits. We ran each experiment for a
total of 20 communication rounds with each client iterating over 20 internal
epochs in each round. All hyperparameters for our experiments have been
described in Table 1, and were fixed across experiments for a fair evaluation.
It should be mentioned that we could not go beyond a batch size of $128$ due
to computational constraints.

SSL Pre-training
---
backbone | ResNet18
learning rate | $0.01$
batch size ($\\#clients\in\\{5,10\\}$) | $128$
batch size ($\\#clients=20$) | $64$
epochs per FL round | $20$
temperature (SimCLR) | $\tau=0.5$
FL setting
$\\#clients$ | $5,10,20$
rounds | $20$
non-i.i.d. split | Dirichlet ($\alpha=0.1$)
Proximal weight (FedProx) | $\mu=0.001$
KNN evaluation
distance metric | Euclidean distance
$k$ | $20$

Table 1: Hyperparameter setting or our experimental protocol.

For evaluation purposes, following recent SSL literature [35, 15] we used a
simple KNN classification protocol ($k=20$) with Euclidean distance metric
[16] on the representations obtained by the frozen encoder network. This
offers a simple benchmarking scheme without many bells and whistles directly
on features learnt in a completely unsupervised manner.

###  4.3 Evaluation metrics

Two standard classification metrics have been used for the performance
evaluation of the algorithms investigated by this study: Accuracy and weighted
F1-Score. For reporting empirical results, we used F1-score and for
qualitative analyses, we used accuracy values. We report the average over
individual client scores for all experiments.

SSL Method | $\\#clients=5$ | $\\#clients=10$ | $\\#clients=20$
---|---|---|---
FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx
Barlow | 0.837 $\pm$ 0.148 | 0.746 $\pm$ 0.240 | 0.854 $\pm$ 0.133 | 0.831 $\pm$ 0.149 | 0.753 $\pm$ 0.221 | 0.810 $\pm$ 0.174 | 0.826 $\pm$ 0.147 | 0.770 $\pm$ 0.199 | 0.829 $\pm$ 0.144
VICReg | 0.869 $\pm$ 0.121 | 0.844 $\pm$ 0.144 | 0.878 $\pm$ 0.113 | 0.872 $\pm$ 0.117 | 0.837 $\pm$ 0.145 | 0.870 $\pm$ 0.116 | 0.819 $\pm$ 0.145 | 0.810 $\pm$ 0.162 | 0.826 $\pm$ 0.145
SimSiam | 0.771 $\pm$ 0.211 | 0.776 $\pm$ 0.206 | 0.771 $\pm$ 0.211 | 0.806 $\pm$ 0.170 | 0.774 $\pm$ 0.200 | 0.807 $\pm$ 0.168 | 0.773 $\pm$ 0.199 | 0.779 $\pm$ 0.188 | 0.774 $\pm$ 0.192
TiCo | 0.826 $\pm$ 0.162 | 0.798 $\pm$ 0.186 | 0.825 $\pm$ 0.164 | 0.792 $\pm$ 0.183 | 0.779 $\pm$ 0.196 | 0.826 $\pm$ 0.154 | 0.772 $\pm$ 0.195 | 0.768 $\pm$ 0.196 | 0.770 $\pm$ 0.197
SimCLR | 0.867 $\pm$ 0.124 | 0.851 $\pm$ 0.140 | 0.879 $\pm$ 0.112 | 0.871 $\pm$ 0.113 | 0.829 $\pm$ 0.151 | 0.878 $\pm$ 0.108 | 0.819 $\pm$ 0.153 | 0.781 $\pm$ 0.187 | 0.819 $\pm$ 0.155

Table 2: $F1-scores$ obtained on Pneumonia-MNIST dataset.

SSL Method | $\\#clients=5$ | $\\#clients=10$ | $\\#clients=20$
---|---|---|---
FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx
Barlow | 0.755 $\pm$ 0.225 | 0.785 $\pm$ 0.197 | 0.755 $\pm$ 0.225 | 0.823 $\pm$ 0.181 | 0.769 $\pm$ 0.258 | 0.822 $\pm$ 0.180 | 0.675 $\pm$ 0.319 | 0.724 $\pm$ 0.291 | 0.675 $\pm$ 0.319
VICReg | 0.757 $\pm$ 0.228 | 0.764 $\pm$ 0.218 | 0.757 $\pm$ 0.228 | 0.803 $\pm$ 0.197 | 0.783 $\pm$ 0.223 | 0.802 $\pm$ 0.197 | 0.711 $\pm$ 0.284 | 0.678 $\pm$ 0.31 | 0.711 $\pm$ 0.284
SimSiam | 0.731 $\pm$ 0.250 | 0.737 $\pm$ 0.256 | 0.731 $\pm$ 0.250 | 0.722 $\pm$ 0.287 | 0.767 $\pm$ 0.247 | 0.721 $\pm$ 0.287 | 0.702 $\pm$ 0.294 | 0.680 $\pm$ 0.307 | 0.702 $\pm$ 0.294
TiCo | 0.731 $\pm$ 0.215 | 0.766 $\pm$ 0.215 | 0.731 $\pm$ 0.248 | 0.748 $\pm$ 0.267 | 0.770 $\pm$ 0.240 | 0.747 $\pm$ 0.266 | 0.696 $\pm$ 0.295 | 0.676 $\pm$ 0.312 | 0.696 $\pm$ 0.294
SimCLR | 0.727 $\pm$ 0.264 | 0.776 $\pm$ 0.215 | 0.727 $\pm$ 0.264 | 0.793 $\pm$ 0.223 | 0.786 $\pm$ 0.232 | 0.792 $\pm$ 0.222 | 0.695 $\pm$ 0.295 | 0.693 $\pm$ 0.292 | 0.695 $\pm$ 0.295

Table 3: $F1-scores$ obtained on Breast-MNIST dataset.

SSL Method | $\\#clients=5$ | $\\#clients=10$ | $\\#clients=20$
---|---|---|---
FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx
Barlow | 0.497 $\pm$ 0.302 | 0.494 $\pm$ 0.300 | 0.500 $\pm$ 0.294 | 0.486 $\pm$ 0.280 | 0.494 $\pm$ 0.261 | 0.486 $\pm$ 0.280 | 0.470 $\pm$ 0.291 | 0.465 $\pm$ 0.280 | 0.470 $\pm$ 0.291
VICReg | 0.535 $\pm$ 0.265 | 0.505 $\pm$ 0.300 | 0.518 $\pm$ 0.279 | 0.506 $\pm$ 0.250 | 0.497 $\pm$ 0.26 | 0.505 $\pm$ 0.249 | 0.474 $\pm$ 0.269 | 0.478 $\pm$ 0.272 | 0.474 $\pm$ 0.269
SimSiam | 0.499 $\pm$ 0.299 | 0.504 $\pm$ 0.294 | 0.523 $\pm$ 0.277 | 0.486 $\pm$ 0.270 | 0.498 $\pm$ 0.258 | 0.485 $\pm$ 0.27 | 0.451 $\pm$ 0.268 | 0.468 $\pm$ 0.255 | 0.451 $\pm$ 0.268
TiCo | 0.519 $\pm$ 0.280 | 0.503 $\pm$ 0.297 | 0.540 $\pm$ 0.261 | 0.500 $\pm$ 0.258 | 0.497 $\pm$ 0.256 | 0.500 $\pm$ 0.257 | 0.448 $\pm$ 0.271 | 0.479 $\pm$ 0.266 | 0.447 $\pm$ 0.270
SimCLR | 0.522 $\pm$ 0.277 | 0.513 $\pm$ 0.286 | 0.518 $\pm$ 0.280 | 0.478 $\pm$ 0.281 | 0.505 $\pm$ 0.250 | 0.477 $\pm$ 0.280 | 0.458 $\pm$ 0.266 | 0.478 $\pm$ 0.253 | 0.458 $\pm$ 0.266

Table 4: $F1-scores$ obtained on Retina-MNIST dataset.

##  5 Results and Discussion

We report the findings of the experimentation across the datasets as metrics
averaged across clients, along with their standard deviation values. As
discussed earlier, our experiments comprise evaluating different SSL
algorithms over three popular FL setups, simulating variation in the number of
clients. The exhaustive results for each dataset have been tabulated in tables
2, 3, 4, 5, 6 and 7.

###  5.1 Across varying dataset sizes

The six datasets utilised in our work fall into two categories: the $3$ small
datasets, Pneumonia ($\approx 5.8$k images), Breast (780 images) and Retina
(1.6k images); and the $3$ big datasets, Organ-A ($\approx 58$k images),
Organ-C ($\approx 23$k images), and Organ-S ($\approx 25$k images). To
investigate further into the effect of dataset size on SSL algorithms under a
federated setup, we compare the performance of the different SSL methods under
study over these two categories of datasets. We analyze the algorithms both in
terms of performance (mean F1-score) as well as stability (standard deviation
of scores across clients).

SSL Method | $\\#clients=5$ | $\\#clients=10$ | $\\#clients=20$
---|---|---|---
FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx
Barlow | 0.677 $\pm$ 0.093 | 0.310 $\pm$ 0.072 | 0.681 $\pm$ 0.086 | 0.606 $\pm$ 0.048 | 0.362 $\pm$ 0.062 | 0.605 $\pm$ 0.048 | 0.540 $\pm$ 0.047 | 0.360 $\pm$ 0.070 | 0.533 $\pm$ 0.047
VICReg | 0.721 $\pm$ 0.085 | 0.414 $\pm$ 0.088 | 0.724 $\pm$ 0.089 | 0.681 $\pm$ 0.052 | 0.399 $\pm$ 0.048 | 0.682 $\pm$ 0.055 | 0.630 $\pm$ 0.044 | 0.512 $\pm$ 0.054 | 0.623 $\pm$ 0.046
SimSiam | 0.441 $\pm$ 0.026 | 0.455 $\pm$ 0.087 | 0.441 $\pm$ 0.081 | 0.406 $\pm$ 0.059 | 0.426 $\pm$ 0.052 | 0.404 $\pm$ 0.056 | 0.405 $\pm$ 0.061 | 0.428 $\pm$ 0.059 | 0.402 $\pm$ 0.062
TiCo | 0.643 $\pm$ 0.082 | 0.534 $\pm$ 0.084 | 0.647 $\pm$ 0.081 | 0.609 $\pm$ 0.034 | 0.512 $\pm$ 0.045 | 0.603 $\pm$ 0.041 | 0.501 $\pm$ 0.049 | 0.459 $\pm$ 0.054 | 0.478 $\pm$ 0.044
SimCLR | 0.653 $\pm$ 0.067 | 0.513 $\pm$ 0.085 | 0.697 $\pm$ 0.086 | 0.669 $\pm$ 0.046 | 0.519 $\pm$ 0.048 | 0.661 $\pm$ 0.051 | 0.588 $\pm$ 0.050 | 0.483 $\pm$ 0.059 | 0.583 $\pm$ 0.047

Table 5: $F1-scores$ obtained on OrganA-MNIST dataset.

SSL Method | $\\#clients=5$ | $\\#clients=10$ | $\\#clients=20$
---|---|---|---
FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx
Barlow | 0.648 $\pm$ 0.069 | 0.291 $\pm$ 0.061 | 0.644 $\pm$ 0.081 | 0.593 $\pm$ 0.076 | 0.327 $\pm$ 0.059 | 0.575 $\pm$ 0.078 | 0.559 $\pm$ 0.067 | 0.320 $\pm$ 0.084 | 0.558 $\pm$ 0.066
VICReg | 0.696 $\pm$ 0.072 | 0.408 $\pm$ 0.080 | 0.692 $\pm$ 0.070 | 0.655 $\pm$ 0.076 | 0.429 $\pm$ 0.073 | 0.653 $\pm$ 0.080 | 0.594 $\pm$ 0.071 | 0.466 $\pm$ 0.082 | 0.600 $\pm$ 0.074
SimSiam | 0.376 $\pm$ 0.077 | 0.374 $\pm$ 0.075 | 0.376 $\pm$ 0.078 | 0.384 $\pm$ 0.079 | 0.395 $\pm$ 0.072 | 0.373 $\pm$ 0.090 | 0.367 $\pm$ 0.089 | 0.389 $\pm$ 0.082 | 0.367 $\pm$ 0.090
TiCo | 0.599 $\pm$ 0.073 | 0.463 $\pm$ 0.074 | 0.600 $\pm$ 0.072 | 0.570 $\pm$ 0.061 | 0.471 $\pm$ 0.072 | 0.572 $\pm$ 0.060 | 0.439 $\pm$ 0.076 | 0.414 $\pm$ 0.080 | 0.436 $\pm$ 0.072
SimCLR | 0.611 $\pm$ 0.132 | 0.495 $\pm$ 0.081 | 0.664 $\pm$ 0.066 | 0.635 $\pm$ 0.070 | 0.523 $\pm$ 0.066 | 0.626 $\pm$ 0.066 | 0.544 $\pm$ 0.069 | 0.440 $\pm$ 0.084 | 0.543 $\pm$ 0.062

Table 6: $F1-scores$ obtained on OrganC-MNIST dataset.

SSL Method | $\\#clients=5$ | $\\#clients=10$ | $\\#clients=20$
---|---|---|---
FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx | FedAVG | FedBN | FedProx
Barlow | 0.556 $\pm$ 0.096 | 0.298 $\pm$ 0.090 | 0.541 $\pm$ 0.096 | 0.545 $\pm$ 0.075 | 0.352 $\pm$ 0.079 | 0.554 $\pm$ 0.071 | 0.498 $\pm$ 0.081 | 0.332 $\pm$ 0.091 | 0.500 $\pm$ 0.081
VICReg | 0.602 $\pm$ 0.102 | 0.365 $\pm$ 0.089 | 0.607 $\pm$ 0.094 | 0.554 $\pm$ 0.100 | 0.401 $\pm$ 0.078 | 0.577 $\pm$ 0.088 | 0.526 $\pm$ 0.085 | 0.423 $\pm$ 0.073 | 0.531 $\pm$ 0.082
SimSiam | 0.368 $\pm$ 0.100 | 0.355 $\pm$ 0.100 | 0.365 $\pm$ 0.099 | 0.380 $\pm$ 0.086 | 0.387 $\pm$ 0.076 | 0.380 $\pm$ 0.092 | 0.351 $\pm$ 0.088 | 0.374 $\pm$ 0.085 | 0.351 $\pm$ 0.086
TiCo | 0.519 $\pm$ 0.097 | 0.415 $\pm$ 0.087 | 0.510 $\pm$ 0.102 | 0.502 $\pm$ 0.062 | 0.431 $\pm$ 0.073 | 0.502 $\pm$ 0.068 | 0.395 $\pm$ 0.085 | 0.387 $\pm$ 0.080 | 0.413 $\pm$ 0.091
SimCLR | 0.527 $\pm$ 0.104 | 0.428 $\pm$ 0.105 | 0.560 $\pm$ 0.099 | 0.578 $\pm$ 0.065 | 0.461 $\pm$ 0.078 | 0.573 $\pm$ 0.064 | 0.480 $\pm$ 0.088 | 0.399 $\pm$ 0.087 | 0.471 $\pm$ 0.084

Table 7: $F1-scores$ obtained on OrganS-MNIST dataset.

Small datasets: Tables 2, 3 and 4 contain the mean and standard deviations of
weighted $F1-scores$ over individual clients for each SSL method over
Pneumonia, Breast and Retina datasets respectively. From Table 2 we can see
that the best performer overall on Pneumonia is the non-contrastive method
VICReg. While SimCLR gets highest scores in some cases under smaller number of
clients, its performance significantly deteriorates as the number of clients
is increased to $20$. In contrast, the performances of the non-contrastive
methods, especially VICReg and Barlow Twins, decrease to a far smaller extent,
making them the best performers under higher numbers of clients. When setting
number of clients to $20$, the batch size is reduced to $64$ (to allow for
lower per-client data availability due to increased number of clients).
Contrastive learning methods like SimCLR are heavily dependent on the number
of negative samples available, and thus perform better under higher batch
sizes [7, 8], while non-contrastive methods, not having such a dependency, are
not affected as much. This trend is more visibly observable in Table 3 and
Table 4 as we enter into datasets having considerably fewer available samples
(Breast and Retina having $780$ and $1600$ respectively compared to
Pneumonia’s $5856$ samples).In both these datasets, the non-contrastive method
(VICReg in most cases) emerges with the best performance under almost every
federated setup examined. We also observe each algorithm having very high
standard deviation in its performance across all $3$ datasets, implying high
variance among the individual clients. This can be attributed to the
non-i.i.d. distribution of samples in a low data setting leading to clients
having severely smaller and possibly more imbalanced datasets and thus, higher
variations in performance.

Big datasets: From tables 5, 6, 7, we once again see VICReg emerging as the
best performer overall. SimCLR too shows considerably better performance on
these datasets compared to the smaller ones. We attribute this improvement to
the greater amount of data available in these $3$ datasets (The $3$ OrganMNIST
datasets have nearly $10$ times more data compared to the smaller datasets
like Breast or Retina), which generally aids all self-supervision methods.
Another important observation from tables 5, 6, 7 is the significantly lower
standard deviation of $F1-scores$ of each algorithm relative to those in
tables 2, 3, 4, which indicate a lower variance of scores among the clients.
With the Organ datasets having a considerably large amount of data, each
client is left with a larger and potentially more balanced set of samples,
even under non-i.i.d. distribution.

###  5.2 Across FL algorithms

We studied the behaviour of non-contrastive methods subjected under the
different federated learning simulations – FedAVG [24], FedBN [19] and FedProx
[18]. The findings have been graphically presented in Figure 3, Figure 4 and
Figure 5 respectively.

(a)

(b)

Figure 3: Performance across various number of clients under FedAVG
simulation.

(a)

(b)

Figure 4: Performance across various number of clients under FedBN simulation.

(a)

(b)

Figure 5: Performance across various number of clients under FedProx
simulation.

FedAVG: Figure 3 shows the performances of each non-contrastive SSL variant
across various numbers of clients under FedAVG [24] scheme. We observe that
there is a general trend of performance deterioration with increase in
clients. This is expected, as more clients imply lesser data in each of them
along with a higher percentage of imbalanced clients due to non-i.i.d.
settings. For the OrganMNIST datasets, the behaviours of Barlow [37], VICReg
[3] and TiCo [40] are very similar in this regard, with VICReg being the best
performer among a majority of cases. On the other hand, among the “small”
datasets (Pneumonia, Breast and Retina), a majority of the algorithms peak in
performance when utilising $10$ clients, followed by a sharp dip when the
number of clients is further increased further to $20$. This may be attributed
to the severe decrease in the local client data (as the datasets are even
smaller; $\approx 10$x smaller than the Organ datasets) which hurts the
learning ability of self-supervision models in general [9, 8].

FedBN: As evident from Figure 4, FedBN simulation results are in contrast to
the expected behaviour, (i.e., with increase in $\\#clients$ the performance
should drop). For VICReg, the general trend across the small datasets is the
aforementioned expected behaviour, however, for big datasets such as Organ-C,
we see an increase in the accuracy values with an increase in $\\#clients$. It
is also important to note that out of the 4 Non-CL algorithms in Figure 4,
TiCo [40] shows the most consistent general trend across all the datasets.
Nonetheless, from the above observations we can conjecture that local Batch
Normalization plays a pivotal role in the variability of the performances of
SSL algorithms for the datasets taken into consideration.

FedProx: Comparing Figure 3 and Figure 5, it can be inferred that FedProx
shows very similar trends to FedAVG across a varying number of clients, which
is reasonable as it only makes lightweight alterations to the latter in terms
of introducing the proximal regularization term [18]. For each SSL method,
accuracy values obtained under FedAVG and FedProx are very close to each
other, with minor performance gains observed for TiCo [40] and VICReg [3]
under FedProx simulation scores.

###  5.3 Contrastive vs Non-Contrastive SSL

Since our study intends to investigate the behaviour of non-contrastive SSL
variants under FL setups, we seek to analyse how they fare against the
contrastive SimCLR [7] – a strong self-supervision baseline for image
classification tasks [2, 31, 13]. To do so, for each of the experimental
setups we compare the accuracy obtained by the best-performing non-contrastive
method (denoted by NCL-Best) with that of SimCLR. The plots depicting these
comparisons have been provided in Figure 6, Figure 7 and Figure 8, distributed
according to the number of clients considered ($5,10$ and $20$). The reason
for such a comparison lies in the fact that it is not possible to find a
_single best_ non-contrastive method for all setups, as well as to make our
study more generalised rather than specific to a particular SSL algorithm.

(a)

(b)

Figure 6: Contrastive vs non-contrastive for $\\#clients=5$.

(a)

(b)

Figure 7: Contrastive vs non-contrastive for $\\#clients=10$.

(a)

(b)

Figure 8: Contrastive vs non-contrastive for $\\#clients=20$.

Number of clients: From the figures, it is evident that NCL-Best outperforms
SimCLR on almost all setups, with several having significant margins of
difference. The trend is best prominent in Figure 8 ($\\#clients=20$), where
the batch size is reduced to $64$ (refer to Table 1 to cater to the low data
regime caused by the increased number of clients. It is well-known [7, 8] that
contrastive approaches work better with higher batch sizes (or greater amount
of data [11]), due to the availability of more negative samples. Moreover,
from the plots, it is clear that the performance of SimCLR decreases with an
increase in clients (i.e. decrease in effective data sizes across clients).
Whereas non-contrastive methods being negative-free do not have such a
dependency and thus are more stable across the number of clients, as well as
expectedly beating the former in the low-data regime. With lesser number of
clients too (Figure 6 and Figure 7), non-contrastive methods mostly prove to
be more effective than SimCLR, but the trend is less prominent compared to
those in Figure 8. One noteworthy observation here is, that under FedBN [19]
setup across a lower number of clients, the contrastive approach seems to be
more favourable compared to the non-contrastive ones, which is _contrary_ to
the overall behaviour observed. However, in the low data regime (Figure 8), it
follows the general trend.

Dataset size: We revisit figures 6, 7 and 8, this time keeping in mind the
dataset sizes (as previously described in Section 4.1). We observe that among
“small” datasets, the difference between NCL-Best and SimCLR is smaller
compared to those in “big” datasets. The gaps, however, increase with the
number of clients and make the trends discussed above more generalised.
Furthermore, SimCLR shows greater stability on the OrganMNIST datasets
compared to the smaller datasets (Pnuemonia, Breast and Retina) when
$\\#clients$ is varied.

From these observations, it is reasonable to infer that non-contrastive self-
supervised methods [37, 3, 40] are more effective than contrastive ones under
federated simulations (especially under low data availability) and thus, are
worthy of greater attention and future research.

###  5.4 Further analysis

SimSiam performs poorly on OrganMNIST: Studying Figure 3 and Figure 5, it can
be clearly seen that SimSiam performs much poorer than its counterparts on the
3 OrganMNIST datasets. As discussed in [17], this can be attributed to SimSiam
succumbing to dimensional collapse resulting from relatively smaller size of
encoder (ResNet-18) and comparably higher data complexity ($\approx 10$x) in
the 3 larger OrganMNIST datasets. This is further borne out by the significant
performance gain we see in Figure 4 caused by a reduction in the local bias
via exclusion of batch normalization layers from global model updates in FedBN
[19].

FedBN aids SimCLR: In Figure 6 and Figure 7, we observe that SimCLR seems to
perform comparatively better (with respect to the best non-contrastive method)
under FedBN compared to the other FL algorithms. While it still doesn’t beat
the best-performing non-contrastive approach in all instances, the gap in
performance is significantly smaller when FedBN is used. However, we also note
that in Figure 8, FedBN behaves in a similar fashion as FedAVG or FedProx,
thus suggesting that the underlying cause maybe due to the batch sizes
considered. We point this out as an intriguing observation which may be
investigated in future studies.

##  6 Conclusion

Despite the great progress in medical image analysis, the concerns regarding
privacy and annotation bottleneck are yet to be fully resolved. In this study,
we take a step forward and tackle these challenges head-on by combining
federated learning with self-supervision for medical image classification. We
experimented with SoTA SSL algorithms exhaustively across various FL setups,
simulated by varying the number of clients, to conduct realistic non-i.i.d
simulations on the MedMNIST data suite. Our results suggest the high
applicability of non-contrastive SSL methods to such tasks, which are found to
outperform the contrastive baseline by fair margins. With a systematic
analysis of our findings, we show the trends of different algorithms and
simulations across different dataset sizes, number of clients, etc. Our
holistic evaluation and benchmarking is the first of its kind, which we feel
should be of great assistance to other researchers working in related domains.
The inferences drawn from our findings, as well as some of the “anomalies”
(e.g. surprisingly poor behaviour of SimSiam, contrasting trends of FedBN),
should be of great interest for fellow researchers for further exploration. In
future, we plan to expand the horizon of our investigations by digging deeper
into the nuances of each FL setting to better interpret the observed trends.

## References

  * [1] Abeer Aljuaid and Mohd Anwar.  Survey of supervised learning for medical image processing.  SN Computer Science, 2022.
  * [2] Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, et al.  Big self-supervised models advance medical image classification.  In ICCV, 2021.
  * [3] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  In ICLR, 2022.
  * [4] Hritam Basak, Rajarshi Bhattacharya, Rukhshanda Hussain, and Agniv Chatterjee.  An exceedingly simple consistency regularization method for semi-supervised medical image segmentation.  In IEEE ISBI, 2022.
  * [5] Hritam Basak, Soumitri Chattopadhyay, Rohit Kundu, Sayan Nag, and Rammohan Mallipeddi.  Ideal: Improved dense local contrastive learning for semi-supervised medical image segmentation.  arXiv preprint arXiv:2210.15075, 2022.
  * [6] Hritam Basak, Sagnik Ghosal, and Ram Sarkar.  Addressing class imbalance in semi-supervised image segmentation: A study on cardiac mri.  In MICCAI, 2022.
  * [7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * [8] Ting Chen, Calvin Luo, and Lala Li.  Intriguing properties of contrastive losses.  NeurIPS, 2021.
  * [9] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In CVPR, 2021.
  * [10] Ehsan Degan, Shafiq Abedin, David Beymer, Angshuman Deb, Nathaniel Braman, Benedikt Graf, and Vandana Mukherjee.  Application of federated learning in medical imaging.  In Federated Learning, pages 483–497. Springer, 2022.
  * [11] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, 2020.
  * [12] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CVPR, 2016.
  * [13] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon.  A survey on contrastive self-supervised learning.  Technologies, 2020.
  * [14] Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  IEEE TPAMI, 2020.
  * [15] Anna Kukleva, Moritz Böhle, Bernt Schiele, Hilde Kuehne, and Christian Rupprecht.  Temperature schedules for self-supervised contrastive methods on long-tail data.  In ICLR, 2023.
  * [16] Jorma Laaksonen and Erkki Oja.  Classification with learning k-nearest neighbors.  In ICNN, 1996.
  * [17] Alexander C Li, Alexei A Efros, and Deepak Pathak.  Understanding collapse in non-contrastive siamese representation learning.  In ECCV, 2022.
  * [18] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.  Federated optimization in heterogeneous networks.  In MLSys, 2020.
  * [19] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou.  Fedbn: Federated learning on non-iid features via local batch normalization.  arXiv preprint arXiv:2102.07623, 2021.
  * [20] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez.  A survey on deep learning in medical image analysis.  Medical Image Analysis, 2017.
  * [21] Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng.  Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space.  In CVPR, 2021.
  * [22] Siladittya Manna, Saumik Bhattacharya, and Umapada Pal.  Interpretive self-supervised pre-training: boosting performance on visual medical data.  In ICVGIP, 2021.
  * [23] Siladittya Manna, Soumitri Chattopadhyay, Saumik Bhattacharya, and Umapada Pal.  Swis: Self-supervised representation learning for offline signature verification.  In IEEE ICIP, 2022.
  * [24] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In AISTATS, 2017.
  * [25] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In ECCV, 2016.
  * [26] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In NeurIPS, 2019.
  * [28] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In CVPR, 2016.
  * [29] Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Ehsan Adeli, Li Fei-Fei, and Daniel Rubin.  Rethinking architecture design for tackling data heterogeneity in federated learning.  In CVPR, 2022.
  * [30] Jérôme Rony, Soufiane Belharbi, Jose Dolz, Ismail Ben Ayed, Luke McCaffrey, and Eric Granger.  Deep weakly-supervised learning methods for classification and localization in histology images: a survey.  arXiv preprint arXiv:1909.03354, 2019.
  * [31] Saeed Shurrab and Rehab Duwairi.  Self-supervised learning methods and applications in medical imaging analysis: A survey.  PeerJ Computer Science, 2022.
  * [32] Jiancheng Yang, Rui Shi, and Bingbing Ni.  Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis.  In IEEE ISBI, 2021.
  * [33] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni.  Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification.  arXiv preprint arXiv:2110.14795, 2021.
  * [34] Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu.  A survey on deep semi-supervised learning.  arXiv preprint arXiv:2103.00550, 2021.
  * [35] Chun-Hsiao Yeh, Cheng-Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun.  Decoupled contrastive learning.  In ECCV, 2022.
  * [36] Safoora Yousefi and Yao Nie.  Transfer learning from nucleus detection to classification in histopathology images.  In IEEE ISBI, 2019.
  * [37] J. Zbontar, L. Jing, Ishan Misra, Y. LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In ICML, 2021.
  * [38] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In ECCV, 2016.
  * [39] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra.  Federated learning with non-iid data.  arXiv preprint arXiv:1806.00582, 2018.
  * [40] Jiachen Zhu, Rafael M Moraes, Serkan Karakulak, Vlad Sobol, Alfredo Canziani, and Yann LeCun.  Tico: Transformation invariance and covariance contrast for self-supervised visual representation learning.  arXiv preprint arXiv:2206.10698, 2022.