11institutetext:  LaTIM UMR 1101, Inserm, Brest, France 22institutetext:
University of Western Brittany, Brest, France 33institutetext: IMT Atlantique,
Brest, France 44institutetext: Ophtalmology Department, CHRU Brest, Brest,
France 55institutetext: Lariboisière Hospital, AP-HP, Paris,
France66institutetext: LBAI UMR 1227, Inserm, Brest, France

# Longitudinal self-supervised learning using neural ordinary differential
equation

Rachid Zeghlache 1122 Pierre-Henri Conze 1133 Mostafa El Habib Daho 1122 Yihao
Li 1122 Hugo Le Boité 55 Ramin Tadayoni 55 Pascal Massin 55 Béatrice Cochener
112244 Ikram Brahim 112266 Gwenolé Quellec 11 Mathieu Lamard 1122

###### Abstract

Longitudinal analysis in medical imaging is crucial to investigate the
progressive changes in anatomical structures or disease progression over time.
In recent years, a novel class of algorithms has emerged with the goal of
learning disease progression in a self-supervised manner, using either pairs
of consecutive images or time series of images. By capturing temporal patterns
without external labels or supervision, longitudinal self-supervised learning
(LSSL) has become a promising avenue. To better understand this core method,
we explore in this paper the LSSL algorithm under different scenarios. The
original LSSL is embedded in an auto-encoder (AE) structure. However,
conventional self-supervised strategies are usually implemented in a Siamese-
like manner. Therefore, (as a first novelty) in this study, we explore the use
of Siamese-like LSSL. Another new core framework named neural ordinary
differential equation (NODE). NODE is a neural network architecture that
learns the dynamics of ordinary differential equations (ODE) through the use
of neural networks. Many temporal systems can be described by ODE, including
modeling disease progression. We believe that there is an interesting
connection to make between LSSL and NODE. This paper aims at providing a
better understanding of those core algorithms for learning the disease
progression with the mentioned change. In our different experiments, we employ
a longitudinal dataset, named OPHDIAT, targeting diabetic retinopathy (DR)
follow-up. Our results demonstrate the application of LSSL without including a
reconstruction term, as well as the potential of incorporating NODE in
conjunction with LSSL.

###### Keywords:

Longitudinal analysis longitudinal self supervised learning neural ODE disease
progression diabetic retinopathy

##  1 Introduction

In recent years, the deep learning community has enthusiastically embraced the
self-supervised learning paradigm, by taking advantage of pretext tasks to
learn better representations to be used on a downstream task. Most of the
existing works are based on contrastive learning [4] or hand-crafted pretext
task [20]. When using hand-crafted pretext tasks, the model learns
automatically by obtaining supervisory signals extracted directly from the
nature of the data itself, without manual annotation performed by an expert.
An adequate objective function teaches robust feature representations to the
model, which are needed to solve downstream tasks (e.g., classification,
regression). However, to design an effective pretext task, domain-specific
knowledge is required.

Recently, several approaches that involve pretext tasks in a longitudinal
context have appeared with the purpose of encoding disease progression. These
approaches aim to learn longitudinal changes or infer disease progression
trajectories at the population or patient levels [18, 15, 21, 5].

Longitudinal self-supervised learning (LSSL) was initially introduced in the
context of disease progression as a pretext task by [16], which involved the
introduction of a longitudinal pretext task utilizing a Siamese-like model.
The model took as input a consecutive pair of images and predicted the
difference in time between the two examinations. Since then, more
sophisticated algorithms have been proposed, including the advanced version of
LSSL proposed in [21]. The framework attempted to theorize the notion of
longitudinal pretext task with the purpose of learning the disease
progression. LSSL was embedded in an auto-encoder (AE), taking two consecutive
longitudinal scans as inputs. The authors added to the classic reconstruction
term a cosines alignment term that forces the topology of the latent space to
change in the direction of longitudinal changes.

Moreover, as conducted in [16], [8] employed a Siamese-like architecture to
compare longitudinal imaging data with deep learning. The strength of this
approach was to avoid any registration requirements, leverage population-level
data to capture time-irreversible changes that are shared across individuals
and offer the ability to visualize individual-level changes. Neural Ordinary
Differential Equations (NODEs) is a new core algorithm that has a close
connection to modeling time-dependant dynamics. NODE, introduced in [3], deals
with deep learning operations defined by the solution of an ODE. Whatever the
involved architecture and given an input, a NODE defines its output as the
numerical solution of the underlying ordinary differential equation (ODE). One
advantage is that it can easily work with irregular time series [17], which is
an inherent aspect of the disease progression context. This is possible
because the NODEs are able to deal with continuous time. Additionally, NODEs
leverage the inductive bias that time-series originates from an underlying
dynamical process, where the rate of change of the current state depends on
the state itself. NODEs have been used to model hidden dynamics in disease
progression using neural networks. Authors in [14] have developed a Neural
ODE-based model in order to learn disease progression dynamics for patients
under medication for COVID-19. Thus, Lachinov et al. proposed in [10] a U-Net-
based model coupled with a neural ODE to predict the progression of 3D volumic
data in the context of geographic atrophy using retinal OCT volumes and
predicting the brain ventricle change with MRI for the quantification of
progression of Alzheimer’s disease. Thus, the main objectives of our work are
to examine if:

  1. 1.

By including Neural ODEs, it becomes possible to generate a latent
representation of the subsequent scan without the explicit need for feeding an
image pair to the model. Due to the inherent characteristics of NODE, there is
potential to encode the latent dynamic of disease progression and longitudinal
change. We believe this established a natural connection between NODE and LSSL
algorithms, warranting further investigation to gain a deeper understanding of
these newly introduced frameworks.

  2. 2.

Most of the current self-supervised learning frameworks are embedded in a
Siamese-like paradigm, using only an encoder and optimizing different loss
functions based on various similarity criteria. While the reconstruction term
offers a way to encode anatomical change, successful longitudinal pretext
tasks [16, 7, 8] have only used an encoder in terms of design, which justify
the potential of extending it to LSSL. In order to examine this hypothesis, we
are constructing a Siamese-like variant of the LSSL framework to evaluate the
significance of the reconstruction term within LSSL.

##  2 Method

In this section, we briefly introduce the concepts related to LSSL [21] and
NODE [3], and we investigate the longitudinal self-supervised learning
framework under different scenarios: standard LSSL (Fig.1b), Siamese-like LSSL
(Fig.1a) as well as their NODE-based versions (Fig.1c-d). Let $\mathcal{X}$ be
the set of subject-specific image pairs extracted from the full collection of
color fundus photographs (CFP). $\mathcal{X}$ contains all
$(x^{t_{i}},x^{t_{i+1}})$ image pairs that are from the same subject with
image $x^{t_{i}}$ scanned before image $x^{t_{i+1}}$. These image pairs are
then provided as inputs of an auto-encoder (AE) network (Fig.1). The latent
representations generated by the encoder are denoted by
$z^{t_{i}}=f(x^{t_{i}})$ and $z^{t_{i+1}}=f(x^{t_{i+1}})$ where $f$ is the
encoder. From this encoder, we can define the trajectory vector $\Delta
z=(z^{t_{i+1}}-z^{t_{i}})$. The decoder $g$ uses the latent representation to
reconstruct the input images such that $\tilde{x}^{t_{i}}=g(z^{t_{i}})$ and
$\tilde{x}^{t_{i+1}}=g(z^{t_{i+1}})$.

###  2.1 Longitudinal self-supervised learning (LSSL)

Longitudinal self-supervised learning (LSSL) exploits a standard AE (Fig.1b).
The AE is trained with a loss that forces the trajectory vector $\Delta z$ to
be aligned with a direction that could rely in the latent space of the AE
called $\tau$. This direction is learned through a subnetwork composed of
single dense layers which map dummy data (vector full of ones) into a vector
$\tau$ that has the dimension of the latent space of the AE. Enforcing the AE
to respect this constraint is equivalent to encouraging $\cos\left(\Delta
z,\boldsymbol{\tau}\right)$ to be close to 1, i.e., a zero-angle between
$\boldsymbol{\tau}$ and the direction of progression in the representation
space. With $\mathbf{E}$ being the mathematical expectation, the objective
function is defined as follows:

$\mathbf{E}_{(x^{t_{i}},x^{t_{i+1}})\sim\mathcal{X}}\left(\lambda_{recon}\cdot(\parallel
x^{t_{i}}-\tilde{x}^{t_{i}}\parallel_{2}^{2}+\parallel
x^{t_{i+1}}-\tilde{x}^{t_{i+1}}\parallel_{2}^{2})-\lambda_{dir}\cdot\cos(\Delta
z,\tau)\right)$ (1)

When $\lambda_{dir}=0$ and $\lambda_{recon}>0$, the architecture is reduced to
a simple AE. Conversely, using $\lambda_{dir}>0$ and $\lambda_{recon}=0$
amounts to a Siamese-like structure with a cosine term as a loss function
(Fig.1c).

###  2.2 Neural ordinary differential equations (NODE)

NODEs approximate unknown ordinary differential equations by a neural network
[2] that parameterizes the continuous dynamics of hidden units
$\mathbf{z}\in\mathbb{R}^{n}$ over time with $\mathbf{t}\in\mathbb{R}$. NODEs
are able to model the instantaneous rate of change of $\mathbf{z}$ with
respect to $\mathbf{t}$ using a neural network $u$ with parameters $\theta$.

$\lim_{h\rightarrow
0}\frac{\mathbf{z}_{t+h}-\mathbf{z}_{t}}{h}=\frac{d\mathbf{z}}{dh}=u(t,\mathbf{z},\boldsymbol{\theta})$
(2)

The analytical solution of Eq.2 is given by:

$\mathbf{z}_{t_{1}}=\mathbf{z}_{t_{0}}+\int_{t_{0}}^{t_{1}}f(t,\mathbf{z},\boldsymbol{\theta})\mathrm{d}t=\textrm{ODESolve}(\mathbf{z}(t_{0}),u,t_{0},t_{1},\theta)$
(3)

where $[t_{0},t_{1}]$ represents the time horizon for solving the ODE, $u$
being a neural network, and $\theta$ is the trainable parameters of $u$.

###  2.3 LSSL-NODE

By using a black box ODE solver introduced in [2], we are able to
approximately solve the initial value problem (IVP) and calculate the hidden
state at any desired time using Eq.3. We can differentiate the solutions of
the ODE solver with respect to the parameters $\theta$, the initial state
$\mathbf{z}_{t_{0}}$ at initial time $t_{0}$, and the solution at time $t$.
This can be achieved by using the adjoint sensitivity method [3]. This allows
to backpropagate through the ODE solver and train any learnable parameters
with any optimizer. Typically, NODE is modelized by a feedforward layer, where
we solve the ODE from $t_{0}$ to a terminal time $t_{1}$, or it can be used to
output a series, by calculating the hidden state at specific times
$\\{t_{1},...,t_{i},t_{i+1}\\}$. For a given patient, instead of giving a pair
of consecutive images to the model, we only provide the first image of the
consecutive pair. In our case, through the latent representation of this
image, we define an IVP problem that aims to solve:
$\dot{z}(t)=u(z(t),t,\theta),$ with the initial value $z(t_{i})=z^{{t_{i}}}$.
This results in the following update of the equations from the previous
notation. The latent representations generated by the encoder are denoted by
$z^{t_{i}}=f(x^{t_{i}})$ and
$z^{t_{i+1}}_{node}=\textrm{ODESolve}(z^{{t_{i}}},u,t_{i},t_{i+1},\theta)$
where $f$ is the encoder and $u$ is the defined neural network for our NODE.
From this encoder and the NODE, we can define the trajectory vector
$(\Delta_{z}^{node})=(z^{t_{i+1}}_{node}-z^{t_{i}})$. The same decoder $g$
uses the latent representation to reconstruct the input images such that
$\tilde{x}^{t_{i}}=g(z^{t_{i}})$ and
$\tilde{x}^{t_{i+1}}=g(z^{{t_{i+1}}}_{node})$. The objective function is
defined as:

$\mathbf{E}_{(x^{t_{i}},x^{t_{i+1}})\sim\mathcal{X}}\left(\lambda_{recon}\cdot(\parallel
x^{t_{i}}-\tilde{x}^{t_{i}}\parallel_{2}^{2}+\parallel
x^{t_{i+1}}-\tilde{x}^{t_{i+1}}\parallel_{2}^{2})-\lambda_{dir}\cdot\cos(\Delta_{z}^{node},\tau)\right)$
(4)

Figure 1: Illustration of original LSSL and the proposed extension of the
LSSL, in figure a) the Siamese-like LSSL(S-LSSL), b) original LSSL and in
figure c) and d) their NODE-based version respectively.

##  3 Experiments and Results

###  3.1 Dataset

The proposed models were trained and evaluated on OPHDIAT [12], a large CFP
database collected from the Ophthalmology Diabetes Telemedicine network
consisting of examinations acquired from 101,383 patients between 2004 and
2017. Within 763,848 interpreted CFP images, 673,017 are assigned with a DR
severity grade, the others being nongradable. The image sizes range from 1440
$\times$ 960 to 3504 $\times$ 2336 pixels. Each examination has at least two
images for each eye. Each subject had 2 to 16 scans, with an average of 2.99
scans spanning an average time interval of 2.23 years. The age range of
patients is from 9 to 91. The dataset is labeled according to the
international clinical DR severity scale (ICDR) where the classes include: no
apparent DR, mild non-proliferative diabetic retinopathy (NPDR), moderate
NPDR, severe NPDR, and proliferative diabetic retinopathy (PDR), respectively
labeled as grades 0, 1, 2, 3, and 4. NPDR (grades 1, 2, 3) corresponds to the
early-to-middle stage of DR and deals with a progressive microvascular disease
characterized by small vessel damages.

Image selection. The majority of patients from the OPHDIAT database have
multiple images with different fields of view for both eyes. To facilitate the
selection, we chose to randomly take a single image per eye for each
examination. In addition, we defined two sub-datasets from patients that have
a longitudinal follow-up to fit our experiment setup.

  1. 1.

Pair-wise dataset: From the OPHDIAT database, we selected pairs from patients
with at least one DR severity change in their follow-up. This resulted in
10412 patients and 49579 numbers of pairs.

  2. 2.

Sequence-wise dataset: From the OPHDIAT database, we selected patients that
have at least four visits. This resulted in 7244 patients and 13997 sequences.

For both datasets, the split was conducted with the following distribution:
training (60%), validation (20%), and test (20%) based on subjects, i.e.,
images of a single subject belonged to the same split and in a way that
preserves the same proportions of examples in each class as observed in the
original dataset. We also ensured that for both datasets, there was no
intersection between a patient in training/validation/test sets. Except for
the registration stage, we followed the same image processing performed in
[19].

###  3.2 Implementation details

As conducted in [21, 13, 19], a basic AE was implemented in order to focus on
the methodology presented. We used the same encoder and decoder as used in
[19]. This encoder provides a latent representation of size 64 × 4 × 4. The
different networks were trained for 400 epochs by the AdamW optimizer, with a
learning rate of $5\times 10^{-4}$, OneCycleLR as scheduler, and a weight
decay of $10^{-5}$, using an A6000 GPU with the PyTorch framework and trained
on the pair-wise dataset. The regularization weights were set to
$\lambda_{dir}=1.0$ and $\lambda_{recon}=1.0$. Concerning NODE, we used the
torchdiffeq framework, which provides integration of NODE in Pytorch with the
possibility to use a numeric solver with the adjoint method, which allows
constant memory usage. Our neural ODE function consists of the succession of
dense layers followed by a tanh activation function. We employed the fifth-
order "dopri5" solver with an adaptive step size for the generative model,
setting the relative tolerance to 1e-3 and the absolute tolerance to 1e-4.
When using the dopri solver with the adjoint method, the torchdiffeq package
does not provide support for a batched time solution, so during training for
the forward batch in the NODE we used the workaround introduced in [11] to
enable a batched time solution.

###  3.3 Evaluating the learned feature extractor

To evaluate the quality of the encoder from the different experiments, we will
use two tasks with a longitudinal nature. We only perform fine-tuning
scenarios, as performed in [7] to assess the quality of the learned weights.
We initialize the weights of the encoder that will be used for the two tasks,
with the weights of the longitudinal pre-training method presented in Fig.1.
An additional AE and AE-NODE were added for comparison purposes. Each task was
trained for 150 epochs with a learning rate of $10^{-3}$ and a weight decay of
$10^{-4}$ and OneCycleLR as a scheduler. The first one is the prediction of
the patient’s age in years, using a CFP (this task is called age regression
for the rest of the manuscript), and the second task is the prediction of the
development of the DR for the next visit using the past three examinations
(task called predict next visit). For the age regression, the model developed
is a combination of the implemented encoder and a multi-layer perceptron. The
model is trained with the Mean Squared Error (MSE) on the image selected for
the sequence-wise dataset. For predict next visit, we used a CNN+RNN [6], a
standard approach for tackling sequential or longitudinal tasks. The RNN we
used is a long short-term memory (LSTM). The model is trained using cross-
entropy, on the sequence-wise dataset. We used the Area Under the receiver
operating characteristic Curve (AUC) with three binary tasks which are the
following: predicting at least NPDR in the next visit (AUC Mild+), at least
moderate NPDR in the next visit denoted (AUC Moderate+) and finally at least
severe NPDR in the next visit (AUC severe+) for evaluating the models.

Weights $\lambda_{dir}$ $\lambda_{recon}$ NODE MSE From scratch - - - 0.0070
AE 0 1 No 0.0069 AE 0 1 Yes 0.0069 LSSL 1 1 No 0.0050 LSSL 1 1 Yes 0.0048
Siamese LSSL 1 0 No 0.0049 Siamese LSSL 1 0 Yes 0.0046

Table 1: Results for the age regression task reporting the Mean Squared Error
(MSE) in squared years. Best results for the LSSL and S-LSSL are in underline,
best results for their NODE version are in bold.

Weights $\lambda_{dir}$ $\lambda_{recon}$ NODE AUC Mild+ AUC Moderate + AUC
severe+ From scratch - - - 0.574 0.602 0.646 AE 0 1 No 0.563 0.543 0.636 AE 0
1 Yes 0.569 0.565 0.649 LSSL 1 1 No 0.578 0.618 0.736 LSSL 1 1 Yes 0.604 0.630
0.760 Siamese LSSL 1 0 No 0.578 0.596 0.708 Siamese LSSL 1 0 Yes 0.569 0.549
0.756

Table 2: Results for the predict next visit label for the different
longitudinal pretext task for the three binary tasks using the AUC. Best
results for the LSSL and S-LSSL are in underline, best results for their NODE
version are in bold.

According to the results of both Tab.1 and 2, we observe that longitudinal
pre-training is an efficient pre-training strategy to tackle a problem with a
longitudinal nature compared to training from scratch of classic autoencoder,
which is aligned with the following studies [7, 13, 21, 19]. For the age
regression task, according to the results presented in Tab.1, the best
longitudinal pre-training strategy is the Siamese LSSL version with NODE.
However, the difference in performance is marginal, indicating that the
Siamese LSSL could also be used as pretext task. The cosine alignment term in
the loss function, which is the one responsible for forcing the model to
encode longitudinal information, seems to be beneficial in solving
longitudinal downstream tasks.

For the predict next visit task (Tab.2), the LSSL-NODE version performs better
than the rest. An observation that could explain the difference is that during
the training of the LSSL-NODE, the reconstruction term and the direction
alignment (second term of Eq.4) in the loss function reache 0 at the end of
the training. While for the LSSL, only the reconstruction term converges to 0.
The direction alignment term faces a plateau of around 0.3. One way to explain
this convergence issue is the fact that the longitudinal pair were selected
randomly. Even if it is the same patient, the image may have a very different
field of view which could increase the difficulty for the model to both
minimize the reconstruction loss and cosine alignment term. The LSSL-NODE does
not have this issue since only one image is given to the CNN, which we suspect
ease the problem solved by the LSSL+NODE. Another way to explain this
phenomena is the fact that the LSSL was trained with $\lambda_{recon}$ and
$\lambda_{dir}$ set to 1, a more advanced weight balanced between the two
terms in the loss could reduce this issue. This observation could be the
explanation for the difference in performance when using the LSSL-NODE vs
classic LSSL. Another simple explanation could be the fact that we did not use
any registration method for the longitudinal pairs. Surprisingly, according to
both Tab.1 and 2, the model with the NODE extension performed well compared to
their original version. We believe that the NODE forces the CNN backbone to
learn a representation that is more suitable for modeling the dynamics of the
disease progression.

###  3.4 Analysis of the norm of $\Delta_{z}$

Using the same protocol introduced in [13, 21], we computed the norm of the
trajectory vector. The intuition behind this computation is the following:
$\Delta_{z}$ can be seen as some kind of vector that indicates the speed of
disease progression because it can be regarded as an instantaneous rate of
change of $\mathbf{z}$ normalized by 1. For the different extensions of LSSL,
we evaluate pregnancy factor [9] and type of diabetes [1], which are known
factors that characterize the speed of the disease progression in the context
of DR. This is done to analyze the capacity of $\Delta_{z}$ to capture the
speed of disease progression.

  1. 1.

Pregnancy: Pregnant vs not pregnant (only female). For the pregnant group, we
selected longitudinal pairs from patients that were in early pregnancy and in
close monitoring. For the rest, we selected longitudinal pair from female
patients without antecedent of pregnancy.

  2. 2.

Diabetes type: Patients with known diabetes type. We only selected a
longitudinal pair of patients for known diabetes type 1 or 2.

First, patients present in the training set for the longitudinal pretext task
were not allowed to be selected as part of any group. For the first group, we
randomly selected 300 patients for each category. For the second group, we
randomly selected 2000 patients for the two categories. We applied a
statistical test (student t-test) to explore if the mean value of the norm of
the trajectory vector with respect to both defined factors has a larger mean
value for patients with pregnancy than for patients without pregnancy. And if
the patient with type 1 diabetes had a higher mean than type 2. We observe
that the norm of the trajectory vector $(\Delta z)$ is capable of dissociating
the two types of diabetes (t-test p-value < 0.01) and pregnancy type (t-test
p-value < 0.00001) for all models that have a cosine alignment term (except
the AE and the AE+NODE extension). Regarding the type of diabetes, a specific
study in the OPHDIAT dataset [1] indicated that the progression of DR was
faster in patients with type 1 diabetes than for patients with type 2
diabetes. Furthermore, pregnancy is a known factor [9] in the progression
speed of DR. Those observations are aligned with the expected behavior of the
two factors. In addition, as observed in [13, 19, 21], standard AE is not
capable of encoding disease progression.

###  3.5 Evaluation of the NODE weights

For models trained with a NODE, we design a protocol to assess the capacity of
NODE to learn a meaningful representation related to disease progression. The
protocol to evaluate the weights of the NODE is as follows. We implemented a
NODE classifier (NODE-CLS) illustrated in Fig.2. The NODE classifier is
constructed as the concatenation of a backbone, a NODE, and a multilayer
perceptron (MLP). The architecture of the backbone and the NODE is the same
that was used with the LSSL method. The MLP consists of two fully connected
layers of dimensions 1024 and 64 with LeakyReLU activation followed by a last
single perceptron, which project the last layer representation into the number
of class, and the network is trained using the pairwise dataset. The NODE-CLS
only requires a single CFP denoted $x^{t_{i}}$ and $\Delta_{t}=t_{i+1}-t_{i}$,
which is the time lapse between examinations $x^{t_{i}}$ and $x^{t_{i+1}}$.
The image is first given to the backbone, which produces the latent
representation $z^{t_{i}}$, then this vector is fed to the black box NODE,
using the same IPV defined for the LSSL-NODE we can define the latent
representation of the next visit. Using the predicted latent representation by
the NODE, we predicted the severity grade of the next visit with the MLP. The
same loss and metrics were used for the predict next visit task.

Weights $\lambda_{dir}$ $\lambda_{recon}$ NODE AUC Mild+ AUC moderate+ AUC
severe+ From scratch - - - 0.552 0.600 0.583 AE 0 1 Yes 0.561 0.609 0.590 LSSL
1 1 Yes 0.547 0.609 0.641 Siamese LSSL 1 0 Yes 0.558 0.617 0.670

Table 3: Comparaison of AUC for the NODE classifier using different initialize
weights, best results in bold. Figure 2: Illustration of proposed NODE
classifier for evaluating the weights of the method with a backbone + NODE

For the NODE-CLS, Tab.3 shows a clear performance gain compared to the method
trained from scratch. The best weights are obtained with the Siamese-LSSL. The
classical LSSL also provides descent result, for the different subtask. We
suspect that the LSSL model is more affected by the design of our experiments.
Since we randomly selected images per examination, we did not perform any
registration step. These results suggest that the classical LSSL is more
likely to underperform than the Siamese-like method when the pair are not
registered, aligned with the finding of [8].

##  4 Discussion and conclusion

In this paper, we investigated the use of the LSSL under different scenarios
in order to get a better understanding of the LSSL framework. Our result
demonstrated that the use of Siamese-like LSSL is possible. The S-LSSL might
be more suitable when not registered pairs are given to the model. Moreover,
for the various tasks that were used to evaluate the quality of the weights, a
performance gain was observed when the LSSL and S-LSSL coupled with the NODE
compared to their standard versions, while the reasons for such performance
remain unclear. We hypothesize that adding the NODE in training forces the
linked backbone to provide an enriched representation embedded with
longitudinal information. Thus, since the formulation of the original LSSL
[21] is based on a differential equation, which resulted in the introduction
of the cosine alignment term in the objective function. Including the NODE is
aligned with the formulation of the LSSL problem, which is reflected in our
results. In this direction, we are interested in looking at the representation
of the learned neural ODE, in order to understand if the neural ODE is able to
interpolate the spatial feature and thus give the opportunity not even to need
the registration. If so, the use of NODE with longitudinal self-supervised
learning could also overcome the need of heavy registration step. LSSL
techniques are quite promising and we believe that they will continue to grow
at a fast pace. We would like to extend this study by including more
frameworks [7, 8, 13, 5] as well as more longitudinal datasets.

Some limitations should be pointed out, since we selected a random view of CFP
for each examination, we did not apply any registration step. Using a better
pairing strategy with the right registration step could enhance the results
presented for LSSL. Moreover, we did not perform any elaborate hyperparameter
search to find the correct loss weights in order to reach a complete loss
convergence for the LSSL.

This work also opens up interesting research questions related to the use of
neural ODE. The formulation and hypotheses of the LSSL framework are based on
a differential equation that could be directly learned via a Neural ODE which
could explain the better performance of the LSSL+NODE version. In the future,
we will work on a theoretical explanation as to why this blending works.

In addition, for the models trained with a NODE, our experiments demonstrated
the possibility of using pretext tasks on Neural ODE in order to provide an
enhanced representation. In the classic SSL pretext task paradigm, the goal is
to learn a strong backbone that yields good representation to solve specific
downstream task. The results from the NODE-CLS experiments indicated that pre-
training techniques where a NODE is involved can be reused on longitudinal
downstream tasks and have the ability to enhance the results when a NODE is
part of the model. In the future, we would like to explore time-aware
pretraining in general, with a pretext task that has longitudinal context to
be reused on longitudinal tasks.

Acknowledgements The work takes place in the framework of Evired, an ANR RHU
project. This work benefits from State aid managed by the French National
Research Agency under the “Investissement d’Avenir” program bearing the
reference ANR-18-RHUS-0008.

## References

  * [1] Chamard, C., Daien, V., Erginay, A., Gautier, J.F., Villain, M., Tadayoni, R., Carriere, I., Massin, P.: Ten-year incidence and assessment of safe screening intervals for diabetic retinopathy: the OPHDIAT study. British Journal of Ophthalmology 105(3), 432–439 (Jun 2020). https://doi.org/10.1136/bjophthalmol-2020-316030, https://doi.org/10.1136/bjophthalmol-2020-316030
  * [2] Chen, R.T.Q., Rubanova, Y., Bettencourt, J., Duvenaud, D.: Neural ordinary differential equations (2018). https://doi.org/10.48550/ARXIV.1806.07366, https://arxiv.org/abs/1806.07366
  * [3] Chen, R.T.Q., Rubanova, Y., Bettencourt, J., Duvenaud, D.: Neural ordinary differential equations (2019)
  * [4] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations (2020). https://doi.org/10.48550/ARXIV.2002.05709, https://arxiv.org/abs/2002.05709
  * [5] Couronné, R., Vernhet, P., Durrleman, S.: Longitudinal self-supervision to disentangle inter-patient variability from disease progression (09 2021). https://doi.org/10.1007/978-3-030-87196-3_22
  * [6] Cui, R., Liu, M.: Rnn-based longitudinal analysis for diagnosis of alzheimer’s disease. Computerized Medical Imaging and Graphics 73, 1–10 (2019). https://doi.org/https://doi.org/10.1016/j.compmedimag.2019.01.005, https://www.sciencedirect.com/science/article/pii/S0895611118303987
  * [7] Emre, T., Chakravarty, A., Rivail, A., Riedl, S., Schmidt-Erfurth, U., Bogunović, H.: Tinc: Temporally informed non-contrastive learning for disease progression modeling in retinal oct volumes. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. pp. 625–634. Springer Nature Switzerland, Cham (2022)
  * [8] Kim, H., Sabuncu, M.R.: Learning to compare longitudinal images (2023)
  * [9] Klein, B.E.K., Moss, S.E., Klein, R.: Effect of Pregnancy on Progression of Diabetic Retinopathy. Diabetes Care 13(1), 34–40 (01 1990). https://doi.org/10.2337/diacare.13.1.34, https://doi.org/10.2337/diacare.13.1.34
  * [10] Lachinov, D., Chakravarty, A., Grechenig, C., Schmidt-Erfurth, U., Bogunovic, H.: Learning spatio-temporal model of disease progression with neuralodes from longitudinal volumetric data (2022)
  * [11] Lechner, M., Hasani, R.: Learning long-term dependencies in irregularly-sampled time series. arXiv preprint arXiv:2006.04418 (2020)
  * [12] Massin, P., Chabouis, A., Erginay, A., Viens-Bitker, C., Lecleire-Collet, A., Meas, T., Guillausseau, P., Choupot, G., André, B., Denormandie, P.: Ophdiat©: A telemedical network screening system for diabetic retinopathy in the ile-de-france. Diabetes & metabolism 34, 227–34 (07 2008). https://doi.org/10.1016/j.diabet.2007.12.006
  * [13] Ouyang, J., Zhao, Q., Adeli, E., Sullivan, E.V., Pfefferbaum, A., Zaharchuk, G., Pohl, K.M.: Self-supervised longitudinal neighbourhood embedding
  * [14] Qian, Z., Zame, W.R., Fleuren, L.M., Elbers, P., van der Schaar, M.: Integrating expert odes into neural odes: Pharmacology and disease progression (2021)
  * [15] Ren, M., Dey, N., Styner, M.A., Botteron, K., Gerig, G.: Local spatiotemporal representation learning for longitudinally-consistent neuroimage analysis (2022)
  * [16] Rivail, A., Schmidt-Erfurth, U., Vogel, W.D., Waldstein, S.M., Riedl, S., Grechenig, C., Wu, Z., Bogunovic, H.: Modeling disease progression in retinal octs with longitudinal self-supervised learning. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11843 LNCS, 44–52 (2019). https://doi.org/10.1007/978-3-030-32281-6_5
  * [17] Rubanova, Y., Chen, R.T.Q., Duvenaud, D.: Latent odes for irregularly-sampled time series (2019). https://doi.org/10.48550/ARXIV.1907.03907, https://arxiv.org/abs/1907.03907
  * [18] Vernhet, P., Durrleman, S.: Longitudinal self-supervision to disentangle inter-patient variability pp. 231–241 (2021). https://doi.org/10.1007/978-3-030-87196-3
  * [19] Zeghlache, R., Conze, P.H., Daho, M.E.H., Tadayoni, R., Massin, P., Cochener, B., Quellec, G., Lamard, M.: Detection of diabetic retinopathy using longitudinal self-supervised learning. In: Antony, B., Fu, H., Lee, C.S., MacGillivray, T., Xu, Y., Zheng, Y. (eds.) Ophthalmic Medical Image Analysis. pp. 43–52. Springer International Publishing, Cham (2022)
  * [20] Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14. pp. 649–666. Springer (2016)
  * [21] Zhao, Q., Liu, Z., Adeli, E., Pohl, K.M.: Longitudinal self-supervised learning. Medical Image Analysis 71 (2021). https://doi.org/10.1016/j.media.2021.102051


# FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in
Computational Pathology

###### Abstract

The fusion of complementary multimodal information is crucial in computational
pathology for accurate diagnostics. However, existing multimodal learning
approaches necessitate access to users’ raw data, posing substantial privacy
risks. While Federated Learning (FL) serves as a privacy-preserving
alternative, it falls short in addressing the challenges posed by
heterogeneous (yet possibly overlapped) modalities data across various
hospitals. To bridge this gap, we propose a Federated Multi-Modal (FedMM)
learning framework that federatedly trains multiple single-modal feature
extractors to enhance subsequent classification performance instead of
existing FL that aims to train a unified multimodal fusion model. Any
participating hospital, even with small-scale datasets or limited devices, can
leverage these federated trained extractors to perform local downstream tasks
(e.g., classification) while ensuring data privacy. Through comprehensive
evaluations of two publicly available datasets, we demonstrate that FedMM
notably outperforms two baselines in accuracy and AUC metrics.

Index Terms—  Multimodal Fusion, Federated Learning, Histology Image, Genomic
Signal, Modality Heterogeneity.

##  1 Introduction

Recent advancements in computational pathology have triggered a revolution in
medical diagnostics and research [1], [2]. A critical aspect of this progress
involves leveraging complementary data from diverse sources to improve
diagnosis through machine learning [3], [4]. For instance, researchers can
combine histology data, such as Whole Slide Images (WSI), with genomic data
like Copy Number Variations (CNV) to gain a comprehensive understanding of
cancer subtypes [5]. This multimodal approach leads to enhanced diagnostic
accuracy and personalized treatment options [6].

However, multimodal data fusion raises significant privacy concerns,
particularly when handling medical data [7]. To tackle this issue, Federated
Learning (FL) [8], a privacy-conscious distributed learning framework, has
drawn substantial attention. FL facilitates the local update of models, which
are then uploaded to a central server, thus eliminating the requirement to
share raw data [9]. Although current FL methods prove effective in scenarios
with single-modal FL or multimodal FL with homogeneous modality distributions
(i.e., the input maintains a consistent structure), they fall short in
multimodal FL systems with modality heterogeneity [10].

Fig. 1: The problem of modality heterogeneity in computational pathology
(left) and our solution (right). The aim is to harness distributed privacy-
sensitive data, which might overlap on some modalities across several
hospitals, to train multiple single-modal feature extractors federatedly.
These federated extractors are intended to exhibit superior performance
compared to locally trained extractors.

This modality heterogeneity arises from multiple factors. First, equipment
limitations at different hospitals may restrict the use of multiple diagnostic
devices, leading to a narrow range of available data modalities for diagnosis,
as illustrated in Fig. 1. Second, patient preferences for specific hospitals
may result in some hospitals possessing limited available data modalities for
diagnosing the same disease. Lastly, data housed in various hospitals may be
non-identically and independently distributed (non-i.i.d). Consequently,
collaborative training becomes unfeasible under the conventional FL framework.
Hospitals are thus constrained to local model training, overlooking the
potential gains that could be achieved by leveraging overlapped modalities
across various hospitals (i.e., clients in FL).

To bridge this gap, we propose a Federated Multi-Modal (FedMM) Learning
framework. Unlike traditional FL, which aims to train a multimodal fusion
model, FedMM trains multiple single-modal feature extractors federatedly. This
allows each participating hospital to locally leverage these federated feature
extractors for feature extraction and subsequent classification tasks.
Besides, as these federated feature extractors are trained using distributed
data, their capacity for generalization is significantly enhanced.
Specifically, any hospital can benefit from this framework, no matter its
device availability and sample size. For example, even newly-established
hospitals, constrained by limited samples and devices, can leverage these
federated extractors to aid local model training while ensuring data privacy.
Our key contributions are summarized as follows:

  * •

We leverage multimodal data to enhance performance in classifying diverse
subtypes of lung and kidney cancers.

  * •

We propose FedMM, armed with an innovative dynamic loss function, to address
the issue of modality heterogeneity in computational pathology while ensuring
data privacy.

  * •

Through comprehensive evaluation of two publicly available datasets, we
demonstrate that FedMM significantly outperforms two established baseline
methods.

##  2 Related work

In computational pathology, multimodal information fusion significantly
improved diagnostic accuracy [11], [12]. Wang et al. developed a hybrid
network that combined image and genomics data to improve lung cancer subtype
diagnosis [5]. Chen et al. applied multimodal deep learning to analyze
pathology images and molecular profiles across 14 cancer types [13]. However,
these methods commonly overlooked the privacy-sensitive nature of medical
data. The difficulty in collecting large-scale data for model training further
restricted their real-world applicability [14].

The advent of FL presents a promising distributed training framework that
addresses privacy concerns across different institutions [15], [16]. Lu et al.
showed FL’s effectiveness in computational pathology [17], while Li et al.
proposed a privacy-preserving framework for multi-site fMRI analysis [18].
However, existing solutions focused on single-modal FL, struggling with
heterogeneous data often found across different clients [19], [20], [21]. Very
recently, Yu et al. introduced CreamFL [22], which employs the contrastive
representation ensemble to train a global model with the support of multiple
clients, diverging from our emphasis on client-side performance. Chen et al.
presented FedMSplit [23], which modularized client models into shareable
blocks to accommodate diverse sensor setups. Concurrently to our work, Ouyang
et al. proposed Harmony [24], but in the fusion phase, limited samples impeded
the ideal aggregation of multimodal client classifiers within the same
cluster, owing to the necessity for consistent modality combinations.

To the best of our knowledge, we are the first to tackle the FL issue with
modality heterogeneity in computational pathology.

##  3 Method

###  3.1 Problem Formulation

In a typical FL setup, multiple clients work with one central server to train
a shared model, denoted as $\mathcal{F}(\cdot)=g\left[f(\cdot)\right]$. Here,
$f(\cdot)$ is the feature extractor like a Convolutional Neural Network (CNN),
and $g(\cdot)$ is the classifier like a Multilayer Perceptron (MLP). The goal
is to train a global model $\mathcal{F}(\cdot)$ using distributed data from
different clients and minimize the loss function, which can be formulated as:

$\displaystyle\underset{\mathbf{w}_{i}}{\arg\min}\left\\{\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}\left[\mathcal{F}_{i}(\mathbf{w}_{i};\mathbf{X}_{i}),y_{i}\right]=\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}\left[\mathbf{w}_{i};g_{i}\left[f_{i}(\mathbf{X}_{i})\right],y_{i}\right]\right\\}$
(1)

where $N$ is the total number of clients, $\mathbf{w}_{i}$ represents the
learnable weights associated with the $i$-th client model, $\mathbf{X}_{i}$ is
the input, $y_{i}$ is the corresponding label, and $\mathcal{L}$ denotes the
loss function. This method works well in single-modal FL or multimodal FL with
homogeneous modality distributions (i.e., consistent input structure).

However, it falls short when dealing with heterogeneous modality distributions
across various clients because clients may possess varying numbers and types
of modalities data. Such heterogeneity hampers the optimization of feature
extractor $f(\cdot)$ and classifier $g(\cdot)$ using existing FL methods like
FedAvg [8], FedProx [25], etc.

Thus, the intuition behind our study is to investigate the feasibility of
aggregating feature extractors corresponding to the same modality across all
clients. Our strategy aims to train multiple single-modal feature extractors
federatedly by utilizing privacy-sensitive, small-scale data from various
local clients. Subsequently, each client deploys these federated extractors to
extract features from the corresponding modality data locally. These extracted
features then serve as the input for each client’s classifier. Notably, the
aggregation process excludes the participation of classifiers from different
clients since the nature of heterogeneous inputs is unavoidable.

Specifically, we consider a multimodal FL system comprising $N$ clients and
one central server. Each client possesses up to $M$ different modalities in
their local data, where $M\geq 2$. For any arbitrary client $C_{i}$, where
$1\leq i\leq N$, the local sensitive-privacy dataset is denoted as
$\left\\{\mathbf{X}_{i},y_{i}\right\\}$. Here,
$\mathbf{X}_{i}=\\{\mathbf{x}_{i}^{(k)}\\}_{k=1}^{M_{i}}$ contains the number
of $M_{i}$ modalities. To put it more clearly, for the client $i$, the
available modalities are denoted by $\forall k\in[M_{i}]$, while for the
server, they are $\forall k\in[M]$, and $[M_{i}]\subseteq[M]$.

For the client $C_{i}$, the data of each modality is individually fed into the
corresponding single-modal feature extractor ${f_{i}}^{(k)}(\cdot)$ to
generate $M_{i}$ sets of feature embeddings:

$\\{\mathbf{h}_{i}^{(k)}\\}_{k=1}^{M_{i}}=\\{{f_{i}}^{(k)}[\mathbf{x}_{i}^{(k)}]\\}_{k=1}^{M_{i}}$
(2)

Here, $\mathbf{h}_{i}^{(k)}$ represents the feature embedding extracted by
${f_{i}}^{(k)}(\cdot)$. Subsequently, these diverse extracted single-modal
feature embeddings can be fused (concatenated) to combine complementary
information, provided they originate from the same sample. The resulting
concatenated multimodal feature embedding is then used as input to the local
classifier $g_{i}(\cdot)$ for making prediction $\hat{y}_{i}$, as expressed
below (the concatenated expression $\oplus$ is omitted for brevity):

$\hat{y}_{i}=g_{i}\\{{f_{i}}^{(k)}[\mathbf{x}_{i}^{(k)}]\\}_{k=1}^{M_{i}}=g_{i}\\{\mathbf{h}_{i}^{(k)}\\}_{k=1}^{M_{i}}$
(3)

The multimodal FL model of client $C_{i}$ can be expressed as:

$\mathcal{F}_{i}(\mathbf{w}_{i};\mathbf{X}_{i})=\left\\{\mathbf{w}_{i};g_{i}\\{\mathbf{h}_{i}^{(k)}\\}_{k=1}^{M_{i}}\right\\}$
(4)

where $\mathbf{w}_{i}$ denotes the learnable weights of the model associated
with client $C_{i}$. Thus, the objective of multimodal FL is to train a set of
models $\\{\mathcal{F}_{i}(\cdot)\\}_{i=1}^{N}$. However, the heterogeneous
model architectures result in learnable weights $\mathbf{w}_{i}$ for each
local model having different formats. Thus, these heterogeneous local model
weights $\\{\mathbf{w}_{i}\\}_{i=1}^{N}$ cannot be directly aggregated using
existing FL methods.

More explicitly, our objective in Eq. (5) is to minimize the loss function for
clients with modality heterogeneity. This problem diverges from the
conventional FL problem formulation in Eq. (1).

$\underset{\mathbf{w}_{1},\mathbf{w}_{2},...,\mathbf{w}_{N}}{\arg\min}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}\\{\mathbf{w}_{i};g_{i}\\{{f_{i}}^{(k)}[\mathbf{x}_{i}^{(k)}]\\}_{k=1}^{M_{i}},y_{i}\\}$
(5)

###  3.2 FedMM

To tackle the issue of clients with heterogeneous but possibly overlapped
modalities data, FedMM aims to train multiple single-modal feature extractors
federatedly to help improve subsequent classification performance. This
contrasts with other methods focusing on training a multimodal fusion model.
In FedMM, all clients collaboratively contribute to training multiple
federated single-modal feature extractors. Subsequently, each client can
utilize these federated extractors to extract features and perform
classifications locally.

Specifically, for a multimodal client $C_{i}$ with $M_{i}$ modalities, it
parallelly trains a total of $M_{i}$ distinct single-modal feature extractors
$\\{f_{i}^{(k)}(\cdot)\\}_{k=1}^{M_{i}}$. Beyond using the local true label
for training, we innovatively incorporate the global prototype as a “pseudo-
label”. This innovation allows us to design a dynamic loss function and
overcome the challenge of updating federated feature extractors in the absence
of labels due to privacy constraints.

The global prototype acts as a proxy for a given class and is determined by
averaging the embeddings within the same class and modality across all clients
[26], [27], [28]. Notably, each modality has the same number of prototypes as
the number of classes, and we omit to label the number of prototypes of the
same modality below for brevity. The prototype of $k$-th modality can be
expressed as:

$\bar{\mathbf{p}}^{(k)}=\frac{1}{N_{k}}\sum_{i=1}^{N_{k}}\left\\{\mathbf{h}_{i}^{(k)}\right\\}$
(6)

Here, $N_{k}$ is the number of clients that possess the $k$-th modality (i.e.,
a subset of $N$ clients). Notably, federated feature extractors can be trained
using both local true label $\hat{y}_{i}$ and global prototype
$\bar{\mathbf{p}}^{(k)}$.

The aim of training the feature extractor for the $k$-th modality is to
minimize a loss function consisting of two components. The first term  1
calculates the $L2$ distance loss between the local embedding
$\mathbf{h}_{i}^{(k)}$ and the global prototype $\bar{\mathbf{p}}^{(k)}$. The
second term  2 incorporates the binary cross-entropy ($BCE$) loss between the
local prediction $\hat{y}_{i}$ and the local true label $y_{i}$. The objective
function is formulated as:

$\displaystyle\min\left\\{\underbrace{\lambda(t)\frac{\beta}{D}\left[L2\left(\mathbf{h}_{i}^{(k)},\bar{\mathbf{p}}^{(k)}\right)\right]}_{\clap{\scalebox{0.8}{$\leavevmode\hbox
to8.36pt{\vbox to8.36pt{\pgfpicture\makeatletter\hbox{\hskip
4.17778pt\lower-4.17778pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{3.97778pt}{0.0pt}\pgfsys@curveto{3.97778pt}{2.19688pt}{2.19688pt}{3.97778pt}{0.0pt}{3.97778pt}\pgfsys@curveto{-2.19688pt}{3.97778pt}{-3.97778pt}{2.19688pt}{-3.97778pt}{0.0pt}\pgfsys@curveto{-3.97778pt}{-2.19688pt}{-2.19688pt}{-3.97778pt}{0.0pt}{-3.97778pt}\pgfsys@curveto{2.19688pt}{-3.97778pt}{3.97778pt}{-2.19688pt}{3.97778pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.575pt}{-2.03pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}$}}}+\underbrace{\Bigl{[}1-\lambda(t)\Bigr{]}\Bigl{[}BCE\bigl{(}\hat{y}_{i},y_{i}\bigr{)}\Bigr{]}}_{\clap{\scalebox{0.8}{$\leavevmode\hbox
to8.36pt{\vbox to8.36pt{\pgfpicture\makeatletter\hbox{\hskip
4.17778pt\lower-4.17778pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{3.97778pt}{0.0pt}\pgfsys@curveto{3.97778pt}{2.19688pt}{2.19688pt}{3.97778pt}{0.0pt}{3.97778pt}\pgfsys@curveto{-2.19688pt}{3.97778pt}{-3.97778pt}{2.19688pt}{-3.97778pt}{0.0pt}\pgfsys@curveto{-3.97778pt}{-2.19688pt}{-2.19688pt}{-3.97778pt}{0.0pt}{-3.97778pt}\pgfsys@curveto{2.19688pt}{-3.97778pt}{3.97778pt}{-2.19688pt}{3.97778pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.575pt}{-2.03pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{2}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}$}}}\right\\}$ (7)

In more explicit terms, Eq. (7) is given by:

$\displaystyle\min\left\\{\lambda(t)\frac{\beta}{D}\left\|\mathbf{h}_{i}^{(k)}-\bar{\mathbf{p}}^{(k)}\right\|_{2}\right.$
$\displaystyle\left.+\Bigl{[}1-\lambda(t)\Bigr{]}\Bigl{[}-y_{i}\log(\hat{y}_{i})-(1-y_{i})\log(1-\hat{y}_{i})\Bigr{]}\right\\}$
(8)

Here, $D$ represents the dimensionality of the embedding, while $\beta$ is
introduced to balance the magnitudes of the $L2$ and $BCE$ losses. Initially,
it can be set to 1 and fine-tuned based on the losses observed during initial
training. The dynamic weight $\lambda(t)$ is defined as:

$\lambda(t)=\frac{1}{1+e^{-\alpha(t-t_{0})}}$ (9)

where $t_{0}$ designates the transition point (round) for shifting emphasis
from the $BCE$ loss to the $L2$ loss, and $\alpha$ modulates the sharpness of
this transition.

Fig. 2: An illustration of a multimodal FL system with modality heterogeneity.
It includes $N=3$ clients and one central server with $M=2$ modality
processing components.

During training, it’s crucial to dynamically adjust these losses using
$\lambda(t)$, especially as the initial prototype might not accurately capture
the actual underlying patterns of the data. Giving too much weight to the $L2$
loss early on could make the model overly sensitive to these potentially
inaccurate prototypes.

Hence, $\lambda(t)$ starts by giving more emphasis to the $BCE$ loss,
leveraging the local true label for clearer feedback. As training progresses
and the feature extractor refines its representations, the prototypes likewise
become more accurate and reliable. At this point, $\lambda(t)$ gradually
increases the emphasis on the $L2$ loss. More specifically, the first term  1
aims to ensure the trained feature extractor generates embeddings closely
aligned with the global prototype, and the second term  2 aims to ensure the
trained feature extractor can help make prediction values closer to the label
on the local client.

More explicitly, in the $t$-th round of FedMM training, the procedures for
client update and server update are as follows:

Client Update: Client $C_{i}$ parallelly updates $M_{i}$ single-modal feature
extractors $\\{f_{i}^{(k)}\\}_{k=1}^{M_{i}}$ using its local multimodal data
$\\{\mathbf{x}_{i}^{(k)}\\}_{k=1}^{M_{i}}$ and the received global prototypes
$\\{\bar{\mathbf{p}}^{(k)}\\}_{k=1}^{M_{i}}$ from the server.

$f_{i}^{(k),t}(\cdot)\leftarrow
SGD\left\\{f_{i}^{(k),t-1}(\cdot),(\mathbf{x}_{i}^{(k)},y_{i},\bar{\mathbf{p}}^{(k)})\right\\}$
(10)

Server Update: The server runs $2\times M$ threads performing model
aggregation to update single-modal feature extractors and embedding
aggregation to update global prototypes. Specifically, upon receiving the
model weights from $N_{k}$ clients, the server performs the extractor
aggregation and embedding aggregation as follows:

$\begin{cases}\begin{aligned}
\bar{f}^{(k),t}(\cdot)&\leftarrow\text{$Fed$}\left\\{f_{1}^{(k),t}(\cdot),f_{2}^{(k),t}(\cdot),\ldots,f_{N_{k}}^{(k),t}(\cdot)\right\\}\\\
\bar{\mathbf{p}}^{(k),t}&\leftarrow\frac{1}{N_{k}}\sum_{i=1}^{N_{k}}\left\\{\mathbf{h}_{i}^{(k),t}\right\\}\end{aligned}\end{cases}$
(11)

Here, $Fed(\cdot)$ can encompass any established FL methods, including but not
limited to FedAvg [8] and FedProx [25]. This flexibility allows FedMM to
seamlessly integrate with various state-of-the-art FL methods, thereby
enhancing its potential for real-world applications.

Subsequently, these federated feature extractors are deployed locally to
extract corresponding modality features from small-scale, privacy-sensitive
datasets on each client. These extracted features then serve as the input for
the local classifiers residing on each client.

In sum, this transformation from a complex and heterogeneous multimodal input
into multiple manageable single-modal inputs provides an effective solution to
the challenges posed by heterogeneous modalities within a multimodal FL
system.

##  4 Experiments

###  4.1 Datasets

We evaluated FedMM using two publicly available datasets from The Cancer
Genome Atlas (TCGA) 111https://portal.gdc.cancer.gov/repository. To mirror the
real-world challenge of modality heterogeneity across various hospitals, we
reasonably distributed these data among three hospitals, as shown in Table 1.
The flow of the experiment is illustrated in Fig. 2.

TCGA-NSCLC: This dataset contains data from patients with non-small cell lung
cancer (NSCLC). It contains two subtypes: adenocarcinoma (LUAD) and squamous
cell carcinoma (LUSC).

TCGA-RCC: This dataset includes data from patients with renal cell carcinoma
(RCC), the most prevalent type of kidney cancer. It mainly includes three
subtypes: Kidney Renal Clear Cell Carcinoma (KIRC), Kidney Renal Papillary
Cell Carcinoma (KIRP), and Kidney Chromophobe (KICH). Due to limited KICH
samples, we focus solely on binary classification for KIRC and KIRP subtypes.

Table 1: Sample number details of two public datasets from TCGA.

Dataset | Modality | Class | Hospital 1 | Hospital 2 | Hospital 3 | Total
---|---|---|---|---|---|---
TCGA-NSCLC | WSI | LUAD | 199 | 315 | 0 | 1009
LUSC | 210 | 285 | 0
CNV | LUAD | 0 | 315 | 203 | 1022
LUSC | 0 | 285 | 219
TCGA-RCC | WSI | KIRC | 246 | 291 | 0 | 824
KIRP | 129 | 158 | 0
CNV | KIRC | 0 | 291 | 243 | 825
KIRP | 0 | 158 | 133

###  4.2 Implementation and Reproducibility

WSI Feature Extractor: Owing to the gigapixel size of the WSI (e.g.,
$40,000\times 40,000$ pixels), using a CNN directly for feature extraction was
impractical. To overcome this, we adopted a Multiple Instance Learning (MIL)
strategy [29]. We cropped the WSI into multiple non-overlapped patches, each
measuring $224\times 224$ pixels. After removing the background patches, we
used ResNet-34 [30] for patch feature extraction. Specifically, we froze the
preceding layers, which were pre-trained on ImageNet [31], while fine-tuning
the last convolutional layer in the third residual block. Importantly, tumor
regions were notably large, making up over 80% of the full WSI in both the
TCGA-NSCLC and TCGA-RCC datasets. This allowed us to effectively aggregate
patch-level representations, generating WSI-level representation
$\mathbf{h}_{i}^{\text{(WSI)}}$ using appropriate pooling methods like
attention pooling [32], [33], which was commonly used in MIL.

CNV Feature Extractor: Initial preprocessing of the CNV data was performed
using GISTIC2 [34]. We then used the Self-Normalizing Network (SNN) [35]
comprised of two hidden layers to tackle the issues of high-dimensional
genomic data with limited samples [36]. The final fully connected layer was
responsible for learning the representation $\mathbf{h}_{i}^{\text{(CNV)}}$
[13]. The experiments involved three clients and one central server. The
training was executed on one Nvidia Tesla V100 GPU and three Nvidia Tesla P100
GPUs. We set global training rounds to 100 and the local epoch to 3, with all
clients participating in each round. The learning rate and batch size were set
to 0.001 and 32. $\beta$, $\alpha$, and $t_{0}$ were set to 0.25, 0.05 and 30.
We repeated the experiment 20 times, randomly splitting the dataset into
training and testing at 0.8:0.2.

###  4.3 Experimental Results and Analysis

We evaluated performance using both accuracy and ROC curves for binary
classification tasks, comparing FedMM against two baselines.

Local Training (Baseline 1): Each local model is trained independently,
without considering modality overlaps among clients.

Multi-FedAvg (Baseline 2): All modalities from all clients are considered.
Missing modalities on local clients are set to zero.

FedMM significantly outperforms two baselines across three hospitals, as shown
in Fig. 3. For example, for the TCGA-NSCLC dataset at hospital 2, FedMM
surpasses local training and Multi-FedAvg baselines by AUC values of 0.023
(2.79%$\uparrow$) and 0.065 (8.31%$\uparrow$), respectively. Similarly, for
the TCGA-RCC dataset at the same hospital, FedMM exceeds two baselines by AUC
values of 0.027 (2.97%$\uparrow$) and 0.069 (7.96%$\uparrow$). Notably, a
direct comparison of FedMM and Multi-FedAvg highlights FedMM’s effectiveness
in tackling the issue of modality heterogeneity. Finally, while FedMM
outperforms all baselines with similar training sample sizes (e.g., in
hundreds), collecting more data remains essential for distributing the data
across more clients and further validating FedMM.

(a) Accuracy (TCGA-NSCLC)

(b) ROC Curve (TCGA-NSCLC)

(c) Accuracy (TCGA-RCC)

(d) ROC Curve (TCGA-RCC)

Fig. 3: Comparison of the classification performance between FedMM and
baselines based on two public datasets.

(a) Loss Transition Point

(b) Modality Types

(c) Fusion Mechanisms

Fig. 4: Ablation study of FedMM on the TCGA-NSCLC dataset.

Ablation Study: To rigorously evaluate the impact of various components on the
performance of FedMM, we further conduct a comprehensive ablation study, which
investigates the transition point of the two losses, different modality
inputs, and various fusion mechanisms. We present results on the TCGA-NSCLC
dataset, as shown in Fig. 4, with similar results on the TCGA-RCC dataset.
Notably, centralized fusion is impractical in real-world situations due to
privacy constraints on aggregating data from all hospitals. However, the AUC
value achieved by FedMM closely approaches that of centralized fusion,
highlighting its practical effectiveness.

##  5 Conclusion

In conclusion, we propose a FedMM framework to train multiple single-modal
feature extractors federatedly to help improve subsequent classification
performance. Hospitals can leverage these federated trained extractors to
perform local feature extraction and classification tasks. Our comprehensive
evaluations show that FedMM notably outperforms two baseline methods, proving
its effectiveness in multimodal and privacy-sensitive computational pathology.

##  6 References

## References

  * [1] Rami S Vanguri et al. “Multimodal integration of radiology, pathology and genomics for prediction of response to PD-(L) 1 blockade in patients with non-small cell lung cancer” In _Nature cancer_ 3.10 Nature Publishing Group US New York, 2022, pp. 1151–1164
  * [2] Jana Lipkova et al. “Artificial intelligence for multimodal data integration in oncology” In _Cancer cell_ 40.10 Elsevier, 2022, pp. 1095–1110
  * [3] Ping Luo, Yuanyuan Li, Li-Ping Tian and Fang-Xiang Wu “Enhancing the prediction of disease–gene associations with multimodal deep learning” In _Bioinformatics_ 35.19 Oxford University Press, 2019, pp. 3735–3742
  * [4] Valsamo Anagnostou et al. “Multimodal genomic features predict outcome of immune checkpoint blockade in non-small-cell lung cancer” In _Nature cancer_ 1.1 Nature Publishing Group US New York, 2020, pp. 99–111
  * [5] Xingze Wang et al. “Lung cancer subtype diagnosis by fusing image-genomics data and hybrid deep networks” In _IEEE/ACM Transactions on Computational Biology and Bioinformatics_ IEEE, 2021
  * [6] Jianliang Gao et al. “MGNN: A multimodal graph neural network for predicting the survival of cancer patients” In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2020, pp. 1697–1700
  * [7] Dinh C Nguyen et al. “Federated learning for smart healthcare: A survey” In _ACM Computing Surveys (CSUR)_ 55.3 ACM New York, NY, 2022, pp. 1–37
  * [8] Brendan McMahan et al. “Communication-efficient learning of deep networks from decentralized data” In _Artificial intelligence and statistics_ , 2017, pp. 1273–1282 PMLR
  * [9] Mohammed Adnan et al. “Federated learning and differential privacy for medical image analysis” In _Scientific reports_ 12.1 Nature Publishing Group UK London, 2022, pp. 1953
  * [10] Liangqiong Qu et al. “Rethinking architecture design for tackling data heterogeneity in federated learning” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10061–10071
  * [11] Xuejun Qian et al. “Prospective assessment of breast cancer risk from multimodal multiview ultrasound images via clinically applicable deep learning” In _Nature biomedical engineering_ 5.6 Nature Publishing Group UK London, 2021, pp. 522–532
  * [12] Peter Peneder et al. “Multimodal analysis of cell-free DNA whole-genome sequencing for pediatric cancers with low mutational burden” In _Nature communications_ 12.1 Nature Publishing Group UK London, 2021, pp. 3230
  * [13] Richard J Chen et al. “Pan-cancer integrative histology-genomic analysis via multimodal deep learning” In _Cancer Cell_ 40.8 Elsevier, 2022, pp. 865–878
  * [14] Julián N Acosta, Guido J Falcone, Pranav Rajpurkar and Eric J Topol “Multimodal biomedical AI” In _Nature Medicine_ 28.9 Nature Publishing Group US New York, 2022, pp. 1773–1784
  * [15] Shivam Kalra et al. “Decentralized federated learning through proxy model sharing” In _Nature communications_ 14.1 Nature Publishing Group UK London, 2023, pp. 2899
  * [16] Jean Ogier du Terrail et al. “Federated learning for predicting histological response to neoadjuvant chemotherapy in triple-negative breast cancer” In _Nature medicine_ 29.1 Nature Publishing Group US New York, 2023, pp. 135–146
  * [17] Ming Y Lu et al. “Federated learning for computational pathology on gigapixel whole slide images” In _Medical image analysis_ 76 Elsevier, 2022, pp. 102298
  * [18] Xiaoxiao Li et al. “Multi-site fMRI analysis using privacy-preserving federated learning and domain adaptation: ABIDE results” In _Medical Image Analysis_ 65 Elsevier, 2020, pp. 101765
  * [19] Xiaoxiao Li et al. “Fedbn: Federated learning on non-iid features via local batch normalization” In _arXiv preprint arXiv:2102.07623_ , 2021
  * [20] Qinbin Li, Yiqun Diao, Quan Chen and Bingsheng He “Federated learning on non-iid data silos: An experimental study” In _2022 IEEE 38th International Conference on Data Engineering (ICDE)_ , 2022, pp. 965–978 IEEE
  * [21] Saeed Vahidian, Mahdi Morafah, Mubarak Shah and Bill Lin “Rethinking data heterogeneity in federated learning: Introducing a new notion and standard benchmarks” In _IEEE Transactions on Artificial Intelligence_ IEEE, 2023
  * [22] Qiying Yu et al. “Multimodal Federated Learning via Contrastive Representation Ensemble” In _arXiv preprint arXiv:2302.08888_ , 2023
  * [23] Jiayi Chen and Aidong Zhang “FedMSplit: Correlation-adaptive federated multi-task learning across multimodal split networks” In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2022, pp. 87–96
  * [24] Xiaomin Ouyang et al. “Harmony: Heterogeneous Multi-Modal Federated Learning through Disentangled Model Training” In _Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services_ , 2023, pp. 530–543
  * [25] Tian Li et al. “Federated optimization in heterogeneous networks” In _Proceedings of Machine learning and systems_ 2, 2020, pp. 429–450
  * [26] Yue Tan et al. “Fedproto: Federated prototype learning across heterogeneous clients” In _Proceedings of the AAAI Conference on Artificial Intelligence_ 36.8, 2022, pp. 8432–8440
  * [27] Wenke Huang et al. “Rethinking Federated Learning With Domain Shift: A Prototype View” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023, pp. 16312–16322
  * [28] Philip Chikontwe, Soopil Kim and Sang Hyun Park “CAD: Co-Adapting Discriminative Features for Improved Few-Shot Classification” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 14554–14563
  * [29] Marc-André Carbonneau, Veronika Cheplygina, Eric Granger and Ghyslain Gagnon “Multiple instance learning: A survey of problem characteristics and applications” In _Pattern Recognition_ 77 Elsevier, 2018, pp. 329–353
  * [30] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun “Deep Residual Learning for Image Recognition” In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [31] Jia Deng et al. “Imagenet: A large-scale hierarchical image database” In _2009 IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255 Ieee
  * [32] Maximilian Ilse, Jakub Tomczak and Max Welling “Attention-based deep multiple instance learning” In _International conference on machine learning_ , 2018, pp. 2127–2136 PMLR
  * [33] Yun Wang, Juncheng Li and Florian Metze “A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling” In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 31–35 IEEE
  * [34] Craig H Mermel et al. “GISTIC2. 0 facilitates sensitive and confident localization of the targets of focal somatic copy-number alteration in human cancers” In _Genome biology_ 12 Springer, 2011, pp. 1–14
  * [35] Günter Klambauer, Thomas Unterthiner, Andreas Mayr and Sepp Hochreiter “Self-normalizing neural networks” In _Advances in neural information processing systems_ 30, 2017
  * [36] Richard J Chen et al. “Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis” In _IEEE Transactions on Medical Imaging_ 41.4 IEEE, 2020, pp. 757–770