# EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression
Recognition

Niki Maria Foteinopoulou
Queen Mary University of London
London, UK
Ioannis Patras
Queen Mary University of London
London, UK
<EMAIL_ADDRESS>
###### Abstract

Facial Expression Recognition (FER) is a crucial task in affective computing,
but its conventional focus on the seven basic emotions limits its
applicability to the complex and expanding emotional spectrum. To address the
issue of new and unseen emotions present in dynamic in-the-wild FER, we
propose a novel vision-language model that utilises sample-level text
descriptions (i.e. captions of the context, expressions or emotional cues) as
natural language supervision, aiming to enhance the learning of rich latent
representations, for zero-shot classification. To test this, we evaluate using
zero-shot classification of the model trained on sample-level descriptions on
four popular dynamic FER datasets. Our findings show that this approach yields
significant improvements when compared to baseline methods. Specifically, for
zero-shot video FER, we outperform CLIP by over 10% in terms of Weighted
Average Recall and 5% in terms of Unweighted Average Recall on several
datasets. Furthermore, we evaluate the representations obtained from the
network trained using sample-level descriptions on the downstream task of
mental health symptom estimation, achieving performance comparable or superior
to state-of-the-art methods and strong agreement with human experts. Namely,
we achieve a Pearson’s Correlation Coefficient of up to 0.85 on schizophrenia
symptom severity estimation, which is comparable to human experts’ agreement.
The code is publicly available on: https://github.com/NickyFot/EmoCLIP.

##  1 Introduction

Facial Expression Recognition (FER) is a primary task of affective computing,
with several practical applications in Human-Computer Interaction [6],
education [39] and mental health [11], among others. To formalise the spectrum
of human emotions, researchers have proposed several models. Ekman & Friesen
[9] propose six emotions (later seven with the addition of “contempt”) as the
basis of human emotional expression. This is the most widely accepted model
for FER; however, human emotional experience is significantly more complex and
varied than the seven basic categories, with up to 27 distinct categories for
emotion reported in recent studies [7]. A continuous arousal-valence scale has
been proposed [28] as an alternative to creating emotion categories. However,
the scale is neither objective [12] nor self-explanatory to human readers.
Therefore, as more fine-grained definitions of emotion are proposed and the
categorical models expand, there is also a need for automated systems to
adjust to new definitions, unseen emotions and mental states or compound
emotions. Zero-shot methodologies in Facial Emotion Recognition (FER)
specifically tackle these challenges, providing effective solutions to unseen
emotions and mental states.

Zero-shot Learning (ZSL) is a machine learning paradigm where a model can
recognise and classify objects or concepts it has never been trained on by
leveraging auxiliary information or attributes associated with those unseen
classes [35]. In the context of emotion recognition, ZSL has traditionally
been achieved by some sort of label semantic embedding to produce emotion
prototypes, typically word2vec [43, 4]. The use of hard labels, however, is
failing to include semantically rich information in the prototypes as well as
ignoring the subtle differences in expression between subjects. Recent
developments in Vision-Language Models (VLM) [27, 13, 1], using image-caption
pairs instead of hard-labels have demonstrated superior generalisation and
zero-shot abilities. Furthermore, CLIP [27] has been used as the basis for
several methods in action recognition [42, 33] or video retrieval and
captioning [30, 21, 40, 20]. However, the use of VLMs in dynamic FER remains
relatively unexplored. Recent preliminary works in FER have used video-
language contrastive training [16, 17], which primarily rely on class-level
prompt learning. Nonetheless, this approach is akin to class supervision and
is not designed to generalise to unseen behaviours.

In this work, we propose a novel approach to zero-shot FER from video inputs
by jointly learning video and text embeddings, utilising the contrastive
learning framework with natural language supervision. The architecture of the
network is simple and trains a video and text encoder concurrently, as shown
in Fig. 1. The text and image encoders are initialised by leveraging the
knowledge of large-scale pre-trained CLIP [27], as typical dynamic FER
datasets do not have enough samples to train from scratch, and we train an
additional temporal module from scratch, similar to previous works in action
recognition [18, 33]. Contrary to previous VLM or ZSL works in emotion
recognition [4, 43, 16, 17] we use sample-level descriptions during training
i.e. captions of the subject’s facial expression and context, available in the
MAFW [19] dataset. These act as soft labels and aim to achieve more
semantically rich latent representations of the video samples. Then, during
inference, we use class-level descriptions for each emotion. Specifically, we
generate descriptions of each emotion in relation to the typical facial
expressions associated with it. In the case of compound emotions, we propose
manipulating the latent representation of the categories’ descriptions in the
embedding space rather than creating additional prompts. Specifically, as
compound emotions are combinations of basic emotions, we propose averaging the
embeddings of the components and adding them to the set of embeddings for each
additional compound emotion. We show that the proposed methodology trained on
sample-level descriptions shows generalisation capabilities invariant to
domain shift as we perform zero-shot evaluation on multiple datasets. We also
show that compared to the CLIP and FaRL baselines, the temporal information
and domain-specific knowledge of the sample-level descriptions improve the
zero-shot performance of FER. Finally, to show the generalisation ability of
the representations we obtain from the video encoder, we adapt them to the
domain of mental health. Using a simple MLP, trained in a fully supervised
manner, we achieve results comparable to or outperforming previous state-of-
the-art on estimating non-verbal symptoms of schizophrenia. We see a
significant improvement compared to previous works, particularly on symptoms
associated with affect and expression of emotions as well as total negative
score, which is similar to that of human experts.

Our main contributions can be summarised as follows:

  * •

Our paper introduces a novel paradigm for zero-shot Facial Emotion Recognition
(FER) from video input, employing sample-level descriptions and a dynamic
model. This approach, which leverages CLIP [27], outperforms class-level
descriptions and achieves significant improvements, especially for under-
represented emotions.

  * •

We propose a novel method for representing compound emotions by using average
latent representations of basic emotions, instead of concatenating prompts.
This approach is more intuitive and efficient than prompt engineering but also
shows significant improvements across all metrics compared to prompt
concatenation.

  * •

Our proposed method, EmoCLIP, trained on the MAFW [19] dataset and evaluated
on popular video FER datasets (AFEW [8], DFEW [14], FERV39K [34], and MAFW),
achieves state-of-the-art performance on ZSL. Additionally, we evaluate the
embeddings of the video encoder of EmoCLIP on the downstream task of
schizophrenia symptom estimation using the NESS dataset [25]. We achieve
results comparable to or better than previous state-of-the-art methods and
comparable to human experts with a simple 2-layer MLP.

The remainder of this paper is organised as follows. Section 2 discusses
related literature, Section 3 introduces our methodology, Section 4 reports
the results on the tasks of zero-shot FER for simple and compound emotions,
and the downstream task of schizophrenia symptom estimation, and Section 5
concludes the paper.

##  2 Related Work

In this section, we review previous works on Vision and Language Models (VLM),
and zero-shot learning in FER & Mental health.

###  2.1 Vision and Language Models

The use of large contrastive pre-training for Vision-Language Models (VLM) has
become increasingly popular, as these models have demonstrated impressive
generalisation capabilities [27, 13, 44, 1]. As large VLM require a large
number of data and very high computational resources to achieve those results,
the latest research on VLM has been mostly concentrated on three paths: (a)
latent space manipulation, (b) leveraging pre-trained spatial features and
fine-tuning a temporal module for video input, and (c) prompt learning.

Menon & Vondrick [22] use an ensemble of prompts generated by Large Language
Models (LLMs) containing descriptive features of each class and show
significant improvements in terms of accuracy as well as explainability of
decisions. Ouali  _et al_. [23] propose a method for latent space feature
alignment in a target domain without the need for additional training.
Similarly, Bain  _et al_. [3] propose several methods for temporal pooling of
frames, using pre-trained VLMs with very little or no additional training.
Such approaches, although effective, use no information from the temporal
dimension in video, which is essential in human FER to understand macro and
micro-expressions.

Large VLMs trained on static images have been used for video classification,
particularly for action recognition. Lin  _et al_. [18] propose using a
lightweight Transformer Decoder over the CLIP [27] spatial features, for
downstream classification. Similarly, ActionCLIP [33] class labels are used
for natural language supervision; therefore in both approaches, the open
vocabulary capabilities are lost. CLIP [27] has been used as a backbone in
several video captioning works [20, 38, 21]; however, none of these works has
been evaluated or trained on the domain of FER where the behaviour is in
general not as clearly defined.

Figure 1: Overview of our method, EmoCLIP. During training (a), we use joint
training to optimise the cosine similarity of video-text embedding pairs in
the mini-batch. Sample-specific descriptions of the subject’s facial
expressions are used to train the model. During inference (b), we perform
zero-shot classification using class-level descriptions for each of the
emotion categories.

To overcome the challenges of prompt engineering in VLMs, some works propose
learning a set of tokens [46, 45, 24] to append to the class name, which can
improve performance as the text-encoder acts more like a bag of words [41, 2].
Natural language supervision for facial expression recognition (FER) is a
relatively unexplored idea, but there have been some preliminary works
exploring this approach. For instance, CLIPER [16] has proposed prompt
learning to improve closed dictionary FER. However, these tokens are class-
specific and cannot be used in open dictionary settings for zero-shot
classification.

###  2.2 Zero-Shot Learning in Facial Expression Recognition

Several works [4, 26, 36, 37] have proposed zero-shot frameworks for emotion
recognition or emotional response recognition [43] by aligning class name
embeddings to multi-media embeddings and then evaluating the method on unseen
emotions. These methods, however, still rely on the use of hard labels and
simpler class prototype embeddings (such as word2vec). As such, they do not
take into consideration the intra-class differences or the underlying concepts
in each class and as such, do not capture semantically rich information in the
latent representations.

##  3 Methodology

An overview of the proposed method can be seen in Fig. 1. In a nutshell, we
follow the CLIP [27] contrastive training paradigm to jointly optimise a video
and a text encoder. The video and text encoders of the network are jointly
trained using a contrastive loss over the cosine similarities of the video-
text pairings in the mini-batch. More specifically, the video encoder
($E_{V}$) is composed of the CLIP image encoder ($E_{I}$) and a Transformer
Encoder to learn the temporal relationships of the frame spatial
representations. The text encoder ($E_{T}$) used in our approach is the CLIP
text encoder. The weights of the image and text encoders in our model are
initialised using the large pre-trained weights of CLIP [27] and finetuned on
the target domain, as FER datasets are not large enough to train a VLM from
scratch with adequate generalisation. Contrary to the previous video VLM works
in both action recognition [33, 18] and FER [16, 17], we propose using sample
level descriptions for better representation learning, rather than embeddings
of class prototypes. This leads to more semantically rich representations,
which in turn allows for better generalisation.

We describe how we train the proposed method for dynamic FER in Section 3.1,
and how we use it at inference time for simple and compound emotions in
Section 3.2

###  3.1 Architecture and Training

The CLIP framework, proposed by Radford  _et al_. [27], operates as a
contrastive multi-modal system, encoding image and text features into a shared
space and maximising the cosine similarity of matching image-text pairs
(positives) while minimising the similarity of all other pairs (negatives) by
optimising a cross-entropy loss over the similarity pairs. We adopt the
framework in the dynamic paradigm by introducing a transformer encoder over
the spatial features to learn from the temporal dimension.

More specifically, we utilise two separate encoders for processing video and
text inputs. Given a video-text pair $x=\\{x^{V},x^{T}\\}$, we obtain the
video-text embeddings using the respective encoders so that
$\textbf{z}^{V}=E_{V}(x^{V})$ and $\textbf{z}^{T}=E_{T}(x^{T})$, where
$\textbf{z}^{V},\textbf{z}^{T}\in\mathbb{R}^{D}$. The video and text
embeddings, $\textbf{z}^{V}$ and $\textbf{z}^{T}$, respectively, obtained for
each image-text pair in the mini-batch $B$, are utilised to generate a
$B\times B$ matrix of cosine similarities. The diagonal elements of the matrix
correspond to the $B$ positive pairings, while the remaining elements
represent $B^{2}-B$ negative pairings. A cross-entropy loss is employed to
maximise the similarity between the positives on the diagonal and minimise the
similarity of the negatives.

The video encoder architecture is relatively simple and similar to
architectures proposed for video captioning [21, 30]. Specifically, ($E_{V}$)
is composed of the CLIP image encoder ($E_{I}$) that extracts frame-level
features, which are then fed to a two-layer transformer encoder that acts as a
temporal encoder. The state of the learnable classification token at the
output of the transformer is used as the video embedding $\textbf{z}^{V}$.

While the encoders $E_{V},E_{T}$ could be trained from scratch given a
sufficiently large dataset of video-caption pairs, in the domain of FER, the
only available dataset with such annotations is the MAFW [19] dataset, which
is relatively small. To this end, we leverage the pre-trained CLIP [27] image
and text encoders to initialise the weights of $E_{I}$ and $E_{T}$ in our
architecture and fine-tune on the FER domain.

###  3.2 Inference

During inference, the cosine similarity of text and image embedding in the
joined latent space is used as the basis for the classification. The
prediction probability is then defined as:

$P(y=i|x)=\frac{e^{\langle\textbf{z}^{V},\textbf{z}^{T}_{i}\rangle/\tau}}{\sum_{j=1}^{N}e^{\langle\textbf{z}^{V},\textbf{z}^{T}_{j}\rangle/\tau}},$
(1)

where $\tau$ is a learnable temperature parameter in CLIP and
$\langle\cdot,\cdot\rangle$ is the cosine similarity.

In the case of basic emotions, we provide class descriptions in the form of
natural language obtained from LLMs, rather than using a prompt in the form of
‘an expression of {emotion}’, to match the information-rich sample level
descriptions. We note that these are different descriptions than the ones used
during training, as in the latter, we use sample-level video-text pairs, as
can be seen in Fig. 1. As such, our method is performing zero-shot
classification during inference. Specifically, to obtain the class-level
descriptions, we prompt ChatGPT with the input:

Q: What are the facial expressions associated with {emotion}?

Architecture | Temporal Layer | Prompt Ensemble | Frame Ensemble | Pre-training | UAR | WAR
---|---|---|---|---|---|---
CLIP [27] |  |  | ✓ | Laion-400m | 20.40 | 21.16
CLIP [27] |  | ✓ |  | Laion-400m | 19.77 | 18.64
CLIP [27] |  | ✓ | ✓ | Laion-400m | 19.46 | 17.61
CLIP [27] |  |  |  | Laion-400m | 16.97 | 21.69
EmoCLIP (Frozen backbone) | ✓ |  |  | MAFW | 23.6 | 31.36
EmoCLIP | ✓ |  |  | MAFW | 25.86 | 33.49
Table 1: Performance of the proposed method, EmoCLIP, on the MAFW [19] dataset
on 11-class single expression classification against the baseline, with
different frame aggregation and prompting strategies.

We then curate the generated responses to exclude irrelevant information, such
as body pose or emotional cues (such as “a sad expression” for helplessness).
For example, “A facial expression showing irritation and unrest, with a
wrinkled forehead, narrowed eyes, and tight lips or a frown” is the generated
description for “anger”. Examples of prompts can be seen in Fig. 1 & 2; the
full set of class descriptions used for inference are given in the
Supplementary Material. This strategy is proposed over the prompt templates
used in CLIP [27], as the prompt in the form of ‘an expression of {emotion}’
would imply a universal definition for each emotion, that the text encoder has
learnt. Additionally, while the CLIP prompts have shown impressive results on
clearly defined objects in images, emotions are significantly more vague and
open to interpretation, both in terms of expression and understanding.
Finally, the use of LLM, in this case, ChatGPT, over hand-crafted prompts is
proposed to avoid introducing the authors’ bias in the prompts.

Figure 2: EmoCLIP manipulates the latent space of basic emotions to create
representations for compound emotions. We take the average latent
representation of the components and concatenate it to the set of
representations for each new compound emotion.

Compound emotions are a complex combination of basic emotions, such as
“happily surprised” [7, 19], that are used to identify a wider range of human
facial expressions. As, by definition, compound emotions are combinations of
basic emotions, we propose a new approach to constructing the latent
$\textbf{z}^{T}$ representation. Instead of treating them as independent
emotional states and creating additional prompts, we use the pre-normalised
latent representations of the components to compose the new compound emotion
as shown in Fig. 2.

Formally, for any new compound emotion, we calculate its latent representation
$\textbf{z}_{n}^{T}$ as the average of the latent representations of its $C$
component emotions so that:

$\textbf{z}_{n}^{T}=\frac{1}{C}\sum_{c=0}^{C}\textbf{z}_{c}^{T}$ (2)

The resulting vector representations are then concatenated to the set of class
representations.

##  4 Experimental Results

In this section, we present the experimental set-up (Section 4.1), the
ablation study (Section 4.2), and present the experimental evaluation of the
proposed framework in the Zero-Shot paradigm (Section 4.3) as well as the
Downstream task of schizophrenia symptom estimation (Section 4.4).

###  4.1 Experimental Setup

Datasets: We train on the MAFW [19] dataset, as to our knowledge it is the
only FER dataset that includes sample-level descriptions. The dataset contains
$10k$ audio-video clips with categorical annotations for 11 emotions,
accompanied by short descriptions of facial expressions in two languages,
English and Mandarin Chinese. The dataset is divided into five folds for
evaluation.

We evaluate our method on three additional FER datasets on the 7 basic
emotions. The AFEW [8] dataset contains 1,809 clips from movies, divided into
three subsets. We train the method from scratch and report results on the
validation set. DFEW [14] is composed of 16,000 videos from movies, split into
five folds for cross-validation. FERV39K [34] is a large dataset with around
39,000 clips annotated by 30 annotators, and we report results on the test
set.

Finally, we evaluate the video encoder of EmoCLIP as trained on the MAFW
dataset, on the downstream task of schizophrenia symptom estimation using a
subset of the NESS[25] dataset, as described in [5, 11]. The subset includes
113 in-the-wild baseline clinical interviews from 69 patients and two symptom
scales, PANSS[15] and CAINS[10]. To ensure a fair comparison with previous
works, we used leave-one-patient-out cross-validation. The values of “Total
Negative” and “EXP - Total” in PANSS and CAINS scales respectively, were
scaled during training to match the range of individual symptoms.

Training Details: The CLIP image encoders used in all experiments have a
ViT-B/32 architecture. During training, we apply augmentation to the spatial
dimensions of the video inputs for all datasets. Specifically, we apply a
random horizontal flip to all frames in a sequence with a probability of 50%.
We also apply a random rotation with a range of 6°and random centre crop. In
the temporal dimension, we empirically trim all videos to $T=32$ number of
frames and downsample by a factor of 4 (which results in the entire video
being included for the majority of samples). As the NESS dataset contains
significantly longer sequences, during inference, we average the prediction of
all the clips as described in [11].

For all experiments, we use a Stochastic Gradient Descent optimiser (SGD) with
a learning rate of $10^{-3}$. We finetune the pre-trained parameters of the
CLIP backbone using a different learning rate of $10^{-6}$ for the image and
text encoders.

Mode | Architecture | Contrastive Pre-training | UAR | WAR
---|---|---|---|---
Supervised | C3D [19] | - | 31.17 | 42.25
Resnet18_LSTM [19] | - | 28.08 | 39.38
VIT_LSTM [19] | - | 32.67 | 45.56
C3D_LSTM [19] | - | 29.75 | 43.76
T-ESFL [19] | - | 33.28 | 48.18
EmoCLIP (LP) | - | 30.26 | 44.231
| EmoCLIP (Frozen backbone) | MAFW [class descriptions] | 34.24 | 41.46
Zero-shot | CLIP [27] | Laion-400m | 20.40 | 21.16.
FaRL - ViT-B/16 [44] | Laion Face-20M | 14.07 | 7.70
EmoCLIP | MAFW [sample descriptions] | 25.86 | 33.49
Table 2: Performance of the proposed method on the MAFW [19] dataset on
11-class single expression classification against other SOTA architectures in
a supervised and zero-shot setting.

###  4.2 Ablation Study

In order to examine the effect of different components of EmoCLIP, and
evaluate the prompting strategy used in this work, we perform several
experiments using baseline CLIP and our proposed method, EmoCLIP. More
specifically, as discussed in the Methodology Secion 3, during inference, we
use class descriptions generated with the help of ChatGPT; this is different
to the prompt ensemble of Radford  _et al_. [27]. Furthermore, as CLIP [27] is
a static model, we conduct zero-shot evaluation on the middle frame and by
averaging the latent representations of all frames, i.e. performing frame
ensemble. Finally, we compare the performance of EmoCLIP with a frozen
backbone to that of our proposed method. The results of the ablation study can
be seen in Tab. 1. As the prompt templates used in CLIP [27] are similar to
the format “A photo of class name”, they are more suitable for static images
rather than video. This is corroborated by the performance of the CLIP [27]
baseline using prompt ensemble vs our class descriptions. Additionally, as
emotional expression is not static, we also observe an increase in the
performance of the baseline with the use of frame ensembling. The proposed
architecture, including a temporal module, offers a large increase in
performance for both Weighted Average Recall (WAR) and Unweighted Average
Recall (UAR). By finetuning the backbone to the FER domain, we see a further
increase in performance by approximately 2% for each metric.

###  4.3 Zero-Shot Evaluation

To evaluate the effectiveness of the proposed method, we compare it with pre-
trained CLIP [27] and FaRL [44] models in a Zero-shot setting. As both of
these methodologies are trained on static images, we take the average of the
latent representations of all frames in a video to compute the video embedding
and use that to calculate the cosine similarity with the text embeddings. We
show the performance of our method against the CLIP and FaRL baselines, with a
frozen CLIP backbone and the finetuned image-text encoders on the 11 class
classification of MAFW [19] in Table 2. We note that even though FaRL [44] is
trained on a subset of the Laion dataset [29] filtered to include samples of
faces, the model trained on FaRL performs significantly worse than both the
CLIP baseline and our proposed method. We also note that the FaRL pre-trained
weights are only available for the ViT-B/16 architecture, which may explain
the difference in performance. Furthermore, while FaRL is trained with image-
text pairs of faces, these are not necessarily related to facial expression,
so we hypothesise that the FaRL embedding space is significantly more niche
than CLIP but not in a direction beneficial for zero-shot FER.

To further investigate the improvement of our method vs baseline CLIP, we use
PCA to reduce the high-dimensional latent image vectors to three dimensions
(which explain approximately 70% of the variance) and plot them in a 3D
scatter plot, as shown in Fig. 3. We see that by fine-tuning to the FER
domain, the categorical emotions form more distinct clusters than in CLIP,
which is also reflected in the classification performance of our method.

For reference, we also train our architecture with class-level descriptions,
and using an MLP head with two fully connected layers (Linear probe shown as
LP on the table) over the EmoCLIP video encoder, we show the performance
against several supervised architectures as reported in [19]. We see that the
architecture trained using the class descriptions outperforms previous methods
in terms of UAR and is comparable with others in terms of WAR. These results
indicate that the contrastive vision-language approach leads to more
semantically rich and discriminate latent representations even in a supervised
setting. The difference in the two metrics is somewhat expected, as FER
datasets are typically imbalanced.

Mode | Architecture | Repr. Avg | UAR | WAR | F1 | AUC
---|---|---|---|---|---|---
Supervised |  C3D [19] | - | 9.51 | 28.12 | 6.73 | 74.54
Resnet18_LSTM [19] | - | 6.93 | 26.6 | 5.56 | 68.86
VIT_LSTM [19] | - | 8.72 | 32.24 | 7.59 | 75.33
C3D_LSTM [19] | - | 7.34 | 28.19 | 5.67 | 65.65
T-ESFL [19] | - | 9.15 | 34.35 | 7.18 | 75.63
Zero-shot | Random | - | 2.38 | 7.72 | 0.34 | 50.00
CLIP [27] | ✗ | 4.72 | 5.25 | 2.44 | 51.89
CLIP [27] | ✓ | 4.14 | 5.35 | 2.46 | 53.07
FaRL [44] | ✗ | 3.03 | 4.66 | 2.16 | 51.01
FaRL [44] | ✓ | 4.00 | 5.75 | 2.56 | 51.10
| EmoCLIP | ✗ | 5.24 | 15.34 | 3.80 | 51.30
| EmoCLIP | ✓ | 6.58 | 18.53 | 4.78 | 52.59
Table 3: Zero-shot classification on the 43 compound expressions of the MAFW
[19] dataset. Supervised methods are included as a reference.

(a) CLIP Latent Image Representations

(b) EmoCLIP Latent Image Representations

Figure 3: Latent representation of the MAFW dataset using CLIP (a) and EmoCLIP
(b) for each emotion category.

Furthermore, we present experimental results on the classification of 43
compound emotions in the MAFW dataset in Table 3. We evaluate the performance
of our proposed method, EmoCLIP, against a baseline approach of using
concatenated prompts, as well as CLIP and FaRL baselines. Specifically, we
concatenate the class descriptions for each compound emotion and use this as
class prompt input. We demonstrate that EmoCLIP outperforms the baseline
approach for all metrics. Moreover, we note that in the 43 emotions
classification, both CLIP and FaRL perform significantly worse than EmoCLIP
and have performance comparable to random (where only the majority class is
predicted). We theorise this is due to the lack of temporal understanding of
the static models. Furthermore, without fine-tuning on the target domain, the
class descriptions are not discriminate enough for the zero-shot
classification of emotions, as previously discussed. However, the
representation average method on both baselines improves the performance of
the static models on the compound emotions task. We also report the results of
several supervised methods for reference, which perform significantly better
than zero-shot approaches as expected.

Finally, we evaluate the performance of our proposed method using sample-level
descriptions from MAFW [19] on four widely used video FER datasets and compare
it with the CLIP baseline as shown in Table 4. Additionally, in line with
previous works in zero-shot emotion classification [4, 36, 43, 26], we train
our architecture using class-level descriptions and evaluate using leave-one-
class-out (loco) cross-validation. We note that we cannot directly compare
with these architectures, as they involve either different modalities (e.g.
audio, pose) [4, 36, 26, 37] or a different task [43], we adopt however, their
experimental set-up using our architecture to show how natural language
supervision and semantically rich class descriptions can help improve zero-
shot FER performance.

|  |  |  | DFEW
---
(7 classes)
| AFEW
---
(7 classes)
| FERV39K
---
(7 classes)
| MAFW
---
(11 classes)
| Architecture | Training labels | UAR | WAR | UAR | WAR | UAR | WAR | UAR | WAR
Supervised | EmoCLIP | [class description] | 58.04 | 62.12 | 44.32 | 46.19 | 31.41 | 36.18 | 34.24 | 41.46
EmoCLIP (LP) | [class] | 50.29 | 62.09 | 33.74 | 38.85 | 30.58 | 43.54 | 30.26 | 44.21
Zero-shot |  CLIP [27] | [image caption] | 19.86 | 10.60 | 23.05 | 11.80 | 20.99 | 17.10 | 20.04 | 21
EmoCLIP (leave-one-class-out) | [class description] | 22.85 | 24.96 | 35.11 | 27.57 | 39.35 | 41.60 | 24.12 | 24.74
EmoCLIP | [video caption] | 36.76 | 46.27 | 36.13 | 39.90 | 26.73 | 35.30 | 25.86 | 33.49
Table 4: Evaluation of EmoCLIP using sample descriptions vs class-level
description as natural language supervision, on four video FER datasets.

We observe that the EmoCLIP trained on MAFW [19] sample-level descriptions
show impressive generalisation ability on all datasets that we evaluate.
Specifically, for AFEW [8], MAFW [19] and DFEW [14], we see that the EmoCLIP
model is outperforming both the loco experiment and the CLIP [27] baseline.
Furthermore, the generalisation of the method is resistant to domain shift
from unseen datasets, as we observe from the significant performance increase
between the CLIP [27] baseline and EmoCLIP. We note that for FERV39K [34], the
loco experiment has a higher performance than the sample-wise training.
However, it is very important to stress that the FERV39K [34] is significantly
larger than the base dataset (over 3x more samples); therefore, methods
trained on it would have an advantage, particularly as in the loco experiment,
there is no domain shift.

For reference and to provide context, we include the performance of the
architecture in a supervised setting on all four datasets. We want to point
out that the architecture trained using class descriptions is significantly
outperforming the linear probe architecture, particularly in terms of UAR,
showing again that natural language supervision can provide significant
advantages even in a fully supervised setting.

###  4.4 Downstream Task

| N3: Poor Rapport | N6: Lack of Spont. | N1: Blunted Affect | Total Negative Score
---|---|---|---|---
| MAE | RMSE | PCC | MAE | RMSE | PCC | MAE | RMSE | PCC | MAE | RMSE | PCC
Tron  _et al_. [31] | 0.98 | 1.31 | .20 | 1.37 | 1.69 | .13 | 0.90 | 1.28 | .37 | - | - | -
Tron  _et al_. [32] | 1.01 | 1.26 | .15 | 1.32 | 1.62 | .09 | 0.99 | 1.36 | .11 | - | - | -
SchiNet [5] | 0.85 | 1.20 | .27 | 1.25 | 1.51 | .25 | 0.84 | 1.18 | .42 | 3.30 | 4.17 | .29
Relational [11] | 0.87 | 1.16 | .78 | 0.74 | 0.95 | .47 | 0.64 | 0.87 | .56 | 2.80 | 3.78 | .71
EmoCLIP (LP) | 0.89 | 1.28 | .72 | 0.91 | 1.15 | .27 | 0.56 | 0.79 | .63 | 2.31 | 3.01 | .85
Table 5: Performance on the downstream task against other SoTA (PANSS-NEG). | Facial Exp. | Vocal Exp. | Expr. Gestures | Quant. of Speech | EXP-Total Score
---|---|---|---|---|---
| MAE | RMSE | PCC | MAE | RMSE | PCC | MAE | RMSE | PCC | MAE | RMSE | PCC | MAE | RMSE | PCC
Tron  _et al_. [31] | 0.80 | 1.03 | .37 | 0.87 | 1.23 | .23 | 0.85 | 1.19 | .36 | 1.09 | 1.43 | .27 | - | - | -
Tron  _et al_. [32] | 0.75 | 1.07 | .36 | 0.86 | 1.22 | .26 | 0.91 | 1.22 | .38 | 1.02 | 1.36 | .25 | - | - | -
SchiNet [5] | 0.66 | 0.93 | .46 | 0.77 | 1.10 | .27 | 0.90 | 1.15 | .36 | 0.98 | 1.30 | .30 | 2.67 | 3.34 | .45
Relational [11] | 0.56 | 0.72 | .75 | 0.65 | 0.89 | .71 | 0.71 | 0.89 | .76 | 0.60 | 0.82 | .54 | 1.88 | 2.60 | .77
EmoCLIP (LP) | 0.49 | 0.65 | .77 | 0.59 | 0.83 | .74 | 0.66 | 0.85 | .74 | 0.64 | 0.89 | .50 | 1.32 | 2.52 | .83
Table 6: Performance on the downstream task against other SoTA (CAINS-EXP).

We evaluate the representations obtained by the proposed method on the
downstream task of schizophrenia symptom estimation on two scales, namely the
PANSS [15] and CAINS [10] scales. As symptoms in both scales have ordinal
labels, we address the problem of symptom estimation as a multi-label
regression. We evaluate the proposed method in terms of Mean Absolute Error
(MAE), Root Mean Squared Error (RMSE) and Pearson’s Correlation Coefficient
(PCC) for a fair comparison with previous works in the literature. We train
the linear probe architecture, with the video encoder obtained through
contrastive pre-training on the MAFW [19] frozen, and updating only the MLP
weights. The results against previous state-of-the-art (SoTA) can be seen on
Tables 5 & 6 for the PANSS [15] and CAINS [10] scales respectively. The
proposed method, pre-trained contrastively on sample level descriptions,
outperforms or is comparable to previous methods on all symptoms, particularly
for “N1: Blunted Affect” and “Facial Expression” on the PANSS and CAINS
scales, which are by definition most related to FER.

As the NESS [25] dataset is annotated by multiple healthcare professionals, we
can compare the PCC achieved by the method to the annotators’ PCC which has a
mean value of 0.85 across all symptoms [5, 25]. The proposed method on the
downstream task achieves performance comparable to that of human experts on
the total scores of both scales.

##  5 Conclusion

In this work, we presented a novel contrastive pre-training paradigm for video
FER, trained on video-text pairs with sample-level descriptions without any
class-level information. We evaluate our method, EmoCLIP, on four popular FER
video datasets [8, 14, 34, 19] and test using zero-shot evaluation on the
basic emotions as well as compound emotions. Our method outperforms the CLIP
baseline by a large margin and shows impressive generalisation ability on
unseen datasets and emotions. To our knowledge, this is the first work to
train with sample-level descriptions for FER and to propose zero-shot
evaluation using semantically rich class descriptions in the domain. We also
evaluated the EmoCLIP video encoder features on schizophrenia symptom
estimation, outperforming previous state-of-the-art methods and achieving
performance comparable to human experts in terms of PCC.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.  Flamingo: a visual language model for few-shot learning.  In Advances in Neural Information Processing Systems, 2022.
  * [2] Piyush Bagad, Makarand Tapaswi, and Cees G. M. Snoek.  Test of Time: Instilling Video-Language Models with a Sense of Time.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 2023.
  * [3] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  A CLIP-Hitchhiker’s Guide to Long Video Retrieval, May 2022\.
  * [4] Abhishek Banerjee, Uttaran Bhattacharya, and Aniket Bera.  Learning Unseen Emotions from Gestures via Semantically-Conditioned Zero-Shot Perception with Adversarial Autoencoders.  Proceedings of the AAAI Conference on Artificial Intelligence, 36(1):3–10, June 2022.  Number: 1.
  * [5] Mina Bishay, Petar Palasek, Stefan Priebe, and Ioannis Patras.  Schinet: Automatic estimation of symptoms of schizophrenia from facial behaviour analysis.  IEEE Transactions on Affective Computing, 12(4):949–961, 2019.
  * [6] M Kalpana Chowdary, Tu N Nguyen, and D Jude Hemanth.  Deep learning-based facial emotion recognition for human–computer interaction applications.  Neural Computing and Applications, pages 1–18, 2021.
  * [7] Alan S. Cowen and Dacher Keltner.  Self-report captures 27 distinct categories of emotion bridged by continuous gradients.  Proceedings of the National Academy of Sciences, 114(38):E7900–E7909, 2017.
  * [8] Abhinav Dhall, Roland Goecke, Simon Lucey, Tom Gedeon, et al.  Collecting large, richly annotated facial-expression databases from movies.  IEEE multimedia, 19(3):34, 2012.
  * [9] Paul Ekman and Wallace V. Friesen.  Facial action coding system: Investigator’s guide.  Consulting Psychologists Press, 1978.
  * [10] Courtney Forbes, Jack J. Blanchard, Melanie Bennett, William P. Horan, Ann Kring, and Raquel Gur.  Initial development and preliminary validation of a new negative symptom measure: The Clinical Assessment Interview for Negative Symptoms (CAINS).  Schizophrenia Research, 124(1-3):36–42, Dec. 2010.
  * [11] Niki Maria Foteinopoulou and Ioannis Patras.  Learning from Label Relationships in Human Affect.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 80–89, Lisboa Portugal, Oct. 2022. ACM.
  * [12] Niki Maria Foteinopoulou, Christos Tzelepis, and Ioannis Patras.  Estimating continuous affect with label uncertainty.  In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 1–8, Nara, Japan, Sept. 2021. IEEE.
  * [13] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [14] Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu.  Dfew: A large-scale database for recognizing dynamic facial expressions in the wild.  In Proceedings of the 28th ACM international conference on multimedia, pages 2881–2889, 2020.
  * [15] S. R. Kay, A. Fiszbein, and L. A. Opler.  The Positive and Negative Syndrome Scale (PANSS) for Schizophrenia.  Schizophrenia Bulletin, 13(2):261–276, Jan. 1987.
  * [16] Hanting Li, Hongjing Niu, Zhaoqing Zhu, and Feng Zhao.  CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression Recognition, Feb. 2023.  arXiv:2303.00193 [cs].
  * [17] Yande Li, Mingjie Wang, Minglun Gong, Yonggang Lu, and Li Liu.  FER-former: Multi-modal Transformer for Facial Expression Recognition, Mar. 2023.  arXiv:2303.12997 [cs].
  * [18] Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Frozen CLIP Models are Efficient Video Learners.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, Lecture Notes in Computer Science, pages 388–404, Cham, 2022. Springer Nature Switzerland.
  * [19] Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan.  MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild.  In Proceedings of the 30th ACM International Conference on Multimedia, MM ’22, pages 24–32, New York, NY, USA, Oct. 2022. Association for Computing Machinery.
  * [20] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning.  Neurocomputing, 508:293–304, Oct. 2022.
  * [21] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji.  X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval.  In Proceedings of the 30th ACM International Conference on Multimedia, MM ’22, pages 638–647. Association for Computing Machinery, Oct. 2022.
  * [22] Sachit Menon and Carl Vondrick.  Visual classification via description from large language models.  ICLR, 2023.
  * [23] Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos.  Black Box Few-Shot Adaptation for Vision-Language models, Apr. 2023.
  * [24] Sarah Parisot, Yongxin Yang, and Steven McDonagh.  Learning to Name Classes for Vision and Language Models.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 2023.
  * [25] Stefan Priebe, Mark Savill, Til Wykes, RP Bentall, Ulrich Reininghaus, Christoph Lauber, Stephen Bremner, Sandra Eldridge, and Frank Röhricht.  Effectiveness of group body psychotherapy for negative symptoms of schizophrenia: multicentre randomised controlled trial.  The British Journal of Psychiatry, 209(1):54–61, 2016.
  * [26] Fan Qi, Xiaoshan Yang, and Changsheng Xu.  Zero-shot Video Emotion Recognition via Multimodal Protagonist-aware Transformer Network.  In Proceedings of the 29th ACM International Conference on Multimedia, MM ’21, pages 1074–1083, New York, NY, USA, Oct. 2021. Association for Computing Machinery.
  * [27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 18–24 Jul 2021.
  * [28] James A. Russell.  A circumplex model of affect.  Journal of Personality and Social Psychology, 39(6):1161–1178, 1980\.
  * [29] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs, Nov. 2021.  arXiv:2111.02114 [cs].
  * [30] Mingkang Tang, Zhanyu Wang, Zhenhua LIU, Fengyun Rao, Dian Li, and Xiu Li.  CLIP4Caption: CLIP for Video Caption.  In Proceedings of the 29th ACM International Conference on Multimedia, MM ’21, pages 4858–4862, New York, NY, USA, Oct. 2021. Association for Computing Machinery.
  * [31] Talia Tron, Abraham Peled, Alexander Grinsphoon, and Daphna Weinshall.  Automated facial expressions analysis in schizophrenia: A continuous dynamic approach.  In International Symposium on Pervasive Computing Paradigms for Mental Health, pages 72–81. Springer, 2015.
  * [32] Talia Tron, Abraham Peled, Alexander Grinsphoon, and Daphna Weinshall.  Facial expressions and flat affect in schizophrenia, automatic analysis from depth camera data.  In 2016 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), pages 220–223. IEEE, 2016.
  * [33] Mengmeng Wang, Jiazheng Xing, and Yong Liu.  ActionCLIP: A New Paradigm for Video Action Recognition, Sept. 2021.  arXiv:2109.08472 [cs].
  * [34] Yan Wang, Yixuan Sun, Yiwen Huang, Zhongying Liu, Shuyong Gao, Wei Zhang, Weifeng Ge, and Wenqiang Zhang.  Ferv39k: a large-scale multi-scene dataset for facial expression recognition in videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20922–20931, 2022.
  * [35] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata.  Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.  IEEE transactions on pattern analysis and machine intelligence, 41(9):2251–2265, 2018.
  * [36] Xinzhou Xu, Jun Deng, Nicholas Cummins, Zixing Zhang, Li Zhao, and Björn W. Schuller.  Exploring Zero-Shot Emotion Recognition in Speech Using Semantic-Embedding Prototypes.  24, 2022.  Conference Name: IEEE Transactions on Multimedia.
  * [37] Xinzhou Xu, Jun Deng, Zixing Zhang, Zhen Yang, and Björn W Schuller.  Zero-shot speech emotion recognition using generative learning with reconstructed prototypes.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [38] Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo.  CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment, Sept. 2022.  arXiv:2209.06430 [cs].
  * [39] Elaheh Yadegaridehkordi, Nurul Fazmidar Binti Mohd Noor, Mohamad Nizam Bin Ayub, Hannyzzura Binti Affal, and Nornazlita Binti Hussin.  Affective computing in education: A systematic review and future research.  Computers & Education, 142:103649, 2019.
  * [40] Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid.  Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning, Feb. 2023.  arXiv:2302.14115 [cs].
  * [41] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.  When and why Vision-Language Models behave like Bags-of-Words, and what to do about it?  In International Conference on Learning Representations, 2023\.
  * [42] Giacomo Zara, Subhankar Roy, Paolo Rota, and Elisa Ricci.  AutoLabel: CLIP-based framework for Open-set Video Domain Adaptation, Apr. 2023.  arXiv:2304.01110 [cs].
  * [43] Chi Zhan, Dongyu She, Sicheng Zhao, Ming-Ming Cheng, and Jufeng Yang.  Zero-shot emotion recognition via affective structural embedding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1151–1160, 2019.
  * [44] Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen.  General Facial Representation Learning in a Visual-Linguistic Manner.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18676–18688, June 2022.
  * [45] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [46] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  International Journal of Computer Vision (IJCV), 2022.


11institutetext: The Pennsylvania State University 11email:
<EMAIL_ADDRESS>22institutetext: Visa Research 22email:
<EMAIL_ADDRESS>
# Rethinking Personalized Federated Learning with Clustering-based Dynamic
Graph Propagation

Author Name  Jiaqi Wang This work was done when Jiaqi Wang interned at Visa.11
Yuzhong Chen 22 Yuhang Wu 22 Mahashweta Das 22 Hao Yang 22 Fenglong Ma 11

###### Abstract

Most existing personalized federated learning approaches are based on
intricate designs, which often require complex implementation and tuning. In
order to address this limitation, we propose a simple yet effective
personalized federated learning framework. Specifically, during each
communication round, we group clients into multiple clusters based on their
model training status and data distribution on the server side. We then
consider each cluster center as a node equipped with model parameters and
construct a graph that connects these nodes using weighted edges.
Additionally, we update the model parameters at each node by propagating
information across the entire graph. Subsequently, we design a precise
personalized model distribution strategy to allow clients to obtain the most
suitable model from the server side. We conduct experiments on three image
benchmark datasets and create synthetic structured datasets with three types
of typologies. Experimental results demonstrate the effectiveness of the
proposed FedCedar.

###### Keywords:

Federated learning, Model personalization and distribution

##  1 Introduction

Federated learning (FL) [18] enables different data holders to cooperatively
train machine learning models without sharing data. However, data
heterogeneity poses a significant challenge in FL. To tackle this issue,
personalized FL (PFL) has been proposed. Among the various research tracks in
PFL, local model personalization through parameter decoupling [21] and
clustering [5, 27] have been explored. Parameter decoupling-based PFL has
several drawbacks: (1) It demands intensive computational resources and is
complex to implement. (2) It encounters the challenge of dividing the private
or federated parameters [1]. In contrast, clustering-based approaches do not
have such additional requirements or limitations. However, existing
clustering-based approaches overlook the hidden relations between clusters and
may require additional designs to handle the intricate mechanism [2], which
complicates implementation and deployment on other frameworks. Consequently, a
challenging yet practical question arises: _Is it possible to design a simple
yet effective PFL framework that can address the data heterogeneity problem
while considering the capture of hidden relations across clients?_

To address this question, we encounter several non-trivial challenges: C1:
Hidden relation capturing. In real-world applications, there may exist a
physical topology among clients. Utilizing this topology effectively would
contribute to local model updating. However, clients are unable to share data
or access global topology information. Accurately capturing the hidden
relations among clients becomes a challenging task. C2: Clustering-based
knowledge sharing. In FL frameworks, a random subset of clients participates
in iterative model learning. Learning client-level hidden relations can be
inefficient. Also, propagating under-trained models among all clients can lead
to failure in FL model training. Clustering approaches allow similar clients
to cooperate, which helps avoid such issues. However, most research focuses
solely on gathering knowledge within clusters, disregarding information across
multiple clusters. Extracting appropriate knowledge to facilitate model
updates across all clusters becomes an urgent matter to address. C3: Fitted
model acquisition. Distributing the most fitting models back to clients as
their initialization in the next communication round is not trivial. Designing
an accurate model distribution strategy is crucial to maintain and inherit the
benefits of iterative training, maximizing the utilization of model updates
and effectively supporting model training in subsequent rounds. Moreover, the
proposed approach should be easy to implement without requiring specific or
complicated tuning.

To address all the aforementioned challenges, we propose a simple yet
effective personalized federated learning framework, denoted as FedCedar. The
framework is shown in Figure 1. In our approach, we first perform local model
training using the respective client’s data and upload the model parameters to
the server. On the server side, we cluster the collected models into different
groups based on their parameters. To enhance our framework, we propose a
method incorporating dynamic graph construction and weighted knowledge
propagation. Each cluster center is treated as a node with associated model
parameters, and a graph is constructed connecting these nodes with weighted
edges. Subsequently, the model parameters stored at each node are updated
across the entire graph by leveraging information from other nodes through the
weighted edges. Throughout the federated learning iterations, the active
clients may vary, causing the clusters and graphs to change dynamically in
each communication round. Finally, we design a simple yet precise personalized
model distribution strategy to maintain the benefits of iterative training and
further enhance model personalization. Depending on whether an active client
was selected in the previous update iteration, we assign either the
personalized cluster center models or the aggregated ones to the clients.

##  2 Related Work

FL is initially proposed in [18], which proposes a classical algorithm named
FedAvg. Till this research, FL has recently been explored in multiple
directions[16, 24, 20, 17, 10, 23, 12, 22, 11, 13]. Considering our proposed
work, we discuss related works of model personalization and graph in FL in the
following. Personalized Federated Learning: Personalized FL cares more about
each local model’s performance, which is more sufficient and practical under
the non-IID setting. One research track to solve this problem is named
FedAvg+, which focuses on how to balance the global aggregation and local
training[9, 4, 21]. Besides the research track that we discussed, some
research works focus on parameter decoupling [21] and clustering [5, 27] to
conduct local model personalization in FL. However, all discussed
personalization FL works ignore the hidden relations between the clients or
the clusters. Graph in Federated Learning: Most research works, which get the
graph involved in FL, focus on training GNN (Graph Neural Network) models with
graph structure data distributedly[26, 6, 28, 7, 15]. In a recent work [3],
authors treat each local client as a node to conduct GCN learning at the
server. But the number of models maintained at the server is the same as the
client number, which would take intensive computation resources. Besides that,
the adjacency matrix is required to be known or learned in their proposed
work, which affects its practicability.

##  3 Methodology

###  3.1 Model Overview

Figure 1: Overview of the proposed FedCedar ($K=4$ as an example). Note that
clients selected at time $t$ may be different from those that are selected at
time $t+1$. Condition 1 means that the clients are selected at time $t$ as
well as $(t+1)$, and Condition 2 means that the clients are not selected at
time $t$.

Figure 1 shows the overview of the proposed framework, which consists of two
main modules, i.e., _local update_ and _server update_. In the local update
step, we first train a local model $\mathbf{w}_{n}$ with local data
$\mathcal{D}_{n}$, where
$\mathcal{D}_{n}=\\{(x_{i}^{n},y_{i}^{n})\\}_{i=1}^{H_{n}}$ and $H_{n}$ is the
number of data examples at client $n$. Specifically, we use the cross entropy
(CE) loss to train $\mathbf{w}_{n}$ as follows: $\mathcal{L}_{n}={\rm
CE}\left(f\left(\mathbf{x}^{n};\mathbf{w}_{n}\right),\mathbf{y}^{n}\right)$,
where $f(\cdot;\cdot)$ represents the neural network, $\mathbf{x}^{n}$
represents the client data representations, and $\mathbf{y}^{n}$ is the
corresponding label vector. Following the training procedure of general
federated learning models such as FedAvg [18], a small subset of clients with
size $B$ will be randomly selected to conduct the local model learning at the
$t$-th communication round. The outputs from the local client update are a set
of trained model parameters
$\\{\mathbf{w}_{t,1},\mathbf{w}_{t,2},\cdots,\mathbf{w}_{t,B}\\}$, which will
be uploaded to the server for the server update.

There are four critical components including _client model clustering_ ,
_dynamic graph construction_ , _knowledge propagation_ , and _precise
personalized model distribution_. After receiving
$\\{\mathbf{w}_{t,1},\mathbf{w}_{t,2},\cdots,\mathbf{w}_{t,B}\\}$, FedCedar
will conduct the client model clustering using the K-means algorithm, i.e.,
dividing the $B$ uploaded clients into $K$ clusters, where $K$ is the
predefined number of clusters. The outputs of this step are a set of cluster
centers
$\mathcal{V}_{t}=\\{\mathbf{w}_{s,t}^{2},\mathbf{w}_{s,t}^{1},\cdots,\mathbf{w}_{s,t}^{K}\\}$,
where $\mathbf{w}_{s,t}^{k}$ denotes the center of the $k$-th cluster at the
$t$-th communication round. In the dynamic graph construction step, FedCedar
treats each cluster center as a node and constructs a dynamic weighted graph
$\mathcal{G}_{t}=(\mathcal{V}_{t},\mathcal{E}_{t},\mathcal{R}_{t})$, where
$\mathcal{E}_{t}\subseteq\mathcal{V}_{t}\times\mathcal{V}_{t}$ is the set of
edges between nodes, and $\mathcal{R}_{t}$ is the set of weights on edges.
FedCedar then executes the knowledge propagation step across the whole graph
$\mathcal{G}_{t}$ to update node representations, i.e., model personalization
learning. Finally, FedCedar distributes the learned personalized models
$\\{\mathbf{\hat{w}}_{s,1}^{1},\mathbf{\hat{w}}_{s,1}^{2},\cdots,\mathbf{\hat{w}}_{s,t}^{K}\\}$
back to the corresponding clients with the precise model distribution
strategy. Next, we describe the details of the four components in the server
update step.

###  3.2 Client Model Clustering

At the $t$-th communication round, the $B$ selected clients will upload their
models $\\{\mathbf{w}_{t,1},\cdots,\mathbf{w}_{t,B}\\}$ to the server. These
models will be grouped into $K$ clusters using K-means [14][5] by optimizing
the following loss function:

$J_{t}=\sum_{k=1}^{K}\sum_{b=1}^{B}\delta_{t,b}^{k}(||\mathbf{w}_{s,t}^{k}-\mathbf{w}_{t,b}||^{2}),$
(1)

where $||\mathbf{w}_{s,t}^{k}-\mathbf{w}_{t,b}||^{2}$ is the Euclidean
distance between the $k$-th center $\mathbf{w}_{s,t}^{k}$ and the client model
$\mathbf{w}_{t,b}$. $\delta_{t,b}^{k}$ is the indicator. If the $b$-th client
belongs to the $k$-th cluster, then $\delta_{t,b}^{k}=1$. Otherwise,
$\delta_{t,b}^{k}=0$. Note that for each communication round $t$, we will
generate a new set of $K$ cluster centers
$\\{\mathbf{w}_{s,t}^{1},\cdots,\mathbf{w}_{s,t}^{K}\\}$.

###  3.3 Dynamic Weighted Graph Construction

The clustering process helps local clients with similar data distributions or
characteristics to aggregate together based on their model parameters.
However, simply utilizing the results of the clustering algorithm to conduct
the personalization in FL [27, 5] has several limitations. On the one hand,
the cluster centers $\\{\mathbf{w}_{s,t}^{1},\cdots,\mathbf{w}_{s,t}^{K}\\}$
are obtained by averaging the client parameters within each cluster. In other
words, the personalization information is only from the clients within each
cluster by modeling the _inner-cluster_ characteristics, which ignores that
_between clusters_. On the other hand, the performance of existing clustering
approaches is mainly determined by the quality of the cluster assignment. One
low-quality cluster may lead to slow convergence and poor overall performance.
When modeling the relations between clusters, high-quality information can
flow to other clusters, which may neutralize the negative effect of low-
quality clusters. Thus, it is essential to model the hidden relations between
clusters.

Towards this end, we build a dynamic weighted graph to capture the hidden
relations between the local models to enhance the mode updates further.
However, it is highly non-trivial to design a novel mechanism satisfying the
following conditions and requirements. First, in the FL setting, it would be
better not to increase the network transmission load due to the constraints of
communication cost. Second, in a typical FL framework, we randomly sample
clients at each communication round, which means different batches of clients
contribute to the updates. Then the center model $\mathbf{w}_{s,t}^{k}$ in
each cluster carries different information with huge differences. How to
leverage the iterative heterogeneous knowledge appropriately is a challenge.
FedAvg-based aggregation approach [18] enables rough information sharing by
obtaining one global model, which cannot handle the data or model
heterogeneity. Thus, solving such challenges requires a more reliable and
convincing aggregation strategy to maintain the personalization property for
each local client.

Specifically, we use
$\mathcal{G}_{t}=(\mathcal{V}_{t},\mathcal{E}_{t},\mathcal{R}_{t})$ to denote
a graph, where
$\mathcal{V}_{t}=\\{\mathbf{w}_{s,t}^{1},\cdots,\mathbf{w}_{s,t}^{K}\\}$ is
the node set, and each node corresponds to a cluster.
$\mathcal{E}_{t}\subseteq\mathcal{V}_{t}\times\mathcal{V}_{t}$ is the set of
edges, where any pair of nodes exists an edge with a corresponding weight.
$\mathcal{R}_{t}$ is the set of edge weights. The weight is defined as the
normalized cosine similarity between two cluster centers
$\mathbf{w}_{s,t}^{i}$ and $\mathbf{w}_{s,t}^{j}$, where
$\sum_{j=1}^{K}r^{i,j}_{t}=1$.

###  3.4 Knowledge Propagation and Aggregation

After we build the graph with the weighted edges, we are able to reveal and
describe the hidden relations between cluster centers corresponding to a set
of model parameters. The next question is how we could utilize the connection
to help the model update across the graph appropriately. To answer this
question, we design an effective approach to enable the model parameters saved
at each node to gather information from other nodes for knowledge propagation
across the whole graph.

Mathematically, given a constructed graph
$\mathcal{G}_{t}=(\mathcal{V}_{t},\mathcal{E}_{t},\mathcal{R}_{t})$, for
$\forall k\in K$, we update the center in a weighted sum way as follows:

$\begin{split}\mathbf{\hat{{w}}}_{s,t}^{k}&=g([\mathbf{w}_{s,t}^{1,P-1},\cdots,\mathbf{w}_{s,t}^{K,P-1}],[r_{t}^{k,1},\cdots,r_{t}^{k,K}],P)=\sum_{i=1}^{K}r_{t}^{k,i}\mathbf{w}_{s,t}^{i,P-1}\\\
&=\sum_{i=1}^{K}r_{t}^{k,i}\sum_{j=1}^{K}r_{t}^{i,j}\mathbf{w}_{s,t}^{j,P-2}=\cdots=\underbrace{\sum_{i=1}^{K}r_{t}^{k,i}\sum_{j=1}^{K}r_{t}^{i,j}\cdots\sum_{z=1}^{K}r_{t}^{u,z}\mathbf{w}_{s,t}^{z,0}}_{k\leftarrow
i\leftarrow j\leftarrow\cdots\leftarrow u\leftarrow z(\text{\\# path =
$P$})},\end{split}$ (2)

where $g$ is the function to conduct the knowledge propagation with the
updated model parameters, and $P$ denotes the times we repeat the propagation
process. $\mathbf{w}_{s,t}^{i,P-1}$ is the updated model parameter of node $i$
after $P-1$ rounds of propagation and aggregation, $r_{k,i}$ is the weight
between node $k$ and $i$. When $P=1$,
$\mathbf{w}_{s,t}^{z,0}=\mathbf{w}_{s,t}^{z}$, which is the output from
Section 3.3. After this procedure, we have a set of new models named
$\\{\mathbf{\hat{w}}_{s,t}^{1},\mathbf{\hat{w}}_{s,t}^{2},\cdots,\mathbf{\hat{w}}_{s,t}^{K}\\}$
with the weighted information from in-cluster clients and out-of-cluster
nodes.

###  3.5 Precise Personalized Model Distribution

After we get $K$ customized models at the server side, it is challenging to
select and distribute the most fitted model to each individual active client
to achieve the optimal local personalization performance at the next
communication round. There are two reasons: (1) To simulate the real-world
scenario, we randomly sample clients at each communication round with the
sample ratio $\gamma$. In this case, each sampled client set could vary a lot.
(2) We still expect to maintain the model generalization and personalization
with the benefit of the iterative training mechanism in FL. To solve the above
challenges, after obtaining
$\\{\mathbf{\hat{w}}_{s,t}^{1},\mathbf{\hat{w}}_{s,t}^{2},\cdots,\mathbf{\hat{w}}_{s,t}^{K}\\}$,
we design a precise personalized model distribution strategy.

Specifically, we need to consider two conditions with respect to the client
sampling at two consecutive communication rounds $t-1$ and $t$.
Correspondingly, there are sampled client set named $\mathcal{C}_{t-1}$ and
$\mathcal{C}_{t}$ at time $t-1$ and $t$, respectively. Given
$\mathcal{C}_{t-1}$ and $\mathcal{C}_{t}$, for client $n$ at communication
round $t$, different model distribution strategies are implemented under the
following two conditions:

  * •

Condition 1: if $n\in\mathcal{C}^{*}=\mathcal{C}_{t-1}\cap\mathcal{C}_{t}$
(_intersection set_) at time $t$, we trace back to the cluster $k$, to which
client $n$ belongs at time $t-1$. Then we distribute model
$\mathbf{\hat{w}}_{s,t-1}^{k}$ to client $n$ as the model initialization at
time $t$;

  * •

Condition 2: if $n\in\mathcal{C}^{\prime}=\mathcal{C}_{t}-\mathcal{C}_{t-1}$
(_difference set_) at time $t-1$, we distribute
$\langle\mathbf{\hat{w}}_{s,t-1}\rangle$ to client $n$ as the initial model
parameters, where $\langle\mathbf{\hat{w}}_{s,t-1}\rangle$ is calcualted as
follows:
$\langle\mathbf{\hat{w}}_{s,t-1}\rangle=\frac{1}{K}(\mathbf{\hat{w}}_{s,t-1}^{1}+\mathbf{\hat{w}}_{s,t-1}^{2}+\cdots+\mathbf{\hat{w}}_{s,t-1}^{K})$.

##  4 Experiment

###  4.1 Experiment Setup

####  4.1.1 Dataset Preparation

We conduct experiments for the image classification task on MNIST, SVHN, and
CIFAR-10 datasets in both IID and non-IID data distribution settings,
respectively. We split the training datasets into $80\%$ for training and
$20\%$ for testing. For the IID setting, the training and testing datasets are
both randomly sampled. For the non-IID setting, we divide the training dataset
following the approach used in [3] and set the shard = 2, which is an extreme
non-IID setting. To test the personalization effectiveness, we sample the
testing dataset following the label distribution as the training dataset.

####  4.1.2 Baselines

The proposed FedCedar is a personalized federated learning algorithm. To
achieve personalization, we adopt the clustering technique and construct
dynamically weighted graphs. To fairly evaluate the proposed FedCedar, we use
the following baselines: _(1) Classical FL Models_ : FedAvg [18] and FedProx
[9]; _(2) Personalized FL Models_ : pFedMe [21] and pFedBayes [29]; _(3)
Graph-based FL Model_ : SFL [3]; _(4) Clustering-based FL Models_ : IFCA [5]
and FedSem[27].

Table 1: Performance comparison with baselines.

Category | Dataset | MNIST | SVHN | CIFAR-10
---|---|---|---|---
Setting | IID | non-IID | IID | non-IID | IID | non-IID
Classical | FedAvg | 96.77% | 91.02% | 82.65% | 81.04% | 68.93% | 59.85%
FedProx | 97.87% | 93.98% | 83.09% | 82.68% | 71.07% | 64.07%
Personalization | Per-FedAvg | 96.57% | 93.56% | 87.74% | 87.09% | 71.32% | 75.56%
pFedMe | 96.90% | 93.10% | 87.79% | 86.37% | 73.92% | 77.21%
pFedBayes | 97.23% | 92.08% | 90.69% | 88.03% | 80.45% | 79.05%
Graph | SFL | 96.88% | 93.10% | 86.97% | 86.15% | 79.92% | 75.44%
Cluster | IFCA | 97.96% | 92.09% | 84.44% | 83.26% | 79.21% | 73.08%
FedSem | 96.52% | 93.05% | 84.75% | 84.96% | 79.87% | 71.22%
Ours | FedCedar | 97.90% | 95.96% | 91.19% | 88.76% | 82.80% | 81.53%

####  4.1.3 Implementation Details

In our experiments, we leverage convolutional neural networks (CNNs) as our
basic models for the three image datasets. For the MNIST and SVHN datasets,
the network structure consists of three convolutional layers and two fully-
connected layers. For the CIFAR-10 dataset, we have six convolutional layers
and two fully-connected layers. All the baselines and FedCedar use the same
networks to keep the comparison fair. Following [25], the total client number
is $N=100$, and the active client ratio in each communication round is 30%.
The total communication round between the server and local clients is $T=100$.
The local training batch size is 16, the local training epoch is 5, and the
local training learning rate is 0.01. The number of knowledge propagation $P$
is set to 2. The cluster number $K$ is set to 5. We provide the hyperparameter
study to explore how the key hyperparameters affect the performance of the
proposed FedCedar in the subsection 4.5.

###  4.2 Performance Evaluation

We run each approach three times and report the average accuracy in Table 1.
There are several observations and discussions as below: (1) Overall, our
proposed FedCedar outperforms baselines on MNIST, SVHN, and CIFAR-10 datasets
under both IID and non-IID settings except for the result of IFCA on MNIST
under the IID setting. This is due to the fact that the MNIST dataset under
the IID setting is a relatively easy task where all approaches achieve
comparable performance, compared with other datasets and settings; (2) If we
focus on SVHN and CIFAR-10, we find that the advantage of our algorithm is
becoming dominant, with the dataset being more complicated under the non-IID
setting. Compared with FedAvg under the non-IID setting, our approach
increases the accuracy rate by 9.53% and 36.22%, respectively. Compared with
the best performance of other algorithms on SVHN (88.03%) and CIFAR-10
(79.05%) under the non-IID setting, our algorithm boosts the performance to
88.76% ($\uparrow 0.73\%$) and 81.53%($\uparrow 2.48\%$), respectively; (3)
Among all baselines, pFedBayes performs the best on both SVHN and CIFAR-10
datasets. pFedBayes is specifically tailored for personalization. However, it
requires the exchange of data distribution information between the clients and
the server, which raises concerns about communication costs and privacy
leakage.

###  4.3 Ablation Study

In this subsection, we conduct the ablation study to investigate the
contribution of key modules in our proposed model FedCedar. We use the
following reduced or modified models as baselines: AS-1: _Without clustering_.
We treat each client as a node to build the graph and conduct the knowledge
propagation. Compared with [3], we do not have the complicated GCN learning
process or the optimal global model but distribute its own model via our
precise model distribution strategy. AS-2: _Without building a graph or
knowledge propagation_. We only conduct client clustering, and other modules
are kept. Different from [5], we only conduct one clustering process rather
than an iterative optimization to estimate the cluster identities. AS-3:
_Wthout precise personalized model distribution_. After conducting dynamic
graph construction and knowledge propagation, we aggregate the models from
each node into one single model and distribute it back to the clients.

Table 2: Ablation study.

Dataset | MNIST | SVHN | CIFAR-10
---|---|---|---
Setting | IID | non-IID | IID | non-IID | IID | non-IID
AS-1 | 96.79% | 92.16% | 82.66% | 82.11% | 70.10% | 72.96%
AS-2 | 96.80% | 92.53% | 82.71% | 81.39% | 70.45% | 69.15%
AS-3 | 97.04% | 92.01% | 83.66% | 81.05% | 71.08% | 65.37%
FedCedar | 97.90% | 95.96% | 91.19% | 88.76% | 82.80% | 81.53%

Table 2 shows the experimental results, where we can observe that reducing one
or more key modules in FedCedar leads to performance degradation. There are
several observations and discussions as below: (1) The results compared with
AS-1 show that treating each client as a node in the graph may not be able to
extract enough supporting knowledge from other clients. It also shows that the
help of similar clients within one cluster can enhance the model performance;
(2) In AS-2, we construct the clusters and maintain the precise personalized
distribution strategy. The reduction of the performance can tell the
effectiveness of utilizing the hidden relations between each cluster via graph
construction and knowledge propagation. It reveals that the extra weighted
knowledge is able to lift the model performance; (3)The performance of AS-3 is
the worst compared with AS-1 and AS-2. If we directly aggregate all the models
at the nodes into one global model via doing an average, it essentially can be
simplified as FedAvg.

###  4.4 Case Study

To further explore the effectiveness of the proposed FedCedar, we design a
case study to demonstrate the process of the graph formulation with respect to
the communication round. We are interested in testing if our model is able to
capture and reconstruct the hidden relations between local clients through
iterative updates in FedCedar framework.

####  4.4.1 Structured Data Construction

To achieve our target, we structure a synthetic dataset via MNIST by building
known relations between the nodes shown in Figure 2, and the goal is to test
whether our graph construction method has the capability of identifying these
pre-designed relations. For each node in the graph, we have 20 clients
equipped with designed label distributions. If there are common labels among a
pair of nodes, an edge connects them. For example, in Topology 1, label 2
(denoted as L2) is the common label between node 1 and node 2, so there is an
edge connecting the two nodes. In this case study, we construct three
topologies with 3 nodes, 4 nodes, and 5 nodes, respectively.

Figure 2: Structured datasets with synthetic topologies.

####  4.4.2 Periodic Activation Strategy

For each node in the graph, we have 20 clients, where each client has the
corresponding labels from the node as we discussed. Given a periodic round set
$\mathcal{Q}$, if $t\in\mathcal{Q}$, we sample the clients from each node at
the activation ratio $\gamma$ to formulate the active client set, which
guarantees the formulated client set covers the label information from all the
structured nodes in the graph. At other communication rounds
$t\not\in\mathcal{Q}$, we randomly sample $B$ clients from $N$ clients without
considering where the clients come from, where $B=N*\gamma$. Thus, the total
number of active clients is defined in this simulation experiment as follows:
$\text{if $t\in\mathcal{Q}$: }\mathcal{C}_{t}=\sum_{i=1}^{K}(S_{i}*\gamma)$;
$\text{if $t\not\in\mathcal{Q}$: }\mathcal{C}_{t}=B$, where $S_{i}$ is the
client set at node $i$. Later, we track the graph construction performance at
the round set $\mathcal{Q}$ through the communication rounds compared with the
synthetic topology structure.

####  4.4.3 Evaluation Metrics

We leverage the random index [19] as the evaluation metric to represent the
performance of the constructed clusters across the iterative update process.
It measures the similarity of the two clustering outcomes by considering all
pairs of samples: it counts pairs that are assigned in the same or different
clusters in the generated and ground-truth clusters [8]. Specifically in our
case, given a set
$\\{\mathbf{\hat{w}}_{s,t}^{1},\mathbf{\hat{w}}_{s,t}^{2},\cdots,\mathbf{\hat{w}}_{s,t}^{K}\\}$,
assuming we have two partitions
$\mathcal{A}=\\{\mathit{A}_{1},\mathit{A}_{2},...,\mathit{A}_{a}\\}$ and
$\mathcal{B}=\\{\mathit{B}_{1},\mathit{B}_{2},...,\mathit{B}_{b}\\}$, where
there are $a$ and $b$ subsets in each partition respectively, we have:
$\mathbf{R}=(\alpha+\beta)/\binom{K}{2}=2\times(\alpha+\beta)/K(K-1)$, where
$\alpha$ and $\beta$ are the number of agreements between the two partitions
that the pairs belong to the same subsets or different subsets. Intuitively,
it tells the occurrence or the probability that the partitions agree on the
data sample pairs. Thus, the score $\mathbf{R}=1$ means the perfect
clustering, and $\mathbf{R}=0$ means poor clustering.

####  4.4.4 Results Analysis

Figure 3: Random index.

We run the case study with different cluster numbers $K=3,4$, and $5$. We
choose to activate the targeted clients from the designed topology every 5
communication rounds, which means we have 40 data points during the total 200
communication rounds ($\mathcal{Q}=\\{5,10,\cdots,200\\}$). To have a better
visualization, we plot the data with the scatter every 2 data points, which
means we totally have 20 markers along with each line.

The results are shown in Figure 3. In the figure, the black, blue, and red
demonstrate the results with the number of clusters equal to 3, 4, and 5,
respectively. When $K=3$, which is Topology 1, we can observe that there is
very limited fluctuation at the first initial communication rounds. The lowest
score is 0.73 and the score converges to 1 after several updates. For $K=4$,
the convergence speed is slower than the setting when $K=3$, but it is close
to 1 finally. Besides that, when $K=5$, its convergence speed and score are
not as good as when $K=3$ or $4$. There are several possible reasons: (1) the
complexity of the topology will increase the difficulty of generating the
correct clusters and graphs; (2) with more nodes in our setting, there are
more labels and clients involved in the updates, which increases the
heterogeneity of the datasets. Generally, we observe that the random index
score under three settings all reach convergence and converge to a high value,
demonstrating the effectiveness and interpretability of our proposed
algorithm.

###  4.5 Hyperparameter Study

In this subsection, we investigate how the hyperparameters $P$ and $K$ affect
the performance of FedCedar. $P$ controls how many times we conduct knowledge
propagation across the graph at each communication round in Eq. (2). $K$
controls the number of clusters when we group the clients by optimizing Eq.
(1). To explore the parameter sensitivity, we alter $P$ and $K$ as
$\\{1,2,3,4,5\\}$ and $\\{3,4,5,6,7\\}$, respectively.

Figure 4: Hyperparameter study.

We report the three-time average experiment results in Figure 4. For the
experiment results on SVHN, we observe: (1) The highest accuracy is $88.83\%$
appearing at $\\{P=2,K=4\\}$. (2) In general, with the increase of $P$, the
accuracy shows non-monotonous behavior by firstly going up and then going
down. For example, given $K=4,5,6,$ or $7$, the best performance appears at
the setting with $P=2$ or $3$. (3) With the cluster number $K$ becomes larger,
appropriate increase of $P$ is able to boost the performance. Specifically,
when the cluster number $K$ increase from 3 to 6, the best performance appears
at the point when $P=1,2$ and $3$, respectively.

##  5 Conclusion

We design a personalized federated learning framework FedCedar equipped with
the functionalities of clustering, dynamic weighted graph learning, and
precise personalized model distribution to solve the data heterogeneity
challenge. In particular, we build dynamic weighted graphs to conduct model
aggregation and enable clients to obatain models precisely with our designed
strategy. Our proposed FedCedar outperforms state-of-the-art baselines on
three datasets for the image classification task. To our best knowledge, this
is the first FL research work to embed the clustering and dynamic weighted
graph construction to explore the hidden relations between clients and thus
obtain the optimal personalized local models via precise personalized model
distribution strategies. In future works, we would like to provide theoretical
convergence and generality analysis of our model under more flexible real-
world settings.

## References

  * [1] Arivazhagan, M.G., Aggarwal, V., Singh, A.K., Choudhary, S.: Federated learning with personalization layers. arXiv preprint arXiv:1912.00818 (2019)
  * [2] Briggs, C., Fan, Z., Andras, P.: Federated learning with hierarchical clustering of local updates to improve training on non-iid data. In: 2020 International Joint Conference on Neural Networks. pp. 1–9. IEEE (2020)
  * [3] Chen, F., Long, G., Wu, Z., Zhou, T., Jiang, J.: Personalized federated learning with graph. arXiv preprint arXiv:2203.00829 (2022)
  * [4] Fallah, A., Mokhtari, A., Ozdaglar, A.: Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems 33, 3557–3568 (2020)
  * [5] Ghosh, A., Chung, J., Yin, D., Ramchandran, K.: An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems 33, 19586–19597 (2020)
  * [6] He, C., Ceyani, E., Balasubramanian, K., Annavaram, M., Avestimehr, S.: Spreadgnn: Serverless multi-task federated learning for graph neural networks. arXiv preprint arXiv:2106.02743 (2021)
  * [7] Hu, K., Wu, J., Li, Y., Lu, M., Weng, L., Xia, M.: Fedgcn: Federated learning-based graph convolutional networks for non-euclidean spatial data. Mathematics 10(6), 1000 (2022)
  * [8] Hubert, L., Arabie, P.: Comparing partitions. Journal of classification 2(1), 193–218 (1985)
  * [9] Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems 2, 429–450 (2020)
  * [10] Lin, S., Han, Y., Li, X., Zhang, Z.: Personalized federated learning towards communication efficiency, robustness and fairness. Advances in Neural Information Processing Systems (2022)
  * [11] Liu, C.T., Wang, C.Y., Chien, S.Y., Lai, S.H.: Fedfr: Joint optimization federated framework for generic and personalized face recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 1656–1664 (2022)
  * [12] Liu, Z., Chen, Y., Zhao, Y., Yu, H., Liu, Y., Bao, R., Jiang, J., Nie, Z., Xu, Q., Yang, Q.: Contribution-aware federated learning for smart healthcare. In: Proceedings of the 34th Annual Conference on Innovative Applications of Artificial Intelligence (2022)
  * [13] Long, G., Tan, Y., Jiang, J., Zhang, C.: Federated learning for open banking. In: Federated learning, pp. 240–254. Springer (2020)
  * [14] Long, G., Xie, M., Shen, T., Zhou, T., Wang, X., Jiang, J.: Multi-center federated learning: clients clustering for better personalization. World Wide Web pp. 1–20 (2022)
  * [15] Lou, G., Liu, Y., Zhang, T., Zheng, X.: Stfl: A temporal-spatial federated learning framework for graph neural networks. arXiv preprint arXiv:2111.06750 (2021)
  * [16] Ma, Z., Lu, Y., Li, W., Cui, S.: Beyond random selection: A perspective from model inversion in personalized federated learning. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 572–586. Springer (2023)
  * [17] Marchand, T., Muzellec, B., Beguier, C., Terrail, J.O.d., Andreux, M.: Securefedyj: a safe feature gaussianization protocol for federated learning. arXiv preprint arXiv:2210.01639 (2022)
  * [18] McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. pp. 1273–1282. PMLR (2017)
  * [19] Rand, W.M.: Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association 66(336), 846–850 (1971)
  * [20] Solanki, S., Kanaparthy, S., Damle, S., Gujar, S.: Differentially private federated combinatorial bandits with constraints. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 620–637. Springer (2023)
  * [21] T Dinh, C., Tran, N., Nguyen, J.: Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems 33, 21394–21405 (2020)
  * [22] Wang, J., Ma, F.: Federated learning for rare disease detection: a survey. Rare Disease and Orphan Drugs Journal (2), 1–14 (2023)
  * [23] Wang, J., Qian, C., Cui, S., Glass, L., Ma, F.: Towards federated covid-19 vaccine side effect prediction. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 437–452. Springer (2022)
  * [24] Wang, J., Yang, X., Cui, S., Che, L., Lyu, L., Xu, D., Ma, F.: Towards personalized federated learning via heterogeneous model reassembly. Advances in Neural Information Processing Systems (2023)
  * [25] Wang, J., Zeng, S., Long, Z., Wang, Y., Xiao, H., Ma, F.: Knowledge-enhanced semi-supervised federated learning for aggregating heterogeneous lightweight clients in iot. In: Proceedings of the 2023 SIAM International Conference on Data Mining. pp. 496–504. SIAM (2023)
  * [26] Wu, C., Wu, F., Cao, Y., Huang, Y., Xie, X.: Fedgnn: Federated graph neural network for privacy-preserving recommendation. arXiv preprint arXiv:2102.04925 (2021)
  * [27] Xie, M., Long, G., Shen, T., Zhou, T., Wang, X., Jiang, J., Zhang, C.: Multi-center federated learning. arXiv preprint arXiv:2005.01026 (2020)
  * [28] Zhang, K., Yang, C., Li, X., Sun, L., Yiu, S.M.: Subgraph federated learning with missing neighbor generation. Advances in Neural Information Processing Systems 34, 6671–6682 (2021)
  * [29] Zhang, X., Li, Y., Li, W., Guo, K., Shao, Y.: Personalized federated learning via variational bayesian inference. In: International Conference on Machine Learning. pp. 26293–26310. PMLR (2022)