# DFML: Decentralized Federated Mutual Learning

Yasser H. Khalil  Amir H. Estiri  Mahdi Beitollahi  Nader Asadi  Sobhan Hemati
Xu Li  Guojun Zhang  Xi Chen

###### Abstract

In the realm of real-world devices, centralized servers in Federated Learning
(FL) present challenges including communication bottlenecks and susceptibility
to a single point of failure. Additionally, contemporary devices inherently
exhibit model and data heterogeneity. Existing work lacks a Decentralized FL
(DFL) framework capable of accommodating such heterogeneity without imposing
architectural restrictions or assuming the availability of public data. To
address these issues, we propose a Decentralized Federated Mutual Learning
(DFML) framework that is serverless, supports nonrestrictive heterogeneous
models, and avoids reliance on public data. DFML effectively handles model and
data heterogeneity through mutual learning, which distills knowledge between
clients, and cyclically varying the amount of supervision and distillation
signals. Extensive experimental results demonstrate consistent effectiveness
of DFML in both convergence speed and global accuracy, outperforming prevalent
baselines under various conditions. For example, with the CIFAR-100 dataset
and 50 clients, DFML achieves a substantial increase of +17.20% and +19.95% in
global accuracy under Independent and Identically Distributed (IID) and non-
IID data shifts, respectively.

Machine Learning, ICML

##  1 Introduction

Federated Learning (FL) stands as a promising paradigm in machine learning
that enables decentralized learning without sharing raw data, thereby
enhancing data privacy. Although, Centralized FL (CFL) has been predominant in
the literature (McMahan et al., 2017; Alam et al., 2022; Diao et al., 2020;
Horvath et al., 2021; Caldas et al., 2018), it relies on a central server.
Communication with a server can be a bottleneck, especially when numerous
dispersed devices exist, and a server is vulnerable to a single point of
failure. To avoid these challenges, Decentralized FL (DFL) serves as an
alternative, facilitating knowledge sharing among clients without the need of
a central server (Beltrán et al., 2023; Yuan et al., 2023; Giuseppi et al.,
2022; Li et al., 2021).

Figure 1: Demonstrating the adverse impact of model and data heterogeneity on
global accuracy using decentralized FedAvg. The experiment uses CIFAR-100
dataset with 50 clients. Homogeneous models and IID data signify clients with
identical model architectures and data distributions. In contrast,
heterogeneous models and non-IID data indicate variations in both model
architectures and data distributions among clients. Additional experimental
details can be found in Section 5.1. Figure 2: Our proposed DFML framework. In
each communication round $t$, randomly selected clients (senders) send their
locally trained models $W_{n}$ to another randomly chosen client (aggregator).
Mutual learning takes place at the aggregator using $\alpha^{(t)}$. The
updated models $W^{+}_{n}$ and $\alpha^{(t)}$ are then transmitted back to the
senders. $\alpha^{(t)}$ controls the impact of the loss components in the
objective function (see Section 3), and is computed based on a scheduler
function. $t$ denotes the current communication round. Different shapes and
sizes signify model and data heterogeneity. In this example, clients 2 and 4
act as senders, while client 1 serves as the aggregator.

Federated Averaging (FedAvg) (McMahan et al., 2017) is a widely adopted FL
approach that disperses knowledge by averaging the parameters of models.
Despite its advantages, FedAvg faces a significant limitation: the lack of
support for model heterogeneity. This limitation becomes impractical in real-
world scenarios where devices inherently possess diverse architectures. In a
DFL system with model heterogeneity, FedAvg confines parameter averaging to
models with the same architectures, thereby hindering knowledge sharing among
clients. This problem exacerbates with the presence of heterogeneity in data
between clients, affecting the preservation of global knowledge (Ye et al.,
2023). Figure 1 demonstrates the adverse effects of model and data
heterogeneity using decentralized FedAvg. This entails a need for a novel
framework that better supports model and data heterogeneity in DFL.

Researchers have extended FedAvg to support model heterogeneity, but these
extensions often impose constraints on model architectures (Diao et al., 2020;
Alam et al., 2022; Shen et al., 2020). Another approach that addresses model
heterogeneity in FL involves mutual learning, where models collaboratively
learn by teaching each other a task (Zhang et al., 2018; Li et al., 2021).
Mutual learning enables knowledge transfer as models mimic each other’s class
probabilities. In this process, each client acts as both a student and a
teacher. The objective function comprises a supervision loss component and
another responsible for distilling knowledge from experts (Hinton et al.,
2015). However, existing works utilizing knowledge distillation require a
server or public data (Lin et al., 2020; Li & Wang, 2019; Li et al., 2020).
Reliance on public data can be impractical, especially in sensitive domains
such as health. This makes a solution that avoids additional data more
desirable.

To this end, we propose a Decentralized Federated Mutual Learning (DFML)
framework that is 1) serverless, 2) supports model heterogeneity without
imposing any architectural constraints, and 3) does not require additional
public data. Table 1 highlights the advantages of our proposed DFML over prior
arts. As will be shown in the results section, DFML outperforms other
baselines in addressing the model and data heterogeneity problems in DFL.
Figure 2 depicts our proposed DFML framework. In each communication round,
multiple clients (senders) transmit their models to another client
(aggregator) for mutual learning, effectively handling model heterogeneity.
DFML leverages the aggregator’s data for knowledge distillation, eliminating
the need for extra data. Additionally, DFML addresses data heterogeneity by
employing re-Weighted SoftMax (WSM) cross-entropy (Legate et al., 2023), which
prevents models from drifting toward local objectives.

Furthermore, we observed varying performance levels111Refer to Section 6.2
when using different fixed values to balance the ratio between the supervision
and distillation loss components in the objective function. In response, we
propose a cyclic variation (Loshchilov & Hutter, 2016; Smith, 2017) of the
ratio between these loss components. The cyclical approach iteratively directs
the model’s objective between the aggregator’s data and the reliance on
experts to distill knowledge using that data. Gradually, as the distillation
signal becomes dominant, global knowledge increases and eventually reaches a
peak. This cyclical knowledge distillation further enhances global knowledge
compared to using a fixed ratio between the supervision and distillation loss
components.

In this paper, we empirically demonstrate the superior performance of DFML
over state-of-the-art baselines in terms of convergence speed and global
accuracy. The main contributions of this paper are summarized as follows:

  * •

We propose a novel mutual learning framework that operates in DFL, supports
nonrestrictive heterogeneous models and does not rely on additional data.

  * •

We propose cyclically varying the ratio between the supervision and
distillation signals in the objective function to enhance global performance.

##  2 Related Work

Due to the vastness of the FL literature, we restrict our review to works most
relevant to our research. This includes studies that support homogeneous and
heterogeneous architectures, operate in DFL, address catastrophic forgetting,
and adapt knowledge distillation.

###  2.1 Homogeneous and Restrictive Heterogeneous Support

FL aims to learn global knowledge from distributed clients without sharing
their private data. FedAvg (McMahan et al., 2017) uses a server to perform
parameter averaging on locally updated models, but its support is limited to
homogeneous architectures due to the nature of parameter averaging. In
contrast, decentralized FedAvg (Li et al., 2021; Roy et al., 2019; Giuseppi et
al., 2022; Savazzi et al., 2020) depends on client-to-client communication,
with aggregation occurring on any participating clients. While other methods
support heterogeneous models through parameter averaging, however, their
support is constrained. Methods like Federated Dropout (Caldas et al., 2018),
HeteroFL (Diao et al., 2020), and FedRolex (Alam et al., 2022) employ partial
training with a random, static, and rolling technique for sub-model
extractions, respectively. Lastly, FML (Shen et al., 2020) conducts mutual
learning between personalized (heterogeneous) models, and a global
(homogeneous) model. However, FML assumes the existence of a global model, and
all the global knowledge resides in that model instead of the clients’
heterogeneous models. This differs from our goal of transferring knowledge
directly to the clients’ potentially heterogeneous models.

###  2.2 Nonrestrictive Heterogeneous Support

Works that support model heterogeneity without imposing constraints exist,
however, they require assistance from a server and the availability of public
data (Li & Wang, 2019; Lin et al., 2020). FedMD (Li & Wang, 2019) assumes the
availability of public data on all clients. In FedMD, clients generate
predictions using public data which are then communicated to the server for
averaging. Then, the averaged predictions are communicated back to the clients
and are used to update the heterogeneous models using knowledge distillation.
FedDF (Lin et al., 2020) communicates heterogeneous models to a server, where
prototypes are assumed to exist. The server facilitates parameter averaging of
models with same architectures, and knowledge distillation using unlabelled
public data. The reliance on a server and additional data limits the
applicability of these methods.

###  2.3 Decentralized FL

Def-KT (Li et al., 2021) operates within a DFL framework, where clients
communicate models to other clients (aggregators) for mutual learning. Despite
its serverless nature, Def-KT only supports homogeneous architectures as it
replaces the aggregator’s model with the incoming model. This hinders its
effectiveness in scenarios with model heterogeneity.

###  2.4 Catastrophic Forgetting

Catastrophic forgetting focuses on acquiring new tasks without forgetting
previous knowledge (Wang et al., 2023; De Lange et al., 2021). In the context
of FL among clients with non-IID data distributions, catastrophic forgetting
occurs, preventing models from reaching optimal global performance. To
mitigate this issue, researchers aim to shield the learned representations
from drastic adaptation. Asymmetric cross-entropy (ACE) is applied to the
supervision signal of the objective function to address the representation
drift problem (Caccia et al., 2021). ACE uses masked softmax on the classes
that do not exist in the current data. Another approach, proposed by Legate et
al. 2023, modifies the loss function of each client using re-Weighted SoftMax
(WSM) cross-entropy, with re-weighting based on each client’s class
distribution.

Table 1: Comparison between DFML and other FL methods.

Framework |  No Server |  Nonrestrictive Heterogeneous |  No Additional Data
---|---|---|---
FedAvg | ✗ | ✗ | ✓
HeteroFL | ✗ | ✗ | ✓
FedRolex | ✗ | ✗ | ✓
FML | ✗ | ✗ | ✓
FedDF | ✗ | ✓ | ✗
FedMD | ✗ | ✓ | ✗
Def-KT | ✓ | ✗ | ✓
DFML (Ours) | ✓ | ✓ | ✓

###  2.5 Adaptive Knowledge Distillation

In the literature, existing works have explored scaling the loss function in
knowledge distillation (Zhou et al., 2021; Clark et al., 2019); however, the
advancements in this area have been minimal. WSL (Zhou et al., 2021) handles
sample-wise bias-variance trade-off during distillation, while ANL-KD (Clark
et al., 2019) gradually transitions the model from distillation to supervised
learning. Early in training, the model primarily distills knowledge to
leverage a useful training signal, and towards the end of the training, it
relies more on labels to surpass its teachers. Additionally, although FedYogi
(Reddi et al., 2020) proposes adaptive optimization, but its reliance on a
server limits its applicability.

##  3 Proposed Approach

To begin with, we outline the DFL system within which our proposed DFML
operates. Then, we provide a detailed explanation of DFML without delving into
the discussion of varying the balance between the loss components. Lastly, we
explain cyclical knowledge distillation and peak models.

###  3.1 System Setup

In our DFL setup, there are $N$ clients, each client $n$ is equipped with
local training data $D_{n}\in\\{D_{1},D_{2},...,D_{N}\\}$, regular model
$W_{n}\in\\{W_{1},W_{2},...,W_{N}\\}$, peak model
$\widehat{W}_{n}\in\\{\widehat{W}_{1},\widehat{W}_{2},...,\widehat{W}_{N}\\}$,
and local weight alpha
$\alpha_{n}\in\\{\alpha_{1},\alpha_{2},...,\alpha_{N}\\}$. In each
communication round, a client is chosen at random to act as the aggregator
$a$. The choice of the aggregator is determined by the previous aggregator
from the preceding round. In the initial round, the client with the lowest ID
number is assigned the role of the aggregator. Additionally, several clients
are randomly chosen as senders $S$ to send their models to the aggregator.

Algorithm 1 DFML

Input: Initialize $N$ clients, each client $n$ has data $D_{n}$ and two
models: regular $W_{n}$ and peak $\widehat{W}_{n}$. The local weight alpha of
each client $\alpha_{n}=0$.

for communication round $t=1,2,...,T$ do

Randomly select one aggregator $a\in\\{1,...,N\\}$

Randomly select senders $S\subset\\{1,...,N\\}$, $a\notin S$

Participants $\mathcal{P}=S\cup\\{a\\}$

\\\ Client Side

for all $n\in\mathcal{P}$ do

for all batch $X_{n}\in$ local data $D_{n}$ do

$W_{n}\leftarrow$ locally train $W_{n}$ using Equation (2)

Send locally updated models $W_{s}$ for all $s\in S$ to $a$

\\\ Aggregator Side

$\alpha^{(t)}\leftarrow\text{scheduler}(\cdot)$

for $k$ = 1, 2, …, $K$ do

for all batch $X_{a}\in$ local data $D_{a}$ do

for all $n\in\mathcal{P}$ do

$z_{n}\leftarrow\text{logits}(W_{n},X_{a}$)

for all $n\in\mathcal{P}$ do

Update $W_{n}$ using $\alpha^{(t)}$ and logits according to Equation (1)

Send back updated models $W^{+}_{s}$ and $\alpha^{(t)}$ for all $s\in S$

\\\ Client Side

for all $n\in\mathcal{P}$ do

$W_{n}\leftarrow W^{+}_{n}$ \\\ Update regular model

if $\alpha^{(t)}\geq\alpha_{n}$ then

$\widehat{W}_{n}\leftarrow W^{+}_{n}$ \\\ Update peak model

Set local weight $\alpha_{n}\leftarrow\alpha^{(t)}$

###  3.2 DFML Formulation

The goal of DFML is to enable the exchange of local information among clients
without the need to share raw data. DFML facilitates knowledge sharing across
multiple communication rounds $T$ until convergence. In each round $t\in T$, a
set of participants $\left(\mathcal{P}=S\cup\\{a\\}\right)$ is randomly
selected. For each selected clients $n\in\mathcal{P}$, their regular models
$W_{n}$ undergo local training on their private data $D_{n}$. This ensures
that $W_{n}$ retains its local knowledge, allowing knowledge transfer to other
participants during the aggregation process. The process of distilling
knowledge to other models is referred to as aggregation. Subsequently, all
locally trained models are sent to the aggregator for the aggregation process.
When multiple clients send their models to the aggregator, multiple experts
contribute during aggregation, enhancing the accuracy of the global knowledge
transfer.

DFML employs weighted mutual learning for aggregation to allow models to
collaboratively learn from each other using the aggregator’s data. This
technique ensures that larger models contribute more significantly to
knowledge transfer compared to smaller models, leveraging their finer
knowledge. DFML conducts the aggregation process $K$ times to maximize
knowledge transfer without impacting the communication cost per round.
Following this, all updated models $W^{+}_{n}$ are transmitted back to their
respective senders. Subsequently, each participant $n\in\mathcal{P}$ replaces
its model $W_{n}$ with the updated version $W^{+}_{n}$. This entire process is
repeated until global knowledge has been disseminated across the entire
network. Cyclical knowledge distillation and the peak models are explained in
Section 3.2.1. Algorithm 1 describes our proposed DFML framework. The
objective function of DFML is as follows:

$\mathcal{L}=(1-\alpha)\mathcal{L}_{\text{WSM}}+\alpha\mathcal{L}_{\text{KL}},$
(1)

where $\mathcal{L}_{\text{WSM}}$ represents the supervision loss signal
computed using re-weighted softmax cross-entropy (WSM) (Legate et al., 2023),
and $\mathcal{L}_{\text{KL}}$ represents the distillation loss signal computed
by Kullback–Leibler Divergence (KL). The hyperparameter $\alpha$ controls the
balance between these two loss components. $\mathcal{L}_{\text{WSM}}$ is
defined as follows:

$\mathcal{L}_{\text{WSM}}\\!=\\!-\sum_{x\in
X_{n}}\biggr{[}z_{n}(x)_{y(x)}-\log\biggl{(}\sum_{c\in
C}\beta_{c}e^{z_{n}(x)_{c}}\biggl{)}\biggr{]},$ (2)

where $X_{n}$ represents a data batch drawn from the distribution $D_{n}$ of
client $n$. $z_{n}$ denotes the logits of data sample $x$ with model weights
$W_{n}$, $y(x)$ is the data label, $\beta_{c}$ is a vector representing the
proportion of label $c$ present in the client dataset, and $C$ is the set of
classes in the entire dataset.

The distillation loss component $\mathcal{L}_{\text{KL}}$ is defined as
follows:

$\mathcal{L}_{\text{KL}}=\sum_{x\in X_{a}}\sum_{q\neq
n}^{\mathcal{P}}\biggl{[}\frac{\Phi_{q}}{\sum_{u\neq
n}^{\mathcal{P}}{\Phi_{u}}}\text{KL}\bigl{(}p_{q}(x)\;||\;p_{n}(x)\bigr{)}\biggr{]},$
(3)

where $X_{a}$ corresponds to a data batch drawn from the distribution $D_{a}$
of aggregator $a$. $\mathcal{P}$ denotes the set of participants in mutual
learning, including the senders and the aggregator. $\Phi$ represents the
model size based on the number of trainable parameters. Finally, $p_{q}(x)$
and $p_{n}(x)$ are the teacher $q$ and student $n$ predictions of the data
sample $x$ with model weights $W_{q}$ and $W_{n}$, respectively. $u$ is a
dummy variable indexing all teachers.

The use of WSM in both local training and mutual learning serves as a
protective measure against catastrophic forgetting, which arises from shifts
in data distribution between clients. WSM ensures that the models update
parameters considering the proportion of available labels. This strategy
prevents models from altering their accumulated knowledge on labels that are
not present in the current data distribution, thereby safeguarding against
catastrophic forgetting.

Figure 3: Illustrating the impact of cyclically varying $\alpha$ on global
accuracy. Peak models are updated up to the first $\alpha$ maximum and every
subsequent time $\alpha$ reaches its maximum limit.

####  3.2.1 Cyclic knowledge distillation

Cyclical knowledge distillation is manifested by periodically adjusting the
value of $\alpha$ in the objective function during each communication round.
Inspired by (Loshchilov & Hutter, 2016; Smith, 2017; Izmailov et al., 2018),
we use the cyclical behavior to vary $\alpha$. This dynamic variation in
$\alpha$ occurs with each new aggregator selection, leading to mutual learning
with a distinct $\alpha$ value at each round. The cyclical process,
influencing the balance between the supervision and distillation loss
components, contributes to an overall increase in global knowledge. The global
knowledge is measured by global accuracy throughout training.

Figure 3 depicts the impact of cyclical $\alpha$ on global accuracy. When the
supervision signal dominates, each $W_{n}$ exclusively learns from $D_{a}$
without collaborating with other models (experts). This focus on the
supervision signal directs the model’s objective toward $D_{a}$, causing
$W_{n}$ to lose previously acquired global knowledge from earlier rounds. As
the distillation signal gains prominence, $W_{n}$ begins to reacquire global
knowledge, facilitating simultaneous knowledge distillation among all models.
With a dominant distillation signal, each model exclusively learns from other
experts, maximizing global knowledge in each one.

Therefore, the peak in global performance is reached when the distillation
signal is dominant ($\alpha$ is maximum), and the lowest is attained when the
supervision signal is dominant ($\alpha$ is minimum). We observed that
cyclically changing between the two signals leads to a higher global accuracy
compared to using either one exclusively or a linear combination of them. This
cyclical adjustment of $\alpha$ is crucial for the continuous growth in global
accuracy throughout training, albeit with fluctuations in global accuracy.

####  3.2.2 Peak models

To counteract the undesirable fluctuations in global accuracy resulting from
the cyclical process, we introduce an additional model for each client, termed
the peak model $\widehat{W}_{n}$. The primary role of $\widehat{W}_{n}$ is to
retain the best global parameters of $W_{n}$. Specifically, each
$\widehat{W}_{n}$ is updated whenever $W_{n}$ is aggregated with a dominant
distillation signal. $\widehat{W}_{n}$ are detached from the training process
and are kept in a frozen state, preserving the maximum global accuracy
achieved so far. Also, the peak models are continuously updated from the
initial communication round up to the first $\alpha$ maximum, allowing them to
quickly reach the first global accuracy peak. Thus, the peak models act as a
stabilizing mechanism, retaining the optimal global knowledge attained.

##  4 Experiments

###  4.1 Dataset

We evaluate our proposed DFML against prevalent baselines using CIFAR-10 and
CIFAR-100 datasets (Krizhevsky et al., 2009). The evaluation covers
experiments on two data distribution shifts: Independent and Identically
Distributed (IID) and non-IID. The non-IID distribution involves a heavy label
shift based on the Dirichlet distribution with $\beta=0.1$. The dataset is
distributed among clients by dividing the train set into $N$ splits, either
evenly for IID or utilizing Dirichlet distribution for non-IID. Each split is
further segmented into training and validation sets following an 80:20 ratio.
Local validation sets are employed to assess local performance, while the
entire test set evaluates the global accuracy of the clients. The global
accuracy of DFML is evaluated by examining the peak models unless stated
otherwise. The data partitions for all clients are available in Appendix A.

###  4.2 Implementations

In our experiments, we utilize CNNs (LeCun et al., 1998), ResNets (He et al.,
2016), and ViTs (Dosovitskiy et al., 2020) as our model architectures. The
evaluation of DFML encompasses three architecture modes: homogeneous,
restrictive heterogeneous, and nonrestrictive heterogeneous. We name these
three modes: $H0$, $H1$, and $H2$, respectively. Details of these modes and
the associated model architectures are outlined in Table 7. To ensure a fair
comparison, all experiments including the baselines are run with WSM instead
of Cross-Entropy (CE), unless specified otherwise. Further implementation
details are available in Appendix B.

###  4.3 Baselines

To address the absence of baselines tailored for DFL settings, we derive
baselines by decentralizing some state-of-the-art CFL algorithms, adapting
them to function within our DFL system. We begin our experiments by
constraining model architectures to enable comparison with existing baselines.
The initial experiments focus on homogeneous architectures for direct
comparison with baselines like decentralized FedAvg and Def-KT. Our derived
decentralized version of FedAvg is referred to as decentralized FedAvg. Def-KT
intrinsically operates in a DFL framework.

Subsequently, we conduct experiments with restrictive heterogeneous
architectures, where models have the same number of layers but different
widths. This allows comparison of DFML with our derived decentralized versions
of partial training methods (HeteroFL and FedRolex) alongside FedAvg. Further
details on the derived decentralized baselines are provided in Appendix C.

Following this, we demonstrate the full capabilities of DFML by conducting
experiments with nonrestrictive heterogeneous architectures, which are
incompatible with partial training algorithms. The only baseline available for
this set of experiments is decentralized FedAvg. In this paper, we omit the
comparison of DFML with baselines that require additional data such as (Lin et
al., 2020; Li & Wang, 2019) to ensure fairness.

Figure 4: Demonstrating the global accuracy gain DFML achieves in comparison
with decentralized FedAvg and HeteroFL under model and data heterogeneity.
CIFAR-100 dataset is used with $50$ clients and CNN architectures.

##  5 Results

In this section, we evaluate DFML against other state-of-the-art baselines. We
first demonstrate the effectiveness of DFML in handling model and data
heterogeneity. Moreover, we prove that DFML outperforms all baselines in terms
of final convergence speed and final global accuracy across three architecture
modes: homogeneous, restrictive heterogeneous, and nonrestrictive
heterogeneous architectures.

###  5.1 Model and Data Heterogeneity

Figure 4 demonstrates that our proposed DFML under model and data
heterogeneity, mitigates the impact on global accuracy more effectively than
decentralized FedAvg and HeteroFL. To ensure a fair comparison between
homogeneous and heterogeneous experiments, we maintained the same average
number of parameters in each case. This was achieved by selecting the median
model of the five different architectures from the heterogeneous experiment as
the model used in the homogeneous experiment.

###  5.2 Homogeneous Architectures

Table 2 demonstrates that our DFML outperforms both decentralized FedAvg and
Def-KT in terms of global accuracy across datasets and under two data
distributions. Larger improvements are recorded under non-IID data shifts. To
align the number of communications per round for all baselines, adjustments
were made for Def-KT by setting the number of aggregators to $25$. This is
necessary as in Def-KT each sending model should be received by a different
aggregator.

Table 2: Global accuracy comparison using homogeneous CNN architectures with
$50$ clients and $25$ senders. For Def-KT, $25$ aggregators are selected, and
$1$ aggregator for the other methods.

| CIFAR-10 | CIFAR-100
---|---|---
Method | IID | non-IID | IID | non-IID
Dec. FedAvg | 80.00 $\pm$ 0.19 | 74.68 $\pm$ 0.06 | 48.32 $\pm$ 0.18 | 43.17 $\pm$ 0.16
Def-KT | 79.29 $\pm$ 0.18 | 72.59 $\pm$ 0.34 | 48.43 $\pm$ 0.18 | 43.84 $\pm$ 0.24
DFML (Ours) | 80.96 $\pm$ 0.07 | 76.27 $\pm$ 0.40 | 50.47 $\pm$ 0.29 | 46.41 $\pm$ 0.48

Table 3: Global accuracy comparison using restrictive heterogeneous
architectures with $50$ clients and $25$ senders. Architectures used are CNN,
ResNet, and ViT.

| CNN | ResNet | ViT
---|---|---|---
| CIFAR-10 | CIFAR-100 | CIFAR-10 | CIFAR-100 | CIFAR-10 | CIFAR-100
Method | IID | non-IID | IID | non-IID | IID | non-IID | IID | non-IID | IID | non-IID | IID | non-IID
Dec. FedAvg | 72.09 $\pm$ 0.20 | 59.84 $\pm$ 0.18 | 33.59 $\pm$ 0.06 | 27.03 $\pm$ 0.23 | 79.84 $\pm$ 0.16 | 54.93 $\pm$ 0.76 | 39.98 $\pm$ 0.45 | 28.80 $\pm$ 0.24 | 64.50 $\pm$ 0.13 | 50.76 $\pm$ 0.26 | 27.43 $\pm$ 0.22 | 22.27 $\pm$ 0.03
Dec. HeteroFL | 73.96 $\pm$ 0.21 | 64.21 $\pm$ 0.67 | 37.62 $\pm$ 0.16 | 32.98 $\pm$ 0.10 | 81.70 $\pm$ 0.11 | 68.92 $\pm$ 0.60 | 43.33 $\pm$ 0.34 | 34.92 $\pm$ 0.29 | 65.08 $\pm$ 0.61 | 53.90 $\pm$ 0.09 | 29.60 $\pm$ 0.53 | 25.65 $\pm$ 0.25
Dec. FedRolex | 74.38 $\pm$ 0.02 | 64.30 $\pm$ 0.41 | 37.65 $\pm$ 0.27 | 32.83 $\pm$ 0.16 | 81.70 $\pm$ 0.17 | 69.41 $\pm$ 0.19 | 43.30 $\pm$ 0.20 | 34.94 $\pm$ 0.43 | 65.17 $\pm$ 0.40 | 53.56 $\pm$ 0.61 | 30.44 $\pm$ 0.21 | 25.81 $\pm$ 0.06
DFML (Ours) | 79.02 $\pm$ 0.04 | 73.87 $\pm$ 0.19 | 48.60 $\pm$ 0.03 | 44.26 $\pm$ 0.20 | 85.68 $\pm$ 0.01 | 71.24 $\pm$ 0.02 | 53.32 $\pm$ 0.07 | 46.27 $\pm$ 0.38 | 68.38 $\pm$ 0.03 | 54.50 $\pm$ 0.01 | 36.48 $\pm$ 0.20 | 28.56 $\pm$ 0.60

Table 4: Global accuracy comparison using nonrestrictive heterogeneous
architectures. The experiments are conducted using CNN architectures.
Different numbers of clients $N$ are used, with $50\%\times\\!N$ as senders.

| $\boldsymbol{N:10}$ | $\boldsymbol{N:50}$ | $\boldsymbol{N:100}$
---|---|---|---
| CIFAR-10 | CIFAR-100 | CIFAR-10 | CIFAR-100 | CIFAR-10 | CIFAR-100
Method | IID | non-IID | IID | non-IID | IID | non-IID | IID | non-IID | IID | non-IID | IID | non-IID
Dec. FedAvg | 72.65 $\pm$ 0.17 | 43.78 $\pm$ 0.06 | 33.58 $\pm$ 0.08 | 23.01 $\pm$ 0.09 | 71.51 $\pm$ 0.24 | 59.05 $\pm$ 0.22 | 33.19 $\pm$ 0.19 | 26.27 $\pm$ 0.30 | 71.00 $\pm$ 0.32 | 57.79 $\pm$ 0.11 | 32.30 $\pm$ 0.30 | 26.93 $\pm$ 0.13
DFML (Ours) | 83.87 $\pm$ 0.22 | 74.30 $\pm$ 0.88 | 54.03 $\pm$ 0.15 | 49.67 $\pm$ 0.20 | 81.74 $\pm$ 0.04 | 75.51 $\pm$ 0.12 | 50.39 $\pm$ 0.09 | 46.22 $\pm$ 0.07 | 79.94 $\pm$ 0.03 | 71.75 $\pm$ 0.05 | 47.66 $\pm$ 0.06 | 42.84 $\pm$ 0.23

Table 5: Overall communication rounds DFML requires to attain the same
accuracy as decentralized FedAvg achieves at communication rounds 100 and 500.
Both methods have the same communication cost per round. Different numbers of
clients $N$ are used.

|  | $\boldsymbol{N:10}$ | $\boldsymbol{N:50}$ | $\boldsymbol{N:100}$
---|---|---|---|---
|  Dec. FedAvg Communication Round | CIFAR-10 | CIFAR-100 | CIFAR-10 | CIFAR-100 | CIFAR-10 | CIFAR-100
Method | IID | non-IID | IID | non-IID | IID | non-IID | IID | non-IID | IID | non-IID | IID | non-IID
DFML (Ours) | 100 | 10 | 20 | 20 | 10 | 30 | 40 | 30 | 30 | 50 | 60 | 40 | 50
500 | 20 | 20 | 20 | 20 | 90 | 90 | 90 | 80 | 150 | 190 | 130 | 130

###  5.3 Heterogeneous Architectures with Restrictions

In this set of experiments, some restrictions are applied to heterogeneous
architectures. Table 3 demonstrates that DFML consistently outperforms all
baselines in terms of global accuracy across three different architectures,
two datasets, and both IID and non-IID data distributions. Another observation
is that the partial training baselines outperformed decentralized FedAvg,
which is expected as more knowledge is shared by averaging overlapping
parameters of the models rather than only averaging models with the same
architectures. Furthermore, we implemented decentralized Federated Dropout;
however, the results are not reported as it did not work. The poor performance
is attributed to local models being assigned random parameters from the global
model generated at the aggregator.

###  5.4 Heterogeneous Architectures without Restrictions

After confirming that our proposed DFML competes with the prevalent baselines
in DFL, we showcase the strength of DFML in knowledge transfer using
nonrestrictive heterogeneous models. Additionally, we evaluate DFML under
different $N$ clients. From Table 4, it is evident that across various $N$
clients, datasets, and data distributions, DFML consistently achieves superior
global accuracy compared to the baseline. Table 5 shows that DFML requires
fewer communication rounds to reach the same accuracy as decentralized FedAvg
does at specific rounds. The communication cost per round for all experiments
is $50$, involving the transmission of $25$ models to and from the aggregator.
Appendix D provides further experimental analysis on DFML. A visual
illustration of the convergence speedup achieved by DFML compared to the
baselines is presented in Appendix D.1. Moreover, a comparison between DFML
and a decentralized version of FML is provided in Appendix D.5.

Table 6: Global accuracy comparison between DFML and decentralized FedAvg with
different supervision signals and cyclically varying $\alpha$. The dataset
used is CIFAR-100, with $50$ clients and $25$ senders.

|  |  | CNN | ResNet | ViT
---|---|---|---|---|---
Method | CE / WSM | Cyclical $\boldsymbol{\alpha}$ | IID | non-IID | IID | non-IID | IID | non-IID
DFML (Ours) | CE | ✗ | 47.44 $\pm$ 0.08 | 29.16 $\pm$ 2.27 | 49.13 $\pm$ 0.30 | 18.78 $\pm$ 0.67 | 35.66 $\pm$ 0.02 | 14.02 $\pm$ 0.01
CE | ✓ | 48.44 $\pm$ 0.23 | 41.47 $\pm$ 0.30 | 55.29 $\pm$ 0.13 | 41.07 $\pm$ 1.27 | 36.62 $\pm$ 0.30 | 22.34 $\pm$ 0.12
WSM | ✗ | 47.57 $\pm$ 0.26 | 43.59 $\pm$ 0.35 | 49.77 $\pm$ 0.24 | 30.91 $\pm$ 0.96 | 34.04 $\pm$ 0.21 | 25.04 $\pm$ 0.56
WSM | ✓ | 48.60 $\pm$ 0.03 | 44.26 $\pm$ 0.20 | 53.32 $\pm$ 0.07 | 46.27 $\pm$ 0.38 | 36.48 $\pm$ 0.20 | 28.56 $\pm$ 0.60

##  6 Analysis

In this section, we first demonstrate the effect of using different fixed
$\alpha$ values versus cyclically adjusting it. Second, we present the impact
of cyclical $\alpha$ on global accuracy by evaluating the regular models.
Subsequently, we examine the behavior exhibited by both regular and peak
models as $\alpha$ changes, and highlight the ability of the peak models to
capture the peaks in global performance. Last, we conduct an analysis to
understand the effects of different supervision signals and cyclical $\alpha$
on global accuracy.

Figure 5: Performance comparison of the regular models against cyclically
oscillating $\alpha$ with half cycles. The supervision signal used is CE.
Refer to Appendix E for full-cycle $\alpha$ oscillation.

###  6.1 Regular vs Peak Models

Figure 5 shows the fluctuations in global performance as $\alpha$ is
cyclically varied. When $\alpha=0$, representing only the supervision signal
in the objective function, global accuracy is at its lowest. This is
attributed to models being optimized solely toward the aggregator’s local
data. However, as $\alpha$ increases and eventually reaches its maximum
defined value, models gain knowledge from each other through knowledge
distillation, resulting in a peak in global accuracy. The decline in global
accuracy starts when the distillation signal diminishes, and the supervisory
signal takes over. To maintain global performance at the peaks, the peak
models are used. Initially, the peak models are updated regularly with every
regular model update until the first maximum $\alpha$ value is reached. After
that, the peak models are only updated when the regular models are aggregated
using the highest value of $\alpha$. Figure 6 depicts the global accuracy of
both regular and peak models, highlighting the stability achieved by the peak
models throughout training.

Figure 6: Performance comparison between DFML regular models, that are
communicated and updated in each communication round, and peak models that are
updated only when a peak occurs. The supervision signal used is CE.

###  6.2 Fixed vs Cyclical $\boldsymbol{\alpha}$

As shown in Figure 5, the highest global accuracy is consistently achieved
when $\alpha$ reaches its maximum defined value. Now, we address the impact of
using different fixed values for $\alpha$ and whether setting $\alpha=1$
throughout training yields similar performance as cyclically changing
$\alpha$. Figure 7 demonstrates that using different fixed values of $\alpha$,
under different distribution shifts, results in varied performance levels.
Furthermore, fixing $\alpha=1$ leads to the worst global performance because
when only the distillation signal is present throughout training without any
supervision, noise will be propagated. This results in experts teaching each
other incorrect information.

###  6.3 Supervision Signal and Cyclical $\boldsymbol{\alpha}$

The impact of different supervision signals (CE and WSM) and cyclical $\alpha$
on DFML is presented in Table 6. Results indicate that the use of WSM
primarily enhances the global accuracy in non-IID data distribution shifts.
Furthermore, the addition of cyclical $\alpha$ on top of CE and WSM further
improves global accuracy. The best outcomes are mostly reported when both WSM
and cyclical $\alpha$ are applied. Additional experimental analysis on
cyclical knowledge distillation can be found in Appendix E.

Figure 7: Performance comparison between different fixed $\alpha$ values and
cyclically varying it, under IID and non-IID data distributions. The
supervision signal used is CE.

##  7 Conclusion

We proposed DFML, a framework that supports a decentralized knowledge transfer
among heterogeneous models, without architectural constraints or reliance on
additional data. DFML outperformed state-of-the-art baselines in addressing
model and data heterogeneity in DFL, showcasing better convergence speed and
global accuracy.

## References

  * Alam et al. (2022) Alam, S., Liu, L., Yan, M., and Zhang, M.  Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction.  _Advances in Neural Information Processing Systems_ , 35:29677–29690, 2022.
  * Beltrán et al. (2023) Beltrán, E. T. M., Pérez, M. Q., Sánchez, P. M. S., Bernal, S. L., Bovet, G., Pérez, M. G., Pérez, G. M., and Celdrán, A. H.  Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges.  _IEEE Communications Surveys & Tutorials_, 2023.
  * Caccia et al. (2021) Caccia, L., Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J., and Belilovsky, E.  New insights on reducing abrupt representation change in online continual learning.  _arXiv preprint arXiv:2104.05025_ , 2021.
  * Caldas et al. (2018) Caldas, S., Konečny, J., McMahan, H. B., and Talwalkar, A.  Expanding the reach of federated learning by reducing client resource requirements.  _arXiv preprint arXiv:1812.07210_ , 2018.
  * Clark et al. (2019) Clark, K., Luong, M.-T., Khandelwal, U., Manning, C. D., and Le, Q. V.  Bam! born-again multi-task networks for natural language understanding.  _arXiv preprint arXiv:1907.04829_ , 2019.
  * De Lange et al. (2021) De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(7):3366–3385, 2021.
  * Diao et al. (2020) Diao, E., Ding, J., and Tarokh, V.  Heterofl: Computation and communication efficient federated learning for heterogeneous clients.  _arXiv preprint arXiv:2010.01264_ , 2020.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Giuseppi et al. (2022) Giuseppi, A., Manfredi, S., Menegatti, D., Pietrabissa, A., and Poli, C.  Decentralized federated learning for nonintrusive load monitoring in smart energy communities.  In _2022 30th Mediterranean Conference on Control and Automation (MED)_ , pp. 312–317. IEEE, 2022.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hinton et al. (2015) Hinton, G., Vinyals, O., and Dean, J.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Horvath et al. (2021) Horvath, S., Laskaridis, S., Almeida, M., Leontiadis, I., Venieris, S., and Lane, N.  Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout.  _Advances in Neural Information Processing Systems_ , 34:12876–12889, 2021.
  * Izmailov et al. (2018) Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G.  Averaging weights leads to wider optima and better generalization.  _arXiv preprint arXiv:1803.05407_ , 2018.
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al.  Learning multiple layers of features from tiny images.  2009\.
  * LeCun et al. (1998) LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Lee et al. (2021) Lee, S. H., Lee, S., and Song, B. C.  Vision transformer for small-size datasets.  _arXiv preprint arXiv:2112.13492_ , 2021.
  * Legate et al. (2023) Legate, G., Caccia, L., and Belilovsky, E.  Re-weighted softmax cross-entropy to control forgetting in federated learning.  _arXiv preprint arXiv:2304.05260_ , 2023.
  * Li et al. (2021) Li, C., Li, G., and Varshney, P. K.  Decentralized federated learning via mutual knowledge transfer.  _IEEE Internet of Things Journal_ , 9(2):1136–1147, 2021.
  * Li & Wang (2019) Li, D. and Wang, J.  Fedmd: Heterogenous federated learning via model distillation.  _arXiv preprint arXiv:1910.03581_ , 2019.
  * Li et al. (2020) Li, Q., He, B., and Song, D.  Practical one-shot federated learning for cross-silo setting.  _arXiv preprint arXiv:2010.01017_ , 2020.
  * Lin et al. (2020) Lin, T., Kong, L., Stich, S. U., and Jaggi, M.  Ensemble distillation for robust model fusion in federated learning.  _Advances in Neural Information Processing Systems_ , 33:2351–2363, 2020.
  * Loshchilov & Hutter (2016) Loshchilov, I. and Hutter, F.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * McMahan et al. (2017) McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_ , pp. 1273–1282. PMLR, 2017.
  * Reddi et al. (2020) Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečnỳ, J., Kumar, S., and McMahan, H. B.  Adaptive federated optimization.  _arXiv preprint arXiv:2003.00295_ , 2020.
  * Roy et al. (2019) Roy, A. G., Siddiqui, S., Pölsterl, S., Navab, N., and Wachinger, C.  Braintorrent: A peer-to-peer environment for decentralized federated learning.  _arXiv preprint arXiv:1905.06731_ , 2019.
  * Savazzi et al. (2020) Savazzi, S., Nicoli, M., Rampa, V., and Kianoush, S.  Federated learning with mutually cooperating devices: A consensus approach towards server-less model optimization.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 3937–3941. IEEE, 2020.
  * Shen et al. (2020) Shen, T., Zhang, J., Jia, X., Zhang, F., Huang, G., Zhou, P., Kuang, K., Wu, F., and Wu, C.  Federated mutual learning.  _arXiv preprint arXiv:2006.16765_ , 2020.
  * Smith (2017) Smith, L. N.  Cyclical learning rates for training neural networks.  In _2017 IEEE winter conference on applications of computer vision (WACV)_ , pp. 464–472. IEEE, 2017.
  * Wang et al. (2023) Wang, L., Zhang, X., Su, H., and Zhu, J.  A comprehensive survey of continual learning: Theory, method and application.  _arXiv preprint arXiv:2302.00487_ , 2023.
  * Ye et al. (2023) Ye, M., Fang, X., Du, B., Yuen, P. C., and Tao, D.  Heterogeneous federated learning: State-of-the-art and research challenges.  _ACM Computing Surveys_ , 56(3):1–44, 2023.
  * Yuan et al. (2023) Yuan, L., Sun, L., Yu, P. S., and Wang, Z.  Decentralized federated learning: A survey and perspective.  _arXiv preprint arXiv:2306.01603_ , 2023.
  * Zhang et al. (2018) Zhang, Y., Xiang, T., Hospedales, T. M., and Lu, H.  Deep mutual learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4320–4328, 2018.
  * Zhou et al. (2021) Zhou, H., Song, L., Chen, J., Zhou, Y., Wang, G., Yuan, J., and Zhang, Q.  Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective.  _arXiv preprint arXiv:2102.00650_ , 2021.

|  |
---|---|---
|  |
|  |
|  |
Figure 8: Data partitions based on IID and non-IID distribution with clients
$N=\\{10,50,100\\}$ for both CIFAR-10 and CIFAR-100 datasets. The size of the
red circle represents the magnitude of data samples for each class label in
each client.

##  Appendix A Dataset Distributions

The data partitions for both IID and non-IID distributions of CIFAR-10 and
CIFAR-100 datasets are illustrated in Figure 8.

##  Appendix B Implementation Details

In our setup, we assume a star topology where all clients can send and receive
models from any other client. Each communication round involves randomly
selecting 50% of the clients as senders $S$, with an additional client
randomly chosen as the aggregator $a$ (unless specified otherwise). We utilize
SGD optimizer for each client with momentum 0.9 and weight decay 5e-4. The
learning rate is selected from $\\{0.1,0.01,0.001\\}$. The batch size is set
to 64. For the cyclic $\alpha$ scheduler, we apply cosine annealing. The
initial oscillating period is set to 10 and is incrementally increased after
each completion. $\alpha$ is oscillated from 0 to a maximum value selected
between $\\{0.8,0.9,1.0\\}$. Figure 9 illustrates an example of the behavior
of $\alpha$ throughout training. The number of mutual learning epochs $K$,
performed at the aggregator, is set to 10. Moreover, the temperature is
configured to 1. All experiments are repeated for 3 trials with random seeds.
With the existence of an $\alpha$ and period schedulers, aggregators need to
be aware of communication round $t$, and the round when the period was last
updated to compute the period and $\alpha$ value. To achieve this, each
aggregator needs to communicate these two values to the next aggregator.

Figure 9: Cyclic oscillation of $\alpha$ with incremental period increase.
$\alpha$ ranges from $0\rightarrow 1$.

The architectures employed in our experiments are presented in Table 7. Modes
$H0$, $H1$, and $H2$ refer to homogeneous, restrictive heterogeneous, and
nonrestrictive heterogeneous architectures, respectively. In mode $H1$, the
model rates for different model types are:
$[2^{0},2^{-1},2^{-2},2^{-3},2^{-4}]$. Here, smaller models are scaled
versions of the largest model in terms of width. Mode $H2$ designates
heterogeneous architectures with no constraints, allowing each client to have
a different number of layers and hidden channel sizes. In our $H1$ and $H2$
experiments, the models are evenly distributed among clients.

For CNNs, the values inside the array represent the number of channels in each
layer, and the array’s length corresponds to the number of layers. Each CNN
layer has $5\times 5$ kernels followed by the ReLU activation function,
$2\times 2$ max pooling, and layer normalization. In ResNets, the largest
model is pre-activated ResNet18, while others are scaled versions of ResNet18.
The values inside the array represent the number of channels per layer, and
each layer consists of two blocks. Similarly, for ViTs, the largest model is
comprised of $2$ layers with $512$ channels each; the others are scaled
versions of the largest ViT. Shifted Patch Tokenization (SPT) and Locality
Self-Attention (LSA) (Lee et al., 2021) are used in our ViT architectures to
solve the lack of locality inductive bias and enable us to use non-pretrained
ViTs on small datasets. Additionally, for ViTs, the patch size is set to
$4\times 4$, head dimensions to 64, depth to 2, dropout to 0.1, and embedding
dropout to 0.1.

Table 7: Different architecture modes. Mode Name | Model
---|---
Heterogeneity | Type | Architectures
$H0$ |  Homogeneous | CNN | ${[}32,64{]}$
$H1$ |  Restrictive Heterogeneous | CNN | ${[}128,256{]}$
${[}64,128{]}$
${[}32,64{]}$
${[}16,32{]}$
${[}8,16{]}$
ResNet | ${[}64,128,256,512{]}$
${[}32,64,128,256{]}$
${[}16,32,64,128{]}$
${[}8,16,32,64{]}$
${[}4,8,16,32{]}$
ViT | ${[}512,512{]}$
${[}256,256{]}$
${[}128,128{]}$
${[}64,64{]}$
${[}32,32{]}$
$H2$ |  Nonrestrictive Heterogeneous | CNN | ${[}32,64,128,256{]}$
${[}32,64,128{]}$
${[}32,64{]}$
${[}16,32,64{]}$
${[}8,16,32,64{]}$

##  Appendix C Baselines

###  C.1 Decentralized FedAvg

When senders send their heterogeneous models to the aggregator, decentralized
FedAvg performs parameter averaging exclusively among models with identical
architectures. In the homogeneous scenario, averaging encompasses all
available models. In contrast, in heterogeneous scenarios, clusters of global
models are formed as identical models in each group are averaged together. The
resulting global models are then communicated back to the clients with the
same model architecture. During parameter averaging, weights are assigned
based on the number of data samples in each client. Algorithm 2 provides a
detailed explanation of decentralized FedAvg. Model parameters in FedAvg are
aggregated as follows:

$W_{g}=\frac{1}{\sum_{n\in\mathcal{P}}d_{n}}\sum_{n\in\mathcal{P}}d_{n}W_{n},$
(4)

where $W_{g}$ is the global model and $W_{n}$ is the model of client $n$. The
weight $d_{n}$ is based on the number of data samples in the client.

Algorithm 2 Decentralized FedAvg

Input: Initialize $N$ clients, each client $n$ has a model $W_{n}$ and data
$D_{n}$. All models with the same architectures have the same initialization.

for communication round $t=1,2,...,T$ do

Randomly select one aggregator $a\in\\{1,...,N\\}$

Randomly select senders $S\subset\\{1,...,N\\}$, $a\notin S$

Participants $\mathcal{P}=S\cup\\{a\\}$

\\\ Client Side

for all $n\in\mathcal{P}$ do

for all batch $X_{n}\in$ local data $D_{n}$ do

$W_{n}\leftarrow$ locally train $W_{n}$ using Equation (2)

Send locally updated models $W_{s}$ for all $s\in S$ to $a$

\\\ Aggregator Side

Each cluster $u\in\mathcal{U}$ contains a group of models of same
architectures.

for all $u\in\mathcal{U}$ do

$W_{g}^{u}\leftarrow$ Aggregate homogeneous models, for all $W_{n}\in u$,
according to Equation (4)

for all $u\in\mathcal{U}$ do

for all $W_{n}\in u$ do

$W_{n}\leftarrow W_{g}^{u}$ \\\ Fork $W_{g}^{u}$ into local models

Send back updated models $W_{s}$ for all $s\in S$

###  C.2 Decentralized Partial Training

Similar to DFML and decentralized FedAvg, in decentralized partial training
methods each client owns a local model, and in each communication round
several clients are randomly selected. One client is designated as the
aggregator, while the others act as senders. The senders transmit their models
to the aggregator after training their models locally. At the aggregator, the
largest available model serves as the global model $W_{g}$. Algorithm 3
provides a detailed description of decentralized Federated Dropout, HeteroFL,
and FedRolex. Model parameters in partial training methods are aggregated as
follows:

$W_{g,[i,j]}=\frac{1}{\sum_{n\in\mathcal{P}}d_{n}}\sum_{n\in\mathcal{P}}d_{n}W_{n,[i,j]},$
(5)

where $W_{[i,j]}$ is the $j^{th}$ parameter at layer $i$ of global model
$W_{g}$, while $W_{n,[i,j]}$ is the $j^{th}$ parameter at layer $i$ of client
$n$. The client weight is equal for all clients, $d_{n}=1/|\mathcal{P}|$.

####  C.2.1 Decentralized Federated Dropout

In each communication round $t$ within Federated Dropout (Caldas et al.,
2018), sub-models (local models) are extracted from the centralized $W_{g}$
based on a random selection process. The parameters representing the sub-
models that are randomly chosen through this selection are then transmitted to
clients for local training. After local training, the updated parameters are
sent back to $W_{g}$ for aggregation. The random extraction scheme for layer
$i$ of sub-model $W_{n}$ for client $n$ is extracted from $W_{g}$ as follows:

$\mathcal{X}_{n,i}=\\{j_{s}\;|\;\text{integer}\;j_{s}\in[0,J_{i}-1]\;\text{for}\;1\leq
s\leq\lfloor r_{n}J_{i}\rfloor\\},$ (6)

where $\mathcal{X}_{n,i}$ is the parameter indices of layer $i$ extracted from
$W_{g}$. $r_{n}$ denotes $W_{n}$ rate relative to $W_{g}$. $J_{i}$ denotes the
total number of parameters in layer $i$ of $W_{g}$. A total of $\lfloor
r_{n}J_{i}\rfloor$ is randomly selected from layer $i$ of $W_{g}$ for $W_{n}$.

Our derived decentralized version of Federated Dropout involves generating
random indices (Equation (6)) at the aggregator, guided by the largest
available model ($W_{g}$). Subsequently, each model is assigned a random set
of indices equivalent to its size. The aggregation process is then carried out
using these randomly selected indices. Once the aggregation is finalized, sub-
models are created from $W_{g}$ using the same set of indices. Finally, these
sub-models are transmitted back to the respective participating clients.

####  C.2.2 Decentralized HeteroFL

Unlike Federated Dropout, HeteroFL (Diao et al., 2020) consistently extracts
sub-models from a predefined section of $W_{g}$. Specifically, HeteroFL
extracts sub-models from $W_{g}$ starting from index 0 up to the maximum layer
size of $W_{n}$. The extraction scheme is defined as follows:

$\mathcal{X}_{n,i}=\\{0,1,...\lfloor r_{n}J_{i}\rfloor-1\\},$ (7)

In the decentralized HeteroFL approach, the parameters of each layer in all
sub-models share the same starting point (index 0). Consequently, at the
aggregator, parameter averaging takes place with overlapping indices from the
available models. After aggregation, the updated parameters of each layer from
all models, spanning from index 0 up to the maximum size, are communicated
back to their respective clients.

####  C.2.3 Decentralized FedRolex

In FedRolex (Alam et al., 2022), local clients are initially generated from
the global model beginning from index 0 and extending up to the local models’
capacity. In the first communication round, the sub-models are generated
similarly to HeteroFL. However, in each subsequent communication round, the
starting point of the indices shifts to the right. The sub-model extraction in
FedRolex is defined as follows:

$\mathcal{X}^{t}_{n,i}=\begin{cases}\\{\tilde{t},\tilde{t}+1,...,\tilde{t}+\lfloor
r_{n}J_{i}\rfloor-1\\}\;&\text{if $\tilde{t}+\lfloor r_{n}J_{i}\rfloor\leq
J_{i}$, }\\\
\\{\tilde{t},\tilde{t}+1,...,J_{i}-1\\}\cup\\{0,1,...,\tilde{t}+\lfloor
r_{n}J_{i}\rfloor-1-J_{i}\\}&\text{otherwise},\\\ \end{cases}$ (8)

where $\tilde{t}=t\mod J_{i}$. $t$ is the current communication round and
$J_{i}$ is the size of layer $i$ of $W_{g}$.

In decentralized FedRolex, the size of $W_{g}$ is determined by the largest
available model, and as a result, the rightward shift in indices is computed
based on the current communication round $t$ and $J_{i}$ of the selected
$W_{g}$. The indices are calculated using $\mathcal{X}^{t}_{n,i}$ from
Equation (8). These indices are utilized for aggregation and to extract the
updated local models after aggregation. In decentralized FedRolex, since
aggregators must be aware of $t$ to compute the indices, each aggregator needs
to communicate $t$ to the next aggregator.

Algorithm 3 Decentralized Federated Dropout, HeteroFL, and FedRolex

Input: Initialize $N$ clients, each client $n$ has a model $W_{n}$ and data
$D_{n}$. Each model has a rate $r_{n}$, which is the model’s size rate
compared to the largest model in the network. Models with the same rate have
the same initialization.

for communication round $t=1,2,...,T$ do

Randomly select one aggregator $a\in\\{1,...,N\\}$

Randomly select senders $S\subset\\{1,...,N\\}$, $a\notin S$

Participants $\mathcal{P}=S\cup\\{a\\}$

\\\ Client Side

for all $n\in\mathcal{P}$ do

for all batch $X_{n}\in$ local data $D_{n}$ do

$W_{n}\leftarrow$ locally train $W_{n}$ using Equation (2)

Send locally updated models $W_{s}$ for all $s\in S$ to $a$

\\\ Aggregator Side

Set $W_{g}$ to be like the largest available model

Local models are assigned indices $\mathcal{X}_{n,i}$ for all $i$ and
$n\in\mathcal{P}$, where $\mathcal{X}_{n,i}$ is from Equation (6) or Equation
(7) or Equation (8)

for all $n\in\mathcal{P}$ do

$W_{n}\leftarrow W_{g,\mathcal{X}_{n,i}}$ for all $i$ \\\ Split $W_{g}$ into
local models

Send back updated models $W_{s}$ for all $s\in S$

###  C.3 Decentralized FML

FML (Shen et al., 2020) is a centralized framework that relies on a central
server. In FML, each client owns a local model, which is not transmitted
during training. In each communication round, the global model forks its model
into all participating clients. Shen et al. 2020 named the forked models: meme
models. Subsequently, at each client, the local (heterogeneous) model engages
in mutual learning with the meme (homogeneous) model using their respective
local data. After mutual learning is complete, the meme models are
communicated back to the server for aggregation.

In decentralized FML, two models are dedicated to each client: the first is
the heterogeneous model, and the second is the homogeneous (meme) model. In
each communication round, several clients perform mutual learning between
their heterogeneous and homogeneous models using their local data. Next, the
homogeneous models from all participating clients are transmitted to the
aggregator for aggregation. After aggregation is complete, the aggregated
model is transmitted back to all participating clients. This process repeats
for the remaining communication rounds.

We decentralized FML for comparison with our proposed DFML. It is crucial to
emphasize that decentralized FML and our proposed DFML are two distinct
frameworks. The key differences are as follows: 1) DFML uses one heterogeneous
model per client for training, while decentralized FML uses two models per
client for training; 2) DFML aims to transfer global knowledge to
heterogeneous models, whereas FML treats the heterogeneous models as
personalized models and the homogeneous models to hold the global knowledge;
and 3) FML requires a server, and we derived the decentralized version of FML
to facilitate comparison with our proposed DFML.

##  Appendix D DFML: Further Analysis

###  D.1 Convergence Speedup

We illustrate in Figures 10, 11, and 12 the convergence speedup achieved by
DFML compared to the baselines under three heterogeneity settings: homogeneous
architectures, restrictive heterogeneous architectures, and nonrestrictive
heterogeneous architectures; respectively.

|  |  |
---|---|---|---
Figure 10: Comparison between DFML, decentralized FedAvg, and Def-KT using homogeneous architectures. |  |  |
---|---|---|---
Figure 11: Comparison between DFML, decentralized partial training algorithms, and decentralized FedAvg using restrictive heterogeneous architectures. |  |  |
---|---|---|---
|  |  |
|  |  |
Figure 12: Comparison between DFML and decentralized FedAvg for different
numbers of clients and data distributions using non-restrictive heterogeneous
architectures.

###  D.2 Weighted vs Vanilla Average of KL Divergences (KLs)

In Equation (3), we use a weighted average of KL divergences (KLs) between all
teacher models and the student. The weighting is determined based on the
number of trainable parameters in each teacher model. Figure 15 demonstrates
that the weighted average leads to better convergence speed and global
accuracy compared to vanilla averaging. The reason is that larger models tend
to have a higher probability of possessing finer knowledge, thus giving them
more weight during knowledge distillation results in better knowledge
transfer.

###  D.3 Effect of Increasing Mutual Learning Epochs $\boldsymbol{K}$

Figure 15 shows that increasing the number of mutual learning epochs $K$ at
each aggregator contributes to a faster convergence speed. A higher number of
$K$ enables more knowledge to be distilled from the experts, leading to
improved convergence speed. This, in turn, results in a more efficient
communication cost throughout training.

Figure 13: Weighted vs vanilla average of KL divergences (KLs) in the
distillation signal of the objective function.

Figure 14: Effect of increasing mutual learning epochs $K$.

Figure 15: Evaluating DFML against decentralized with fewer number of senders
$S$.

###  D.4 Different Number of Participants

We investigate the performance of DFML with a varying number of senders $S$ in
each communication round. Figure 15 compares DFML and decentralized FedAvg
with 50%, 20%, and 10% senders. DFML exhibits effective learning even with
fewer participants compared to decentralized FedAvg. Additionally, both
methods with a reduced number of participants demonstrate slower convergence
speed compared to the 50% scenario, which is expected.

Due to the low participation rate, we increase the number of peak updates.
With a limited number of participating models, updating them only when the
maximum $\alpha$ is reached results in slower convergence speeds. Therefore,
we allow multiple updates instead of updating the models solely at the maximum
$\alpha$. We estimate that an appropriate number of peak updates is
$\frac{N}{|S|}$ for $|S|<50\%$ of the clients. Consequently, with $|S|=10\%$,
updates are applied in the largest five $\alpha$ values. If cyclical $\alpha$
is not added to DFML, adjusting the number of peak updates is unnecessary, as
peak models will no longer be needed and only the updated models must be
communicated back to senders.

|  |  |
---|---|---|---
Figure 16: Comparison between our proposed DFML and decentralized FML.

###  D.5 DFML vs Decentralized FML

In decentralized FML, the clients’ heterogeneous models are the same as the
models used in our proposed DFML, which are based on CNN architectures and
mode $H2$ from Table 7. The homogeneous model size used is $[32,64]$, which is
the median of models of the five different architectures. Figure 16 compares
our DFML and decentralized FML. We include the global performance of the local
heterogeneous and the homogeneous meme models. We observe that the homogeneous
meme models have a better convergence speed than our DFML, and in some cases,
lead to better final accuracy. However, the global knowledge performance of
heterogeneous models in decentralized FML is much worse than DFML and is even
lower than decentralized FedAvg. As our goal is to transfer global knowledge
to clients’ heterogeneous models, we consider that our DFML significantly
outperforms the decentralized FML framework.

##  Appendix E Cyclical Knowledge Distillation: Further Analysis

###  E.1 Cyclical Knowledge Distillation

In Figure 17, we illustrate the impact of varying the value of $\alpha$ on
global knowledge throughout training with full-cycle oscillations. Compared to
Figure 5, it is evident that using a half-cycle oscillation results in better
convergence speed. With half-cycle oscillations, less time is spent in the
stage where the supervision signal is dominant, leading to less loss of global
knowledge compared to full-cycle oscillations. When the distillation signal
becomes dominant again, the recovery from the lost global knowledge is faster.

Figure 17: DFML performance against varying $\alpha$. The regular models are
evaluated instead of the peak models and $\alpha$ oscillates with full cycles.
The supervision signal used is CE.

Figure 18: Comparison between different supervision signals.

Figure 19: Comparing various adaptive techniques with our proposed DFML. The
supervision signal used is CE.

Figure 20: Different schedulers assigned to the loss components in the
objective function.

###  E.2 Different Supervision Signals

As previously mentioned, mitigating catastrophic forgetting is crucial in
applications such as FL where data distribution shift exists across clients.
Training models on one dataset can lead to optimizing its objective towards
that specific local data, causing it to forget tasks learned previously from
other clients. ACE (Caccia et al., 2021) and WSM (Legate et al., 2023) are two
approaches to mitigate catastrophic forgetting. Figure 20 illustrates the
improvement in global accuracy when the supervision signal is changed from CE
to ACE or WSM. Results show that WSM as a supervision signal leads to the
highest global accuracy, as it takes into account the number of samples for
each class label in the clients. The maximum range of $\alpha$ oscillation is
tuned for each supervision signal to achieve the best final accuracy. Tuning
the maximum $\alpha$ range is important, particularly in scenarios where the
supervisory signal is non-noisy, such as in IID distribution shift, or when
using ACE or WSM in non-IID settings. Completely diminishing the supervisory
signal (equivalent to setting $\alpha=1$) in such cases would lead to a
performance decline. Therefore, in situation where the supervision signal is
not noisy, the maximum $\alpha$ value is better to be set to 0.8 or 0.9. For
instance, in non-IID cases with CE as the supervision signal, where the signal
is very noisy, setting $\alpha=1$ yields the best performance.

###  E.3 Different Adaptive Techniques

In Figure 20, we compare the performance of different adaptive techniques,
including WSL (Zhou et al., 2021) and ANL-KD (Clark et al., 2019), with our
proposed cyclical DFML framework. The results indicate that WSL and ANL-KD
show negligible improvement compared to DFML with a fixed $\alpha=0.5$.

Figure 21: Comparison between fixing and increasing the period throughout
training. The initial period is set to 10.

###  E.4 Different Schedulers for Loss Components

In Equation (1), the objective function consists of two loss components:
supervision and distillation loss signals. In Equation (9), we examine the
impact of independently varying each loss component on global performance.
Specifically, we compare additional scenarios where one scheduler is assigned
to $\mathcal{L}_{\text{CE}}$ alone, another scenario where a scheduler is
assigned to $\mathcal{L}_{\text{KL}}$ alone, and a third scenario where one
scheduler scales both loss components with the same factor. For this
experiment, we use CE as the supervision signal, as the improvements are more
notable under the non-IID distribution shifts. Figure 20 illustrates the
comparison of assigning independent schedulers to loss components. In this
experiment, $\gamma$ oscillates from $1\rightarrow 0$, and $\alpha$ oscillates
from $0\rightarrow 1$.

$\mathcal{L}=\gamma\mathcal{L}_{\text{CE}}+\alpha\mathcal{L}_{\text{KL}}$ (9)

Scaling the distillation signal alone and leaving the scale of
$\mathcal{L}_{\text{CE}}$ untouched does not yield any advancement in
performance compared to keeping $\alpha$ fixed for both loss components. This
indicates that the supervision signal has a more significant impact than the
distillation signal. Performance gains are observed when the
$\mathcal{L}_{\text{CE}}$ signal is reduced, allowing more influence on the
$\mathcal{L}_{\text{KL}}$ signal. In contrast, when $\mathcal{L}_{\text{CE}}$
oscillates while the $\mathcal{L}_{\text{KL}}$ scale is kept fixed, it results
in the same performance as when both $\mathcal{L}_{\text{CE}}$ and
$\mathcal{L}_{\text{KL}}$ signals are scaled in opposite directions (Equation
(1)). This is because the $\mathcal{L}_{\text{CE}}$ signal is dominant without
any scaling, and as it diminishes it allows the $\mathcal{L}_{\text{KL}}$
signal to take precedence. The peak value is attained when the
$\mathcal{L}_{\text{CE}}$ signal reaches $0$, and the
$\mathcal{L}_{\text{KL}}$ signal’s scale is $1$. Finally, scaling both
$\mathcal{L}_{\text{CE}}$ and $\mathcal{L}_{\text{KL}}$ signals with a common
scheduler leads to inferior performance compared to scaling
$\mathcal{L}_{\text{CE}}$ alone or scaling both signals in opposite
directions. The poor performance of oscillating $\mathcal{L}_{\text{KL}}$
signal alone is attributed to the continuous dominance of
$\mathcal{L}_{\text{CE}}$ signal during mutual learning, causing the models to
drift toward the aggregator’s local data.

###  E.5 Fixed vs Increasing Period

Figure 21 demonstrates that increasing the period over time results in better
convergence speed, higher global accuracy, and enhanced stability. The period
is initially set to 10. In the fixed period experiment, the period is kept
constant, while in the increasing period experiment, the period is
incremented. Starting with a small period is crucial for more frequent peak
updates, which accelerates convergence speed. However, over time, increasing
the period proves beneficial, allowing models to transition from the
supervision to the distillation signal more slowly. This extended time in the
distillation-dominant region enhances global accuracy. For instance, if all
clients are participating and the period is initially set at $100$, better
accuracy is achieved after $100$ communication rounds compared to a constant
period of $10$. However, the convergence speed is notably affected, as a
period of $100$, results in one peak at communication round $100$, while a
period of $10$ leads to $10$ peaks. Further, in scenarios with partial
participation, then after $100$ rounds only the participating clients will be
updated. Whereas a smaller initial period ensures that, on average, all
clients are updated several times within the first $100$ rounds. Therefore, to
reap the benefits of high convergence speed and improved final accuracy, we
set the period to be initially small and increment it gradually.


# Model-based Optimization of Superconducting Qubit Readout

Andreas Bengtsson  Alex Opremcak  Mostafa Khezri  Daniel Sank  Alexandre
Bourassa  Kevin J. Satzinger  Sabrina Hong  Catherine Erickson  Brian J.
Lester  Kevin C. Miao  Google Quantum AI, Santa Barbara, CA  Alexander N.
Korotkov  Google Quantum AI, Santa Barbara, CA  Department of Electrical and
Computer Engineering, University of California, Riverside, CA  Julian Kelly
Zijun Chen  Paul V. Klimov  Google Quantum AI, Santa Barbara, CA

###### Abstract

Measurement is an essential component of quantum algorithms, and for
superconducting qubits it is often the most error prone. Here, we demonstrate
model-based readout optimization achieving low measurement errors while
avoiding detrimental side-effects. For simultaneous and mid-circuit
measurements across 17 qubits, we observe 1.5% error per qubit with a
$500\text{\,}\mathrm{n}\mathrm{s}$ end-to-end duration and minimal excess
reset error from residual resonator photons. We also suppress measurement-
induced state transitions achieving a leakage rate limited by natural heating.
This technique can scale to hundreds of qubits and be used to enhance the
performance of error-correcting codes and near-term applications.

Superconducting qubits have achieved measurement errors below 1% for single
qubits [1, 2, 3, 4] thanks to advancements including dispersive readout [5]
and quantum-limited parametric amplifiers [6]. However, the increasing scale
and complexity of algorithms bring a new set of challenges beyond simple
single-qubit measurements. For example, quantum error correction requires
measurements to be performed simultaneously, in the middle of circuits, and in
a repetitive fashion. Mid-circuit measurements must be fast to avoid
decohering qubits not being measured, but faster readout can increase
measurement and leakage errors. In recent error-correction experiments with
superconducting qubits, readout-induced leakage severely limited performance
[7, 8], and in another, 20% of the total error (twice that of the measurement
itself) was due to qubit idling during measurement [9].

Typically, readout is calibrated in-situ, meaning that control parameters like
pulse amplitude and frequency are varied until the observed measurement error
is minimized. While effective at minimizing measurement error for isolated
qubits, it can fail to capture other destructive processes like leakage or
residual resonator photons. Additionally, effects such as qubit-qubit coupling
impose non-locality in that the optimal values for one qubit depend on the
values of neighboring qubits. In turn, qubits optimized in isolation tend to
perform poorly when measured simultaneously. Thus, optimizing multi-qubit
measurement requires searching a parameter space where the dimension scales
linearly with the number of qubits, while attempting to minimize many metrics
at once. This task rapidly becomes intractable by in-situ parameter sweeps as
the number of qubits grows.

In this Letter, we present an ex-situ (model based) optimization technique for
readout parameters. Ex-situ optimization which allows us to explore a larger
parameter space and minimize errors that are difficult or costly to measure,
compared to in-situ optimization where the speed is limited by the data rate
of the quantum processor. In designing our model-based approach, we tackle
three challenges which are often seen in quantum optimal control [10]. First,
if the models do not accurately capture the present error channels, ex-situ
optimization is likely to perform worse than in-situ. Second, the models must
be evaluated quickly to be able to actually explore a larger space. These two
challenges are conflicting in that more accurate models typically result in
slower evaluation; for instance, a quantum simulation of the system dynamics
would be too slow. Third, we must use an optimization algorithm that can find
a reasonably good minimum without requiring a prohibitively long runtime.

We begin by describing representative models for readout error channels
relevant to a Sycamore processor [11], which consists of superconducting
frequency-tunable transmon qubits, each coupled to their own readout
resonator. The models are quick to evaluate and we demonstrate that they
accurately predict a variety of metrics over a wide range of parameters. We
then use them together with the snake optimizer [12] to minimize the errors
for 17 qubits in a distance-3 surface code. We achieve 1.5% measurement error
per qubit in $500\text{\,}\mathrm{n}\mathrm{s}$ (from the start of the readout
until the system is ready for the next operation), while also reducing any
additional errors like reset and leakage.

Figure 1:  (a) The optimization workflow. We build error models from fixed
parameters like resonator frequency and linewidth, which we then run an
optimizer on to find a set of variable parameters (e.g. pulse amplitude and
length) that gives low errors. The output of the optimizer is a unique pulse
shape for each qubit, as well as qubit frequency (not shown). (b) Examples of
readout error mechanisms. The middle qubit, which is in $|1\rangle$, is
coupled to two other qubits. During readout of the middle qubit its excitation
can relax to $|0\rangle$; swap into $|1\rangle$ of the right qubit; or combine
with the excitation in the left qubit to $|2\rangle$. Additionally, photons in
the readout resonator can excite the qubit up to a high state. (c) Number of
photons in the readout resonator as a function of time, both simulated (solid
line) and measured (circles). The inset shows the simulated values on a
logarithmic scale. Residual photons can cause reset and dephasing errors.

The models fall into two categories: predictive or heuristic. Ideally, we
would only have predictive models (models that accurately predict error
rates), but this is not always feasible as the computation might be too
inaccurate or take too long. In those cases we use heuristic models to steer
the optimizer away from parameter regions where errors are large, but
difficult to quantify. We use predictive models for the signal-to-noise ratio
(SNR), qubit relaxation during readout, and residual resonator photons.
Heuristic models include measurement-induced state transitions [13, 14], and
coupling to neighboring qubits. We sum all models into a single optimization
cost function. The process is illustrated in Fig. 1 (a).

Each model takes one or several input parameters describing the properties of
the qubits, their readout resonators, or the measurement system itself. In
total there are seven such parameters,

  * •

Qubit anharmonicity, $\alpha<0$

  * •

Resonator-qubit coupling, $g(\omega_{q})$

  * •

Bare resonator frequency, $\omega_{r}$

  * •

Measurement efficiency, $\eta$ [5]

  * •

Resonator linewidth, $\kappa$

  * •

Qubit relaxation rate as a function of frequency, $\Gamma_{1}(\omega_{q})$

  * •

A calibrated reference for the readout pulse power at the processor

We characterize these using a suite of metrology experiments [15]. Most of
them are static and characterized just once. However, the qubit relaxation
time is known to fluctuate [16] and is therefore remeasured just before
optimization.

Additionally, there are four parameters which we can tune and optimize,

  * •

Qubit frequency during readout, $\omega_{q}$

  * •

Readout pulse amplitude, $B_{0}$

  * •

Readout pulse length, $t_{p}$

  * •

Readout ringdown length, $t_{r}$

The ringdown length is needed for mid-circuit measurements to allow the
resonator to decay back to its ground state before other operations can
resume. We choose to use a fixed total readout time
($t_{p}+t_{r}=$500\text{\,}\mathrm{n}\mathrm{s}$$), allowing us to synchronize
gates, as well as reduce the number of optimization parameters to three.

A key parameter derived from the model inputs is the separation between
resonator frequencies for the states $|0\rangle$ and $|1\rangle$, i.e. the
dispersive shift $2\chi(\omega_{q})$, given by [17],

$\chi(\omega_{q})=\frac{g(\omega_{q})^{2}\alpha}{(\omega_{q}-\omega_{r})^{2}(1+\alpha/(\omega_{q}-\omega_{r}))}\left(1-\frac{\omega_{q}-\omega_{r}}{\omega_{q}}\right).$
(1)

The shift is tuneable since it depends on the qubit frequency; absent other
constraints, the SNR per measurement photon is maximized when $2\chi=\kappa$.

A second derived parameter is the field in the resonator as a function of
time, $\beta(t)$ which is found by solving

$\frac{d\beta}{dt}=\sqrt{\kappa}B(t)+(i\Delta-\kappa/2)\beta(t),$ (2)

where $B(t)$ is the readout drive amplitude and has the dimension
$\sqrt{\mathrm{photons~{}per~{}time}}$, and $\Delta$ is the frequency
difference between the drive and the dressed resonator. In the rest of this
paper we restrict the drive to be in the center of the resonator frequencies
corresponding to $|0\rangle$ and $|1\rangle$, i.e. $\Delta=\pm\chi$, since
that yields the highest SNR in the parameter regime we are interested in. For
both states, we find the corresponding $\beta_{|0\rangle}(t)$ and
$\beta_{|1\rangle}(t)$ by numerically solving Eq. 2.

The applied readout pulse, together with the noise (assumed to be Gaussian) in
the readout chain, leads to a certain SNR, from which we derive the
corresponding probability to misidentify the state. Given
${\delta\beta(t)=\beta_{|0\rangle}(t)-\beta_{|1\rangle}(t)}$, the SNR is
calculated as

$\text{SNR}=2\eta\kappa\frac{\left|\int_{0}^{t_{p}+t_{r}}\delta\beta(t)w(t)dt\right|^{2}}{\int_{0}^{t_{p}+t_{r}}|w(t)|^{2}},$
(3)

where $w(t)$ is an integration window function, which we set to
$\delta\beta(t)^{*}$ to maximize SNR. From SNR we derive the corresponding
error,

$\epsilon_{\text{separation}}=\frac{1}{2}\text{erfc}\left(\frac{\sqrt{\text{SNR}}}{2}\right).$
(4)

During readout the qubit might decay and potentially cause a measurement
error. To calculate the error rate we need the qubit frequency during readout,
which is changing throughout the process due to the AC-Stark effect, and the
corresponding relaxation rates at those frequencies. The latter is measured by
a standard relaxation experiment versus qubit frequency; while the former can
be found via Eqs. 1 and 2,

$\omega_{q}(t)=\omega_{q}(0)+2|\beta_{|1\rangle}(t)|^{2}\chi(\omega_{q}(0)),$
(5)

where we have assumed that the AC-Stark shift is strictly linear. Given
$\Gamma_{1}(\omega_{q})$, we calculate the relaxation error as

$\epsilon_{\text{relaxation}}=\int_{0}^{T_{0}}\Gamma_{1}(\omega_{q}(t))\mathrm{d}t.$
(6)

We approximate $T_{0}$ to be the point where SNR is half of its maximum, since
a relaxation event beyond that point should not change the measurement
outcome. We also choose to ignore the upward transition rate
$|0\rangle\rightarrow|1\rangle$, since the timescale for that process is much
longer than for relaxation and the readout.

The resonator must be mostly depleted of photons before the next operation can
begin, since any remaining photons cause qubit dephasing. Additionally, in our
architecture such photons directly translate into reset errors since it is
based on the swap interaction between the qubit and its resonator [18], and a
photon in the resonator could be swapped into either $|1\rangle$ or
$|2\rangle$, depending on the state. We use the mean photon number in the
resonator at the end of the readout,

$\epsilon_{\text{photon}}=\frac{|\beta_{|0\rangle}(T)|^{2}+|\beta_{|1\rangle}(T)|^{2}}{2},$
(7)

as the error model to minimize both the reset error and qubit dephasing.

Shown in Fig. 1(c) are the expected photon number,
$|\beta_{|0\rangle}(t)|^{2}$, and the corresponding measured values (via
spectroscopic measurements of the AC-Stark shift [14]). The measurement
technique is not sensitive to the small frequency shifts occurring at the low
photon numbers towards the end of readout; however, since the agreement is
good during the pulse itself we can use the model to infer what the final
photon number is, which in this example is 0.005.

Figure 2: Error models and their dependence on readout parameters. (a)
Separation and relaxation errors, and residual photon number, versus the qubit
frequency with amplitude and length kept fixed. Circles are measured data, and
lines are simulated values. As the qubit frequency changes, we track the
readout pulse frequency to be centered between the two dressed resonator
states. The peak at $6\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$ is due to a
two-level system defect. (b) and (c) show the same models, but versus pulse
amplitude and length, respectively. The total readout time is kept constant,
such that when the pulse length increases, the ringdown time decreases. The
non-swept parameters are kept fixed at the values indicated by the dashed
vertical lines in the respective panels.

In Fig. 2, we show predicted and measured values for
$\epsilon_{\text{separation}}$ and $\epsilon_{\text{relaxation}}$ as a
function of qubit frequency, pulse amplitude, and pulse length. We
additionally show the predicted residual photon number, though we do not have
a sensitive enough technique to reliably measure this quantity at the modeled
levels. Overall, we see good agreement between measured and simulated values,
with the exception for $\epsilon_{\text{relaxation}}$ at low amplitudes and
lengths where $\epsilon_{\text{separation}}$ is large. That parameter regime
should be avoided and accurately predicting $\epsilon_{\text{relaxation}}$
there is less important. The cause of the discrepancy could be due to the
approximations in Eq. 6 or from the inaccuracy in trying to extract a small
error on top of a large error.

We can understand the tradeoffs in readout optimization by studying the
predictive models. If we only consider these three models, the ideal pulse
would be short and with high amplitude, since the SNR approximately scales
quadratically with the amplitude [Fig. 2 (b)] and linearly with the pulse
length [Fig. 2 (c)], while $\epsilon_{\text{photon}}$ also scales
quadratically with the amplitude, but exponentially with the pulse length (for
a fixed total time). Additionally, a short readout pulse minimizes
$\epsilon_{\text{relaxation}}$.

However, a high pulse amplitude can be problematic for several reasons. For
example, it leads to measurement-induced state transitions [13, 14], which
occurs when resonator photons are transferred to the qubit and excite it far
beyond the computational subspace, as illustrated in Fig. 1 (b). While this
high state may lead to a measurement error, it is more importantly immune to
our reset protocol [18], making it particularly destructive for mid-circuit
measurements and quantum error correction [7, 8]. Using the model in Ref.
[14], valid only for $\omega_{q}>\omega_{r}$, we define a heuristic that
constrains the maximum photon number in the resonator,

$\text{max}(|\beta(t)|^{2})<ae^{b(\omega_{q}-\omega_{r})}-\sqrt{ae^{b(\omega_{q}-\omega_{r})}},$
(8)

where $a$ and $b$ are extracted from numerical simulations and only dependent
on $\alpha$ and $g$ [14].

Finally, we introduce a model related to the coupling between qubits. Our
qubits are laid out on a square grid where a pair of qubits have four relevant
coupling channels, $|01\rangle\leftrightarrow|10\rangle$,
$|11\rangle\leftrightarrow|20\rangle$, $|11\rangle\leftrightarrow|02\rangle$,
and $|12\rangle\leftrightarrow|21\rangle$. We heuristically model the errors
associated with these channels using a sum of Lorentizians,

$\epsilon_{\text{coupling}}=\sum_{i}c_{i}\frac{\gamma_{i}}{2}\frac{\pi}{(\omega_{q}-\omega_{i})^{2}+\gamma_{i}^{2}/4},$
(9)

where $c_{i}$, $\gamma_{i}$ and $\omega_{i}$ are the amplitude, width, and
center frequency of each transition. We use a heuristic to avoid having to
model the time dependence of the qubit frequency and its effect on the
measurement errors. That dependence is complicated by the AC-Stark effect,
which imposes both a frequency shift due to mean number of photons in the
resonator [Eq. 5], as well as frequency broadening due to photon number
fluctuations. By using wide and large enough Lorentzians the optimizer avoids
any qubit-qubit interactions. Assuming couplings between nearest and next-
nearest neighbors there are up to 32 frequency collisions for each qubit.

We now continue to the actual optimization. While any global optimizer can be
used, we choose to employ the snake optimizer [12], which has successfully
optimized single and two-qubit gate parameters for a variety of quantum
algorithms [11, 9, 19]. More optimization details are found in Ref. [20]. As
our experimental platform we use 17 qubits in a distance-3 surface code
layout, illustrated in Fig. 3(a). The optimization takes 1 minute and includes
1.7 million evaluations of the cost function. Afterwards, the resulting
parameters are uploaded to the control system, and the only remaining
calibration is to find the discrimination line to distinguish between
$|0\rangle$ and $|1\rangle$ for each qubit. We choose to not model this since
we can efficiently measure it simultaneously across all qubits, and it does
not conflict with the other parameter choices.

We compare three different optimization strategies to evaluate the performance
of our model-based approach. The first strategy is in-situ optimization where
we choose a fixed pulse length ($300\text{\,}\mathrm{n}\mathrm{s}$) and
perform a sequence of 1D sweeps to find the optimal pulse frequency,
amplitude, qubit frequency, and integration window. The second strategy is ex-
situ optimization using a partial cost function consisting of only the
predictive models, i.e. no qubit-qubit coupling or measurement-induced state
transitions models. The third strategy is ex-situ optimization using a
complete cost function consisting of all available models. For each strategy,
we quantify three important aspects: measurement errors, reset errors, and
leakage. Note that we do not benchmark the performance of optimizer itself,
e.g. how well it finds the actual global minimum. The performance aspects of
the snake have been recently studied in Ref. [21].

Figure 3: Benchmarking of the optimized readout performance. We compare three
optimization strategies: in-situ; ex-situ with only predictive models; ex-situ
with predictive and heuristic models. (a) The distance-3 surface code with 9
data qubits (yellow) and 8 measure qubits (blue), used for the benchmarking.
(b) Simultaneous measurement errors for two cases: all qubits, only measure
qubits. We prepare a set of random states across the qubits and perform
simultaneous measurements. The data shows the combination of the two cases.
(c) Reset error added by a preceding measurement, benchmarked on the measure
qubits. The excess reset error is caused by residual photons in the readout
resonator. (d) Average leakage probability in the measure qubits after
preparing $|1\rangle$ and performing $N$ measurements. The dashed line shows
the heating limit where the measurements are replaced by an equivalent amount
of waiting time.

We benchmark measurement errors by preparing 200 random initial states over
all qubits and then sampling 2,000 measurement outcomes for each initial
state. We then repeat the procedure, but this time using only the measure
qubits to mimic the surface code mid-circuit measurements. We compare all
outcomes with the known initial states and extract the errors, seen in Fig. 3
(b), and calculate the measurement error as $\left(P(1|0)+P(0|1)\right)/2$,
where $P(f|i)$ is the probability of preparing $|i\rangle$ and measuring
$|f\rangle$. Note that state preparation errors will show up as measurement
errors in this protocol. The complete ex-situ optimizer achieves an average
measurement error of 1.5% per qubit, while in-situ and partial ex-situ
optimization achieve 1.9% and 4.7%, respectively. Overall, in-situ and
complete ex-situ optimization have similar performance with the exception of a
few high-error outliers for the in-situ optimizer. For instance, the largest
outlier is caused by $|11\rangle\leftrightarrow|02\rangle$ swapping between
two neighboring qubits, which the ex-situ optimizer is able to avoid [20]. Out
of the 1.5% error per qubit, we are able to account for 1.2% when we include
the contributions from state preparation, separation error, and relaxation
error [20]. We estimate that the state preparation error is 0.4%, which, if
accurate, should be subtracted from the values above. However, since state
preparation errors should affect all three strategies the same we have chosen
to be conservative and not do the subtraction.

Next, we benchmark reset errors added by readout for the measure qubits only
(data qubits do not need reset). We prepare $|1\rangle$, perform measurements
immediately followed by reset and another round of measurements. In the case
of no errors we expect the second measurement to yield $|0\rangle$. We also
perform the same sequence but without the first round of measurements and
subtract that to remove the intrinsic reset and measurement errors. The
results are shown in Fig. 3(c). Complete ex-situ optimization adds on average
an additional 0.4% of reset errors, compared to 4.7% and 1.4% for partial and
in-situ, respectively.

For the final benchmark, we quantify qubit leakage. Again, we focus on the
measure qubits and prepare $|1\rangle$ as that makes the qubits more likely to
leak, and then perform a variable number of measurements. We append a final
and different measurement that is able to discriminate if the qubit has left
the computational subspace. Figure 3(d) shows the probability of leakage as a
function of the number of measurements. Complete ex-situ optimization
suppresses leakage down to an average of 0.8% after 20 measurement rounds,
comparable to the heating limit as measured by repeating the experiment with
no readout pulses but an equivalent amount of waiting time. After the same
number of rounds, the partial and in-situ strategies have leakage populations
of 5.7% and 2.2%, respectively.

Comparing the optimization strategies, we see that complete ex-situ using both
predictive and heuristic models outperforms the others in all three
benchmarks. It is able to achieve lower measurement errors, while also adding
less reset and leakage errors for mid-circuit measurements. The partial ex-
situ optimizer generally performs worse than in-situ optimization. This is
likely due to the lack of an amplitude limiting model, which tends to drive
the optimizer towards short and high-amplitude pulses, which in turn leads to
state transitions. This emphasizes that for model-based optimization to work
well, the models have to account for all dominant error mechanisms, even if
only as heuristics.

In conclusion, we demonstrated model-based optimization for superconducting
qubit readout achieving low measurement errors (1.5%) for both mid-circuit and
terminal measurements. For mid-circuit measurements, we also observed
suppressed reset errors (0.4%) and no increase in leakage due to readout. We
accomplished this by overcoming the challenges stated in the introduction: the
presented models accurately capture the relevant error channels, and they can
be evaluated 10,000 times faster (1 minute vs 1 week for the parameter space
used here) than measuring errors directly in hardware, which unlocks the
ability to use a global optimizer. Based on recent work in Ref. [21] we
believe the snake optimizer and these models will scale to at least 1,000
qubits.

Our model-based readout optimization strategy has already been employed in
several large experiments, such as the demonstration of a distance-5 surface
code [9] with a measurement error of 1.9% per qubit, and a 70 qubit random-
circuit sampling experiment [19] with an error of 1.3% per qubit. While the
performance is among the best observed for repetitive and simultaneous
measurements in superconducting qubits, even better performance will be needed
to be well below the error-correcting threshold. In particular, the readout
time has to be shorter to avoid data-qubit idling errors.

We believe that the error rates achieved in this Letter are close to optimal
for the given processor, and that the path to more performant readout is
through longer relaxation times, higher measurement efficiencies, and more
optimized circuit parameters. While we treated the circuit parameters as
fixed, we could include them as optimization parameters to inform the design
of future processors. However, more research is needed to find the optimal
readout circuit for superconducting qubits.

###### Acknowledgements.

We thank the broader Google Quantum AI team for fabricating the processor,
building and maintaining the cryogenic system, and general hardware and
software infrastructure that enabled this experiment. We also thank V. Sivak
and W. Livingston for providing comments on the manuscript, and L. Martin for
helping with the crosstalk analysis.

## References

  * Walter  _et al._ [2017] T. Walter, P. Kurpiers, S. Gasparinetti, P. Magnard, A. Potočnik, Y. Salathé, M. Pechal, M. Mondal, M. Oppliger, C. Eichler, _et al._ , Rapid high-fidelity single-shot dispersive readout of superconducting qubits, Physical Review Applied 7, 054020 (2017).
  * Sunada  _et al._ [2022] Y. Sunada, S. Kono, J. Ilves, S. Tamate, T. Sugiyama, Y. Tabuchi, and Y. Nakamura, Fast readout and reset of a superconducting qubit coupled to a resonator with an intrinsic purcell filter, Physical Review Applied 17, 044016 (2022).
  * Chen  _et al._ [2023] L. Chen, H.-X. Li, Y. Lu, C. W. Warren, C. J. Križan, S. Kosen, M. Rommel, S. Ahmed, A. Osman, J. Biznárová, _et al._ , Transmon qubit readout fidelity at the threshold for quantum error correction without a quantum-limited amplifier, npj Quantum Information 9, 26 (2023).
  * Swiadek  _et al._ [2023] F. Swiadek, R. Shillito, P. Magnard, A. Remm, C. Hellings, N. Lacroix, Q. Ficheux, D. C. Zanuz, G. J. Norris, A. Blais, _et al._ , Enhancing dispersive readout of superconducting qubits through dynamic control of the dispersive shift: Experiment and theory, arXiv preprint arXiv:2307.07765 (2023).
  * Blais  _et al._ [2021] A. Blais, A. L. Grimsmo, S. M. Girvin, and A. Wallraff, Circuit quantum electrodynamics, Reviews of Modern Physics 93, 025005 (2021).
  * Aumentado [2020] J. Aumentado, Superconducting parametric amplifiers: The state of the art in josephson parametric amplifiers, IEEE Microwave magazine 21, 45 (2020).
  * Sundaresan  _et al._ [2022] N. Sundaresan, T. J. Yoder, Y. Kim, M. Li, E. H. Chen, G. Harper, T. Thorbeck, A. W. Cross, A. D. Córcoles, and M. Takita, Matching and maximum likelihood decoding of a multi-round subsystem quantum error correction experiment, arXiv preprint arXiv:2203.07205 (2022).
  * Marques  _et al._ [2023] J. Marques, H. Ali, B. Varbanov, M. Finkel, H. Veen, S. van der Meer, S. Valles-Sanclemente, N. Muthusubramanian, M. Beekman, N. Haider, _et al._ , All-microwave leakage reduction units for quantum error correction with superconducting transmon qubits, Physical Review Letters 130, 250602 (2023).
  * Google Quantum AI [2023] Google Quantum AI, Suppressing quantum errors by scaling a surface code logical qubit, Nature 614, 676 (2023).
  * Koch  _et al._ [2022] C. P. Koch, U. Boscain, T. Calarco, G. Dirr, S. Filipp, S. J. Glaser, R. Kosloff, S. Montangero, T. Schulte-Herbrüggen, D. Sugny, _et al._ , Quantum optimal control in quantum technologies. strategic report on current status, visions and goals for research in europe, EPJ Quantum Technology 9, 19 (2022).
  * Arute  _et al._ [2019] F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, R. Biswas, S. Boixo, F. G. Brandao, D. A. Buell, _et al._ , Quantum supremacy using a programmable superconducting processor, Nature 574, 505 (2019).
  * Klimov  _et al._ [2020] P. V. Klimov, J. Kelly, J. M. Martinis, and H. Neven, The snake optimizer for learning quantum processor control parameters, arXiv preprint arXiv:2006.04594 (2020).
  * Shillito  _et al._ [2022] R. Shillito, A. Petrescu, J. Cohen, J. Beall, M. Hauru, M. Ganahl, A. G. M. Lewis, G. Vidal, and A. Blais, Dynamics of transmon ionization, Physical Review Applied 18, 034031 (2022).
  * Khezri  _et al._ [2023] M. Khezri, A. Opremcak, Z. Chen, K. C. Miao, M. McEwen, A. Bengtsson, T. White, O. Naaman, D. Sank, A. N. Korotkov, _et al._ , Measurement-induced state transitions in a superconducting qubit: Within the rotating-wave approximation, Physical Review Applied 20, 054008 (2023).
  * Sank  _et al._ [2024] D. Sank, A. Opremcak, A. Bengtsson, M. Khezri, Z. Chen, O. Naaman, and A. Korotkov, System characterization of dispersive readout in superconducting qubits, arXiv preprint arXiv:2402.00413 (2024).
  * Burnett  _et al._ [2019] J. J. Burnett, A. Bengtsson, M. Scigliuzzo, D. Niepce, M. Kudra, P. Delsing, and J. Bylander, Decoherence benchmarking of superconducting qubits, npj Quantum Information 5, 54 (2019).
  * Khezri [2018] M. Khezri,  _Dispersive measurement of superconducting qubits_ (University of California, Riverside, 2018).
  * McEwen  _et al._ [2021] M. McEwen, D. Kafri, Z. Chen, J. Atalaya, K. Satzinger, C. Quintana, P. V. Klimov, D. Sank, C. Gidney, A. Fowler, _et al._ , Removing leakage-induced correlated errors in superconducting quantum error correction, Nature communications 12, 1 (2021).
  * Morvan  _et al._ [2023] A. Morvan, B. Villalonga, X. Mi, S. Mandrà, A. Bengtsson, P. Klimov, Z. Chen, S. Hong, C. Erickson, I. Drozdov, _et al._ , Phase transition in random circuit sampling, arXiv preprint arXiv:2304.11119 (2023).
  * [20] See Supplemental Material, which includes Ref. [22].
  * Klimov  _et al._ [2023] P. V. Klimov, A. Bengtsson, C. Quintana, A. Bourassa, S. Hong, A. Dunsworth, K. J. Satzinger, W. P. Livingston, V. Sivak, M. Y. Niu, _et al._ , Optimizing quantum gates towards the scale of logical qubits, arXiv preprint arXiv:2308.02321 (2023).
  * Heinsoo  _et al._ [2018] J. Heinsoo, C. K. Andersen, A. Remm, S. Krinner, T. Walter, Y. Salathé, S. Gasparinetti, J.-C. Besse, A. Potočnik, A. Wallraff, and C. Eichler, Rapid high-fidelity multiplexed readout of superconducting qubits, Physical Review Applied 10, 034040 (2018).