# Federated Learning Optimization: A Comparative Study of Data and Model
Exchange Strategies in Dynamic Networks

Alka Luqman1 Yeow Wei Liang Brandon1&Anupam Chattopadhyay1 1Nanyang
Technological University, Singapore
{alka001, yeow0070<EMAIL_ADDRESS>
###### Abstract

The promise and proliferation of large-scale dynamic federated learning gives
rise to a prominent open question - is it prudent to share data or model
across nodes, if efficiency of transmission and fast knowledge transfer are
the prime objectives. This work investigates exactly that. Specifically, we
study the choices of exchanging raw data, synthetic data, or (partial) model
updates among devices. The implications of these strategies in the context of
foundational models are also examined in detail. Accordingly, we obtain key
insights about optimal data and model exchange mechanisms considering various
environments with different data distributions and dynamic device and network
connections. Across various scenarios that we considered, time-limited
knowledge transfer efficiency can differ by up to 9.08%, thus highlighting the
importance of this work.

##  1 Introduction

Federated Learning (FL) is a decentralized learning paradigm that has gained
significant attention recently due to its potential to address privacy
concerns and data ownership issues in large-scale machine learning
applications. However, the dynamic nature of the participating devices and the
complexities introduced by large-scale, heterogeneous data distributions pose
significant challenges to the effectiveness and efficiency of FL systems. The
amount of data collected daily is about 2,500,000 Terabytes and it is
increasing in volume every day Marr (2023). In 2020, a single self-driving car
was collecting up to 5TB of data a day Miller (2020). We are using more
sensors and cameras to collect data today thanks to the Internet of Things
(IoT), wearable personal devices, connected autonomous vehicles etc. The
increasing infeasibility of processing this amount of data on a single
machine, combined with the private nature of the captured data, makes FL an
apt technique for creating application-specific ML models. Several studies
have investigated various aspects of FL, including communication efficiency,
privacy preservation, and model convergence Kairouz et al. (2019). Recent
advancements have explored the use of synthetic data as an alternative to raw
data exchange Goetz and Tewari (2020); Hu et al. (2022), aiming to mitigate
privacy risks and reduce communication overhead. An optimal strategy for
exchanging knowledge among devices in large dynamic settings remains an open
research question.

##  2 Literature Review

Distributed machine learning (DML) distributes the computational workload of
ML algorithms across multiple machines Kairouz et al. (2019). This intuitively
channels model parallelism, where the ML model or different parts of one ML
model are computed across various devices. Data parallelism, on the other
hand, distributes the dataset across devices such that each device operates on
a smaller batch of data. This is suitable for situations with large amounts of
data Krizhevsky (2014). FL follows such a process of data parallelism to train
ML models on data present at various edge devices, which then share their
model updates (weights or gradients) to be aggregated by a trusted parameter
server. In the absence of a trusted central aggregator, decentralized
coordination can be performed using Peer-to-Peer FL (P2P FL).

McMahan et al. (2016) state that to optimize FL algorithms like FedAvg, the
communication costs significantly outweigh computation costs and that
computational costs become negligible with a sufficiently large number of
clients. In a realistic large-scale peer-to-peer network, especially in IoT,
these communications occur over networks where problems like low client
availability, fluctuating device power, and dynamic network bandwidths are
expected. In such dynamically changing environments, it becomes imperative
that the ML infrastructure also adapts dynamically, decoupled from the
objective of the ML model.

In order to do this, a device participating in training ML models can opt to
implement a DML approach or an FL approach at any point in time. It has the
flexibility to share raw data or model updates with its trusted peers to
create an aggregated global model. Such a model can be extended to untrusted
peers via Fung et al. (2020); Li et al. (2021); Byrd and Polychroniadou (2021)
to mitigate attacks against robustness and privacy. In the case of privacy-
sensitive ML applications, devices can opt to share synthetic data generated
from the raw data they have access to. This approach has been demonstrated by
Goetz and Tewari (2020); Hu et al. (2022) to be equally effective as sharing
raw data.

Figure 1: Workflow depicting Dataset Handler which utilizes an array of
shuffling, sharding and image transformations to systematically generate
realistic client datasets. Each client trains its own generator model to
prevent data leakage while the communication handler dynamically adopts
between data-sharing strategies and model-sharing strategies.

The FL approach may be extended to pre-trained foundational models in order to
combine the strengths of pre-training with the privacy benefits of federated
learning and adaptability to client-specific distributions. Deploying multiple
instances of such large models and fine-tuning the entire models can incur
significant storage and communication costs. A straightforward way to reduce
this cost is to perform low-rank decomposition on the pre-trained weight
matrix using LoRAHu et al. (2021) to split it into frozen layers and trainable
layers. Adapter-tuningHoulsby et al. (2019) and parameter-efficient fine-
tuningDing et al. (2023); He et al. (2022) methods can be adopted to make the
model updates more efficient for FL.

Here we develop an algorithm that identifies the optimal action a device in a
peer-to-peer network can take to enhance communication efficiency without
sacrificing model accuracy. It considers essential factors such as the size of
other connected peers’ datasets, their accuracy, CPU speed, available RAM,
bandwidth, number of connected peers, and privacy constraints related to data
sharing.

##  3 Methodology

In this work, we first want to evaluate the performance of models in dynamic
networks and characterized by varying non-i.i.d data distributions. The
results of these comparative studies will then be used to create an adaptive
algorithm that implements a hybrid transfer model using data and/or model
sharing techniques.

###  3.1 Dataset & Preprocessing

The dataset used for benchmarking was CIFAR-10, which comprises 60,000 32x32
color images in 10 classes, with 6,000 images per class. The training dataset
of 50,000 images was partitioned among multiple devices to simulate a large
dynamic federated learning environment.

Zhao et al. (2018) identified that the test set accuracy can fall by as much
as 55% in a highly skewed scenario where each client only has one class. As
this is a highly probable scenario in FL, the following standard methods from
Kairouz et al. (2019) were used to create non-i.i.d data distribution in our
experiments.

Algorithm 1 Pre-Communication Algorithm

Input: $self$.K_degree $>0$

1: for client in network_graph do

2: peers $\leftarrow$BFS($self$, $self$.K_degree)

3: action_score_tuples $\leftarrow$ Ranking($self$, peers)

4: action_queue $\leftarrow$ PriorityQueue(action_score_tuples)

5: costs, actions_to_execute $\leftarrow$ 0, empty list

6: while costs $<self$.bandwidth do

7: action $\leftarrow$ action_queue.$pop$()

8: if action.cost $>$ ($self$.bandwidth $-$ costs) then

9: break

10: else

11: actions_to_execute.append(action)

12: costs $+=$ action.cost

13: end if

14: end while

15: end for

16: return actions_to_execute

  * •

Prior probability shift \- It occurs across clients $i$ and $j$ when the set
of labels observed by client $i$ may not be the same as the set observed by
client $j$. This is a label distribution skew where $P(y_{i})\neq P(y_{j})$
although $P(x|y)$ is the same across clients. This is referred to as
pathological non-i.i.d data set distribution in McMahan et al. (2016).

  * •

Quantity skew \- It occurs when one client has more data than another. It is
implemented through a sharding function to split the dataset into preset
ratios across a specific number of clients. This function assumes that the
data is already shuffled or sorted for i.i.d data and pathological non-i.i.d
data.

  * •

Co-variate shift \- It occurs when the features representing the same label
across client $i$ and $j$ vary. This is when $P(x_{i})\neq P(x_{j})$ although
$P(y|x)$ is the same. This can typically occur when one camera is slanted or
has a different lens quality. It can also occur when clients belong to
different geographical regions, having different dialects of speech or styles
of writing etc. Concept drift is a similar but related concept which occurs
when the distribution of labels $y$ is the same, yet the distribution of $x$
shifts from client to client across class labels. As this is an alternate
methodology to introduce non-i.i.d behaviour, we refer to this as non-
pathological non-i.i.d data set distribution. This is implemented by applying
common image data augmentation techniques like translation, random crop, image
rotation, application of Gaussian noise, distortion etc.

###  3.2 Federated Learning Framework

Lalitha et al. (2019) and Qin and Kondo (2021) introduced FL as a graph
learning problem using a peer-to-peer architecture. There is no single global
model in the network, but nodes aggregate models from their peers and
eventually converge to a global optimum. We implemented a federated learning
framework using PeerFLLuqman et al. (2024) to create a simulation of peer-to-
peer federated learning scenarios with multiple clients joining dynamic adhoc
networks. Each client receives a subset of the preprocessed dataset, and the
partitions are updated dynamically to simulate real-world scenarios where
devices may join or leave the network.

The computation cost at device $i$ is defined as a function of its battery
level $D_{i}^{pow}$ and available memory $D_{i}^{mem}$ to perform on-device
training.

$\displaystyle\mathcal{C}^{compute}_{i}=f(D_{i}^{pow},D_{i}^{mem})=D_{i}^{pow}*D_{i}^{mem}$
(1)

The communication cost is calculated as a function of the channel bandwidth
$E_{i}^{bw}$ and the size of the message being transmitted over it $S$.

$\displaystyle\mathcal{C}^{comm}_{i}=g(E_{i}^{bw},S)=\frac{\alpha
S}{E_{i}^{bw}}$ (2)

The simulation parameter ranges are sourced from Ng et al. (2021).

Assuming each client state is represented by a set of information and the
other connected clients can access that information by traversing the graph, a
three-step algorithm is proposed in Algorithm 1 for each client to discover
the best actions to take with its $k$th degree peer given constraints. The
constraints are further probed using a ranking function which can be modified
to represent any characteristic in the node such as the local data set
quality, CPU availability for offloading training, availability of GPU, etc.
This step initiates the network exploration and peer prioritization for the
model training.

Algorithm 2 BFS: Breadth-first Search to $k$th degree for Client Information
Retrieval

Input: $client$, history, k$=-1$

1: if k$=-1$ then

2: k$\leftarrow client$.K_degree

3: peer_list $\leftarrow emptylist$

4: peer_queue $\leftarrow[client,$k$]$

5: end if

6: while peer_queue is not empty do

7: peer, current_k $\leftarrow$ peer_queue.$pop$(0)

8: peer_list.append(peer)

9: if current_k $==0$ then

10: continue

11: end if

12: for $child$ in peer.peers do

13: if $child$ in peer_list then

14: continue

15: end if

16: peer_queue.append(($child$, k-1))

17: end for

18: end while

19: return peer_list

Algorithm 3 Ranking: Client-level Communication Decision Algorithm

Input: $client$, peers

1: actionsToRun $\leftarrow emptylist$

2: for peer in peers do

3: if peer.accuracy $>client$.accuracy then

4: runFedAvg $\leftarrow$ True

5: for recent_actions in $client$.action_history do

6: if recent_actions contains FedAvg then

7: runFedAvg $\leftarrow$ False

8: end if

9: end for

10: if runFedAvg then

11: reward $\leftarrow$ Mean($client$.accuracy, peer.accuracy)

12: actionsToRun.add($runShareModel$($self$, peer), reward)

13: end if

14: end if

15:

16: if $client$ in peer.trustedPeers then

17: dataset_type $\leftarrow$ Synthetic

18: else

19: dataset_type $\leftarrow$ Sampled

20: end if

21: reward $\leftarrow$ scoreContext($client$, peer)

22: actionsToRun.add($runShareData$($client$, peer, dataset_type), reward)

23: end for

24: return actionsToRun

The mechanism chosen for network exploration is a breadth-first search. At
each client we apply Algorithm 2 recursively to traverse and retrieve
parameters of all connected peers to the $k$th degree. Limiting the peer
exploration to a finite $k$ neighbours helps prevent the algorithm from
getting stuck in infinite loops in the case of cyclic graphs, and is memory-
efficient in the case of very large graphs. Algorithm 3 is the communication
handler, which decides the action to be performed based on the characteristics
of peers and clients.

The complexity of Algorithm 2 is $O(K\cdot(V+E))$, influenced by $V$, the
number of peers and $E$, the connections between peers, limited by $K$ which
is the maximum degree of exploration. The ranking operation in Algorithm 3
contributes complexity of $O(V\log V)$. Therefore the overall complexity of
Algorithm 1 is $O(K\cdot(V+E)+V\log V)$. While this performs well in most
cases, the density of device connections has a significant impact in the worst
case. Increasing the exploration depth $k$ leads to a higher communication
cost in practice in dense graphs (with average node degree $>\frac{2E}{V}$),
even though the asymptotic time complexity remains comparable to sparse
network graphs.

###  3.3 Data & Model Exchange Strategies

  1. 1.

Exchanging Raw Data: In this approach, the raw data samples are exchanged
among the peers. Each client trains a local model on its dataset and sends the
updated model to its peer or cluster head. This device aggregates the model
updates to with its own local model and propagates this further to the rest of
the network.

  2. 2.

Exchanging Synthetic Data: Synthetic data generation techniques, such as
Variational AutoEncoders Kingma and Welling (2013) and Vector-Quantisation
Variational Autoencoders Oord et al. (2017), are used to create artificial
data samples based on the local data of each client. VAEs are chosen
specifically since they create lightweight models suitable to run on hardware-
constrained devices. The synthetic data samples are then exchanged among the
peers, and the federated learning process proceeds similarly to the raw data
exchange approach.

  3. 3.

Exchanging Model Updates: Instead of exchanging raw or synthetic data, only
the model updates (i.e. gradients) are exchanged among the devices. Each
client computes the model gradients based on its local dataset and sends these
gradients to be aggregated via FedAvgMcMahan et al. (2017) and processed
further.

  4. 4.

Using Foundational Models: This method is a variation of strategy 3, i.e.
exchanging model updates, but the models used are foundational models which
are further fine-tuned at each client and the model weights of trainable
parameters are aggregated via averaging. We experimented with YOLOv8Jocher et
al. (2023) models pretrained on COCO and applied to object detection on the
KITTI datasetGeiger et al. (2013).

Each client, apart from storing its private data and model, also maintains its
own synthetic data generator, trained on its respective dataset partition.
This generator is used to craft samples instead of sharing raw data samples
when the network is untrusted. This generation is performed on the fly at each
client to preserve device-level privacy. Backbone models of VGG16Simonyan and
Zisserman (2014) and SqueezeNetIandola et al. (2016) are compared using
initial learning rates of 0.01 and 0.001 respectively on batch sizes of 256.

###  3.4 Metrics

To evaluate the performance of the different data exchange strategies, we
considered the following metrics:

  1. 1.

Model Accuracy: The accuracy of the best model at each device on the test
dataset.

  2. 2.

Communication Overhead: The amount of data exchanged between the clients and
the central server.

  3. 3.

Convergence Speed: The number of communication rounds required for the model
to converge.

For each peer we formulate the resultant ML model accuracy as a reward that
each choice of exchange strategy results in.

$\displaystyle Reward(Action)=E(AccuracyIncrease|Action)$ (3)

This score is weighted by the cost of communication (Eq 2), which is the
amount of data each action costs.

For object detection on KITTI, MAP is evaluated with IoU threshold of 50%. The
experiments were designed to compare the performance of the different data
exchange strategies under varying conditions, such as different network
topologies and communication frequencies. Each experiment was repeated
multiple times to ensure the robustness and reliability of the results.

Figure 2: Synthetically generated samples of 5 images across classes of VAEs
used in the Generator.

##  4 Experimental Studies

Model backbone | Model packet size (MB)
---|---
SqueezeNet | 7.396
VGG16 | 60.771
YOLOv8 | 27.890
Client’s percentage | Data packet
of dataset | size (MB)
10% | 58.785
20% | 117.569
30% | 176.354
60% | 352.707
80% | 470.276
Table 1: Average size of compressed model weight packets and data packets (in MB) during transmission based on which Communication Handler chooses a strategy. Data Distribution | Strategy 1 | Strategy 2 | Strategy 3 | Strategy 4 | Most used | Ideal
---|---|---|---|---|---|---
at clients | only (Acc%) | only (Acc%) | only (Acc%) | only (MAP%) | strategy | for
I.I.D | 95.3 | 95.1 | 93.1 | 76.7 |  |
Pathological | 85.6 | 85.1 | 70.1 | 51.7 | 1,2 | High idle
non I.I.D |  |  |  |  |  | time
Non Pathological | 82.8 | 89.2 | 83.0 | 71.6 | 2,4 | High local
non I.I.D |  |  |  |  |  | compute
Quantity skew | 94.6 | 97.2 | 72.8 | 64.5 | 1,2 first |
$<$5% data with quantity | 54.3 | 53.1 | 66.4 | 38.2 | 3 first | Dense
skew at each client |  |  |  |  |  | network graph
Table 2: Consolidated insights on optimum choice of strategy from Section 3.3.
Figure 3: Results of 3 clients trained from scratch using model sharing
strategy. Pathological non-i.i.d distribution seems to worsen all FedAvg,
mostly due to missing classes being difficult to learn across multiple clients
simultaneously. For client C however, it performs worse consistently across
all distributions. Model sharing seems to contribute most when data samples
are low and the data set may be more difficult, as seen from the larger
improvement.

The experiments were conducted to evaluate the performance of different data
exchange strategies in large-scale dynamic federated learning settings. The
experimental setup is inspired by the methodologies used in previous studies
on federated learning and data exchange strategies McMahan et al. (2017);
Bonawitz et al. (2019) and presented in Figure 1.

Both VGG16 and SqueezeNet performed considerably better when initialized using
pre-trained weights from tiny ImageNet, as opposed to random initialization
and training only on the local datasets. For all experiments excluding
SqueezeNet, the initial probing rounds were leveraged to also perform model
sharing, to make maximum use of the bandwidth consumed. This practice resulted
in locally converged models, the accuracy of which could be later improved
using other exchange strategies.

A stark contrast in quality across generator models was observed by utilizing
a discrete latent space (VQVAE, CVQVAE) over a Gaussian latent space (VAE,
CVAE). A few synthetically generated samples are visualized in Figure 2.
Conditional VAEs are cheaper to run compared to VQVAEs which require a larger
model to generate noise for sampling, and thus are suited for memory-
constrained devices.

Table 1 shows that sharing data consumes a higher bandwidth cost but takes
fewer communication rounds to converge while model weights incur a lower
bandwidth cost but require multiple successive communication rounds to
improve. The general trend noticed is that model-sharing strategies create a
stable model while data-sharing strategies have faster convergence. Performing
model-sharing first shows the ability to pull a model out of local minima with
continuous adoption of data-sharing strategies in later stages.

With pre-trained foundational models, the law of diminishing effects is
observed and sharing raw or synthetic data mostly leads to overfitting, and
the clients benefit better from model sharing. But on clients that have
pathological non-I.I.D data, model sharing is rarely seen to be a good choice.
They mostly benefit from data sharing, a sample subset of the raw data from
trusted peers, or synthetic data in the case of non-trusted peers. The
insights from our experiments are tabulated in Table 2 for easy inference.

A surprising observation was that sharing synthetic data was as effective as
using pre-trained models to improve accuracy. This is attributed to two
reasons - the imbalanced data distribution among the devices, introduced
through quantity skew, and the distributional bias introduced in the raw data
by the co-variate shift operations.

Learning from indirect data representations in highly non-I.I.D settings
provides a semblance of privacy since the representations cannot be directly
linked to individual data points. This statistical anonymity aligns with the
principles of differential privacy (DP), to create secure ML models but unlike
DP, these do not provide quantifiable guarantees of privacy.

##  5 Conclusion

This study establishes the advantage of combining various learning strategies
at the framework level to cater to robust real-world ML applications. Some of
the key takeaways are:

  * •

Model sharing can be seamlessly integrated into network exploration with
minimal bandwidth overhead, making it an ideal first step for assessing
feasibility among different strategies.

  * •

Clients with limited data should solicit additional data from peers,
leveraging VAEs for privacy-preserving synthetic data generation, which
significantly enhances performance without overshadowing the original
datasets.

  * •

Employing synthetic data sharing as a supplementary measure post-convergence
significantly boosts performance without compromising the integrity of the
original datasets and works to revitalize model performance if it plateaus.

  * •

In larger networks with high levels of non-I.I.D skews, learning from
synthetic data is ideal for either balancing the data distribution or
mitigating the bias inherent in noisy raw data.

Techniques like DP and secure multi-party computation (MPC) can complement the
proposed data and model-sharing strategies for more robust privacy protection
in sensitive settings. This work explored using partial and complete data or
model exchange strategies in dynamic federated learning settings to combat the
problem of knowledge transfer efficiency even in skewed data distributions.
This underscores the need for learning strategies to adapt to their
environments and participants in order to maximize effectiveness and ensure
optimal outcomes.

## Acknowledgments

This research is supported by the National Research Foundation, Singapore
under its Strategic Capability Research Centres Funding Initiative. Any
opinions, findings and conclusions or recommendations expressed in this
material are those of the author(s) and do not reflect the views of National
Research Foundation, Singapore.

## References

  * Bonawitz et al. [2019] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečnỳ, Stefano Mazzocchi, Brendan McMahan, et al.  Towards federated learning at scale: System design.  Proceedings of Machine Learning and Systems, 1:374–388, 2019.
  * Byrd and Polychroniadou [2021] David Byrd and Antigoni Polychroniadou.  Differentially private secure multi-party computation for federated learning in financial applications.  In Proceedings of the First ACM International Conference on AI in Finance, ICAIF ’20, New York, NY, USA, 2021. Association for Computing Machinery.
  * Ding et al. [2023] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al.  Parameter-efficient fine-tuning of large-scale pre-trained language models.  Nature Machine Intelligence, 5(3):220–235, 2023.
  * Fung et al. [2020] Clement Fung, Chris J. M. Yoon, and Ivan Beschastnikh.  Mitigating sybils in federated learning poisoning, 2020.
  * Geiger et al. [2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets Robotics: The KITTI Dataset.  International Journal of Robotics Research (IJRR), 2013.
  * Goetz and Tewari [2020] Jack Goetz and Ambuj Tewari.  Federated learning via synthetic data, 2020.
  * He et al. [2022] Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang.  Parameter-efficient fine-tuning for vision transformers.  arXiv preprint arXiv:2203.16329, 3, 2022.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International conference on machine learning, pages 2790–2799. PMLR, 2019.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * Hu et al. [2022] Shengyuan Hu, Jack Goetz, Kshitiz Malik, Hongyuan Zhan, Zhe Liu, and Yue Liu.  Fedsynth: Gradient compression via synthetic data in federated learning, 2022.
  * Iandola et al. [2016] Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ¡0.5mb model size, 2016.
  * Jocher et al. [2023] Glenn Jocher, Ayush Chaurasia, and Jing Qiu.  Ultralytics YOLO, January 2023.
  * Kairouz et al. [2019] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A. Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D’Oliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaïd Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao.  Advances and open problems in federated learning.  CoRR, abs/1912.04977, 2019.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2013.
  * Krizhevsky [2014] Alex Krizhevsky.  One weird trick for parallelizing convolutional neural networks, 2014.
  * Lalitha et al. [2019] Anusha Lalitha, Osman Cihan Kilinc, Tara Javidi, and Farinaz Koushanfar.  Peer-to-peer federated learning on graphs.  CoRR, abs/1901.11173, 2019.
  * Li et al. [2021] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma.  Anti-backdoor learning: Training clean models on poisoned data.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 14900–14912. Curran Associates, Inc., 2021.
  * Luqman et al. [2024] Alka Luqman, Shivanshu Shekhar, and Anupam Chattopadhyay.  Peerfl: A simulator for peer-to-peer federated learning at scale, 2024.
  * Marr [2023] B. Marr.  How much data do we create every day? the mind-blowing stats everyone should read.  Forbes, 2023.  Accessed: 25 April 2024.
  * McMahan et al. [2016] H. Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera y Arcas.  Federated learning of deep networks using model averaging.  CoRR, abs/1602.05629, 2016.
  * McMahan et al. [2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-Efficient Learning of Deep Networks from Decentralized Data.  In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 1273–1282. PMLR, 20–22 Apr 2017.
  * Miller [2020] Rich Miller.  Rolling zettabytes: Quantifying the data impact of connected cars, 01 2020.
  * Ng et al. [2021] Jer Shyuan Ng, Wei Yang Bryan Lim, Zehui Xiong, Xianbin Cao, Jiangming Jin, Dusit Niyato, Cyril Leung, and Chunyan Miao.  Reputation-aware hedonic coalition formation for efficient serverless hierarchical federated learning.  IEEE Transactions on Parallel and Distributed Systems, 33(11):2675–2686, 2021.
  * Oord et al. [2017] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning, 2017.
  * Qin and Kondo [2021] Yang Qin and Masaaki Kondo.  Mlmg: Multi-local and multi-global model aggregation for federated learning.  In 2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops), pages 565–571, 2021.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2014.
  * Zhao et al. [2018] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra.  Federated learning with non-iid data.  2018\.