XX Month, XXXX XX Month, XXXX XX Month, XXXX XX Month, XXXX XX Month, XXXX
OJIM.2022.1234567

# Federated Large Language Models for Swarm Intelligence: A Survey

Youyang Qu1  Member  IEEE

###### Abstract

Federated learning (FL) offers a compelling framework for training large
language models (LLMs) while addressing data privacy and decentralization
challenges. This paper surveys recent advancements in the federated learning
of large language models, with a particular focus on machine unlearning—a
crucial aspect for complying with privacy regulations like the Right to be
Forgotten. Machine unlearning in the context of federated LLMs involves
systematically and securely removing individual data contributions from the
learned model without retraining from scratch. We explore various strategies
that enable effective unlearning, such as perturbation techniques, model
decomposition, and incremental learning, highlighting their implications for
maintaining model performance and data privacy. Furthermore, we examine case
studies and experimental results from recent literature to assess the
effectiveness and efficiency of these approaches in real-world scenarios. Our
survey reveals a growing interest in developing more robust and scalable
federated unlearning methods, suggesting a vital area for future research in
the intersection of AI ethics and distributed machine learning technologies.

Federated Learning (FL), Large Language Models (LLMs), Swarm Intelligence,
Efficiency, Pre-trained Models, Privacy and Security

##  1 INTRODUCTION

In the rapid evolution of artificial intelligence has led to significant
breakthroughs in various domains, particularly with the advent of Large
Language Models (LLMs) like GPT [1] and BERT [2]. These models have
revolutionized the way machines understand and generate human language,
enabling applications ranging from automated customer support to advanced
natural language processing tasks [3, 4]. Concurrently, there has been a
growing interest in swarm intelligence, a field inspired by the collective
behavior of decentralized, self-organized systems found in nature, such as ant
colonies or bee swarms [5]. Swarm intelligence leverages the simple rules
followed by individual agents interacting locally with their environment to
solve complex problems, a concept that is inherently robust and scalable [6].

The convergence of LLMs with the principles of swarm intelligence introduces a
novel approach to enhancing decentralized decision-making processes [7]. By
integrating these two technologies, it becomes possible to process and
generate language-based data under the constraints of decentralized and
privacy-preserving environments [8]. This integration is especially pertinent
in scenarios where centralizing sensitive data is impractical or undesirable,
such as in healthcare or financial services, where privacy concerns are
paramount.

Federated learning (FL) emerges as a critical enabler in this context. It
allows for the collaborative training of LLMs across multiple decentralized
nodes without the need to share their actual data [9, 10]. This approach not
only preserves the privacy and security of the data but also harnesses the
collective intelligence of the swarm. By distributing the learning process
across numerous nodes, federated learning enhances the robustness and
adaptability of the models, making them more resilient to attacks and capable
of generalizing across diverse datasets [11].

However, integrating LLMs with swarm principles in a federated setting
introduces unique challenges. These include managing heterogeneous data
sources, ensuring consistency in learning outcomes across diverse nodes, and
developing efficient communication protocols that adhere to the lightweight
nature of swarm agents. Addressing these challenges requires innovative
solutions that balance the scalability and flexibility of swarm intelligence
with the advanced processing capabilities of LLMs.

Motivated by this, we aim to provide a systematic view of the current research
status of federated LLMs for swarm intelligence from various perspectives,
including architecture, efficiency, scalability, pre-trained models, security,
and privacy. The main contributions of this survey are as follows.

  * •

Synthesis of Federated Learning and Swarm Intelligence: We provide a
systematic review of the methodologies that integrate federated learning with
large language models, specifically tailored for swarm intelligence
applications. This includes discussions on the adaptations necessary for
handling decentralized, autonomous agents in a swarm.

  * •

Analysis of Techniques and Performance: The paper evaluates various strategies
and modifications to traditional federated learning algorithms to enhance
their applicability and efficiency in swarm-based systems, emphasizing real-
world implementations and case studies.

  * •

Strategic Insights and Frameworks: We compare a set of criteria and frameworks
for designing and deploying federated LLMs within swarm intelligence contexts,
offering insights into achieving scalability, robustness, and real-time
responsiveness.

The remainder of the survey is as follows. Section 2 provides a detailed
exploration of the foundations of swarm intelligence and its integration with
large language models, highlighting the synergies and potential enhancements.
Section 3 examines the specific adaptations and innovations in federated
learning algorithms that facilitate efficient and effective training of LLMs
across distributed swarm agents. Section 4 presents a critical analysis of
current implementations and discusses the technical challenges, practical
limitations, and potential solutions. Section 5 concludes the survey with a
discussion on future research directions and the broader implications of
federated LLMs for swarm intelligence in various applications.

Figure 1: Federated LLMs for Swarm Intelligence Architecture

##  2 Foundations and Integration of SI and LLM

###  2.1 Introduction to Swarm Intelligence

####  2.1.1 Definition and Historical Context

Swarm intelligence (SI) refers to the collective behavior of decentralized,
self-organized systems, typically composed of a population of simple agents
interacting locally with their environment and each other [12]. The agents
follow very simple rules, and although there is no centralized control
structure dictating how individual agents should behave, local interactions
between such agents lead to the emergence of “intelligent” global behavior,
unknown to the individual agents.

The concept of swarm intelligence originated from biological studies of
colonies of ants and swarms of bees in the late 1980s and early 1990s [13]. In
nature, these organisms exhibit behaviors that can be extraordinarily complex,
such as finding the shortest path to a food source or constructing complex
structures, despite the simplicity of individual behaviors and the lack of
central control. This paradox, where complex global behavior arises from
simple local interactions, has inspired various algorithms and techniques in
artificial intelligence.

####  2.1.2 Key Principles of Swarm Behavior

Swarm behavior is underpinned by several key principles that enable the
robust, scalable, and flexible characteristics of natural swarms [14]:

Decentralization: There is no single controlling entity in swarm systems;
instead, control is distributed across the individual agents comprising the
swarm. This decentralization contributes to the robustness and fault tolerance
of the system, as there is no single point of failure.

Simple Rules: Each agent in a swarm follows simple rules, and the interaction
of these rules across the entire swarm leads to the emergence of complex
behavior. These rules are typically based on the local perception of the
environment, without a global view.

Local Interaction: Agents primarily rely on local information or interactions
with their neighbors. This limitation is crucial for scalability, as it
reduces the communication overhead and computational complexity that would
arise from managing global information.

Emergence: The global behavior of the swarm emerges from the interactions of
individuals without being explicitly programmed. Emergent behavior is a
hallmark of swarm intelligence, providing solutions that are often more
adaptive and resilient than those devised through central planning.

Adaptability and Learning: Swarm systems are adaptive; they can change their
behavior based on feedback from the environment. This learning can occur at
the level of the individual or the swarm, enhancing the system’s ability to
respond to dynamic or uncertain environments.

###  2.2 Fundamentals of Large Language Models

####  2.2.1 Architecture and Functionality

Large Language Models (LLMs) are a type of artificial intelligence model
designed to understand, generate, and manipulate human language by learning
from vast amounts of textual data [15]. These models are based on the
architecture of deep neural networks, particularly Transformer [16] models,
which have revolutionized the field of natural language processing (NLP) since
their introduction.

The core component of a Transformer, and consequently of many LLMs, is the
self-attention mechanism. This mechanism allows the models to weigh the
importance of different words in a sentence, regardless of their distance from
each other in the text [17]. This feature enables the model to capture complex
linguistic structures and contextual nuances, making them highly effective for
a wide range of NLP tasks, from machine translation and summarization to
question answering and text generation.

LLMs such as GPT (Generative Pretrained Transformer) [1], BERT (Bidirectional
Encoder Representations from Transformers) [2], and their variants are
pretrained on diverse internet-scale datasets using tasks like masked language
modeling (where random words are masked out and the model learns to predict
them based on the surrounding context) and next-token prediction (predicting
the next word in a sequence). This pretraining step allows the models to
develop a broad understanding of language and context, which can then be fine-
tuned to specific applications with relatively little additional training data
[18].

####  2.2.2 Evolution of LLMs in Natural Language Processing

The evolution of LLMs has marked several milestones in the field of NLP,
starting from simpler models like RNNs (Recurrent Neural Networks) [19] and
LSTMs (Long Short-Term Memory networks) [20] to more sophisticated
architectures like the aforementioned Transformers. The ability of LLMs to
handle longer sequences of text and their parallel processing capabilities (a
significant advantage over RNNs and LSTMs) have dramatically improved
efficiency and scalability in training and inference processes.

The progression from models that could handle only specific tasks (task-
specific models) to models that can be applied to virtually any text-based
task (general-purpose models) is another crucial development [21]. This shift
has been driven by the realization that a deep, contextual understanding of
language can be more effectively achieved through large-scale, unsupervised
learning from diverse and extensive textual corpora.

As LLMs have grown in size—from millions to billions of parameters—their
ability to model and generate human-like text has improved. However, this
increase in size comes with its own set of challenges, such as higher
computational costs, greater energy consumption, and the need for substantial
hardware resources, prompting ongoing research into more efficient model
architectures and training methods [22].

###  2.3 Integrating Swarm Intelligence with LLMs

####  2.3.1 Potential Benefits and Synergistic Effects

Integrating swarm intelligence (SI) principles with large language models
(LLMs) offers unique opportunities to enhance the adaptability and efficiency
of distributed computing systems, especially in processing and generating
language-based data. This combination leverages the decentralized nature of SI
to manage and operate LLMs across multiple nodes, potentially reducing the
computational load and data privacy concerns associated with centralized
training and inference.

One of the main benefits of this integration is the enhancement of robustness
and fault tolerance in language processing tasks. By distributing the LLM’s
functionalities across a swarm of agents, the system can continue to operate
effectively even if some nodes fail or are compromised, as the collective
intelligence of the swarm compensates for individual failures.

Another significant benefit is scalability. Swarm intelligence inherently
supports scaling to large numbers of simple agents or nodes, which is
particularly useful when deploying LLMs in environments with resource
constraints or where real-time responses are crucial. This scalability also
allows for more extensive data processing without the need for proportionally
increased computational power at a single point.

####  2.3.2 Case Studies and Successful Implementations

Several case studies highlight the practical applications and benefits of
integrating swarm intelligence with large language models:

Decentralized Content Moderation: In social media platforms, a swarm of agents
each running smaller segments of an LLM can collaboratively monitor and
moderate content in real-time, ensuring compliance with community guidelines
while maintaining user privacy.

Multi-Agent Translation Systems: For multinational corporations, swarm-based
LLMs can provide real-time translation services across different regions,
where each agent handles language tasks specific to its local region,
improving the speed and accuracy of services.

Distributed Sentiment Analysis: In market research, a swarm of LLM-equipped
agents can perform sentiment analysis on customer feedback data distributed
across various servers or locations, synthesizing insights locally and
aggregating them to derive global sentiment trends without transferring
sensitive data centrally.

##  3 Federated LLMs for Smarm Intelligence

###  3.1 LLM Architectures in Federated Settings

Table 1: Overview of Federated Large Language Model Architectures No. |  Contributions |  Technique |  LM used |  Dataset |  Limitations
---|---|---|---|---|---
1 |  Implementation of N-gram models in federated settings, dealing with distributed data. |  Federated learning |  N-gram RNN |  Not specified |  High communication overhead, data privacy concerns.
2 |  Enhances privacy and reduces server communication through localized computations. |  Federated Reconstruction |  LSTM-based |  MovieLens |  Complexity in local computations, potential performance trade-offs.
3 |  Framework for training and deploying LLMs in federated settings, addressing scalability and efficiency. |  FederatedScope |  LLaMa-7B |  HumanEval, HELM, etc |  Scalability issues, computational demands.
4 |  Integration of LLMs with edge computing for autonomous decision-making in connected environments. |  Edge AI, Federated Learning |  GPT-3 |  Synthetic challenging user request dataset |  Resource constraints on edge devices, deployment challenges.
5 |  Techniques for reducing LLM parameter count in federated settings, ensuring efficiency. |  Model pruning, Knowledge distillation |  Bert-based |  IMDP, Yelp |  Compromise on model performance, complexity in model reduction techniques.

Chen et al. [23] explore the adaptation of N-gram language models to federated
learning environments. It discusses the challenges of maintaining model
performance when training data is distributed and not centrally stored. The
primary focus is on how to effectively aggregate learned N-grams from multiple
nodes while preserving privacy and minimizing communication overhead.

Singhal et al. [24] introduce a novel approach named ”Federated
Reconstruction,” which enhances privacy in federated learning by allowing more
computation to be performed locally. The paper particularly emphasizes its
application to large language models, discussing how it can reduce the amount
of data required to be shared with the server, thus enhancing data privacy and
reducing bandwidth requirements.

Kuang et al. [25] present FEDERATEDSCOPE, a framework designed to facilitate
the training and deployment of large language models in a federated setting.
It offers insights into the architectural adjustments and optimizations
necessary to handle the complexities associated with training large models
across decentralized data sources. The paper also explores scalability and
efficiency challenges, providing solutions to maintain robust model
performance.

Shen et al. [26] discusses the integration of large language models (LLMs)
with edge computing devices in a federated learning context. It focuses on how
LLMs can empower edge devices for autonomous decision-making, highlighting the
potential for LLMs to enhance connected intelligence across various
applications. The paper addresses the challenges of deploying complex models
on resource-constrained edge devices.

Focusing on the issue of model size, Jiang et al. [27] propose techniques for
reducing the parameter count of large language models in federated settings.
It explores methods like model pruning and knowledge distillation to maintain
model efficacy with fewer parameters, which is critical for efficient data
transmission and quick adaptation in federated networks.

These papers collectively cover a spectrum of considerations for deploying
large language models in federated learning environments. Starting from the
basic implementation of N-gram models in a federated manner, the survey
transitions to advanced strategies like Federated Reconstruction for enhancing
privacy and computational efficiency. It then moves into a discussion on
frameworks and architectural adjustments necessary for scaling LLMs across
decentralized networks. The integration with edge AI represents an
application-focused advancement, providing a practical viewpoint on deploying
reduced-parameter models in real-world settings. This logical progression from
foundational concepts to practical implementations and optimizations
illustrates a comprehensive view of current research in the domain of
federated large language models.

The integration of large language models (LLMs) in federated settings, as
explored in these papers, significantly advances the field of swarm
intelligence by leveraging the collective learning capabilities of
decentralized networks. The first paper’s exploration of N-gram models in
federated environments lays the foundational understanding of how individual
nodes, like agents in a swarm, can contribute localized knowledge to build a
cohesive and comprehensive model. The introduction of [23] in the second paper
enhances this concept by minimizing central coordination, thereby promoting
more autonomous local decision-making—akin to the independent yet coordinated
behavior seen in natural swarms. The [24] framework further capitalizes on
this by scaling up the capabilities of LLMs across diverse and distributed
datasets, mirroring how swarms adapt to varied environments. [25] brings these
concepts into the realm of edge computing, where LLMs empower edge devices to
function like intelligent agents that perform tasks and make decisions in real
time, enhancing the swarm’s overall responsiveness and flexibility. Lastly,
the techniques discussed in [26] for reducing the parameter count of LLMs
ensure that the computational load is manageable even in resource-constrained
environments, which is crucial for maintaining the efficiency and agility of a
swarm. Together, these studies demonstrate how federated learning models can
emulate and enhance swarm-like intelligence, promoting robust, scalable, and
decentralized problem-solving capabilities in artificial systems.

###  3.2 Efficient Fine-tuning of LLM

Table 2: Summary of Efficient Fine-Tuning of LLMs in Federated Settings No. |  Contributions |  Technique |  LM used |  Dataset |  Limitations
---|---|---|---|---|---
1 |  Integrates differential privacy efficiently in federated LLM training with minimal performance loss. |  Differential privacy, noise addition, parameter clipping |  CIFG 19M, Transformer 21M, etc. |  WordPiece |  Potential impact on model accuracy, complexity in tuning privacy parameters.
2 |  Utilizes prompt tuning and adaptive optimization to reduce parameters and adapt learning rates dynamically. |  Prompt tuning, adaptive optimization |  GPT-2, LLaMa, etc. |  MRPC |  May not reach the full model’s potential, depends heavily on prompt design.
3 |  Optimizes prompt tuning in federated settings with synchronization for stable and accurate model updates. |  Federated prompt tuning, synchronization mechanisms |  Roberta |  GLUE |  Challenges in synchronization across diverse networks, prompt design limitations.
4 |  Benchmarks federated learning models and systems at scale, identifying and addressing bottlenecks. |  Benchmarking, model compression, robust aggregation methods |  Albert |  Reddit |  Scaling issues, may require significant computational resources.
5 |  Reduces data transmission with forward gradient method, enhancing privacy and efficiency. |  Forward gradient technique, efficient gradient aggregation |  LLaMa, Roberta, etc. |  AGNEWS, Yelp, etc. |  Potential loss of gradient information, requires careful implementation.

Ro et al. [28] address the challenges of applying differential privacy in
federated settings, particularly when fine-tuning large language models
(LLMs). It introduces a novel architecture that balances the trade-offs
between privacy, efficiency, and performance. The proposed model uses
lightweight mechanisms to ensure differential privacy without significantly
degrading the model’s utility. Key techniques include noise addition and
parameter clipping during the training process to maintain privacy. The
architecture is designed to be modular, allowing easy integration with
existing federated learning frameworks. This study is pivotal as it
demonstrates how privacy considerations can be seamlessly integrated into the
federated fine-tuning of LLMs, ensuring that user data remains confidential
while still contributing to the collective learning process.

Che et al. [29] explore the use of prompt tuning as a parameter-efficient
method for fine-tuning large language models in federated learning
environments. Prompt tuning involves adjusting a small set of parameters
(prompts) while keeping the majority of the model fixed, significantly
reducing the computational and communication overhead typically associated
with training large models. Additionally, the paper introduces an adaptive
optimization technique that dynamically adjusts learning rates based on the
network conditions and convergence rates of the participating nodes. This
approach not only enhances the efficiency of the federated learning process
but also ensures that the model adapts optimally across diverse and
potentially noisy datasets.

Building on the concept of prompt tuning, Zhang et al. [10] specifically
tailor this technique for federated settings. The paper presents a framework
that optimizes the deployment of prompt tuning across a distributed network,
ensuring that all nodes contribute effectively to the model’s learning process
without overwhelming the network’s bandwidth. The framework also includes a
novel synchronization mechanism that aligns the updates from all nodes to
improve the overall stability and accuracy of the model. By focusing on the
efficient distribution and synchronization of prompts, this study further
refines the practical application of prompt tuning in federated environments.

Lai et al. [30] offer a comprehensive benchmarking study that evaluates the
performance of various federated learning models, including LLMs, across
different scales and settings. The paper identifies key bottlenecks in scaling
federated learning systems and proposes solutions to overcome them. Among the
highlighted solutions are strategies for efficient data sampling and
distribution, model compression techniques, and robust aggregation methods
that can handle large-scale deployments. This benchmarking is crucial for
understanding how different models and systems perform in real-world
scenarios, providing a foundation for further optimization of federated
learning frameworks.

Xu et al. [31] introduce a forward gradient technique that optimizes the
gradient computation and transmission in federated learning of LLMs. By
calculating and sharing forward gradients instead of the full gradients, this
method significantly reduces the amount of data that needs to be transmitted
between nodes and the central server. This is particularly beneficial in
scenarios with limited bandwidth or where data privacy is a concern. The
technique also includes mechanisms for aggregating these gradients
efficiently, ensuring that the model converges quickly without compromising on
performance.

Together, these papers present a comprehensive view of current innovations in
the efficient fine-tuning of large language models within federated learning
frameworks. Starting from foundational privacy-preserving techniques, the
discussion progresses through enhancements in parameter efficiency via prompt
tuning, system-wide optimizations for handling scale, and innovative gradient
management for improved communication efficiency. Each paper contributes to a
layered understanding of how to optimize the training and deployment of LLMs
across distributed networks, ensuring both performance and practical viability
in real-world applications. This narrative arc not only highlights individual
advancements but also illustrates the synergistic potential of combining these
techniques to address the multifaceted challenges of federated learning.

In the realm of swarm intelligence within federated learning environments, the
contributions of these papers are pivotal in demonstrating how decentralized
systems can collaboratively refine and fine-tune large language models (LLMs)
efficiently. [28] introduces an architecture that incorporates differential
privacy directly into the federated training process, reflecting the swarm
intelligence concept of achieving collective goals while maintaining
individual privacy. [29] and [10] further the narrative by focusing on prompt
tuning—modifying a small subset of model parameters—thus reducing the
complexity and computational load of federating large-scale models. This
approach allows each node, akin to an agent in a swarm, to contribute more
effectively and efficiently to the collective intelligence, optimizing the
system’s overall learning capability.

[30] provides benchmarking insights that help understand the performance and
system constraints when scaling up federated learning—akin to assessing the
operational capacity of a swarm over varied environments. Lastly, [31]
enhances the communication efficiency between nodes using a forward gradient
method, significantly reducing the bandwidth needed for model updates. This
mirrors a swarm’s efficiency in using limited resources to maintain robust
communication across the group. Collectively, these studies exemplify how
federated learning can harness swarm intelligence principles to achieve
decentralized problem-solving and learning, effectively managing resources and
maintaining synchronization across distributed agents to optimize collective
outcomes in the training and deployment of LLMs.

###  3.3 Pre-Training of LLM in Federated Learning

Table 3: Pre-Training of LLM in Federated Learning No. |  Contributions |  Technique |  LM used |  Dataset |  Limitations
---|---|---|---|---|---
1 |  Adapts BERT pre-training to federated learning, maintaining data privacy across distributed datasets. |  Federated learning adaptations, privacy enhancements |  BERT |  GLUE |  Uneven data distribution, increased complexity in training.
2 |  Explores practical deployment of pretrained models in federated learning, addressing data heterogeneity. |  Dynamic update rates, selective parameter updating |  DistiBERT, BART |  SST2, OntoNotes, etc |  Sync issues, managing diverse data distributions.
3 |  Enhances multilingual understanding in federated learning using pretrained models across diverse languages. |  Multilingual model adaptation |  Multilingual BERT |  MTNT |  Requires careful handling of linguistic diversity, potential bias in language representation.
4 |  Reduces communication overhead by employing parameter-efficient fine-tuning techniques in federated settings. |  Layer-wise relevance propagation, parameter-efficient fine-tuning |  Bert-based |  GLUE |  Possible reduction in model accuracy, limited updates to model parameters.

Tian et al. [32] explore the feasibility of applying federated learning to the
pre-training phase of BERT (Bidirectional Encoder Representations from
Transformers), a popular language model. Named FedBERT, this approach adapts
the pre-training process to work across distributed data sets without
compromising data privacy. The study demonstrates how federated learning can
be effectively utilized to pre-train BERT with data from multiple
decentralized sources, ensuring that the private data remains local while
still contributing to the training of a robust model. The paper highlights
modifications to the BERT training algorithm to accommodate the federated
setting, including adjustments to handle the uneven distribution of data
across nodes.

Focusing on the practical aspects, Agarwal et al. [33] provide a detailed
examination of deploying pretrained language models in a federated learning
framework. It discusses the challenges and solutions related to
synchronization, data heterogeneity, and maintaining model performance when
the pretrained model is adapted to new datasets across distributed
environments. The paper proposes several optimization techniques to enhance
the efficiency and effectiveness of federated learning for pretrained models,
such as dynamic update rates and selective parameter updating to cope with the
diverse data distributions typically encountered in federated settings.

Addressing the challenge of multilingual contexts, Weller et al. [34]
introduce techniques for leveraging pretrained language models to enhance
language understanding across different languages in a federated learning
scenario. It explores the application of multilingual BERT models, adapting
them to federated settings to improve model training across linguistically
diverse data. This approach not only broadens the applicability of federated
learning but also enhances the inclusivity of language technologies, allowing
for effective model training without centralizing data from various language
communities.

Malaviya et al. [35] address one of the significant challenges in federated
learning: the high communication overhead. It presents strategies for reducing
bandwidth consumption during the pre-training of language models in federated
setups. By employing parameter-efficient techniques such as layer-wise
relevance propagation and other fine-tuning methods that minimize the number
of parameters that need to be updated and transmitted, the paper successfully
decreases the network load, which is crucial for practical implementations of
federated learning where bandwidth may be limited.

These papers collectively cover a spectrum of methodologies for integrating
pre-training phases of large language models with federated learning
principles. Starting from adapting existing models like BERT to federated
settings, moving through practical deployment considerations, addressing
multilingual training needs, reducing communication overhead, and finally
discussing comprehensive lifecycle approaches, these studies showcase a
progressive enhancement in the techniques and applications of federated
learning. Each paper builds on the notion that federated learning can be
effectively scaled and adapted to pre-train language models in a way that
respects privacy, manages resources efficiently, and embraces linguistic
diversity, thus advancing the field towards more inclusive and technologically
feasible solutions.

The four papers significantly advance swarm intelligence principles in
federated learning by demonstrating how large language models can be
collaboratively pre-trained across distributed nodes. Each paper introduces a
different facet of decentralized intelligence: [32] adapts BERT for privacy-
preserving collaborative training, [33] tackles practical deployment
challenges such as dynamic updating to handle data heterogeneity, [34] extends
federated learning to multilingual contexts enhancing linguistic inclusivity,
and [35] reduces communication overhead with parameter-efficient techniques.
Together, these studies exemplify how federated learning can emulate swarm
behavior, optimizing collaborative tasks through decentralized interactions
and contributing to the development of robust, scalable models without
centralized control.

###  3.4 Scalable LLM via Federated Learning

Table 4: Scalable LLM via Federated Learning No. |  Contributions |  Technique |  LM used |  Dataset |  Limitations
---|---|---|---|---|---
1 |  Develops FATE-LLM, a framework for scalable federated learning of LLMs, with robustness against node dropout and data discrepancies. |  Hierarchical aggregation |  LLaMa, GPT-2, etc. |  AdvertiseGen |  May struggle with extremely large datasets or very high node dropout rates.
2 |  Enhances federated learning from pre-trained models using contrastive learning to maintain model quality with non-IID data. |  Contrastive learning |  Not specified |  DomainNet, Office-10, etc. |  Requires careful tuning to avoid negative impacts on convergence.
3 |  Investigates the impact of large cohorts on federated learning efficiency, proposing dynamic cohort size adjustments. |  Dynamic cohort sizing |  Not specified |  Stack Overflow, SHAKESPEARE, etc. |  Communication overhead can still be significant in very large networks.
4 |  Presents an architecture for efficiently fine-tuning LLMs in federated settings, reducing resource demands. |  Selective parameter updating |  DistilBERT |  IMDB, AG News, etc. |  Limited by the granularity of parameter selection and update mechanisms.
5 |  Discusses methodologies for training larger models in cross-device federated learning through model splitting and layered updates. |  Model splitting, layered updates |  LSTM, Transformer, etc. |  Stack Overflow |  May face challenges in ensuring model consistency and managing update synchronization.

Fan et al. [36] introduce FATE-LLM, a framework designed for training large
language models using federated learning across highly distributed and
heterogeneous environments. It focuses on overcoming the scalability
challenges inherent in coordinating and aggregating updates across numerous
devices. FATE-LLM employs a hierarchical aggregation strategy that reduces the
communication burden and latency, enabling effective scalability without
compromising the learning efficiency or model accuracy. The framework also
includes robustness against node dropout and data discrepancies, making it
particularly suitable for real-world applications where network conditions and
data distributions can be highly variable.

Tan et al. [37] explore the use of contrastive learning techniques to enhance
the effectiveness of federated learning when starting from pre-trained
language models. By leveraging contrastive learning, the approach aims to
maximize the relevance of local updates to the global model, thus enhancing
the overall model performance even when individual clients have sparse or non-
IID (non-independent and identically distributed) data. The paper discusses
how this technique can be particularly useful in maintaining model quality
while scaling up the number of participants in a federated learning setup,
addressing common issues such as catastrophic forgetting and model divergence.

Charles et al. [38] investigate the effects of increasing the cohort size (the
number of participating nodes in each training round) on the efficiency and
effectiveness of federated learning systems. It proposes a method for
dynamically adjusting the cohort size based on real-time assessments of
network bandwidth and participant availability, optimizing resource allocation
and training speed. The paper provides empirical evidence that larger cohorts
can lead to faster convergence and improved model performance, provided that
the communication and aggregation protocols are efficiently managed.

Hilmkil et al. [39] address the challenge of fine-tuning large pre-trained
language models in a federated setting. It presents a novel architecture that
allows for efficient distribution of model parameters and selective updating
of those parameters most relevant to specific tasks or data types. The
approach helps mitigate the high resource demands typically associated with
large model fine-tuning, making it feasible to scale up federated learning to
handle large language models across extensive networks of distributed nodes.

Focusing on cross-device scenarios, Ro et al. [40] explore methodologies to
scale up the size of language models that can be effectively trained in a
federated learning framework involving a wide array of devices. It introduces
techniques such as model splitting and layered updates to manage the
computational and memory constraints of devices, allowing even those with
limited capabilities to participate in the training process. The paper
emphasizes the scalability and flexibility of federated learning approaches in
accommodating large models without requiring high-end hardware on each client
device.

These papers collectively advance the field of federated learning by
addressing various aspects of scalability when training large language models.
From hierarchical aggregation and contrastive learning approaches that enhance
communication efficiency and model relevance, to dynamic cohort management and
architectural innovations for fine-tuning, each paper contributes to
overcoming the significant challenges of scaling LLMs in distributed
environments. By implementing these methodologies, federated learning can
harness the power of swarm intelligence, where many distributed agents
(devices) work together to solve complex problems, leading to the development
of more robust and scalable language models that can operate effectively
across diverse and widespread data sources.

These papers provide significant insights into how swarm intelligence
principles can be applied to scale large language models (LLMs) in federated
learning environments. Each paper introduces innovative techniques that
enhance collaborative learning across distributed networks, reflecting key
aspects of swarm behavior such as decentralized decision-making and collective
problem-solving. The first paper [36], with its hierarchical aggregation
framework, exemplifies how individual nodes can efficiently contribute to a
global model without centralized oversight, much like how insects in a swarm
interact locally with simple rules that lead to complex group behavior. [37]’s
use of contrastive learning further leverages local computations to maintain
the relevance and accuracy of the global model, ensuring that each node’s
update significantly contributes to the collective knowledge. [38]’s dynamic
cohort sizing adapts to changing network conditions and resources, optimizing
the learning process similar to how swarms dynamically adjust to environmental
changes. Lastly, the paper [39] on model splitting and layered updates allows
even resource-constrained devices to participate in the learning process,
promoting inclusivity and resilience, akin to a swarm’s ability to adapt and
thrive despite individual limitations. Collectively, these studies advance the
implementation of swarm intelligence in federated settings, demonstrating how
decentralized, collaborative efforts can lead to robust, scalable, and
efficient outcomes in the training of sophisticated models.

###  3.5 Security and Privacy in Federated LLM

Table 5: Security and Privacy in Federated LLM No. |  Contributions |  Technique |  LM used |  Dataset |  Limitations
---|---|---|---|---|---
1 |  Analyzes potential privacy leaks and suggests enhanced cryptographic measures and robust aggregation algorithms. |  Cryptographic measures, robust aggregation |  GPT-2, RoberTa, etc. |  IMDB, Yelp, etc. |  May involve high computational overhead and complexity in implementation.
2 |  Introduces an open-source framework incorporating SMPC and homomorphic encryption to enhance security. |  SMPC, homomorphic encryption |  GPT-4, LLaMa-2, etc. |  Alpaca, UltraFeedback, etc. |  Potential performance degradation due to encryption overhead.
3 |  Demonstrates practical attacks for recovering private text and discusses countermeasures. |  Attack simulation, countermeasure development |  GPT-2 |  WikiText-103 |  Countermeasures may not fully prevent leakage and could affect model efficiency.
4 |  Investigates unintended memorization and proposes best practices to reduce data leakage. |  Analysis of memorization, training modifications |  Bert-based |  Stack Overflow |  Best practices may not be universally effective across all model types and scenarios.

In this section, we explore the challenges and solutions associated with
maintaining privacy and security while training large language models using
federated learning.

Vu et al. [41] provide an in-depth analysis of how privacy can be compromised
during the federated training of large language models. It identifies specific
vulnerabilities where sensitive data could potentially be extracted by
adversaries through model inversion attacks and other inference techniques.
The study systematically examines the types of information that can be leaked
and under what conditions, offering insights into the limitations of current
privacy-preserving methods like differential privacy. The paper suggests
enhanced cryptographic measures and more robust aggregation algorithms to
mitigate these risks, setting a foundational context for the need for advanced
privacy-preserving techniques in federated learning.

Ye et al. [42] introduce an open-source framework specifically designed to
support the development and deployment of federated learning models with an
emphasis on privacy and security. This paper describes the architecture of the
framework, which includes built-in support for secure multi-party computation
(SMPC) and homomorphic encryption, technologies that allow computations to be
performed on encrypted data. The framework aims to provide a practical
solution to the privacy concerns highlighted in the first paper, facilitating
the secure aggregation of updates from multiple clients without exposing their
raw data, thereby enhancing the overall security of the federated learning
process.

Building on the concerns about privacy leaks, Gupta et al. [43] demonstrate a
practical attack scenario where sensitive information is extracted from a
federated language model. The researchers show how certain training techniques
and model configurations can inadvertently lead to the memorization of private
text, which can then be recovered by malicious participants. The paper tests
various scenarios and configurations, providing empirical evidence of the
risks involved. It also discusses potential countermeasures, such as more
stringent data sanitization and the use of privacy-preserving architectures,
to prevent such vulnerabilities.

Thakkar et al. [44] complement the third by further investigating how language
models, especially those trained under federated learning conditions, may
unintentionally memorize and expose sensitive data. It delves into the
mechanics of memorization in neural networks, exploring how different factors
like model size, dataset diversity, and training duration affect the risk of
data leakage. The study proposes a set of best practices and modifications to
the training algorithms that can help reduce the likelihood of such
memorization without significantly impacting model performance.

Together, these papers create a layered understanding of the privacy and
security challenges in federated learning of LLMs. Starting from identifying
potential privacy leaks, moving to a framework designed to address these leaks
through encryption and secure computation, and finally, demonstrating
practical attack vectors and offering solutions to mitigate these issues, the
narrative crafted by these studies underscores the complexity of securely
training language models in a distributed manner. Each paper builds on the
insights of its predecessors, collectively advancing the field towards more
secure and private federated learning environments. This progression not only
highlights the vulnerabilities but also charts a path toward resolving them,
ensuring that federated learning can be safely employed in sensitive
applications.

The papers in this section contribute to the principles of swarm intelligence
in the context of federated learning for large language models by addressing
complex challenges related to security and privacy—key aspects for
collaborative and distributed systems. First, by analyzing potential privacy
leaks, the initial paper lays the groundwork for understanding vulnerabilities
akin to assessing environmental risks in a swarm. [42] introduces a robust
framework with advanced encryption methods, enhancing collective security
without centralized control, much like a swarm’s distributed processing
enhances collective resilience. [43] showcases practical attack simulations
and mitigations, reinforcing the swarm’s adaptability and response to threats.
Finally, [44]’s investigation into unintended data memorization offers
strategies to minimize such risks, promoting safer learning environments.
Collectively, these studies embody swarm intelligence by improving the
collective learning and decision-making processes in a decentralized, secure,
and private manner, ensuring the swarm (network of nodes) remains robust
against internal and external adversarial influences.

##  4 Learned Lessons and Open Challenges

In this section, we show the learned lessons, open challenges, and future
research directions derived from the review and analysis.

###  4.1 Learned Lessons

Efficiency in Distributed Environments: Implementing LLMs within federated
frameworks has demonstrated significant improvements in computational
efficiency. Techniques such as model compression and data pruning have proven
effective in reducing the computational load on individual nodes, thereby
enhancing overall system efficiency.

Enhanced Scalability: The application of swarm principles has shown that LLMs
can be effectively scaled across a larger network of distributed nodes. This
scalability is crucial in handling vast amounts of decentralized data,
ensuring the LLMs remain effective even as the network expands.

Emergence of Swarm Intelligence: The self-organizing nature of swarm
intelligence has been successfully harnessed to improve collective decision-
making processes in LLMs. This emergence has been pivotal in enhancing the
adaptability of models to new, unforeseen conditions without centralized
control.

Improved Privacy Protections: Federated learning inherently supports privacy
by design, allowing for the training of LLMs without requiring the sharing of
raw data. Advanced cryptographic techniques and secure multi-party computation
have further bolstered these protections, ensuring data remains confidential
even in collaborative environments.

Robustness and Fault Tolerance: The decentralized approach has inherently
improved the robustness and fault tolerance of systems. Swarm-based LLMs are
less susceptible to single points of failure, maintaining functionality across
the network even when individual nodes encounter issues.

###  4.2 Open Challenges and Future Directions

Communication and Coordination Overhead: As the number of nodes increases, so
does the complexity of managing communication and coordination among them.
Future research needs to focus on developing more efficient communication
protocols and algorithms that minimize latency and bandwidth usage while
maintaining model accuracy.

Consistency in Learning Across Nodes: Ensuring consistent learning outcomes
across heterogeneous nodes remains challenging. Investigating new methods for
achieving better model convergence and addressing the non-IID nature of
distributed data sources is critical.

Security Against Adversarial Attacks: While privacy is enhanced, security
against adversarial attacks in a federated setting continues to be a
challenge. Research into more robust defense mechanisms to protect against
such attacks, particularly in the context of LLMs, is urgently needed.

Handling of Data Heterogeneity: The diverse and distributed nature of data in
swarm-based systems presents significant challenges in training uniform and
unbiased models. Advanced strategies for effective data sampling and
aggregation are required to address these issues.

Exploration of Hybrid Models: There is potential in exploring hybrid models
that combine the strengths of both centralized and decentralized learning
frameworks. Such models could potentially leverage the efficiency of
centralized processing with the robustness and privacy of decentralized
approaches.

##  5 CONCLUSION

This survey has examined the innovative convergence of swarm intelligence,
federated learning, and LLMs, revealing a transformative approach to
decentralized AI systems. We have demonstrated that the synergy between these
technologies not only enhances the efficiency and scalability of AI models but
also significantly bolsters their privacy and security features. While the
journey thus far has illuminated various promising strategies and noteworthy
achievements, it has also highlighted substantial challenges that remain in
communication efficiency, data consistency, and security. The future of
integrating LLMs with swarm intelligence in a federated context looks bright
but requires continued interdisciplinary efforts to refine the technologies
and methodologies. As this field evolves, it holds the potential to
revolutionize how AI systems are designed and implemented, promising more
robust, adaptive, and privacy-preserving models that are capable of operating
autonomously across distributed environments.

## ACKNOWLEDGMENT

TBA

## References

  * [1] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” _OpenAI Blog_ , 2019. [Online]. Available: https://openai.com/blog/better-language-models/
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423
  * [3] M. Shanahan, K. McDonell, and L. Reynolds, “Role play with large language models,” _Nature_ , vol. 623, no. 7987, pp. 493–498, 2023.
  * [4] L. Sun, Y. Han, Z. Zhao, D. Ma, Z. Shen, B. Chen, L. Chen, and K. Yu, “Scieval: A multi-level large language model evaluation benchmark for scientific research,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 38, no. 17, 2024, pp. 19 053–19 061.
  * [5] S. Warnat-Herresthal, H. Schultze, K. L. Shastry, S. Manamohan, S. Mukherjee, V. Garg, R. Sarveswara, K. Händler, P. Pickkers, N. A. Aziz _et al._ , “Swarm learning for decentralized and confidential clinical machine learning,” _Nature_ , vol. 594, no. 7862, pp. 265–270, 2021.
  * [6] J. Tang, H. Duan, and S. Lao, “Swarm intelligence algorithms for multiple unmanned aerial vehicles collaboration: A comprehensive review,” _Artificial Intelligence Review_ , vol. 56, no. 5, pp. 4295–4327, 2023.
  * [7] T. Alqahtani, H. A. Badreldin, M. Alrashed, A. I. Alshaya, S. S. Alghamdi, K. bin Saleh, S. A. Alowais, O. A. Alshaya, I. Rahman, M. S. Al Yami _et al._ , “The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and research,” _Research in Social and Administrative Pharmacy_ , 2023.
  * [8] B. D. Lund, T. Wang, N. R. Mannuru, B. Nie, S. Shimray, and Z. Wang, “Chatgpt and a new academic reality: Artificial intelligence-written research papers and the ethics of the large language models in scholarly publishing,” _Journal of the Association for Information Science and Technology_ , vol. 74, no. 5, pp. 570–581, 2023.
  * [9] W. He, H. Yao, T. Mai, F. Wang, and M. Guizani, “Three-stage stackelberg game enabled clustered federated learning in heterogeneous uav swarms,” _IEEE Transactions on Vehicular Technology_ , 2023.
  * [10] Z. Zhang, Y. Yang, Y. Dai, Q. Wang, Y. Yu, L. Qu, and Z. Xu, “Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models,” in _Annual Meeting of the Association of Computational Linguistics 2023_. Association for Computational Linguistics (ACL), 2023, pp. 9963–9977.
  * [11] D. Chai, L. Wang, L. Yang, J. Zhang, K. Chen, and Q. Yang, “A survey for federated learning evaluations: Goals and measures,” _IEEE Transactions on Knowledge and Data Engineering_ , 2024.
  * [12] J. Tang, G. Liu, and Q. Pan, “A review on representative swarm intelligence algorithms for solving optimization problems: Applications and trends,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 8, no. 10, pp. 1627–1643, 2021\.
  * [13] J. Kennedy, “Swarm intelligence,” in _Handbook of nature-inspired and innovative computing: integrating classical models with emerging technologies_. Springer, 2006, pp. 187–219.
  * [14] M. G. Hinchey, R. Sterritt, and C. Rouff, “Swarms and swarm intelligence,” _Computer_ , vol. 40, no. 4, pp. 111–113, 2007.
  * [15] A. Birhane, A. Kasirzadeh, D. Leslie, and S. Wachter, “Science in the age of large language models,” _Nature Reviews Physics_ , vol. 5, no. 5, pp. 277–280, 2023.
  * [16] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang, “Transformer in transformer,” _Advances in neural information processing systems_ , vol. 34, pp. 15 908–15 919, 2021.
  * [17] P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [18] S. M. Xie, H. Pham, X. Dong, N. Du, H. Liu, Y. Lu, P. S. Liang, Q. V. Le, T. Ma, and A. W. Yu, “Doremi: Optimizing data mixtures speeds up language model pretraining,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [19] Z. Ma, H. Zhang, and J. Liu, “Mm-rnn: A multimodal rnn for precipitation nowcasting,” _IEEE Transactions on Geoscience and Remote Sensing_ , 2023\.
  * [20] Z. Wang, X. Su, and Z. Ding, “Long-term traffic prediction based on lstm encoder-decoder architecture,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 10, pp. 6561–6571, 2020.
  * [21] B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,” _ACM Computing Surveys_ , vol. 56, no. 2, pp. 1–40, 2023.
  * [22] M. Xu, H. Du, D. Niyato, J. Kang, Z. Xiong, S. Mao, Z. Han, A. Jamalipour, D. I. Kim, X. Shen _et al._ , “Unleashing the power of edge-cloud generative ai in mobile networks: A survey of aigc services,” _IEEE Communications Surveys & Tutorials_, 2024.
  * [23] M. Chen, A. T. Suresh, R. Mathews, A. Wong, C. Allauzen, F. Beaufays, and M. Riley, “Federated learning of n-gram language models,” in _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , 2019, pp. 121–130.
  * [24] K. Singhal, H. Sidahmed, Z. Garrett, S. Wu, J. Rush, and S. Prakash, “Federated reconstruction: Partially local federated learning,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 11 220–11 232, 2021.
  * [25] W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y. Xie, Y. Li, B. Ding, and J. Zhou, “Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,” _arXiv preprint arXiv:2309.00363_ , 2023.
  * [26] Y. Shen, J. Shao, X. Zhang, Z. Lin, H. Pan, D. Li, J. Zhang, and K. B. Letaief, “Large language models empowered autonomous edge ai for connected intelligence,” _IEEE Communications Magazine_ , 2024.
  * [27] J. Jiang, X. Liu, and C. Fan, “Low-parameter federated learning with large language models,” _arXiv preprint arXiv:2307.13896_ , 2023.
  * [28] J. H. Ro, S. Bhojanapalli, Z. Xu, Y. Zhang, and A. T. Suresh, “Efficient language model architectures for differentially private federated learning,” _arXiv preprint arXiv:2403.08100_ , 2024.
  * [29] T. Che, J. Liu, Y. Zhou, J. Ren, J. Zhou, V. Sheng, H. Dai, and D. Dou, “Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization,” in _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023, pp. 7871–7888.
  * [30] F. Lai, Y. Dai, S. Singapuram, J. Liu, X. Zhu, H. Madhyastha, and M. Chowdhury, “Fedscale: Benchmarking model and system performance of federated learning at scale,” in _International conference on machine learning_. PMLR, 2022, pp. 11 814–11 827.
  * [31] M. Xu, Y. Wu, D. Cai, X. Li, and S. Wang, “Federated fine-tuning of billion-sized language models across mobile devices,” _arXiv preprint arXiv:2308.13894_ , 2023.
  * [32] Y. Tian, Y. Wan, L. Lyu, D. Yao, H. Jin, and L. Sun, “Fedbert: When federated learning meets pre-training,” _ACM Transactions on Intelligent Systems and Technology (TIST)_ , vol. 13, no. 4, pp. 1–26, 2022.
  * [33] A. Agarwal, M. Rezagholizadeh, and P. Parthasarathi, “Practical takes on federated learning with pretrained language models,” in _Findings of the Association for Computational Linguistics: EACL 2023_ , 2023, pp. 454–471.
  * [34] O. Weller, M. Marone, V. Braverman, D. Lawrie, and B. Van Durme, “Pretrained models for multilingual federated learning,” in _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2022, pp. 1413–1421.
  * [35] S. Malaviya, M. Shukla, and S. Lodha, “Reducing communication overhead in federated learning for pre-trained language models using parameter-efficient finetuning,” in _Conference on Lifelong Learning Agents_. PMLR, 2023, pp. 456–469.
  * [36] T. Fan, Y. Kang, G. Ma, W. Chen, W. Wei, L. Fan, and Q. Yang, “Fate-llm: A industrial grade federated learning framework for large language models,” _arXiv preprint arXiv:2310.10049_ , 2023.
  * [37] Y. Tan, G. Long, J. Ma, L. Liu, T. Zhou, and J. Jiang, “Federated learning from pre-trained models: A contrastive learning approach,” _Advances in neural information processing systems_ , vol. 35, pp. 19 332–19 344, 2022.
  * [38] Z. Charles, Z. Garrett, Z. Huo, S. Shmulyian, and V. Smith, “On large-cohort training for federated learning,” _Advances in neural information processing systems_ , vol. 34, pp. 20 461–20 475, 2021.
  * [39] A. Hilmkil, S. Callh, M. Barbieri, L. R. Sütfeld, E. L. Zec, and O. Mogren, “Scaling federated learning for fine-tuning of large language models,” in _International Conference on Applications of Natural Language to Information Systems_. Springer, 2021, pp. 15–23.
  * [40] J. Ro, T. Breiner, L. McConnaughey, M. Chen, A. Suresh, S. Kumar, and R. Mathews, “Scaling language model size in cross-device federated learning,” in _Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022)_ , 2022, pp. 6–20.
  * [41] M. Vu, T. Nguyen, M. T. Thai _et al._ , “Analysis of privacy leakage in federated large language models,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2024, pp. 1423–1431.
  * [42] R. Ye, W. Wang, J. Chai, D. Li, Z. Li, Y. Xu, Y. Du, Y. Wang, and S. Chen, “Openfedllm: Training large language models on decentralized private data via federated learning,” _arXiv preprint arXiv:2402.06954_ , 2024.
  * [43] S. Gupta, Y. Huang, Z. Zhong, T. Gao, K. Li, and D. Chen, “Recovering private text in federated learning of language models,” _Advances in neural information processing systems_ , vol. 35, pp. 8130–8143, 2022.
  * [44] O. D. Thakkar, S. Ramaswamy, R. Mathews, and F. Beaufays, “Understanding unintended memorization in language models under federated learning,” in _Proceedings of the Third Workshop on Privacy in Natural Language Processing_ , 2021, pp. 1–10.