# Model Agnostic Hybrid Sharding for Heterogeneous Distributed Inference

Claudio Angione  Yue Zhao  Harry Yang  Ahmad Farhan  Fielding Johnston
James Buban  Patrick Colangelo

###### Abstract

The rapid growth of large-scale AI models, particularly large language models
has brought significant challenges in data privacy, computational resources,
and accessibility. Traditional centralized architectures often struggle to
meet required data security and scalability needs which hinders the
democratization of AI systems. Nesa introduces a model-agnostic sharding
framework designed for decentralized AI inference. Our framework uses
blockchain-based sequential deep neural network sharding to distribute
computational tasks across a diverse network of nodes based on a personalised
heuristic and routing mechanism. This enables efficient distributed training
and inference for recent large-scale models even on consumer-grade hardware.
We use compression techniques like dynamic blockwise quantization and mixed
matrix decomposition to reduce data transfer and memory needs. We also
integrate robust security measures, including hardware-based trusted execution
environments to ensure data integrity and confidentiality. Evaluating our
system across various natural language processing and vision tasks shows that
these compression strategies do not compromise model accuracy. Our results
highlight the potential to democratize access to cutting-edge AI technologies
by enabling secure and efficient inference on a decentralized network.

##  1 Introduction

Centralized AI inference poses significant risks related to data privacy,
computational bottlenecks, deterministic output, and single points of failure.
The prohibitive cost and scarcity of high performance computing resources
prevent the mass adoption of the only counter option to centralization, which
is open-source decentralized AI. These challenges further limit the ability to
contribute to training, fine-tuning, and AI inference at scale. This severely
prohibits the adoption of state-of-the-art AI models for enterprises and
developers, as well as shared research around the world.

Recent large language models are available with more than 100 billion
parameters [1, 2, 3, 4], which makes it important to train them on powerful
and costly accelerated hardware such as GPUs and TPUs. [5]. Several approaches
can be used to make these models more accessible, especially during the fine-
training and inference phase. Using the APIs is one possible approach, which
allows quicker inference passes from pre-trained models, but offers little
customisation capacity, and no options to change or optimise the training
process [2, 6]. A second solution is offloading, where the model components
are moved to slower memory (e.g. RAM or SSD) [7]. Then, only the relevant
portion of the model is iteratively moved to the available GPU, allowing it to
run on less expensive hardware. However, this process involves frequent data
transfers and can be extremely slow for larger models.

Collaborative efforts to AI execution today are further hindered by the
challenges around security [8]. Techniques like continuous and domain
adaptation offer partial solutions by enabling model sharing without direct
data exchange [9]. However, these methods are prone to backdoor attacks and
often result in sub-optimal model performance due to the limitations of semi-
supervised or unsupervised fine-tuning. Collectively, these issues have real-
world implications for businesses requiring critical inference. For example,
Traditional centralized systems fail to meet the strict requirements for data
privacy in the financial domain, where institutions analyze vast amounts of
sensitive transactional data [10]. Hence, existing decentralized systems fail
to achieve the need for both confidentiality and verifiability.

To address these challenges and enable AI democratization, Nesa introduces the
first model-agnostic hybrid sharding approach layered onto a hardware-based
and software-based privacy co-optimization protocol. This solution distributes
computational loads across multiple nodes on a decentralized inference
network. Nesa’s distributed inference protocol delivers privacy-preserving
data processing while facilitating the scalable and auditable execution of AI
inference. The result is a network where anyone can fine-tune models and
perform inference queries without substantial investments in computational
infrastructure [11]. This new protocol, based on its dynamic routing
mechanism, greatly lowers the barrier to entry for participating nodes by
accommodating various levels of computational capabilities, which is
particularly relevant given the high hardware requirements of competing
systems that restrict inference participation only to entities with access to
top-tier GPUs.

Figure 1: High-level overview of the Nesa network.

We integrate recent model optimization techniques like Zero Redundancy
Optimizer (ZeRO), quantization [12], Parameter-Efficient Fine-Tuning (PEFT)
[13], Compression and Low-rank Adaptation (LoRA) [14], to further improve the
computational efficiency of the distributed network and to optimize memory
usage and processing capabilities of nodes responsible for serving the model
segments. This deployment ensures high model accuracy and throughput across
the distributed network, while model shards are secured against tampering via
end-to-end encrypted data sharing and aggregation, compliant with regulatory
standards for all domains of research and enterprise [15, 16].

##  2 Distributed training and inference

Distributed inference and training across multiple nodes are essential due to
the exponential increase in the size of models, their complexity and the
scarcity of computational resources available to process them [17]. The
essence of both of these tasks lies in the necessity to handle vast
computational loads more efficiently and to reduce the latency involved in
generating predictions and updating model parameters [18]. Our approach uses
the collective computational resources and memory available across several
processing nodes. This makes it possible to handle larger models or increased
batch sizes without a proportional increase in inference or training time. One
of the critical aspects of efficient distributed inference and training is
partitioning the computational graph of any neural network. This allows the
nodes with limited resources to only handle a segment of the model during the
inference or training phase. The computational graph represents all the
operations and data flows within the model from input to output. By
Partitioning this graph we divide the model’s computations so that they can be
processed in parallel or in sequence across different nodes [19, 20]. We
integrate optimisation techniques to parallelize the gradient computation and
parameter updates which ultimately reduces the speed of the training process.
Specifically, we use the ZeRO (Zero Redundancy Optimizer) to split the
optimizer state, gradients, and parameters across nodes which reduces the
memory footprint and increases the overall scalability. [21].

###  2.1 Nesa’s model partitioning approach

Due to the novelty of running deep learning algorithms on the blockchain,
Nesa’s partitioning mechanism aims to minimize the amount of data that must be
transferred between different nodes, thus reducing the latency and bandwidth
requirements for both inference and training phases. The overall throughput of
the inference and training tasks is limited by the slowest part of the system,
known as the bottleneck stage [22]. We therefore introduce mechanisms to
balance the workload and avoid bottlenecks. The partitioning must balance the
computational load across all nodes to maximize throughput. This balance
ensures that all parts of the system work at their full potential without any
single stage becoming a drag on performance.

We prioritize fast memory like Static Random-Access Memory (SRAM) for
distributed inference and training across multiple nodes which is essential
for storing intermediate model outputs, i.e., activations, and parameter
weights [23, 24]. SRAM is significantly faster than conventional memory
solutions but is also more expensive and limited in size. Each node in the
network contributes its SRAM, which multiplies the available fast memory
during the distributed inference and training session. This increase in
capacity allows for caching more model parameters in fast memory.

###  2.2 Blockchain-based sequential deep neural network sharding

Nesa developed a new approach for network topology-informed model sharding,
named Blockchain-based Sequential deep neural network Sharding (BSNS). Our
approach establishes and solves an optimization problem that can find a
sequence of nodes able to run an inference request. Each node will be
typically a distributed machine on the network, which will normally run a
block or a contiguous sequence of blocks of a deep learning architecture. It
is crucial that for each inference session involving block sharding, a chain
of nodes is found that can collectively reconstruct the layers of the full
model [25].

The key innovation of our approach is that the selection of nodes for sharding
is informed by: (i) the topology of the network; (ii) our persistent homology
metrics based on graph embeddings and (iii) network-based variables, including
latency and geographical distance between the nodes. Taken together, this will
constitute a full end-to-end neural network that performs inference across the
blockchain at the optimal speed, fully embedding the topology and the security
components of the blockchain.

####  2.2.1 Swarm creation and dynamic rebalancing

The BSNS framework allows arbitrary deep neural networks to be distributed for
training, but with specific focus given to LLMs and transformer-based
architectures. The swarm is formed based on a heuristic that selects the
optimal set from the pool of available nodes. Each node inside the swarm
processes a shard of the network architecture and communicates through a
sequence of remote procedure calls (RPC). Each $N_{i}$ denotes a node within
the network which is responsible for the $i$-th shard of the model. Whereas,
$p$ represents the total number of shards distributed across the swarm. The
selection process heuristic considers different network topology features,
such as node and edge structure, bandwidth, and computational capabilities,
therefore ensuring optimal efficiency in data processing across the swarm.
This has strong practical benefits for our clients, as distance and latency
heavily affect the performance when computing using a blockchain.

Given a network $A$ with $n$ nodes that need to execute $p\leq n$ shards of a
model, and assuming that each block is held by one network node, the task
involves finding a sequence $\mathcal{S}$ of nodes

$\mathcal{S}:(A_{i})_{i\in{1,\ldots,p}}.$ (1)

We model this as a recursive sequence finding problem. Hence, the selection of
each node depends on the previous node(s) shortlisted based on the selection
heuristic parameters.

$A(i)=A_{i}=f((A_{i-1-\alpha},\ldots,A_{i-1});\ \text{network
parameters}(\alpha)),$ (2)

Figure 2: Orchestrator is chosen from the pool of accessible nodes and picks
nodes to build a whole set of blocks (e.g., transformer blocks) according to
the model architecture that needs to be run. To enable direct routing between
blocks, each node has a unique section of the computational graph. Initiating
from the orchestrator, the inference is routed to the first block and then
through each remaining block directly in order.

where $\alpha<i-1$ sets the context length for network inference and affects
the computation of the network parameters. If $\alpha=0$, the network
parameters only involve distance metrics between the current node and the
previous node, while for $\alpha>0$, more than one node is considered. The
BSNS framework introduces a dynamic rebalancing algorithm that adjusts based
on node performance and adapts to adverse network conditions. This algorithm
monitors the network state and status which allows redistribution of model
shards among nodes over time. The swarm rebalance is given by:

${R:S\times P\rightarrow S^{\prime}}$ (3)

where $R$ represents the rebalancing function that transforms the current
sequence of nodes $S$ based on the network parameters $P$. Like the selection
heuristic, this function considers features such as hardware capacity,
throughput, and bandwidth, which influence data transmission speeds between
nodes.

###  2.3 Cache optimization to enhance efficiency

BSNS uses a Key Value (KV) cache for scaling LLMs across a distributed
network. This minimizes the computational overhead associated with token
generation in LLMs when dealing with models that operate on a token-by-token
basis [26, 27]. This mechanism uses the inherent characteristics of
transformer models by caching the key and value vectors after their initial
computation to prevent redundant calculations in subsequent generations. This
also decreases the load on the network’s nodes and allows the processing of
longer sequence lengths in LLMs [28].

Caching improves efficiency by ensuring that each node within a swarm can
generate tokens by utilizing pre-computed key and value vectors. This is a
core factor that increases the overall throughput of the system. Moreover, it
enables the system to scale to models with extensive context sizes by removing
the memory and computational constraints that typically hinder the operation
of large LLMs.

$\text{Cache Size per Token}=2\cdot\text{head\\_dim}\cdot n\\_heads\cdot
n\\_layers,$ (4)

where head_dim represents the dimensionality of the key and value vectors,
$n\\_heads$ is the number of attention heads, and $n\\_layers$ is the number
of transformer layers within the model. This highlights the direct
relationship between the model’s complexity and the size of the KV cache which
shows the need for effective cache management strategies [29].

###  2.4 BSNS with parameter-efficient fine-tuning via adapters

BSNS uses Low-rank adaption techniques to enable fine-tuning capabilities for
arbitrary language models [14]. The adapters are inserted between the
transformer layers which allows learning the downstream tasks without
retraining the entire network as it can be computationally expensive[14, 30].

####  2.4.1 Operational framework for adapters in Nesa

Adapters are small neural network modules inserted between the layers of a
pre-trained model. They allow task-specific training with minimal parameter
updates [14]. The operation of an adapter within a transformer layer is given
as:

$h^{\prime}_{i}=\text{LayerNorm}(h_{i}+W_{2}\phi(W_{1}h_{i})),$ (5)

where $h_{i}$ is the input to the adapter at layer $i$, $W_{1}$ and $W_{2}$
are trainable weights of the adapter, $\phi$ represents a non-linear
activation function, and $h^{\prime}_{i}$ is the output of the adapter [31].

The adaptation of a weight matrix $W$ in a transformer can be modeled as

$W^{\prime}=W+BA,$ (6)

where $W$ is the original weight matrix of the model, $A\in\mathbb{R}^{r\times
n}$ and $B\in\mathbb{R}^{n\times r}$ are the low-rank matrices introduced by
LoRA with $r\ll n$, and $W^{\prime}$ is the adapted weight matrix [14]. Upon
initiating a fine-tuning session, each participating node initializes adapter
modules according to the specified configuration aligned with the shard of the
LLM it is responsible for.

####  2.4.2 Node synchronization

Nodes collaboratively fine-tune their adapter modules using gradients from
task-specific data processed through their portion of the LLM. The process
includes three main steps:

1\. Forward pass. Nodes perform a forward pass through transformer layers and
adapters which generates activations based on the input tensors.

2\. Backward pass. From the final layer of a large language model, gradients
are propagated by comparing predictions with true labels. Gradients for
corresponding adapters are used to perform the gradient descent update.

3\. Synchronization and update. Nodes synchronize their updates across the
network to ensure consistency and convergence

The parameter updates for the adapters, denoted as $\theta_{i}$ for the $i$-th
node, follow the rule:

$\theta^{(t+1)}_{i}=\theta^{(t)}_{i}-\eta\nabla_{\theta_{i}}L(\theta^{(t)}_{i})$
(7)

where $t$ is the current iteration, $\eta$ is the learning rate, and $L$ is
the loss function.

Along with these synchronized update rules, BSNS uses a consensus mechanism
for agreeing on fine-tuning objectives, data distribution, and synchronization
intervals. This setup allows for the sharing and reuse of fine-tuned adapters
and allows nodes to build on existing adaptations for new tasks.

###  2.5 Dynamic sharding of neural networks

The BSNS method works well for LLMs as they have common blocks that can be
distributed on a sequence of nodes. However, for arbitrary layered neural
networks like diffusion or sequence models [32, 33], this is not possible
because of wide architectures and multimodal prediction heads. Nesa utilizes a
dynamic sharding scheme to optimally partition the network’s computation
graph, $G=(V,E)$, across multiple nodes. This graph comprises operations $V$
and data flow edges $E$. Each operation $v\in V$ outputs a tensor, forming an
edge $(u,v)\in E$. It includes all computational tasks from basic arithmetic
to layer-specific matrix multiplications, each with unique computational and
memory requirements: execution time $\text{work}(v)$, parameter memory
$\text{sizeparam}(v)$, and output size $\text{sizeout}(v)$ [21].

Partitioning this graph involves dividing $V$ into $k$ distinct blocks such
that each block can be processed on a different node in a swarm under the
constraint that the induced quotient graph of $G$ remains acyclic. This
division aims to maximize throughput while minimizing inter-node communication
subjected to the bandwidth $B$ between nodes, with the I/O cost from node $S$
to node $T$ given by:

$\text{io}(S,T)=\frac{1}{B}\sum_{v\in N^{-}(T)\cap S}\text{sizeout}(v),$ (8)

where $N^{-}(T)$ represents the set of nodes whose outputs are consumed by
block $T$.

The main challenge is effectively distributing the model’s parameters and
activations across each node’s fast memory (e.g., SRAM) [34]. Parameters that
exceed the fast memory capacity are streamed from slower storage and add
latency. The overflow cost, representing the time to stream parameters beyond
the fast memory limit $M$, is calculated as:

$\text{overflow}(S)=\left(\text{sizeparam}(S)+\text{peak}(S)-M\right)+\frac{\text{peak}(S)}{B},$
(9)

where $\text{peak}(S)$ denotes the peak memory requirement for activations
within block $S$.

The block cost, $f(S)$, combines the costs of receiving input tensors,
executing the block’s operations (including any overflow cost from streaming
parameters), and sending output tensors downstream:

$f(S)=\text{io}(V\setminus S,S)+\sum_{v\in
S}\text{work}(v)+\text{overflow}(S)+\text{io}(S,V\setminus S).$ (10)

The goal of this partitioning, defined by the Max-Throughput Partitioning
Problem (MTPP) [20], is to minimize the maximum cost across all blocks,
optimizing the throughput of the entire pipeline. Formally, MTPP seeks a
partition $P^{*}$ that minimizes the bottleneck cost:

$P^{*}=\text{argmin}_{P\in P_{k}(G)}\left\\{\max_{i\in[k]}f(P_{i})\right\\},$
(11)

where $P_{k}(G)$ denotes the set of all possible partitions of $G$ into $k$
blocks, and $\text{cost}^{*}$ is the minimum achievable bottleneck cost across
these partitions.

###  2.6 Enhanced MTPP slicing of topological order

The MTPP algorithm is key in the optimization of computation graphs for neural
network inference across multiple systems. It is known to be NP-hard, as shown
by its relation to the minimum makespan scheduling problem [35]. This makes
fully polynomial approximations unlikely. Instead, we use a heuristic approach
to handle this complexity and maintain high throughput. For non-transformers
based architectures, our method combines Kahn’s algorithm [36] for topological
sorting with dynamic programming and segment trees. Kahn’s algorithm ensures
correct topological order for acyclic partitioning. Dynamic programming then
calculates the best partitioning and Segment trees quickly adjust partitions
during optimization. This combination helps in distributing the computational
load evenly and reduce communication delays [37, 38].

Our approach centres around the segment cost data structure which calculates
costs based on computation and communication for each segment. We start with a
computation graph $G$ and a topological order $\pi$. The SliceGraph algorithm
uses this to divide the graph into throughput-optimized segments. It
dynamically splits the graph into blocks which distributes workload evenly and
reduces communication between nodes.

Algorithm 1 Enhanced MTPP Slicing of Topological Order $\pi$ into at most $k$
Blocks

1:function EnhancedSliceGraph($G,k,\pi$)

2: Initialize enhanced_segment_cost data structure for $(G,\pi)$ using
heuristic-based weighting

3: return EnhancedDP(enhanced_segment_cost, $n$, $k$)

4:end function

5:

6:function EnhancedDP(enhanced_segment_cost, $r$, $k^{\prime}$)

7: if $k^{\prime}=1$ then

8: return enhanced_segment_cost.Query$(1,r)$ // Leverage the enhanced query
for improved cost estimation

9: end if

10: $ans\leftarrow\infty$

11: for $\ell=1$ to $r-1$ do // Iterate through the nodes

12: $a\leftarrow$ EnhancedDP(enhanced_segment_cost, $\ell$, $k^{\prime}-1$)

13: $b\leftarrow$ enhanced_segment_cost.Query$(\ell+1,r)$

14: $ans\leftarrow\min(ans,\textit{heuristic\\_adjustment}(\max(a,b)))$

15: end for

16: return $ans$

17:end function

The initialization of the segment cost data structure for a graph with $n$
nodes is $O(n^{2})$ which allows us to pre-compute all possible segment costs.
This computation is important for the correct operation of the SliceGraph
algorithm [39]. Initially, the algorithm’s time complexity is
$O(n^{2}k+m\log^{2}(n))$, where $k$ is the number of partitions, and $m$ is
the edge count. This includes the dynamic programming process and cost
queries. Applying the convex hull trick reduces the complexity to $O(nk\log
n)$ and speeds up optimal partitioning.

####  2.6.1 Biased random-key genetic algorithm

BRKGA improves our approach by using stochastic and evolutionary methods which
overcomes the limitations of heuristic-based techniques [40, 41]. This
algorithm enables broad exploration of partition configurations, essential for
solving the NP-hard MTPP. Over generations, BRKGA evolves partitioning
strategies to discover solutions that balance computational load and reduce
communication overhead and achieves optimal throughput in distributed deep
neural network inference [20].

BRKGA evolves node priority vectors $\mathbf{x}\in[0,1]^{n}$ and links these
priorities directly to partitioning quality for adjustments. The process
starts by creating a population of random vectors. These vectors help setup
orders in the SliceGraph algorithm. Each vector’s quality is assessed for
throughput optimization and latency reduction which determines its fitness.
BRKGA refines the strategies until convergence criteria for node priorities is
met. This process involves selection, crossover
$\mathit{new}=\alpha\mathit{parent}_{1}+(1-\alpha)\mathit{parent}_{2}$, and
mutation to generate near optimal offsprings.

The BRKGA approach offers many advantages over sequential sharding methods for
partitioning neural networks. Sequential methods follow a predetermined path
which perhaps can miss more efficient configurations that could improve
computational throughput and reduce communication latency. This adaptability
is important in distributed environments with varying network topologies and
node capabilities.

###  2.7 Swarm topology

Due to the dynamic topology of the swarm network, we further extend our
computational efficiency by monitoring and classifying changes and the current
state of our swarm topology over time. In conjunction, we build several pre-
computed sharding schemas based on the commonly observed swarm topologies as
we monitor our networks. In doing so, we can quickly select among sharding
schemes to find one that is near-optimal for the current swarm topology. In
the following section, we outline how we monitor and analyze the swarm
topology.

####  2.7.1 Topological forecasting and monitoring

Dynamic sharding is computationally expensive as finding an optimal sharding
scheme is known to be NP-hard [20]. We mitigate this expense by precomputing
highly efficient sharding schemas for different topological states of our
network on a per-model basis. As the network topology itself is dynamic, we
periodically monitor our network using techniques from (weighted) persistent
homology [42, 43], to give snapshots of topological characteristics to guide
the applications of our pre-computed sharding schemas. The main goal of this
section will be to introduce many of the concepts behind persistent homology
(which is not a mainstream topic) to guide intuition on how this computational
tool can be leveraged to make dynamic sharding optimization much more
tractable.

As a basic introduction to the implementation of this framework, we briefly
illustrate the mathematical technique of simplicial homology. This uses the
simplicial complex to calculate the homology groups – these groups are
formally Abelian groups but they can be thought of as vector spaces. A
simplicial complex is a collection of simplices that are the convex hull of a
group of vertices. More explicitly, an $n$-simplex $\Delta^{n}$ contains $n+1$
vertices $\\{v_{0},v_{1},\dots,v_{n}\\}$ and is the collection of points in
$\mathbb{R}^{n}$ such that:

$\Delta^{n}=\left\\{k_{0}v_{0}+\cdots+k_{n}v_{n}\ |\
\sum_{i=0}^{n}k_{i}=1\textrm{ and }k_{i}\geq 0\textrm{ for }0\leq i\leq
n\right\\}.$ (12)

Thus, a 0-simplex $\Delta^{0}$ is a point, a 1-simplex $\Delta^{1}$ is a line
segment, a 2-simplex $\Delta^{2}$ is a triangle, and so on. To build a
simplicial complex out of a collection of simplices, we need to define how to
glue the pieces together [44]. Mathematically it is given as (linear) boundary
maps $\partial_{n}^{i}:C^{n}\to C^{n-1}$ which map each $n$-simplex
$\Delta^{n}_{i}$ to its boundary which lives in one dimension lower (if it
exists). Here, were are using the notation $C^{n}$ to indicate the collection
of all $n$-simplices in our complex, sums of which are often referred to as
$n$-chains or simply chains. In general, we can (abstractly) represent the
entire data of the simplicial complex by:

$\cdots\xrightarrow[]{\partial_{n+2}}C^{n+1}\xrightarrow[]{\partial_{n+1}}C^{n}\xrightarrow[]{\partial_{n}}C^{n-1}\xrightarrow[]{\partial_{n-1}}\cdots$
(13)

Referring back to the boundary maps $\partial_{n}^{i}:C^{n}\to C^{n-1}$, the
image of $\partial_{n}^{i}(\Delta^{n}_{i})$ will be the (simplicial) boundary
of $\Delta^{n}_{i}$. We assign an orientation to the simplices to make
boundary calculations consistent. For example, with the 1-simple as the convex
hull of $\\{v_{0},v_{1}\\}$ the image of the boundary map would be defined as
$v_{1}-v_{0}$, with such an orientation induced by imaging the 1-simple is a
directed edge from the point $v_{0}$ to the point $v_{1}$. More explicitly:

$\partial_{1}(\Delta_{0}^{1})=v_{1}-v_{0},$ (14)

where we have intentionally denoted $\Delta_{0}^{1}$ as the simplex
constructed from the vertices $\\{v_{0},v_{1}\\}$. Looking at Figure 3, and
considering the triangle at the left, the collection of 1-chains contains
three edges, which are the convex hulls of the sets, $\\{v_{0},v_{1}\\}$,
$\\{v_{1},v_{2}\\}$ and $\\{v_{0},v_{2}\\}$. The arrows give the orientation
of each edge, which guides the boundary map calculation. We already calculated
the boundary map for the first edge. The remaining two edges, $\Delta_{1}^{1}$
and $\Delta_{2}^{1}$ map can be calculated as:

$\displaystyle\partial_{1}(\Delta_{1}^{1})$ $\displaystyle=v_{2}-v_{1}$ (15)
$\displaystyle\partial_{1}(\Delta_{2}^{1})$ $\displaystyle=v_{0}-v_{2}$

By linearity, we can compute the boundary for the full chain,
$\Delta_{0}^{1}+\Delta_{1}^{1}+\Delta_{2}^{1}$ as:

$\displaystyle\partial_{1}(\Delta_{0}^{1}+\Delta_{1}^{1}+\Delta_{2}^{1})$
$\displaystyle=\partial_{1}(\Delta_{0}^{1})+\partial_{1}(\Delta_{1}^{1})+\partial_{1}(\Delta_{2}^{1})$
(16) $\displaystyle=(v_{1}-v_{0})+(v_{2}-v_{1})+(v_{0}-v_{2})$
$\displaystyle=0.$

Figure 3: Toy example for simplicial homology: the left simplicial complex
consists of three vertices (0-simplices) and three edges (1-simplices) whereas
the right consists of four vertices and four edges. Topologically, these two
complexes are identical from the viewpoint of simplicial homology, but they
are distinguishable via persistent homology.

Thus, the chain $\Delta_{0}^{1}+\Delta_{1}^{1}+\Delta_{2}^{1}$ has no formal
boundary (to be more clear this chain is what most people identify as the
triangle itself). More importantly. This example demonstrates that a cycle (a
chain without a boundary) has a trivial image. This will correlate with some
topological holes in the simplicial complex. Thus, cycles correspond to
elements of the kernel of the boundary map (a kernel of a map is simply all
elements that map to zero). Formally, we calculate the $n$th-homology group
$H_{n}(C^{\star})$ of a simplicial complex $C^{\star}$ as

$H_{n}(C^{\star})=\frac{Z_{n}(C^{\star})}{B_{n}(C^{\star})},$ (17)

where $Z_{n}(C^{\star})$ is the group of $n$-chains that are cycles and
$B_{n}(C^{\star})$ are the group of $n$-chains that are boundaries.

Thinking of these groups as vector spaces with each dimension corresponding to
a simplex and with the operation being vector addition, this definition allows
one to identify chains where the difference is a boundary (a chain that has a
non-trivial image). In other words, adding a boundary chain to a cycle does
not change its class under homology. This is demonstrated in Figure 3 where
the two diagrams are equivalent since the difference in $C^{1}$ is the edge
between $\\{v_{2},v_{3}\\}$. Note that this is a boundary because the value
under the boundary map is $v_{3}-v_{2}\neq 0$, according to the indicated
orientation. It should be further noted that points never have boundaries so
$H_{0}(C^{\star})$ does not count holes, but counts the number of connected
components. Overall, this mathematical formalism allows one to unambiguously
tabulate the topological characteristics of a wide variety of different
geometric objects.

Persistent homology [45] (PH) is based on introducing a filtration of the
topological space as a sequence of simplicial complexes. For our purposes, the
complexes will be derived from the graph corresponding to our network. For
clarity, we will refer to the persistent $n$th homology group for a space $M$
as $PH_{n}(M)$. For our application, the embedding of this graph into
Euclidean space is based on the geographic locations and the edge weighting is
a function of individual node internet connectivity and node computational
power.

Figure 4: Visualizing persistent homology for a simple graph of 3 vertices on
an equilateral triangle: the top row illustrates 3 snapshots of the filtration
sequence of three circles of growing radii (filtration parameter). The bottom
row demonstrates the respective persistence diagrams. Note that in $H_{0}(M)$,
the initial 3 dots are shifted slightly to aid the eye but they would overlap
precisely in reality.

The filtering process in persistent homology for a graph resembles inflating
circles around each network node (vertex) and capturing data about how the
resulting topology changes as the circles begin to merge. One can gain
intuition behind this idea by imagining three points on the vertices of a
triangle (see Figure 4). A scale parameter increases the radius of each point
uniformly. Each step along this parameter change introduces a new step in the
filtration of complexes. Until the circles grow enough to overlap, there will
be three separate components so the zeroth homology group which measures the
number of connected components will have dimension 3 and higher order groups
which measure the number of higher dimensional holes will be trivial (left
schematic in Figure 4). As the radii grow they eventually grow to overlapping
circles. Immediately after the circles overlap there will still be a gap
between the three circles centered in the middle of the triangle, so now
they’re one hole and thus the first homology group suddenly becomes non-
trivial with dimension 1 (center schematic in Figure 4). At the same time, the
number of connected components collapses to 1 so the zeroth homology group
will change from three dimensional to 1 dimensional. As the circles continue
to grow, the gap in the middle will fill, closing the hole and the first
homology group will become trivial again, but the zeroth homology group will
continue to have dimension 1 (right schematic in Figure 4). No further
meaningful changes will occur to the persistent diagram after this point.

The changes illustrated above are typically captured using a persistence
diagram which simply introduces a point for each birth-death pair, where the
x-axis is the birth time and the y-axis is the death time. Here time is just
the scaling parameter. Since births always occur before deaths, all the points
on the diagram will be above the main diagonal line $y=x$. Following the toy
example above, one can imagine that smaller holes will have quicker deaths
since they do not persist long, so they will be near the main diagonal.
Conversely, bigger holes will have a longer lifetime. Intuitively, this gives
information about the network latency, connectivity and possible bottlenecks.
The lifetimes of the connected components, measured by $PH_{0}(M)$, give a
complementary view of these characteristics. This data from both $PH_{0}(M)$
and $PH_{1}(M)$ persistence can be fed into a dedicated ML classifier to
predict which precomputed sharding scheme will be most efficient based on the
current network topology. As calculating persistent homology is not NP-hard
[46] even in dimensions much higher than those used to model graphs, this
gives us an avenue to select a near-optimal precomputed sharding scheme at a
reasonable computational cost.

##  3 Homology enabled routing mechanism

Nesa’s inference framework is based on a Distributed Hash Table (DHT) for
routing data and tasks on the distributed blockchain network [47]. This is a
critical factor in task allocation, and fault tolerance as well as ensuring
node information can be accessed globally. The DHT is immediately updated when
a node becomes unreliable to change and update the heuristic parameters that
allow the network the ability to redeploy without intervention. In the BSNS
framework, this becomes even more important as tasks are distributed across a
swarm of nodes.

The effective routing of data and tasks is crucial in a distributed blockchain
network designed for neural network computations. The routing decision
specifies a subset of nodes that are chosen to serve as the computational
resources and, thus influence system-wide capacity efficiency, latency and
reliability. Nesa’s routing mechanism is dynamically updated based on the node
features and is given as:

Figure 5: Overview of genetic algorithms for optimization and a decision-
making framework based on multi-objective optimization metrics for selecting
nodes in a network.

$A_{i}=\text{argmin}_{A\in\mathcal{N}}\left\\{\sum_{j=1}^{i-1}\left[\left(\frac{D_{j,A}}{B_{j,A}}+L_{j,A}\right)^{\gamma}\right]+\left(\frac{G_{A}}{C_{A}}\right)^{\beta}+\left(\frac{1}{R_{A}+1}\right)^{\alpha}\right\\}$
(18)

We use multiple key metrics which consider the data processing factors,
computational load ratio and reliability of the nodes for forming the swarm
and its dynamic rebalancing. The key data processing factors are as follows:

1\. Data Size ($D_{j,A}$). It measures the size of the payload that needs to
be transferred from node $j$ to node $A$. This factor affects multiple
components including model parameters, activations of intermediate layers and
final outputs.

2\. Bandwidth ($B_{j,A}$). This metric measures the maximum data transfer
capacity between two nodes per unit of time. Better bandwidth indicates a
higher capacity for handling large data volumes and reduces the likelihood of
bottlenecks.

3\. Latency ($L_{j,A}$). Latency is the delay experienced in communication
between two nodes. It can be influenced by physical distances and network
congestion. Lower latency is important in enabling faster fine-tuning and
inference.

These factors measure the combined cost of data transfer and latency. The
hyperparameter $\gamma$ is used for non-linear scaling of this cost and
weights the impact of these factors on overall efficiency. The parameter
$\gamma$ is tuned using genetic algorithms and represents the relative
importance of minimizing data transfer time and latency in different
scenarios.

Along with these data processing factors we use computational load ratio. This
term expresses the balance between available hardware resources and the
current computational load. A higher ratio indicates that a node has more
resources available relative to its load which makes it a better candidate for
selection in a swarm.

1\. GPU Availability ($G_{A}$). This metric represents the available GPU
resources at a node. Nodes report their total GPU capacity and current usage
calculated as follows:

$G_{A}=\text{total GPU capacity}-\text{current GPU usage}$

This helps identify nodes with sufficient GPU resources to handle inference
and fine-tuning tasks without significant delays.

2\. Computational Load ($C_{A}$). This measures the current computational
burden on a node, including CPU and memory usage. It reflects the node’s
active tasks and overall workload. Monitoring $C_{A}$ is important to prevent
overburden and inference failures.

For computational load ratio, the parameter $\beta$ adjusts the sensitivity of
this ratio, prioritizing nodes with sufficient GPU resources for compute-
intensive models. We also consider a resource reputation metric ($R_{A}$) that
assesses the reliability of a node. It incorporates factors such as historical
uptime, failure rates, and maintenance history. A node with high uptime and
low failure rates is considered more reliable. The calculation
$R_{A}=1-\text{Uptime}$ provides a simple measure of reliability, where uptime
is reported as a percentage. The parameter $\alpha$ controls the weight of
reliability in the overall cost calculation, with higher values increasing the
importance of selecting reliable nodes.

###  3.1 Persistent homology integration

Persistent homology in our routing mechanism provides insights into the
swarm’s structure including the stability and connectivity of nodes. It
analyzes the persistence of topological features like connected components and
cycles, which helps to identify stable regions in the network important for
the node selection process. Routing uses persistent homology for network
analysis, filtration and node selection. A filtration is constructed by adding
simplices in a specific order. This process shows how different features of
the network emerge or disappear as the scale changes. For example, at a small
scale, only the strongest connections (shortest edges) are included as they
represent the core structure of the network. As the scale increases, weaker
connections are added.

Nodes that are part of long-lived features are considered more stable and
reliable. For example, a node consistently appearing in a connected component
with a high $\beta_{0}$ value across multiple scales will likely reside in a
stable network region. This stability influences the adjustment of $R_{A}$
values in the routing mechanism and favors nodes in these regions. If
persistent homology shows that a cluster of nodes consistently forms a
connected component, these nodes are deemed more reliable, impacting their
$R_{A}$ values.

###  3.2 Evolutionary algorithm for parameter tuning

The parameters $\gamma$, $\beta$, and $\alpha$ are critical in balancing the
cost components in the routing mechanism. We use a genetic algorithm [48] to
optimize these parameters which allows routing to adapt to varying network
conditions and workloads.

1\. Initialization. The algorithm starts with a population of candidate
parameter sets, each represented by a tuple $(\gamma,\beta,\alpha)$. These
candidates are initialized randomly in a predefined range for a diverse
starting point for the optimization process.

2\. Fitness Evaluation. The fitness of each candidate is evaluated based on
the total cost $F(\gamma,\beta,\alpha)$. The fitness function is inversely
related to the cost where lower costs indicate higher fitness. This step
involves calculating the objective function for each candidate, using real-
time data from the network.

$F(\gamma,\beta,\alpha)=\sum_{i=1}^{n}\left\\{\sum_{j=1}^{i-1}\left[\left(\frac{D_{j,A}}{B_{j,A}}+L_{j,A}\right)^{\gamma}\right]+\left(\frac{G_{A}}{C_{A}}\right)^{\beta}+\left(\frac{1}{R_{A}+1}\right)^{\alpha}\right\\}$
(19)

Where:

  * •

$F(\gamma,\beta,\alpha)$ is the total cost function.

  * •

$\gamma,\beta,\alpha$ are the parameters being optimized.

  * •

$D_{j,A},B_{j,A},L_{j,A}$ are data size, bandwidth, and latency metrics,
respectively.

  * •

$G_{A},C_{A},R_{A}$ are GPU availability, computational load, and resource
reliability metrics, respectively.

3\. Selection. Candidates with higher fitness are more likely to be selected
for reproduction. We use roulette wheel selection [49] to allocate selection
probability proportional to fitness.

4\. Crossover and Mutation. Selected candidates undergo crossover, combining
their parameter values to create offspring. This process introduces new
combinations of $\gamma$, $\beta$, and $\alpha$ and allows the algorithm to
explore the parameter space. Additionally, mutation introduces random changes
to some offspring, preventing premature convergence.

$\theta_{\text{child}}=\eta\theta_{\text{parent1}}+(1-\eta)\theta_{\text{parent2}}$
(20)

Here, $\theta$ represents the parameters $\gamma$, $\beta$, and $\alpha$.
Where $\eta$ is a randomly selected value between 0 and 1. This convergence
returns the optimal values of these hyperparameters that are used for
decision-making related to routing, swarm creation and rebalancing.

##  4 Compression techniques

Fast inference of large models like Mixtral [1] and Llama [50] variants
present scalability challenges due to their high memory demands. Consumer-
grade GPUs often lack sufficient memory to store models with over 100 billion
parameters, such as Llama-3.1. This accentuates the need for distributed
inference and efficient management to minimize communication overhead. We
implemented two main compression strategies to optimize hardware use

1\. Dynamic Quantization. This technique compresses the hidden states
exchanged between nodes during the model inference process. By applying
dynamic blockwise quantization, we can significantly reduce the data that
needs to be transmitted and halve the bandwidth requirements. This reduction
is important for maintaining efficient operation, especially in environments
with constrained bandwidth or increased latency due to the geographical
distribution of nodes. This results in an efficient data transfer process,
which reduces overall system latency and enhances throughput.

2\. Mixed matrix decomposition. We use a mixed matrix decomposition method to
quantize the model weights from 16-bit to 8-bit precision which further
reduces the memory footprint. This method involves decomposing the weight
matrices into 8-bit values while retaining a small fraction of critical
weights at 16-bit precision. For example, for Mixtral 8x22B, it’s compute-
intensive to store all layers at full precision and can be significantly
downsized in terms of GPU requirements by a factor of 30% with this
quantization.

##  5 Benchmarks

Currently, our framework enables running a large number of machine learning
models, which comprehensively span a variety of learning tasks. The models we
host on our platform range from standard deep learning to large language-based
models and vision transformers. Overall, tasks can be performed with multiple
models, depending on the user requirements. We currently support:

1\. Language-related tasks including token classification, question answering,
translation, text classification, and summarization.

2\. Machine vision tasks like instance segmentation, object detection,
panoptic segmentation, and image classification.

3\. General purpose machine learning tasks including feature extraction,
causal generation.

Nesa’s distributed inference system can also handle state-of-the-art LLMs and
their variants, such as Mixtral, Llama Bloom etc. even on consumer-grade GPUs.
This flexibility ensures broader accessibility and scalability, enabling
deployments across diverse environments without the need for high-end,
specialized hardware.

Model | Bits | HellaSwag | Lambada | Causal | Disambig- | Logical
---|---|---|---|---|---|---
|  |  | OpenAI | Judgement | uation QA | Deduction
Llama 8B | 16 | 0.76 ± 0.01 | 0.75 ± 0.03 | 0.63 ± 0.02 | 0.64 ± 0.04 | 0.40 ± 0.03
8 | 0.76 ± 0.01 | 0.74 ± 0.03 | 0.62 ± 0.03 | 0.63 ± 0.04 | 0.39 ± 0.02
Mixtral 7x8B | 16 | 0.78 ± 0.01 | 0.76 ± 0.03 | 0.65 ± 0.02 | 0.66 ± 0.03 | 0.42 ± 0.02
8 | 0.77 ± 0.01 | 0.75 ± 0.02 | 0.64 ± 0.03 | 0.65 ± 0.03 | 0.41 ± 0.03
Lexi 7B | 16 | 0.75 ± 0.02 | 0.74 ± 0.02 | 0.62 ± 0.03 | 0.63 ± 0.04 | 0.39 ± 0.02
8 | 0.74 ± 0.02 | 0.73 ± 0.03 | 0.61 ± 0.04 | 0.62 ± 0.03 | 0.38 ± 0.02
Table 1: Compression impact on Natural language tasks for distributed LLMs.

To assess the performance of our system, we evaluated it on a variety of
natural language processing tasks that test different aspects of language
understanding and reasoning. These tasks provide a comprehensive benchmark for
measuring the capabilities of our models across various domains.

1\. HellaSwag This is one of the benchmarks in commonsense reasoning, which
requires models to choose a follow up continuation event from multiple option.
This task is crucial for assessing a model’s ability to understand and predict
everyday situations and requires deep contextual comprehension [51].

2\. Lambada OpenAI focuses on language modeling, specifically on the model’s
ability to predict the last word of a sentence based on the preceding context.
This is a key test to evaluate the model’s understanding of linguistic
intuitions and coherence [52].

3\. Causal Judgement measures the model’s capacity to grasp causal
relationships of events. This task is essential for gauging the model’s
ability to identify and reason through cause-and-effect scenarios [53].

4\. Disambiguation QA measures how well the model resolved ambiguities in
questions given some relevant context. This task tests how well the model
understands and can differentiate between multiple potential interpretations,
demonstrating its contextual abilities [53].

5\. Logical Deduction involves evaluating the model’s ability to infer
conclusions based on given premises and tests its reasoning and inference
skills[54].

###  5.1 Impact of compression on performance

Our results, detailed in Table 1, demonstrate that reducing precision from
16-bit to 8-bit has a minimal impact on task performance. For instance, in the
HellaSwag task, Mixtral 56B performed equally well under both precisions which
indicates that reasoning and understanding quality was kept intact at a lesser
memory cost. Tasks such as Lambada and Causal Judgement are relatively
unaffected as well, demonstrating that the compression operations do little to
compromise the model’s abilities.

Table 2 shows the impact of network conditions and batch sizes on two LLMs
including Mixtral and LLama. Round-trip time and bandwidth directly impact the
tokens per second speed, which is expected over a distributed swarm. Both LLMs
have 32 transformer blocks, served by 6 nodes chosen based on the previously
discussed routing mechanism. Three out of six machines are using Nvidia L24
GPUs, while the other three are using a consumer-grade GPU i.e. Nvidia GTX
1650. Nesa supports inference on both CPUs and GPUs where slightly more
latency is expected on the CPU. We can observe that token generation speed
increases as batch size increases, demonstrating that parallel computation
enhances network throughput; this behavior is crucial for large-scale, fast
inference.

Model | RTT | Bandwidth | Batch Size | Generation (steps/s) | Tokens/sec
---|---|---|---|---|---
|  |  |  | 64 | 1024 | 64 | 1024
Llama-3 8B | <5 ms | 1 Gbit/s | 1 | 1.20 | 1.10 | 8 | 6.5
32 | 1.15 | 1.08 | 28 | 26.4
64 | 1.10 | 1.05 | 56 | 52.5
<10 ms | 100 Mbit/s | 1 | 0.85 | 0.80 | 6 | 5
32 | 0.80 | 0.75 | 22 | 20
64 | 0.75 | 0.70 | 44 | 40
Mixtral 7x8B | <5 ms | 1 Gbit/s | 1 | 1.30 | 1.25 | 6 | 5.4
32 | 1.25 | 1.20 | 24 | 22.8
64 | 1.20 | 1.15 | 49 | 45.5
<10 ms | 100 Mbit/s | 1 | 0.90 | 0.85 | 5 | 4.5
32 | 0.85 | 0.80 | 20 | 18.5
64 | 0.80 | 0.75 | 40 | 37
Table 2: Performance Metrics for Llama and Mixtral in a swarm of six nodes Category | Model | Fairness | Quality | Creativity | Knowledge | Performance
---|---|---|---|---|---|---
Stable Diffusion | v1.4 | 0.68 | 0.86 | 0.68 | 0.68 | 0.85
v1.5 | 0.54 | 0.73 | 0.21 | 0.50 | 0.81
v2 base | 0.51 | 0.85 | 0.20 | 0.39 | 0.88
Anime-Style | kivotos-xl-2.0 | 0.77 | 0.87 | 0.91 | 0.72 | 0.81
holodayo-xl-2.1 | 0.79 | 0.89 | 0.94 | 0.74 | 0.83
Debiasing | mobius | 0.82 | 0.71 | 0.86 | 0.77 | 0.87
Table 3: Model performance metrics by category for text-to-image models

In addition to evaluating LLMs, we also focus on text-to-image architectures
including diffusion models, Anime generators and debiasing generation models
[32, 55]. These models are assessed across various metrics to evaluate their
effectiveness and reliability. We consider the following metrics [56]:

1\. Fairness evaluates if the image models are biased towards a gender or a
specific ethnicity.

2\. Quality measures the ability of models to generate aesthetic images. This
metric focuses on different aspects including clarity and detail.

3\. Creativity evaluates if the images generated are unique and free from
noise. This also measures if there is any repetition trend in the generation
when similar prompts are used.

4\. Knowledge measures if the model has sufficient knowledge to represent real
word facts. This checks its ability to draw historical figures, major cities
or well-known events

5\. Performance includes metrics such as inference speed and computational
efficiency which are important for fast inference.

##  6 Security and privacy in Nesa’s system

Our BSNS distributed inference protocol offers major advantages for
distributed inference, but it also requires cutting-edge approaches to
security and privacy (S&P) enhancement [57]. Specifically, hardware-based and
software/algorithm-based solutions need to be integrated to achieve co-
optimization, each selected and optimized for varying scenarios within our
ecosystem.

Potential concerns linked to distributed BSNS inference may appear in
different forms. For instance:

  * •

Users may wish to protect their input data and the inference results.

  * •

Node owners might seek to protect the confidentiality of their model
parameters in certain cases.

  * •

Meanwhile, the users want to ensure that the models executed by the nodes are
verifiable — namely, the designated ML models need to generate the inference
results without unexpected changes.

We list below the key steps we carry out towards ensuring the security of the
BSNS protocol. The reader is referred to the companion Security paper for more
details on Nesa’s implementation, which ensures that both user input for AI
inference and private model parameters are protected during the execution of
distributed models.

###  6.1 S&P requirements

In summary, there are two core S&P aspects we identify in decentralized,
private inference: (i) model verification to prove the nodes execute the
designated models, e.g., LLaMA 3 [15], for a user, where adversarial
participants may return random results or even malicious results and (ii) data
encryption [58] to protect the user’s data from being revealed during the
inference. Based on these requirements, we develop a suite of solutions to
ensure S&P in Nesa’s system.

###  6.2 Overview of our hardware-software co-optimization solution

To address both model verification and data encryption jointly, we design an
integrated approach to achieve leading S&P performance in our system.
Specifically, we design a combined strategy of the robust, hardware-centric
protections of Trusted Execution Environments (TEEs) [59] and the advanced
algorithmic approaches, including Zero-Knowledge Machine Learning (ZKML) [60]
and Consensus-based Distribution Verification (CDV) [61] for verification and
Split-Learning (SL) [62] for encryption, to provide the highest level of S&P
in our system.

In short, TEEs provide a secure area within a processor that ensures the
confidentiality and integrity of the code and data loaded within it, thus
supplying robustness from the hardware level. On the software/algorithm side,
ZKML can provide the means to confirm the authenticity and integrity of the
models run by nodes without revealing any other information. Due to its
computational cost, we currently leverage it for private, small models
requiring the highest security level. As an efficient alternative for public,
large models, CDV is a novel algorithm we propose that ensures that the
inference nodes execute the correct model by measuring their output
distribution consensus, while SL protects user data by only transferring the
intermediate computational embeddings other than the raw data. Collectively,
this hardware-software integrated solution guarantees high S&P in Nesa’s
system.

###  6.3 BSNS-compatible data encryption

In decentralized inference systems like Nesa’s BSNS, it is crucial to protect
user data. BSNS distributes computational tasks across various nodes, each
potentially operated by different entities, which in principle increases the
risk of exposing sensitive user data during the inference process intensifies.
To protect against this, we have designed an innovative encryption method
called Sequential Vector Encryption (SVE), developed as a more efficient
version of homomorphic encryption, and applied on select linear layers rather
than the full model based on a sequence of vector space operators. SVE
randomly transforms the outputs of every operator so that the intermediate
vector representations can no longer be interpretable using a given method
developed for the original model’s vector representations. These
representations are then transformed to the original representation space
before feeding to the next operator to render the original model still usable.

####  6.3.1 Split learning

Recognizing the challenges posed by encrypting data for use in decentralized
inference systems, Nesa adopts Split Learning (SL) as a pragmatic solution to
facilitate secure and efficient computation on encrypted data [63, 64].
Traditional encryption methods, while securing data at rest and in transit,
render it unusable for direct computation by obscuring its format and
structure. This limitation is particularly problematic for processing with
LLMs within a decentralized framework, where data privacy cannot be
compromised.

Split Learning [62] addresses these concerns by partitioning the computational
model, allowing for data to be processed in parts without revealing sensitive
information. In essence, the user data is protected by not being directly
transmitted to any nodes – only the data embeddings are being passed around,
and each node will only be accessing the embeddings of certain layers.

Consider a neural network model $\mathcal{N}$, such as Llama 2 [15] composed
of a sequence of 32 layers $\\{L_{1},L_{2},\ldots,L_{32}\\}$, each with its
own set of parameters $\Theta_{i}$ and activation function $\sigma_{i}$. The
input to the network is $X$, and the output of the $i$-th layer, given input
$x_{i}$, can be mathematically described as:

$a_{i}=L_{i}(x_{i};\Theta_{i})=\sigma_{i}(W_{i}x_{i}+b_{i})$ (21)

where $W_{i}$ and $b_{i}$ are the weight matrix and bias vector of the $i$-th
layer, respectively, and $\sigma_{i}$ is a nonlinear activation function such
as ReLU, sigmoid, or tanh.

Assuming the model is split at layer $k$, where the client handles layers
$\\{L_{1},\ldots,L_{k}\\}$ and the server handles layers
$\\{L_{k+1},\ldots,L_{32}\\}$. The client computes the intermediate
representation $Z$ as follows:

$Z=\sigma_{k}(W_{k}\cdot\sigma_{k-1}(\ldots\sigma_{1}(W_{1}X+b_{1})\ldots)+b_{k})$
(22)

This intermediate representation $Z$ is then transmitted to the server, which
continues the computation:

$Y=\sigma_{32}(W_{32}\cdot\sigma_{31}(\ldots\sigma_{k+1}(W_{k+1}Z+b_{k+1})\ldots)+b_{32})$
(23)

The loss function $\mathcal{L}(Y,Y_{true})$ computes the error between the
network output $Y$ and the true labels $Y_{true}$, and the gradient of the
loss with respect to the model’s parameters through backpropagation:

$\frac{\partial\mathcal{L}}{\partial\Theta_{i}}=\text{ChainRule}\left(\frac{\partial\mathcal{L}}{\partial
Y},\frac{\partial Y}{\partial a_{32}},\ldots,\frac{\partial
a_{i}}{\partial\Theta_{i}}\right)$ (24)

For privacy concerns during the transmission of $Z$ from client to server,
differential privacy methods may be applied [65]. Defining a privacy metric
$\mathcal{P}$ that quantifies the information leakage from the intermediate
representation $Z$, a proof of privacy preservation could demonstrate that for
any $\epsilon$-differential privacy guarantee, the information leakage remains
below a threshold:

$\mathcal{P}(Z)\leq\epsilon$ (25)

It is noted that by using differential privacy with SL, the security will be
improved at the cost of inference quality [66]. Thus, in Nesa’s framework,
this is defined as a tunable parameter to be decided, given the user
requirements.

By leveraging Split Learning, Nesa effectively navigates the complexities of
data encryption within its decentralized inference system for LLMs. This
approach not only preserves the confidentiality and integrity of user data but
also ensures the operational feasibility of complex model computations,
demonstrating a sophisticated balance between privacy preservation and
computational pragmatism.

##  7 Conclusion

Our paper introduces a model-agnostic hybrid sharding method, complemented by
a comprehensive security and privacy framework designed for distributed AI
inference. This approach strategically distributes computational tasks across
a decentralized network, enabling scalable AI execution through the sequential
consumption of shards on low-powered nodes. First, our BSNS optimizes network
topology. Equipped with dynamic network rebalancing and KV caching mechanisms,
the BSNS design is adept at facilitating scalable operations for large models
and provides standardization across all variants where the base architecture
is the same.

Next, we proposed an integrated security and privacy framework featuring co-
optimized hardware-based TEEs with CDV and SL. Long-term techniques for the
protocol are also explored which could potentially have significant impacts in
this field. Collectively, our system removes the prohibitive costs associated
with needing to own GPUs, making AI inference execution accessible to the
general population. Democratizing AI requires technology like BSNS in order to
provide the opportunity for participation to anyone with a computer at home,
while upholding an essential commitment to safety and privacy throughout the
inference execution process.

## References

  * [1] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.  Mixtral of experts.  arXiv preprint arXiv:2401.04088, 2024.
  * [2] OpenAI.  Chatgpt: Optimizing language models for dialogue, 2023.
  * [3] Anthropic.  Claude: An ai assistant, 2023.
  * [4] Google DeepMind.  Gemini: A language model by google deepmind, 2023.
  * [5] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel.  Scaling data-constrained language models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [6] Replicate.  Replicate: Apis for large language models, 2023.
  * [7] Artyom Eliseev and Denis Mazur.  Fast inference of mixture-of-experts language models with offloading.  arXiv preprint arXiv:2312.17238, 2023.
  * [8] Stacey A Tovino.  The hipaa privacy rule and the eu gdpr: illustrative comparisons.  Seton Hall L. Rev., 47:973, 2016.
  * [9] Liu X, Fangxu Xing, Chao Yang, El Fakhri, G, and Jonghye Woo.  Adapting off-the-shelf source segmenter for target medical image segmentation.  In MICCAI, pages 549–559, 2021.
  * [10] Meng-Chieh Lee, Yue Zhao, Aluna Wang, Pierre Jinghong Liang, Leman Akoglu, Vincent S Tseng, and Christos Faloutsos.  Autoaudit: Mining accounting and time-evolving graphs.  In 2020 IEEE International Conference on Big Data (Big Data), pages 950–956. IEEE, 2020.
  * [11] Xuhui Chen, Jinlong Ji, Changqing Luo, Weixian Liao, and Pan Li.  When machine learning meets blockchain: A decentralized, privacy-preserving and secure design.  In 2018 IEEE international conference on big data (big data), pages 1178–1187. IEEE, 2018.
  * [12] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704–2713, 2018.
  * [13] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International conference on machine learning, pages 2790–2799. PMLR, 2019.
  * [14] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [15] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [16] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.
  * [17] Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat.  When scaling meets llm finetuning: The effect of data, model and finetuning method.  arXiv preprint arXiv:2402.17193, 2024.
  * [18] Sergio Moreno-Álvarez, Mercedes E Paoletti, Gabriele Cavallaro, and Juan M Haut.  Enhancing distributed neural network training through node-based communications.  IEEE Transactions on Neural Networks and Learning Systems, 2023\.
  * [19] Haiyang Lin, Mingyu Yan, Xiaochun Ye, Dongrui Fan, Shirui Pan, Wenguang Chen, and Yuan Xie.  A comprehensive survey on distributed training of graph neural networks.  Proceedings of the IEEE, 2023.
  * [20] Aaron Archer, Matthew Fahrbach, Kuikui Liu, and Prakash Prabhu.  Pipeline parallelism for dnn inference with practical performance guarantees.  arXiv preprint arXiv:2311.03703, 2023.
  * [21] Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin.  Zero bubble pipeline parallelism.  arXiv preprint arXiv:2401.10241, 2023.
  * [22] Mincong Huang, Chao Wang, Chi Ma, Yineng Zhang, Peng Zhang, and Lei Yu.  Re-evaluating the memory-balanced pipeline parallelism: Bpipe.  arXiv preprint arXiv:2401.02088, 2024.
  * [23] Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, and François Fleuret.  Faster causal attention over large sequences through sparse flash attention.  arXiv preprint arXiv:2306.01160, 2023.
  * [24] Jiange Li, Yuchen Wang, Jinghui Zhang, Jiahui Jin, Fang Dong, and Lei Qian.  Pipepar: A pipelined hybrid parallel approach for accelerating distributed dnn training.  In 2021 IEEE 24th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pages 470–475. IEEE, 2021.
  * [25] Zhaoxin Yang, Meng Li, Ruizhe Yang, F Richard Yu, and Yanhua Zhang.  Blockchain sharding strategy for collaborative computing internet of things combining dynamic clustering and deep reinforcement learning.  In ICC 2022-IEEE International Conference on Communications, pages 2786–2791. IEEE, 2022.
  * [26] Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen.  Get more with less: Synthesizing recurrence with kv cache compression for efficient llm inference.  arXiv preprint arXiv:2402.09398, 2024.
  * [27] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [28] Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang.  Qaq: Quality adaptive quantization for llm kv cache.  arXiv preprint arXiv:2403.04643, 2024.
  * [29] Youpeng Zhao, Di Wu, and Jun Wang.  Alisa: Accelerating large language model inference via sparsity-aware kv caching.  arXiv preprint arXiv:2403.17312, 2024.
  * [30] Yu Yu, Chao-Han Huck Yang, Tuan Dinh, Sungho Ryu, Jari Kolehmainen, Roger Ren, Denis Filimonov, Prashanth G Shivakumar, Ankur Gandhe, Ariya Rastow, et al.  Investigating training strategies and model robustness of low-rank adaptation for language modeling in speech recognition.  arXiv preprint arXiv:2401.10447, 2024.
  * [31] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms.  Advances in Neural Information Processing Systems, 36, 2024.
  * [32] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion models in vision: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10850–10869, 2023.
  * [33] Stephen Grossberg.  Recurrent neural networks.  Scholarpedia, 8(2):1888, 2013.
  * [34] Barna Saha and Christopher Ye.  The i/o complexity of attention, or how optimal is flash attention?  arXiv preprint arXiv:2402.07443, 2024.
  * [35] David Tremblet, Simon Thevenin, and Alexandre Dolgui.  Makespan estimation in a flexible job-shop scheduling environment using machine learning.  International Journal of Production Research, 62(10):3654–3670, 2024.
  * [36] Renkun Liu.  A low complexity topological sorting algorithm for directed acyclic graph.  International Journal of Machine Learning and Computing, 4(2):194, 2014.
  * [37] Yang Xia, Peng Jiang, Gagan Agrawal, and Rajiv Ramnath.  End-to-end lu factorization of large matrices on gpus.  In Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, pages 288–300, 2023.
  * [38] Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodriguez.  Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl.  In International Conference on Machine Learning, pages 38989–39007. PMLR, 2023.
  * [39] José Bernardo Barros, Daniela Da Cruz, Pedro Rangel Henriques, and Jorge Sousa Pinto.  Assertion-based slicing and slice graphs.  Formal Aspects of Computing, 24:217–248, 2012.
  * [40] SN Sivanandam, SN Deepa, SN Sivanandam, and SN Deepa.  Genetic algorithms.  Springer, 2008.
  * [41] Mariana A Londe, Luciana S Pessoa, Carlos E Andrade, and Mauricio GC Resende.  Biased random-key genetic algorithms: A review.  European Journal of Operational Research, 2024.
  * [42] Zhenyu Meng, D Vijay Anand, Yunpeng Lu, Jie Wu, and Kelin Xia.  Weighted persistent homology for biomolecular data analysis.  Scientific reports, 10(1):2079, 2020.
  * [43] Paul G Goerss and John F Jardine.  Simplicial homotopy theory.  Springer Science & Business Media, 2009.
  * [44] Sigurdur Albertsson, Sigurdur Hafstein, Peter Giesl, and Skuli Gudmundsson.  Simplicial complex with approximate rotational symmetry: A general class of simplicial complexes.  Journal of Computational and Applied Mathematics, 363:413–425, 2020\.
  * [45] Gunnar Carlsson and Afra Zomorodian.  The theory of multidimensional persistence.  In Proceedings of the twenty-third annual symposium on Computational geometry, pages 184–193, 2007.
  * [46] Jean-Daniel Boissonnat and Siddharth Pritam.  Computing persistent homology of flag complexes via strong collapses.  Leibniz International Proceedings in Informatics (LIPIcs), 2018\.
  * [47] Frank Frank Edward Dabek.  A distributed hash table.  PhD thesis, Massachusetts Institute of Technology, 2005.
  * [48] Annu Lambora, Kunal Gupta, and Kriti Chopra.  Genetic algorithm-a literature review.  In 2019 international conference on machine learning, big data, cloud and parallel computing (COMITCon), pages 380–384. IEEE, 2019.
  * [49] Adam Lipowski and Dorota Lipowska.  Roulette-wheel selection via stochastic acceptance.  Physica A: Statistical Mechanics and its Applications, 391(6):2193–2196, 2012.
  * [50] Meta.  Meta-llama, 2024.
  * [51] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  arXiv preprint arXiv:1905.07830, 2019.
  * [52] Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling.  Do not have enough data? deep learning to the rescue!  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7383–7390, 2020.
  * [53] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al.  Challenging big-bench tasks and whether chain-of-thought can solve them.  arXiv preprint arXiv:2210.09261, 2022.
  * [54] Ahmad Ghazal, Todor Ivanov, Pekka Kostamaa, Alain Crolotte, Ryan Voong, Mohammed Al-Kateb, Waleed Ghazal, and Roberto V Zicari.  Bigbench v2: The new and improved bigbench.  In 2017 IEEE 33rd International Conference on Data Engineering (ICDE), pages 1225–1236. IEEE, 2017.
  * [55] Cemre Efe Karakas, Alara Dirik, Eylül Yalçınkaya, and Pinar Yanardag.  Fairstyle: Debiasing stylegan2 with style channel manipulations.  In European Conference on Computer Vision, pages 570–586. Springer, 2022.
  * [56] Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al.  Holistic evaluation of text-to-image models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [57] Chuan Ma, Jun Li, Kang Wei, Bo Liu, Ming Ding, Long Yuan, Zhu Han, and H Vincent Poor.  Trusted ai in multi-agent systems: An overview of privacy and security for distributed learning.  arXiv preprint arXiv:2202.09027, 2022.
  * [58] Raphael Bost, Raluca Ada Popa, Stephen Tu, and Shafi Goldwasser.  Machine learning classification over encrypted data.  Cryptology ePrint Archive, 2014.
  * [59] Patrick Jauernig, Ahmad-Reza Sadeghi, and Emmanuel Stapf.  Trusted execution environments: properties, applications, and challenges.  IEEE Security & Privacy, 18(2):56–60, 2020.
  * [60] Haochen Sun, Tonghe Bai, Jason Li, and Hongyang Zhang.  Zkdl: Efficient zero-knowledge proofs of deep learning training.  arXiv preprint arXiv:2307.16273, 2023.
  * [61] Li Shichao and Qin Jiwei.  Aggregation algorithm based on consensus verification.  Scientific Reports, 13(1):12923, 2023.
  * [62] Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar.  Split learning for health: Distributed deep learning without sharing raw patient data.  arXiv preprint arXiv:1812.00564, 2018.
  * [63] Ngoc Duy Pham, Alsharif Abuadbba, Yansong Gao, Tran Khoa Phan, and Naveen Chilamkurti.  Binarizing split learning for data privacy enhancement and computation reduction.  IEEE Transactions on Information Forensics and Security, 2023.
  * [64] Tanveer Khan, Khoa Nguyen, and Antonis Michalas.  Split ways: Privacy-preserving training of encrypted data using split learning.  arXiv preprint arXiv:2301.08778, 2023.
  * [65] Cynthia Dwork.  Differential privacy.  In International colloquium on automata, languages, and programming, pages 1–12. Springer, 2006.
  * [66] Rouzbeh Behnia, Mohammadreza Reza Ebrahimi, Jason Pacheco, and Balaji Padmanabhan.  Ew-tune: A framework for privately fine-tuning large language models with differential privacy.  In 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pages 560–566. IEEE, 2022.