# Aggressive or Imperceptible, or Both:
Network Pruning Assisted Hybrid Byzantines
in Federated Learning

Emre Özfatura2, Kerem Özfatura1, Alptekin Küpçü1, Deniz Gündüz2  1KUIS AI LAB
Koç University  2IPC LAB
Imperial College London

###### Abstract

Federated learning (FL) has been introduced to enable a large number of
clients, possibly mobile devices, to collaborate on generating a generalized
machine learning model thanks to utilizing a larger number of local samples
without sharing to offer certain privacy to collaborating clients. However,
due to the participation of a large number of clients, it is often difficult
to profile and verify each client, which leads to a security threat that
malicious participants may hamper the accuracy of the trained model by
conveying poisoned models during the training. Hence, the aggregation
framework at the parameter server also needs to minimize the detrimental
effects of these malicious clients. A plethora of attack and defence
strategies have been analyzed in the literature. However, often the Byzantine
problem is analyzed solely from the outlier detection perspective, being
oblivious to the topology of neural networks (NNs).

In the scope of this work, we argue that by extracting certain side
information specific to the NN topology, one can design stronger attacks.
Hence, inspired by the sparse neural networks, we introduce a hybrid sparse
Byzantine attack that is composed of two parts: one exhibiting a sparse nature
and attacking only certain NN locations with higher sensitivity, and the other
being more silent but accumulating over time, where each ideally targets a
different type of defence mechanism, and together they form a strong but
imperceptible attack. Finally, we show through extensive simulations that the
proposed hybrid Byzantine attack is effective against 8 different defence
methods. Our code is available at https://github.com/CRYPTO-KU/FL-Byzantine-
Library

##  I Introduction

Federated learning (FL) [31] become de-facto collaborative learning framework
where a large number of clients optimize their model, e.g., neural network
(NN) weights, locally by utilizing their personal data and then seek consensus
on the global model through the exchange of local models, where this process
is often performed through multiple consecutive iterations. While local
optimization helps keep the personal data private, the consensus phase helps
reduce the generalization error. In this work, we focus on centralized and
synchronous FL setup [31, 26], where a central parameter server (PS)
orchestrates the consensus phase by executing an aggregation policy for the
local models received from all the participating clients and following the
aggregation, the new model is conveyed to the clients for the next round of
local optimization/training.

In practice, FL is designed to facilitate collaborative learning over a large
number of clients, where it is often challenging to profile and verify
clients, especially when the variance among the updates from the clients’ is
very high [19] or when the identification metric is solely based on geometric
distance [38]. To be more specific, this can be due to non-identically
distributed client data or because some of the clients may act maliciously and
aim to sabotage the learning process either individually or by colluding with
other malicious clients. In the literature, this type of attempt is referred
to as a poisoning attack [1, 4, 17, 41], where the malicious clients send a
poisoned model to the PS in compliance with a predefined attack mechanism.
These attacks can be targeted  [47, 1, 4] or untargeted [6, 48], based on
whether the objective is the model to fail only certain tasks, e.g., miss-
classification of certain classes, or fail to perform any task [4, 17, 41, 2].
Besides, the attack is not always visible during the training; the model
performs well for the training data but lacks generalization at test time,
particularly performing poorly on samples with a certain triggering pattern,
often referred to as a backdoor attack [1].

In the scope of this work, we limit our focus to untargeted model poisoning
attacks, often known as Byzantine attacks, in FL. The design of the Byzantine
attacks, under various assumptions, e.g., the training data that adversaries
can access or the information they can infer, has been extensively studied in
the literature [2, 14, 39, 50]. To overcome the detrimental impact of
malicious clients, the design of a robust aggregator that neutralizes the
poisoned models from different aspects has also been studied in the literature
[6, 36, 52]. Many of the defence methods consider poisoned models as outliers
and suggest eliminating statistically suspicious ones before using
conventional aggregation functions like averaging [12, 6, 33], or some [52, 8]
of them directly redesign the aggregation function to generate a good
representation model for the benign clients, e.g., geometric mean, and finally
some of them, rather than eliminating suspicious models, sanitize all the
models before the executing the aggregation function [24, 37, 10, 21]. Apart
from these methods, some works in the literature suggest more complicated
frameworks, where the defence strategy either utilizes extra data at the PS
such as [7, 49] or profiles the clients over time and assigns a trust score to
each of them [33, 4]. Recent work also explores a game-theoretic approach to
dynamically selecting a different robust aggregator in each update between PS
and clients. [50].

From the design perspective, an effective attack should be strong and
imperceptible [2, 39, 14]. Hence, attacks often take advantage of the curse of
dimensionality (that is, NNs have a large number of weights) and the variation
among the benign models due to the data distribution among the clients since,
in practice, the data is distributed in a non-iid manner. As the variation
among the benign models increases, the feasible space for the Byzantines,
where they do not look like outliers, also expands. Thus, as a counter
strategy, some of the recent works utilize variance reduction techniques
designed for distributed learning in order to shrink the feasible space of
Byzantine attacks [25, 24, 18, 35, 45, 53, 34]. One of the popular approaches
is to employ momentum SGD as a local optimizer and perform aggregation over
the momentum terms; it has been shown that the use of local momentum enhances
the robustness against the Byzantine attacks [24, 54, 15, 11].

Although there are various successful Byzantine attack designs, there is still
a lack of a universal attack design that is effective against different types
of defence mechanisms simultaneously. To be more precise, as we discuss in
more detail later, while some Byzantine designs are more effective against
defences that investigate index-wise outlier values, some are more effective
against defence mechanisms that utilize geometric distance as a metric to
detect anomalies. Therefore, in the scope of this work, we seek an answer to
whether it is possible to design a universally effective yet computationally
low-cost Byzantine attack strategy. Another important aspect often ignored in
the design of Byzantine attacks is the structure of the NN. To be more
precise, certain weights might be more vulnerable against perturbations, and
this prior knowledge can be utilized to enhance attack. To the best of our
knowledge, this is the first work that utilizes the inherited information from
the neural network topology to modify Byzantine attacks. The contributions of
this work can be summarized as follows

  * •

We introduce a novel Byzantine attack framework consisting of two parts, each
designed to avoid either index-wise elimination or geometric distance based
investigation; hence, their combination is effective against both types of
defences and does not require prior knowledge regarding the defence mechanism.

  * •

Contrary to the existing Byzantine designs, we utilize side information
extracted from the network architecture through a network pruning framework,
which locates the possible sensitive weights and enhances this side
information to generate stronger yet imperceptible attacks.

  * •

Finally, through extensive simulations over different NN networks, datasets,
and defence mechanisms, we show the effectiveness of our approach by reducing
test accuracy up to 60% in IID simulations and diverging the model training
completely in the non-IID simulations.

Algorithm 1 FL with Robust Aggregator and Byzantines

1:Input: $\eta$, $AGG(\cdot)$ , $Attack(\cdot)$

2:Output: Consensus model: $\mathbf{\theta}_{T}$

3:for $t=1,\ldots,T$ do

4: Client side:

5: for $i=1,\ldots,k$ do in parallel

6: Receive: $\mathbf{\theta}_{t-1}$ from PS

7: if $i\in\mathcal{K}_{b}$ then

8: Update local model: $\mathbf{\theta}_{i,t}\leftarrow\mathbf{\theta}_{t-1}$

9: Compute SGD:
$\mathbf{g}_{i,t}\leftarrow\nabla_{\mathbf{\theta}}f_{i}(\mathbf{\theta}_{i,t},\zeta_{i,t})$

10: Update Momentum:
$\mathbf{m}_{i,t}=(1-\beta)\mathbf{g}_{i,t}+\beta\mathbf{m}_{i,t-1}$

11: else

12: $\mathbf{m}_{i,t}\leftarrow Attack(\mathcal{H}_{t})$ Server side:

13: Aggregate local updates: $\tilde{\mathbf{m}}_{t}\leftarrow
AGG(\mathbf{{m}}_{1,t},\ldots,\mathbf{m}_{k,t})$

14: Update the server model:
$\mathbf{\theta}_{t}\leftarrow\mathbf{\theta}_{t-1}-\eta_{t}\tilde{\mathbf{m}}_{t}$

###  I-A Federated Learning (FL) with Byzantines

Notation: We use bold to denote vectors, i.e., $\bm{v}$ and capital
calligraphic letters, e.g., $\mathcal{V}$, to denote sets. When we have an
ordered set of vectors
$\mathcal{V}=\left\\{\bm{v}_{1},\ldots,\bm{v}_{i}\ldots,\bm{v}_{k}\right\\}$,
we use subscript index to identify $i^{th}$ vector in the set, $i\in[k]$, and
use double subscript $\mathbf{v}_{i,t}$, particularly when it is changing over
time/iteration. For slicing operation, we use $[\cdot]$, such as
$\mathbf{v}[j]$ for selecting $j^{th}$ index of a vector. We use $||\ \cdot\
||_{p}$ to denote the $p$-norm of a vector and we use $||\ \cdot\ ||$ without
$p$ we refer $l_{2}$ norm. We use $<\cdot\ ,\cdot>$ for the inner product
between two vectors. In Table I, we list the widely used variables in this
paper.

TABLE I: Notations

Notation | Description
---|---
$\mathcal{K}_{b}$, $\mathcal{K}_{m}$ and $\mathcal{K}=\mathcal{K}_{b}\bigcup\mathcal{K}_{m}$ | Set of benign, malicious, and all clients, respectively
$k_{b}$,$k_{m}$ and $k$ | Number of benign, malicious, and all clients, respectively
$\bm{\theta}_{i,t}$, $\mathbf{g}_{i,t}$ , and $\mathbf{m}_{i,t}$ | Model, gradient, and momentum vector of client $i$ at iteration $t$, respectively.
$\tilde{\mathbf{m}}_{t}$ $\hat{\mathbf{m}}_{t}$ $\bar{\mathbf{m}}_{t}$, | Reference, aggregated, and benign consensus momentum at iteration $t$, respectively
$\delta$, $\mathbf{c}$ | Sparsity ratio and the corresponding mask vector, respectively

The objective of FL is to solve the following parameterized optimization
problem over $k$ clients in a distributed manner

$\min_{\bm{\theta}\in\mathbb{R}^{d}}f(\bm{\theta})=\frac{1}{k}\sum^{k}_{i=1}\underbrace{\mathds{E}_{\zeta_{i}\sim\mathcal{D}_{i}}f(\bm{\theta},\zeta_{i})}_{\mathrel{\mathop{:}}=f_{i}(\bm{\theta})},$
(1)

where $\bm{\theta}\in\mathbb{R}^{d}$ denotes the model parameters, e.g.,
weights of a neural network, $\zeta_{i,t}$ is the randomly sampled mini-batch,
$\mathcal{D}_{i}$ denotes the dataset of client $i$, and $f$ is the problem-
specific empirical loss function. At each iteration of FL, each client aims to
minimize its local loss function $f_{i}(\bm{\theta})$ using SGD. Then, the
clients seek a consensus on the model with the help of the PS.

In this work, following the previous works [24, 54], we focus on a particular
FL framework where the consensus step is performed by aggregating the local
momentum of the clients at the end of each optimization step $t$, which is
later used by PS to update the global model.

In this setup, the flow of the overall framework with Byzantines performing an
attacking strategy, denoted by $Attack(\cdot)$, and the robust aggregator,
denoted by $AGG(\cdot)$ that is implemented at the server, is illustrated in
Algorithm 1. In general, the attack $Attack(\cdot)$ can be considered as a
mapping

$Attack(\cdot):\mathcal{H}_{t}\xrightarrow{}(\mathbf{m}_{i,t}\
i\in\mathcal{K}_{mal})\ \in\mathbb{R}^{d},$ (2)

where $\mathcal{H}_{t}$ is referred to as history and includes all the
available information to Byzantines until $t$.

Adversary Model: We assume the adversary controls up to $k_{m}$ out of $k$
total clients, called malicious clients. We assume that the number of
malicious clients is less than the number of benign clients, i.e., $(k_{m}/k)$
$<$ 0.5; otherwise, no Byzantine-robust aggregator will be able to defeat
poisoning attacks. Following the previous works [39, 1, 2, 4, 14, 48], we
assume that the adversary can access the global model parameters broadcast in
each epoch and can directly manipulate the gradients on malicious devices.
Further, Byzantines can also compute the momentum of the benign clients. On
the other hand, Byzantines form their attacks in an agnostic manner with
respect to the aggregator. In other words, the Byzantines do not know the
robust aggregator or the defence mechanism used by the PS. Hence, the attack
should be universal; that is, it should be effective against all known defence
mechanisms without any aggregator-specific calibration.

##  II Preliminaries and Related Work

###  II-A Analyzing Existing Defence Methods

Various defence mechanisms against Byzantines have been introduced recently to
ensure the learned model’s reliability. Often the underlying defence mechanism
is to detect Byzantines utilizing different measures and ignore their model
update. However, some works directly sanitize each user’s model update rather
than filtering the attackers.

TABLE II: Comparison of the defence methods Defence mechanism | Identification Metric | Defence
---|---|---
Centered Clipping [24] | Euclidean distance | Sanitization with clipping
Krum & Multi-Krum [6] | Euclidean distance | Elimination
Bulyan [12] | Euclidean distance | Elimination
Trimmed mean [52] | Index-wise statistics | Elimination
Centered median [52] | Index-wise statistics | Index-wise Median
SignSGD [3, 23] | Index-wise statistics | Majority voting
RFA [36] | Euclidean distance | Geometric median

In a broad sense, we can classify Byzantine detection strategies into three
main categories based on the identification measures used for outliers:
geometric distance, index-wise statistics, and angular variance.

####  II-A1 Geometric distance

In this section, we analyze the norm-based defences where robust aggregators
consider the geometric distances to identify outliers.

Multi Krum: The underlying idea behind the Krum mechanism [6] is to identify
where the benign vectors accumulate in close proximity. Under the assumption
of $k_{m}$ Byzantines, considered as outliers, the first step of the Krum is
to form a set of neighbouring nodes $\mathcal{S}_{i}$, for each vector
$\mathbf{m}_{i}$, with cardinality $k^{\star}=k-k_{m}+2$ such that

$\mathcal{S}^{\star}_{i}=\min_{\mathcal{S},|\mathcal{S}|=k^{\star}}\sum_{j\in\mathcal{S}}\left\|\mathbf{m}_{i}-\mathbf{m}_{j}\right\|^{2}.$
(3)

Then, Krum identifies the vector $\mathbf{m}_{i}$, that has the minimum
average distance $d_{i}$ to vectors in its closest neighbourhood
$\mathcal{S}^{\star}_{i}$ as the true center for the benign vectors, i.e.,

$argmin_{i\in\mathcal{K}}~{}~{}d_{i}=\sum_{j\in\mathcal{S}^{\star}_{i}}\left\|\mathbf{m}_{i}-\mathbf{m}_{j}\right\|^{2}$
(4)

Multi-krum is built on the same principle but chooses $n$ vectors instead of a
single one and outputs their average.

Centered Clipping (CC): One of the major shortcomings of the robust
aggregators that rely on outlier elimination is the assumption that the number
of Byzantines is known in advance. Even under this assumption, such methods
suffer from miss-classification of benign clients as outliers or under-
utilization of benign models [25, 2]. Hence, in [24], the authors argue to
utilize all the available model updates but first sanitize them. The
underlying idea behind the CC is to perform sanitization by projecting model
update $\mathbf{m}$ into a ball with radius $\tau$ and centered at reference
model $\tilde{m}$, accordingly, the clipping function $f_{cc}()$ can be
formulated as

$f_{CC}(\mathbf{m}|\tilde{\mathbf{m}},\tau)=\tilde{\mathbf{m}}+\min\left\\{1,\frac{\tau}{||\tilde{\mathbf{m}}-\mathbf{m}||}\right\\}(\mathbf{m}-\tilde{\mathbf{m}}).$
(5)

Further, they argue that by choosing the previous aggregate model update as
the reference model for the current step, i.e.,
$\tilde{\mathbf{m}}_{t}=\tilde{\mathbf{m}}_{t-1}$, it is possible to sanitize
poisoned model updates successfully. However, in [54], the authors show that
when Byzantines are not oblivious to $\tilde{\mathbf{m}}_{t}$ and modify their
adversarial perturbation accordingly, CC becomes ineffective. As we further
discuss in section II-B, CC has certain other weaknesses.

Algorithm 2 Robust aggregation with CC

1:Inputs: $\bm{\tilde{m}}_{t}$,
$\left\\{\mathbf{m}_{i,t}\right\\}_{i\in\mathcal{K}},\tau$

2:for $i=1,\ldots,k$ do in parallel

3:
$\tilde{\mathbf{m}}_{i,t}=f_{CC}(\mathbf{m}_{i,t}|\tilde{\mathbf{m}}_{t},\tau)$

4:$\tilde{\mathbf{m}}_{t}=\frac{1}{k}\sum_{i\in\mathcal{K}}\tilde{\mathbf{m}}_{i,t}$

RFA: is a geometric median-based robust aggregation method [36] such that
given set of $n$ vectors
$\left\\{\mathbf{m}_{1},\dotso,\mathbf{m}_{n}\right\\}$ the consensus model is
solution, $\mathbf{m}^{\star}$, of the following optimization problem;

$RFA(\mathbf{m}_{1},\dotso,\mathbf{m}_{n})=\underset{\mathbf{m}^{\star}}{argmin}\sum\limits_{i=1}^{n}||\mathbf{m}^{\star}-\mathbf{m}_{i}||_{2}.$
(6)

It reduces the impact of the contaminated updates by approximating the
geometric median in place of the weighted arithmetic mean, resulting in an
aggregated global model update that looks robust to outliers.

####  II-A2 Index-wise statistics

Numerous algorithms based on stochastic gradient descent (SGD) have been
proposed to solve Eqn. 1 under these distributed optimization frameworks.
Performing updates based on the mean of the local stochastic directions is the
fundamental concept. On the other hand, there are prominent and interesting
algorithms whose updates are not directly related to the mean of the local
updates.

SignSGD. [3, 23] It lowers communication overheads by updating the parameters
according to a majority vote of momentum signs. It also allows fault tolerance
by employing majority voting. More formally, we can define SignSGD as;

$SignSGD(\mathbf{m}_{1},\dotso,\mathbf{m}_{k})=\mathrm{sign}(\sum_{i}^{k}\mathrm{sign}(\mathbf{m}_{i})).$
(7)

CM (coordinate-wise median) [52], employs the index-wise median of momentums
to evaluate the mean of the local updates in order to ensure robustness
against Byzantine workers.

$\hat{\mathbf{m}}=CM(\mathbf{m}_{1},\dotso,\mathbf{m}_{k}):\hat{\mathbf{m}}[j]=\mathrm{median}(\mathbf{m}_{1}[j],\ldots,\mathbf{m}_{k}[j])$
(8)

TM (Trimmed Mean) [52], is proposed as a more robust Byzantine resilient
aggregator to CM by discarding some of the outlier indices and then taking the
mean of the remaining ones. It offers less statistical error under the
assumption that the number $k_{m}$ of Byzantines is known.

In TM; for each coordinate, $j$, let $\mathcal{S}_{j}$ denote the ordered set
formed by permuting the elements in set $[n]$, such that the order is
determined by sorting the $j$ coordinate values. We compute the average after
discarding the $k_{m}$ largest and smallest values; hence the name trimming
[52].

$[TM(\mathbf{m}_{1},\dotso,\mathbf{m}_{n})]_{j}=\frac{1}{k-2k_{m}}\sum\limits_{i=k_{m+1}}^{k-k_{m-1}}[\mathbf{m_{\mathcal{S}_{j}(i)}}]_{j}$
(9)

Based on the variance and skewness of the various workers, the authors [52]
bound the error rate that their method ensures. As such, the larger the
variance and skewness, the higher the error rate the attacker can achieve.
Nevertheless, regardless of the variance, Byzantine clients can steadily
accumulate errors as long as staying close to the $\bar{\mathbf{m}}$ [14, 39,
2].

Bulyan repeatedly employs TM and Multi-Krum to select client momentum to
defend against Byzantine attacks. The underlying idea in Bulyan is that, with
a single gradient dimension that is extremely large, a malicious gradient can
stay near benign gradients and impede the global model’s convergence [12].

GAS highlights that the existing defence mechanisms suffer from the curse of
dimensionality and argues that it is possible to enhance the robustness of
existing aggregators by dividing the whole update vector into $p$ sub-vectors
and performing aggregation for each sub-vector independently. The hyper-
parameter $p$ is chosen according to the size of the underlying NN
architecture, and often larger values are employed for better robustness.

###  II-B Analyzing Existing Attack Methods

ALIE: [2] Traditional aggregators such as Krum [6], TM [52], and Bulyan [12]
assume that the selected set of momentums will lie within a ball centered at
the real mean within a radius, which is a function of the number of benign
clients. The attacker in [2] utilizes index-wise mean ($\bar{\mathbf{m}}$) and
standard deviation ($\bar{\bm{\sigma}}$) vectors of the benign clients to
induce small but consistent perturbations to the parameters. By keeping the
momentum values close to $\bar{\mathbf{m}}$, ALIE can steadily achieve an
accumulation of error while concealing itself as a benign client during
training. To avoid detection and stay close to the center of the ball, ALIE
scales $\bar{\bm{\sigma}}$ with a $z$ parameter, which is calculated based on
the numbers of benign and Byzantine clients. As such, let $s$ be the minimal
number $s=\left[\frac{k}{2}+1\right]-k_{m}$ of benign clients that are
required as supporters. The attacker will then use the properties of the
normal distribution, specifically the cumulative standard normal function
$\phi(z)$, and look for the maximum value of $z$, denoted by $z^{max}$, such
that $s$ benign clients will have a greater distance to the mean compared to
the Byzantine clients, such that, those $s$ clients are more likely to be
classified as Byzantines. At a high level, $z^{max}$ can be calculated as:

$z^{max}=\mathrm{max}\left\\{z:\phi(z)<\frac{k-k_{m}-s}{k-k_{m}}\right\\}$
(10)

Ultimately, $z^{max}$ is employed as a scaling parameter for the standard
deviation to perturb the mean of the benign clients in set $\mathcal{K}_{b}$:

$\mathbf{m}_{i}=\bar{\mathbf{m}}-z^{max}\mathbf{\bm{\bar{\sigma}}}\>,i\in\mathcal{K}_{m},$
(11)

where $\mathcal{K}_{m}$ is the set of Byzantine clients. Each individual
Byzantine client generates an attack with a momentum value near
$\bar{\mathbf{m}}$, following (11).

IPM: [48] From the perspective of convergence analysis, the authors [48]
highlight the required condition that the inner product between the benign
gradient $\bar{\mathbf{g}}$ and the output of the robust estimator should be
positively aligned, i.e.,

$\langle\bar{\mathbf{g}},AGG(\mathbf{g}_{i}:i\ \epsilon\
\mathcal{K})\rangle\geq 0,$ (12)

which ensures that the loss is steadily minimized over iterations. Hence, IPM
generates poisoned model updates at each iteration with an objective that the
condition in eqn (12) is not met. To this end, IPM chooses the benign gradient
with the inverse sign, $-\bar{\mathbf{g}}$, as its base111 When the
aggregation is performed over momentum terms the attack is performed as
$-z\bar{\mathbf{m}}$. for the attack and choose a proper scaling parameter $z$
for imperceptibility so that the final version of the attack
$-z\bar{\mathbf{g}}$ is not easily spotted but hinders the convergence. One of
the major drawbacks of IPM is being easily spotted by an angular investigation
and the need for a large ratio of $\frac{k_{m}}{k_{b}}$ actually to prevent
eqn 12.

Adaptive Optimized Attacks: ALIE and IPM attacks utilize a fixed pre-
determined scaling parameter throughout the training process. In [39], the
authors utilize the ALIE and IPM framework to obtain the base form of the
perturbation. However, instead of fixed scaling parameter $z$, they adaptively
change it by solving an optimization problem with the objective of maximizing
$z$ under certain geometric distance based imperceptibility constraints at
each iteration.

##  III Sparse Byzantine Attacks

This section highlights the key design aspects of our sparse Byzantine attacks
and provides the intuition behind each aspect. Before introducing our proposed
Byzantine attack, we revisit the ALIE attack, which will be the base for our
design due to certain important features we identify.

###  III-A Revisiting ALIE attack from the defence perspective

The key design objective of the ALIE is to find the adversarial model update
values, $\mathbf{m}_{adv}$, such that: 1) the adversarial values are not
statistical outliers, 2) the adversarial values strong enough to shift
distribution. To achieve this target, ALIE utilizes the index-wise statistics,
i.e.,

$\mathbf{m}_{adv}=\bar{\mathbf{m}}-z\bar{\bm{\sigma}}$ (13)

where $\bar{\bm{\sigma}}$ is the vector of index-wise standard deviation and
$z^{max}$ is the scaling factor adjusted according to the Byzantine ratio for
the imperceptibility. From the construction, one can easily observe how ALIE
can escape from index-wise statistical examinations and the corresponding
defence methods such as TM.

Further, as discussed in [2], ALIE is also effective against defence
mechanisms that utilize geometric distance as a metric to identify outliers,
such as Krum, where the aggregation rule selects a model update from the
available model updates, including both clean and Byzantine updates, which
achieves the minimum value of the neighbourhood distance measure.222Multi-Krum
relies on the same measurement; however, instead of performing an aggregation
rule that directly chooses a representative model, Multi-Krum first eliminates
the outliers and then performs averaging as an aggregation rule for the
remaining ones. We emphasize the following observation: the structure of both
Krum and TM defence mechanisms are based on outlier elimination, and even if
they successfully detect Byzantines, they also falsely classify some benign
updates as outliers, which is more frequent when the data is distributed in a
non-iid fashion. In [25], it has been further shown that this type of defence
mechanism deteriorates the performance even when there are no Byzantines or
Byzantines just mimicking the benign clients as an attack.

Therefore, as an alternative, sanitization-based defences first sanitize all
the model updates to minimize the impact of Byzantines and then perform an
aggregation over all the sanitized models in order to avoid falsely discarding
the true benign clients. Indeed, as we later numerically illustrate through
extensive simulations, CC framework, which sanitizes models through vector-
wise clipping, is quite effective against the ALIE attack. However, in [54],
the authors also identify certain limitations of the CC. To be more precise,
CC only scales the deviation from the reference model (aggregate momentum in
the previous round), and thus, angular invariant to the direction of the
perturbation. To better illustrate, when a Byzantine client follows ALIE
strategy, it sends $\bar{\mathbf{m}}-z\bar{\bm{\sigma}}$ as a poisoned model
update, and CC mechanism first measures the drift with respect to the
reference update $\tilde{\mathbf{m}}_{t}$

$\mathbf{\Delta}^{ref}_{t}=\tilde{\mathbf{m}}_{t}-\bar{\mathbf{m}}_{t}+z\bar{\mathbf{\sigma}}_{t},$
(14)

then clip the drift $\mathbf{\Delta}^{ref}_{t}$ if its norm is larger than
$\tau$, i.e.,

$\tilde{\mathbf{\Delta}}^{ref}_{t}=\min\left\\{1,\frac{\tau}{||\mathbf{\Delta}^{ref}_{t}||}\right\\}\mathbf{\Delta}^{ref}_{t},$
(15)

which we refer to as effective perturbation, and as one can observe, the
clipping only depends on the norm of $\mathbf{\Delta}^{ref}_{t}$. Hence, when
ALIE utilizes a larger scaling parameter $z$, CC can detect and clip the
attack. However, such clipping does not necessarily change the direction of
the adversarial perturbation nor prevents their correlation over time, which
enhances the time-coupled nature of the ALIE, as illustrated in [54], and
plays a key role in diverging the model.

To clarify the discussion above, we conduct an experiment where we employ ALIE
with scaling parameters $z\in\left\\{0.25,0.5,1,1.5\right\\}$ and CC with
$\tau=1$ as a robust aggregator, and then for each scaling value we measure
the average norm of the drift $||\mathbf{\Delta}^{ref}_{t}||$, the expected
angle between the effective perturbation $\tilde{\mathbf{\Delta}}^{ref}_{t}$,
$\angle(\tilde{\mathbf{\Delta}}^{ref}_{t},\tilde{\mathbf{m}}_{t})$, and the
reference update $\tilde{\mathbf{m}}_{t}$, and finally temporal correlation on
the effective perturbation, that is, the cosine similarity between the
consecutive effective perturbations,
$\mathrm{cos}(\tilde{\mathbf{\Delta}}^{ref}_{t},\tilde{\mathbf{\Delta}}^{ref}_{t-1})$.
The results are illustrated in Table III, which clearly demonstrates that
being clipped by the CC does not imply being sanitized properly. To be more
precise, as the scaling parameter $z$ is increased, CC starts clipping the
perturbation $\mathbf{\Delta}^{ref}_{t}$, as the norm goes beyond $\tau=1$.
But under such clipping, the orthogonality of the perturbation is still
preserved, and as highlighted by [54], this is quite important for derailing
the model. The key observation here is that as the scaling parameter $z$
increases, the time-coupled nature of the ALIE, measured through cosine
similarity, becomes more and more visible. Thus, we conclude that the CC
strategy gives a false sense of security against ALIE, and its performance
highly depends on the choice of $z$.

TABLE III: ALIE attack with different scaling values against CC aggregator on IID CIFAR-10 dataset trained on Resnet-20 architecture with $k=$ 25 and $k_{m}=$ 5. z | Angle | Cos. Sim. | Norm | Accuracy %
---|---|---|---|---
0.25 | 92 | 0.82 | 0.4 | 82
0.5 | 88 | 0.93 | 0.7 | 66
1 | 77 | 0.97 | 1.2 | 50
1.5 | 76 | 0.98 | 1.7 | 50

Next, we move our focus to the robust aggregators that utilize index-wise
outlier investigation, such as CM and TM, and analyze their performance with
respect to the scaling parameter $z\in\left\\{0.25,0.5,1,1.5,2\right\\}$. For
the analysis, we again consider the same simulation setup, but this time we
measure the final test accuracy and the escape ratio, which is the average
percentage of the indices where the Byzantine model evades the index-wise
statistical outlier investigation; in the case of CM, this means the median is
equal to the poisoned model. The results are exhibited in Table IV. One
immediate observation from the results is that, up to a certain level in the
experiment, the best performance is observed when $z=0.5$, and increasing the
value of $z$ may lead the attack to be more visible but also have a stronger
perturbation. In ALIE, the authors have stated, ”We would like to induce
directed small changes to many parameters, instead of large changes to a few”,
whereas we argue both can be performed at the same time.

Our key design aspect: Based on the aforementioned discussion and
observations, we argue that the ultimate objective should not always be being
invisible, and sometimes, the attack can seek a balance between being
invisible and having a stronger perturbation. The more interesting
observation, which inspired our proposed Byzantine design, is that in terms of
the final test accuracy, effectively attacking only a certain portion of the
indices but with a higher perturbation might be more effective than attacking
all indices with relatively smaller perturbation. Hence, we utilize a hybrid
design that combines two different types of attacks, one being more
conservative and imperceptible and the other being more aggressive but less
imperceptible, which can be simply obtained by choosing two different scaling
$z$ values. Later in section III-B, we further explain how these two sub-
attacks are generated and combined to form the final version of our attack.

Until this point, we have shown that when a sufficiently large $z$ value is
chosen, both CC and index-wise defence methods, TM and CM, are not effective
against ALIE. We then further discuss why larger $z$ is not desired from the
geometric-distance based defence strategies like Bulyan and Krum. To visualize
the discussion, we again follow the same simulation setup, with Bulyan and
Krum as the robust aggregators, and the results are illustrated in Table V. We
emphasize that, unlike the previous robust aggregators, with Bulyan and Krum,
we observe a drastic change in the accuracy when the value of $z$ goes beyond
a certain level; in our simulation, it appears at $z=1.5$. In contrast to our
previous observations, in the case of geometric distance-based defences, when
Byzantines are spotted, they immediately become ineffective since, unlike
index-wise investigation, these defences consider all indices, either
Byzantine or benign. This observation further supports our claim regarding the
need for a hybrid attack. To be more precise, by attacking aggressively, with
larger $z$, to only certain indices rather than all, the imperceptibility of
the attack against geometric-distance based investigation can be preserved. In
the next section, we formalize the structure of the proposed hybrid attack
based on the discussion above.

TABLE IV: Index-wise escape ratios for the ALIE attack with different $z$ scales on the CM and TM aggregators. Acc. corresponds to IID CIFAR-10 test accuracy trained on Resnet-20 with $k=$ 25 and $k_{m}=$ 5. z | CM | TM
---|---|---
Escape ratio (%) | Acc. | Escape ratio (%) | Acc.
0.25 | 66 | 45.43 | 99.9 | 72.7
0.5 | 14 | 39.4 | 98 | 36.35
1 | 0 | 48.3 | 63 | 47.9
1.5 | 0 | 47.5 | 25 | 45.1
2 | 0 | 49.3 | 7.5 | 42.9
TABLE V: Escape ratios for the Multi-Krum and Bulyan for ALIE attack. Acc. corresponds to IID CIFAR-10 test accuracy trained on Resnet-20 with $k=$ 25 and $k_{m}=$ 5. z / AGGR | Krum | Bulyan
---|---|---
Client% | Indices % | Acc. | Client % | Indices % | Acc.
0.25 | 100 | - | 55.75 | 100 | 73 | 47.13
0.5 | 100 | - | 45.15 | 100 | 60 | 47.43
1 | 100 | - | 45.32 | 100 | 29 | 32.79
1.5 | 0 | - | 87.4 | 0 | 0 | 86.54

###  III-B Our Hybrid Sparse Attack Design

In our sparse Byzantine attack design, our objective is to introduce a
framework that can escape from both index-wise and Euclidean distance-based
sanitization methods simultaneously. In the previous section, we highlighted
that although index-wise statistically imperceptible attacks, e.g., ALIE, are
effective against index-wise defence strategies, such as TM, thanks to their
time coupling nature, they often become impotent against Euclidean distance-
based defence mechanisms, such as CC.

Based on these observations, we propose to utilize a hybrid byzantine attack
that can be decomposed into two parts; each part targets a different type of
defence mechanism, i.e.,

$\bm{\Delta}=\bm{\Delta}_{1}+\bm{\Delta}_{2},$ (16)

where $\bm{\Delta}_{1}$ is formed by following a similar strategy to ALIE,
that is, the poisoned update $\bar{\mathbf{m}}_{t}+\bm{\Delta}_{t,1}$ is
statistically imperceptible to index-wise inspection, with respect to the
$\bar{\mathbf{m}}_{t}$, i.e.,

$\bar{\mathbf{m}}_{t}[i]+\bm{\Delta}_{t,1}[i]\in(\bar{\mathbf{m}}_{t}[i]-\alpha\sigma_{i,t},\bar{\mathbf{m}}_{t}[i]-\alpha\sigma_{i,t}),$
(17)

However, while still being bounded, adversarial perturbation
$\bm{\Delta}_{t,1}$ accumulates over time to inhibit the convergence.

On the other hand, for $\bm{\Delta}_{2,t}$, we focus on an Euclidean distance-
based constraint rather than the index-wise constraint given in eqn. (17),
i.e.,

$||\bm{\Delta}_{2,t}||_{2}<<||\bar{\mathbf{m}}_{t}||_{2}.$ (18)

Hence, when $\bm{\Delta}_{2,t}$ is employed alone, it can easily escape
defences that utilize geometric distance as an anomaly metric, and further, it
cannot be detected with angular inspection as well.

The key challenge here is how to preserve the aforementioned features of each
attack when they are linearly combined to form the final adversarial
perturbation $\bm{\Delta}_{t}$. To this end, we consider the following
strategy that let $\mathcal{M}_{1}\bigcup\mathcal{M}_{2}=[d]$ and
$\mathcal{M}_{1}\cap\mathcal{M}_{2}=\emptyset$, then if $i\in\mathcal{M}_{1}$
$\bm{\Delta}_{2,t}[i]=0$ and $\bm{\Delta}_{1,t}[i]=0$, otherwise. Further,
when $|\mathcal{M}_{1}|<<d$, we can assure eqn (17) for most of the indices,
hence still achieving index-wise imperceptibility. Accordingly, we generate
the Byzantine perturbation $\mathbf{\Delta}_{t}$ in the following way: We
first generate a sparse binary mask
$\mathbf{c}\in\mathcal{C}\triangleq\left\\{0,1\right\\}^{d}$ (later we further
explain how to obtain such mask), and then generate a pair of scaling
parameters $(z_{1},z_{2})$ which are used to form the adversarial perturbation
in the following way

$\mathbf{\Delta}_{t}=(z_{1}(1-\mathbf{c})+z_{2}\mathbf{c})\odot\bm{\sigma}_{t},$
(19)

and then the poisoned model is simply obtained by adding $\mathbf{\Delta}_{t}$
to the benign update $\bar{\mathbf{m}}_{t}$. Hence, the key design problem
regarding our sparse Byzantine attack is generating the sparsity mask
$\mathbf{c}$ and finding the proper pair of scaling values $(z_{1},z_{2})$.
Assume that $\mathbf{c}$ is known for now. Since from the construction, we
expect that the part corresponding to $\bm{\Delta}_{t,1}=z_{1}(1-\mathbf{c})$
to be imperceptible against index-wise investigation, we set
$z_{1}=z^{max}_{1}$, that is the maximum scaling value obtained with respect
to the Byzantine ratio according to scheme introduced in ALIE.

Once $\mathbf{c}$ and $z_{1}$ are fixed, the challenge boils down to selecting
the appropriate value for $z_{2}$. We have already highlighted the trade-off
between the imperceptibility and the strength of perturbation; thus, choosing
an arbitrarily large $z_{2}$ value may not be effective. Accordingly, we have
utilize Chebyshev’s inequality as a guide for limiting the value of $z_{2}$,
that is, Chebyshev’s inequality states

$Pr(|\bar{\mathbf{m}}[i]-\mathbf{m}[i]|>z_{2}\mathbf{\sigma}[i])\leq\frac{1}{z^{2}_{2}},$
(20)

hence, we argue that any choice of $z_{2}>\sqrt{2}\approx 1.5$ may make the
perturbation highly visible from the statistical perspective, indeed this
argument is highly aligned with our initial numerical experiments, in which we
investigate the visibility of the perturbation as illustrated in Table IV and
Table V with respect to various values of $z$, where we observe scaling $z$
beyond $1.5$ make the attack to be easily spotted by known defence mechanisms.
Here, we also note that rather than fixing $z_{1}=z^{max}_{1}$, it is possible
to seek an adaptive strategy such that the optimal value of $z_{1}$ is reset
at each iteration with respect to the benign updates
$\mathbf{M}_{t}\triangleq\left\\{\mathbf{m}_{i,t};\right\\}_{i\in\mathcal{K}_{b}}$,
as previously argued in [39]. The overall hybrid Sparse Byzantine attack
mechanims is illustrated in Algorithm 3.

Algorithm 3 Hybrid Sparse Attack

1:for t=1,…T do

2: if t == 1 then

3: Initialization Inputs: Colluded dataset $D_{col}$, target sparsity
$\delta$, pruning policy $\Pi$, initial model weights
$\bm{\theta}_{0}\in\mathbb{R}^{d}$

4: Initialization Phase:

5: Generate mask $\mathbf{c}$ for given $D_{col},\Pi,\bm{\theta}_{0},\delta$

6: Inputs: Benign updates $\mathbf{M}_{t}$, bening statistics
$(\bm{\sigma}_{t},\bar{\bm{m}}_{t})$, $z^{max}_{1}$, $z^{max}_{2}$

7: Generate pair of scaling parameters $(z^{(t)}_{1},z^{(t)}_{2})$

8: Option 1: Fixed policy

9: Use fixed scaling $z^{(t)}_{1}=z^{max}_{1}$ and $z^{(t)}_{2}=z^{max}_{2}$

10: Option 2: Semi-adaptive policy

11: Use fixed $z^{(t)}_{2}=z^{max}_{2}$ and optimize $z^{(t)}_{1}$ acc. MinSum
Policy [39]

12: Generate perturbation:
$\mathbf{\Delta}_{t}=(z^{(t)}_{1}(1-\mathbf{c})+z^{(t)}_{2}\mathbf{c})\odot\bm{\sigma}_{t}$

13: Poisoned model:
$\mathbf{m}_{adv}=\bar{\mathbf{m}}_{t}-\mathbf{\Delta}_{t}$

###  III-C Generating Sparse Mask

In the previous part, we explained how one can generate a Byzantine attack for
a given sparsity mask $\mathbf{c}$. In this section, investigate how to
generate a sparsity mask $\mathbf{c}$. The vanilla approach to generate a
sparsity mask is random sparsity; that is, for a given sparsity ratio
$\delta$, we can randomly generate a binary value for each index i.e., for
each $i\in[d]$,

$\mathbf{c}[i]=\left\\{\begin{array}[]{cl}0,&\text{if }\ w.p.~{}\delta\\\
1,&\text{if }\ w.p.~{}1-\delta\end{array}\right\\}.$ (21)

The previous studies on neural network pruning have shown that, for a moderate
sparsity level $\delta$, random sub-networks often achieve the performance of
dense networks [16]. Thus, by establishing an analogy between a sparse attack
and a sparse network, we argue that a random sparsity pattern, that is, the
number of NN weights being attacked at each layer is proportional to the
dimension of the corresponding layer, would be a good candidate for the sparse
Byzantine attack. Indeed, as we later demonstrate, random sparsity mask works
well against various defence mechanisms.

However, regarding the robustness of the trained model, the impact of each
layer may not be identical, and from the network pruning perspective, one may
need to assign a different sparsity ratio to each layer. To clarify, having a
set of $l$ layers $\mathcal{L}=[l]$, one may consider a different sparsity
ratio $\delta_{l}$ that satisfies the overall sparsity constraint
$\frac{\sum_{i\in\mathcal{L}}\delta_{l}d_{i}}{d}=\delta$. In the literature,
there are different approaches to distribute the sparsity among different
layers; for instance, in [13], the authors utilize Erdos-Renyi-Kernel (ERK)
formulation. Another important aspect often emphasized in the literature on
sparse networks is certain sensitive layers, which we further clarify next,
being excluded from the network pruning since these parts of the NN model are
highly sensitive [9, 42]. Therefore, from the attack perspective, we argue
that we should attack these sensitive layers particularly. Accordingly, we
make all the corresponding locations in $\mathbf{c}$ have a value of $1$ and
consider the sparsity ratio $\delta$ for the remaining layers.

We remark that the approaches or rules mentioned above omniscient to target
dataset and do not induce a high computational complexity. On the other hand,
it has been known that, in high-sparsity regimes, randomly generated sub-
networks do not perform well. Equivalently, if we want to design an attack
with a higher sparsity pattern, the randomly generated $\mathbf{c}$ may not be
a good option. Therefore, in that case, we can further utilize the small
portion of the training data, that is, the data gathered by the colluding
Byzantines, and perform network pruning to identify those important weights to
be attacked. Further discussions on the performance of the aforementioned
approaches, as well as different sparsity ratios, will be presented in the
next section. Further, later in section III-C1, we revisit some of the network
pruning strategies, which work well with limited or no data, to form our
sparse attack. From the design perspective, if one aims to make the attack
even more imperceptible with higher sparsity, then we numerically show that
the use of a network pruning strategy has a clear advantage compared to the
randomly generated sparsity.

####  III-C1 Sparsity with Network Pruning

The overall objective of NN pruning is to find a small subset of NN parameters
$\mathcal{M}$ with a size of $\kappa<<d$ such that the NN model with only the
subset of the parameters $\bm{\theta}_{\mathcal{M}}$ is sufficient to achieve
a certain loss over the given dataset $D$. Formally speaking, the objective
can be formulated as

$\begin{matrix}\underset{\mathbf{c},\bm{\theta}}{\text{min}}~{}F(\mathbf{c}\odot\bm{\theta};D)=\frac{1}{|D|}\sum_{\zeta\in
D}\quad f(\mathbf{c}\odot\bm{\theta};\zeta)\\\
\text{s.t.}:||\mathbf{c}||_{0}\leq\kappa\text{ and
}\bm{\theta}\in\mathbb{R}^{d}.\end{matrix}$ (22)

In our work, we mainly focus on NN pruning strategies that are performed with
a limited dataset since the Byzantines may have access to a limited number of
data samples to form a sparsity pattern. In SNIP [29], the authors highlight
that, for a given $\bm{\theta}$, the additional loss incurred due to the
pruning of the $i^{th}$ weight, i.e.,

$\Delta
F_{i}=F(\mathbf{1}\odot\bm{\theta};D)-F((\mathbf{1}-\mathbf{e}_{i})\odot\bm{\theta};D)$
(23)

where $\mathbf{e}_{i}$ is a one-hot vector where the $i^{th}$ value is one and
zeros everywhere except the $i^{th}$ index. It can be approximated by gradient
with respect to the $\mathbf{c}$, such that,

$\displaystyle\Delta F_{i}\approx$ $\displaystyle\frac{\partial
F(\mathbf{c}\odot\bm{\theta};D)}{\partial c_{i}}|_{\mathbf{c}=\mathbf{1}}=$
(24) $\displaystyle
lim_{\delta\xrightarrow{}0}\frac{F(\mathbf{c}\odot\bm{\theta};D)-F((\mathbf{c}-\delta\mathbf{e}_{i})\odot\bm{\theta};D)}{\delta}|_{\mathbf{c}=\mathbf{1}}.$

The term above implies a directional derivative, a sparse direction, and thus
can be simply written in the following form:

$\bm{\theta}_{i}\frac{\partial F(\bm{\theta};D)}{\bm{\theta}_{i}}.$ (25)

Hence, SNIP concludes that $s_{i}\triangleq|\bm{\theta}_{i}\frac{\partial
F(\bm{\theta};D)}{\bm{\theta}_{i}}|$ can be used as a salience measure for the
connection sensitivity. Though building on the same idea, FORCE [9] argues for
measuring the connection sensitivity after the pruning rather than before the
pruning. To be more specific, FORCE examines the gradient for the foresight
sparse model, i.e,

$\frac{\partial
F(\mathbf{c}\odot\bm{\theta};D)}{\partial\mathbf{c}}|_{\mathbf{c}=\hat{\mathbf{c}}},~{}||\hat{\mathbf{c}}||_{0}=\kappa,\hat{\mathbf{c}}\in\mathcal{C}$
(26)

Accordingly, FORCE searches for the binary sparse mask $\mathbf{c}$ that
maximize the following salience measure:

$S(\bm{\theta},\mathbf{c})=\sum_{i:\mathbf{c}[i]=1}\underbrace{|\bm{\theta}\odot\nabla
F(\mathbf{c}\odot\bm{\theta};D)|}_{\mathbf{s}(\bm{\theta},\mathbf{c})}[i]$
(27)

where $\nabla F(\mathbf{c}\odot\bm{\theta};D)=\left(\frac{\partial
F(\bar{\bm{\theta}};D)}{\partial\bar{\bm{\theta}}}|_{\mathbf{c}}\right)$,
$\bar{\bm{\theta}}=\mathbf{c}~{}\odot~{}\bm{\theta}$, from equation (5) in
[9]. However, such a strategy that measures connection sensitivity after
pruning is practically not feasible due to its combinatorial complexity; that
is, ideally, we need to compute the salience for all possible $d\choose\kappa$
sparsity patterns. To mitigate the combinatorial complexity, FORCE argues to
perform pruning in an iterative manner, under the assumption that the change
of the gradient can be neglected, rather than checking each possible sparsity
pattern individually; for the details, refer to [9].
In our design, we utilize the FORCE framework since it does not require the
whole dataset, which is aligned with our problem setup, where Byzantines can
access a small portion of the dataset. Although, there is a strong analogy
between NN pruning and sparse Byzantine attacks, when we implement the FORCE
framework for high sparsity regime, such as $\delta\leq\%1$, we observe that
non-sparse positions are accumulated in certain layers of the network, namely
early convolutional layers and penultimate fully connected layer on the CNN
architectures. To better visualise, we plot the percentage of the remaining
(non-pruned) weights for each layer of the ResNet Architecture after pruning
with FORCE, which is shown in Fig. 1 as the colour orange. From the Byzantine
perspective, targeting only certain layers may not be effective, especially
when the employed robust aggregator performs a layer-wise defence
investigation [43, 30]. Therefore, we modify the FORCE framework with certain
layer-wise sparsity constraints so that the corresponding attack is more
homogeneously distributed among the layers; in other words, we aim to prevent
certain layers from being overemphasized. Accordingly, we introduce the term
$\delta_{max}$ to impose that the sparsity ratio of a layer cannot be larger
than $\delta_{max}$. Our experiments show that pruning methods often
overemphasize certain layers, particularly the penultimate FC layer, as
illustrated in Fig. 1. Hence, we argue that imposing constraints for these
particular layers is often sufficient for our purpose. The sparsity pattern
obtained by setting $\delta^{max}_{FC}=0.25$ is illustrated in Fig. 1 as the
colour blue, where we observe while the penultimate FC layer becomes more
sparse the number of non-sparse positions in early convolutional layers
increases.

TABLE VI: Final test accuracy results of training ResNet-20 architecture with
Cifar-10 dataset distributed IID over $k=25$ client $k_{m}=5$ of them being
malicious. The training is performed over 100 epochs and repeated for 8
different aggregation mechanisms and under 8 different Byzantine attack
strategies. The reported results are obtained by averaging 3 number of
independent trials. We use  blue,  red and  green to highlight the best
results for fixed policy attacks, ALIE, Bit Flip, IPM, Label Flip, ROP
dynamically optimized attacks, Min-Sum, Min-Max and proposed sparse hybrid
attack variations with different sparsity constraint $\delta$, respectively,
and we underline the best attack result against each defence mechanism.

Method | Bulyan | CC | CM | M-Krum | RFA | TM | GAS-KRUM | GAS-Bulyan
---|---|---|---|---|---|---|---|---
No ATK | 87.63 $\pm$ 0.96 | 87.42 $\pm$ 0.45 | 88.03 $\pm$ 0.14 | 89.24 $\pm$ 0.67 | 88.2 $\pm$ 0.31 | 87.76 $\pm$ 0.37 | 87.7 $\pm$ 0.31 | 87.58 $\pm$ 0.41
ALIE | 47.13 $\pm$ 16.1 | 80.73 $\pm$ 1.41 | 45.09 $\pm$ 9.35 | 55.76 $\pm$ 13.78 | missing32.26 $\pm$ 16.08 | 72.78 $\pm$ 2.49 | 80.96 $\pm$ 0.92 | 81.06 $\pm$ 0.62
Bit Flip | 86.18 $\pm$ 0.57 | 82.25 $\pm$ 0.66 | 80.86 $\pm$ 0.78 | 87.82 $\pm$ 0.31 | 87.5 $\pm$ 0.65 | 80.08 $\pm$ 0.88 | 86.52 $\pm$ 0.49 | 86.79 $\pm$ 0.84
IPM | 70.01 $\pm$ 0.88 | 85.85 $\pm$ 0.33 | 54.52 $\pm$ 0.75 | 83.79 $\pm$ 0.11 | 66.36 $\pm$ 0.67 | 84.9 $\pm$ 0.1 | 85.26 $\pm$ 0.17 | 85.63 $\pm$ 0.32
Label Flip | 85.78 $\pm$ 0.3 | 86.38 $\pm$ 0.55 | 79.89 $\pm$ 0.14 | 87.68 $\pm$ 0.14 | 86.28 $\pm$ 0.08 | 78.51 $\pm$ 0.26 | 87.11 $\pm$ 0.69 | 86.88 $\pm$ 0.63
ROP | 41.45 $\pm$ 1.98 | 47.38 $\pm$ 0.9 | 79.03 $\pm$ 0.4 | 88.22 $\pm$ 0.5 | 43.25 $\pm$ 2.23 | 62.72 $\pm$ 1.28 | 61.97 $\pm$ 1.57 | 63.55 $\pm$ 0.7
Best Attack * | 41.45 $\pm$ 1.98 | 47.38 $\pm$ 0.9 | 45.09 $\pm$ 9.35 | 55.76 $\pm$ 13.78 | 32.26 $\pm$ 16.08 | 62.72 $\pm$ 1.28 | 61.97 $\pm$ 1.57 | 63.55 $\pm$ 0.7
Min-Sum | 41.13 $\pm$ 0.51 | 55.46 $\pm$ 5.21 | 47.6 $\pm$ 0.37 | 55.12 $\pm$ 1.32 | 49.4 $\pm$ 8.64 | 47.48 $\pm$ 7.44 | 55.12 $\pm$ 3.03 | missing49.85 $\pm$ 11.81
Min-Max | 67.17 $\pm$ 4.44 | 40.32 $\pm$ 7.81 | 47.97 $\pm$ 1.83 | 61.59 $\pm$ 11.95 | 48.7 $\pm$ 2.75 | 42.87 $\pm$ 6.23 | 47.09 $\pm$ 3.93 | 86.6 $\pm$ 0.19
$\Pi_{r}(\delta=5\times 10^{-3})$ | 42.98 $\pm$ 17.99 | 79.27 $\pm$ 0.4 | 39.97 $\pm$ 16.02 | 54.87 $\pm$ 5.29 | 42.67 $\pm$ 6.28 | 70.58 $\pm$ 1.26 | 78.93 $\pm$ 0.96 | 78.46 $\pm$ 1.29
$\Pi_{r}(\delta=5\times 10^{-2})$ | 47.62 $\pm$ 3.08 | 75.08 $\pm$ 0.76 | 45.94 $\pm$ 5.94 | 14.14 $\pm$ 2.01 | 48.41 $\pm$ 11.4 | 70.16 $\pm$ 1.93 | 75.85 $\pm$ 0.58 | 75.7 $\pm$ 1.3
$\Pi_{r}(\delta=2\times 10^{-1})$ | 40.25 $\pm$ 0.5 | 41.62 $\pm$ 2.5 | 49.27 $\pm$ 8.52 | 12.1 $\pm$ 0.68 | 59.43 $\pm$ 0.62 | 61.9 $\pm$ 2.36 | 65.03 $\pm$ 1.28 | 66.26 $\pm$ 2.17
$\Pi_{r}(\delta=5\times 10^{-1})$ | 28.17 $\pm$ 1.86 | 46.84 $\pm$ 2.76 | 42.13 $\pm$ 6.49 | missing11.4 $\pm$ 1.23 | 47.82 $\pm$ 5.88 | 40.02 $\pm$ 5.52 | 48.29 $\pm$ 0.79 | 86.83 $\pm$ 0.53
$\Pi_{r}^{+}(\delta=5\times 10^{-3})$ | 37.85 $\pm$ 10.49 | 48.91 $\pm$ 20.78 | 45.5 $\pm$ 9.69 | 30.08 $\pm$ 11.79 | 45.21 $\pm$ 5.61 | 63.66 $\pm$ 11.99 | 80.49 $\pm$ 0.31 | 79.04 $\pm$ 1.24
$\Pi_{r}^{+}(\delta=5\times 10^{-2})$ | 37.53 $\pm$ 5.3 | 50.14 $\pm$ 9.82 | 49.62 $\pm$ 9.44 | 14.23 $\pm$ 0.7 | 50.03 $\pm$ 1.94 | 61.39 $\pm$ 5.87 | 72.17 $\pm$ 1.1 | 73.14 $\pm$ 1.1
$\Pi_{r}^{+}(\delta=2\times 10^{-1})$ | 26.84 $\pm$ 1.21 | 43.04 $\pm$ 1.28 | 45.47 $\pm$ 5.19 | 14.75 $\pm$ 1.39 | 39.31 $\pm$ 10.38 | 46.25 $\pm$ 2.4 | 61.51 $\pm$ 6.0 | 60.67 $\pm$ 4.96
$\Pi_{r}^{+}(\delta=5\times 10^{-1})$ | 27.59 $\pm$ 2.56 | 38.86 $\pm$ 0.63 | 43.1 $\pm$ 10.64 | 11.46 $\pm$ 0.9 | 36.84 $\pm$ 2.91 | 31.34 $\pm$ 9.72 | missing39.34 $\pm$ 7.34 | 86.15 $\pm$ 0.09
$\Pi_{lr}(\delta=5\times 10^{-3})$ | 44.55 $\pm$ 17.42 | 79.24 $\pm$ 0.3 | 36.72 $\pm$ 1.78 | 54.74 $\pm$ 4.75 | 52.97 $\pm$ 3.84 | 71.66 $\pm$ 3.55 | 81.23 $\pm$ 1.03 | 80.2 $\pm$ 0.89
$\Pi_{lr}(\delta=5\times 10^{-2})$ | 53.15 $\pm$ 3.91 | 75.44 $\pm$ 1.95 | 36.28 $\pm$ 5.56 | 26.7 $\pm$ 11.08 | 53.92 $\pm$ 5.87 | 67.88 $\pm$ 2.93 | 74.24 $\pm$ 2.61 | 74.69 $\pm$ 0.7
$\Pi_{lr}(\delta=2\times 10^{-1})$ | 36.27 $\pm$ 2.55 | 40.99 $\pm$ 2.14 | 50.32 $\pm$ 5.19 | 13.32 $\pm$ 1.89 | 59.37 $\pm$ 2.0 | 62.09 $\pm$ 2.91 | 64.92 $\pm$ 3.19 | 61.87 $\pm$ 3.6
$\Pi_{lr}(\delta=5\times 10^{-1})$ | 26.33 $\pm$ 2.59 | 37.52 $\pm$ 8.25 | 50.58 $\pm$ 3.41 | 13.27 $\pm$ 3.62 | 48.44 $\pm$ 5.17 | 39.66 $\pm$ 1.44 | 48.03 $\pm$ 1.78 | 86.82 $\pm$ 0.45
$\Pi_{lr}^{+}(\delta=5\times 10^{-3})$ | 32.78 $\pm$ 4.38 | 74.71 $\pm$ 5.22 | 45.29 $\pm$ 6.27 | 13.83 $\pm$ 0.96 | 45.58 $\pm$ 4.33 | 56.61 $\pm$ 18.7 | 78.89 $\pm$ 1.23 | 77.27 $\pm$ 1.82
$\Pi_{lr}^{+}(\delta=5\times 10^{-2})$ | 32.03 $\pm$ 5.44 | 45.11 $\pm$ 2.04 | 35.94 $\pm$ 7.81 | 16.5 $\pm$ 2.91 | 56.84 $\pm$ 2.83 | 58.99 $\pm$ 7.14 | 73.16 $\pm$ 0.64 | 69.88 $\pm$ 1.91
$\Pi_{lr}^{+}(\delta=2\times 10^{-1})$ | missing26.03 $\pm$ 3.01 | 34.33 $\pm$ 7.28 | 48.05 $\pm$ 1.81 | 17.69 $\pm$ 1.69 | 46.71 $\pm$ 4.82 | 42.3 $\pm$ 7.16 | 58.74 $\pm$ 6.04 | 56.08 $\pm$ 8.72
$\Pi_{lr}^{+}(\delta=5\times 10^{-1})$ | 26.27 $\pm$ 3.35 | 37.63 $\pm$ 7.02 | 45.28 $\pm$ 3.51 | 13.51 $\pm$ 1.92 | 42.7 $\pm$ 2.77 | missing30.57 $\pm$ 0.49 | 39.18 $\pm$ 2.52 | 87.0 $\pm$ 0.38
$\Pi_{ERK}^{+}(\delta=5\times 10^{-3})$ | 26.66 $\pm$ 1.09 | 40.96 $\pm$ 5.2 | 44.35 $\pm$ 13.51 | 16.26 $\pm$ 1.34 | 45.31 $\pm$ 12.67 | 72.06 $\pm$ 0.6 | 77.97 $\pm$ 0.55 | 73.33 $\pm$ 1.58
$\Pi_{ERK}^{+}(\delta=5\times 10^{-2})$ | 57.42 $\pm$ 0.43 | 65.18 $\pm$ 9.33 | 47.09 $\pm$ 4.81 | 61.39 $\pm$ 7.84 | 69.86 $\pm$ 0.67 | 62.86 $\pm$ 10.84 | 70.8 $\pm$ 3.51 | 73.59 $\pm$ 1.02
$\Pi_{ERK}^{+}(\delta=2\times 10^{-1})$ | 58.49 $\pm$ 0.9 | 50.74 $\pm$ 11.08 | 55.35 $\pm$ 1.73 | 56.7 $\pm$ 6.78 | 65.59 $\pm$ 1.28 | 58.29 $\pm$ 4.5 | 61.13 $\pm$ 4.51 | 71.46 $\pm$ 2.1
$\Pi_{ERK}^{+}(\delta=5\times 10^{-1})$ | 85.32 $\pm$ 0.74 | 52.6 $\pm$ 10.18 | 50.1 $\pm$ 2.66 | 55.0 $\pm$ 15.12 | 59.69 $\pm$ 1.55 | 49.23 $\pm$ 2.96 | 51.88 $\pm$ 2.74 | 86.32 $\pm$ 0.21
$\Pi_{F}^{+}(\delta=5\times 10^{-3})$ | 40.65 $\pm$ 1.71 | 39.61 $\pm$ 3.61 | 39.46 $\pm$ 9.21 | 15.47 $\pm$ 1.16 | 52.79 $\pm$ 10.89 | 53.53 $\pm$ 16.59 | 74.55 $\pm$ 3.95 | 74.82 $\pm$ 3.34
$\Pi_{F}^{+}(\delta=5\times 10^{-3},$ $\delta^{max}_{FC}$ = 0.25) | 31.34 $\pm$ 2.67 | missing33.92 $\pm$ 2.88 | missing35.71 $\pm$ 4.46 | 16.33 $\pm$ 2.06 | 42.7 $\pm$ 10.29 | 33.44 $\pm$ 3.26 | 59.51 $\pm$ 17.81 | 52.54 $\pm$ 8.34
$\Pi_{F}^{+}(\delta=1\times 10^{-2},\ \delta^{max}_{FC}$ = 0.25) | 28.39 $\pm$ 1.89 | 38.69 $\pm$ 0.99 | 42.27 $\pm$ 17.84 | 16.33 $\pm$ 2.06 | 39.9 $\pm$ 9.44 | 47.54 $\pm$ 6.68 | 67.75 $\pm$ 8.23 | 59.34 $\pm$ 9.52

##  IV Numerical Results

###  IV-A Simulation Setup

For the simulation setup, we consider a synchronous and centralized FL
framework with $k$=25 participating clients. We set the Byzantine ratio
$\gamma=0.2$, i.e., $k_{m}$=5 clients are malicious, which is the common
scenario in the literature [24, 54]. For the training, we follow a similar
setup as in [24, 54], where we train our neural networks for 100 epochs with a
local batch size of 32 and an initial learning rate of $\eta=0.1$, which is
reduced at epoch 75 by a factor of $0.1$. In all the simulations, we utilize
local momentum with $\beta=0.9$ since the use of local momentum reduces the
variance and enhances the robustness against Byzantines.

####  IV-A1 Datasets and Networks

For the image classification task, we consider MNIST[28], FMNIST[46] and
CIFAR-10 [27] datasets and train them with a 2-layer MLP, 2-layer
convolutional neural network (CNN), and Resnet-20 [22] architectures,
respectively.

We consider two scenarios where we distribute the data among the clients in
IID and non-IID manners, respectively. In the IID scenario, we distribute the
training data homogeneously among the clients with respect to their labels. In
the non-IID scenario, similar to previous works [40, 44, 20, 51], the dataset
is divided among the clients according to Dirichlet distribution [32] with
parameter $\alpha$, denoted by $\mathrm{Dir}(\alpha)$, where $\alpha$
parameter controls the skewness of the probability distribution with respect
to the class labels. In our experiments, we set $\alpha=1$, which is argued to
be valid to simulate practical scenarios.

####  IV-A2 Aggregators

We consider Bulyan, CC, CM, MultiKrum, RFA, SignSGD, and TM for the robust
aggregators. We also employ GAS with Bulyan and MultiKrum since they are the
best-performing base aggregators with the GAS framework[30]. For CC, we employ
a fixed ball radius of $\tau=1$ since a higher radius is more susceptible to
Byzantine attacks [37, 54], and lower $\tau$ performs suboptimally in non-IID
settings [54] and the number of clipping iterations to $l=1$. For GAS, we set
$p=1000$ on ResNet-20 and 2-Layer CNN, while on 2-layer MLP, we set $p=100$
due to the smaller network size. Our $p$ values are proportional to the best
$p$ parameter that authors employed in their works for their NN size [30]. For
SignSGD, we employ an initial learning rate of $0.01$, which makes learning
more stable.

####  IV-A3 Attacks:

For non-omniscient attacks, we consider the bit-flip and label-flip [5, 17]
attacks. In the bit-flip attack, Byzantine clients flip the signs of their own
gradient values, whereas, in the label-flip attack, Byzantine clients flip the
label of a sample by subtracting it from the total number of image classes in
the dataset. For the omniscient model poisoning attacks, we consider ALIE [2],
IPM [48], and ROP [54]. In ROP, we employ the original parameter values given
in [54]. For IPM, previous works have considered different $z$ values, i.e.,
in [24] $z=0.1$ and in [54] $z=0.2$. In our early experiments, we tried
different $z$ values, including the previously used ones $z=0.2$ and $z=0.2$.
We observe that IPM performs its best with $z=0.4$; hence, in our results, we
report the performance with $z=0.4$. For the optimized attacks, we use
aggregator agnostic Min-sum and Min-max attacks [39] using the
$\mathbf{\sigma}_{t}$ as the initial perturbation vector.

####  IV-A4 Hybrid Sparse Attacks:

In our experiments, we consider four different variations of our sparse
Byzantine attack framework. First, we consider base strategy $\Pi_{r}(\delta)$
that generates a sparse mask with a sparsity ratio of $\delta$, randomly
without any constraint; for the second strategy, we impose a constraint that
the sparsity ratio of $\delta$ is achieved at each layer, which is denoted by
$\Pi_{lr}(\delta)$. Finally, for the last two strategies, we modify the first
two strategies by inducing the constraint that the critical layers should be
non-sparse, i.e., $\delta_{l}=1$, for $l\in\mathcal{L}_{critical}$, which are
denoted with $\Pi_{r}^{+}(\delta)$ and $\Pi_{lr}^{+}(\delta)$, respectively.
In addition to random sparsity, we also consider ERK [13] method to obtain
sparsity patterns for each individual layer, denoted by
$\Pi^{+}_{ERK}(\delta)$, and finally, utilize the network pruning-based
attack, using FORCE algorithm, which is denoted by $\Pi^{+}_{FC}(\delta)$.

###  IV-B Experimental Results

To analyze the effectiveness of our proposed solution, we have performed
extensive simulations by measuring its impact on existing SoTA robust
aggregators, namely, TM[52], CC[24], Bulyan[12], Multi-Krum [6], RFA[36],
SignSGD (Majority Voting)[3] and two different implementations (utilize Krum
or Bulyan) of recent SoTA defence mechanism GAS[30]. For the fair performance
comparison, we consider the existing attacks under two categories based on
whether they are dynamically optimized at each round or utilize a fixed
policy. In the first category, the attacks with fixed policy, we consider bit-
flip, label-flip, ALIE [2], IPM [48] and ROP [54]. In the second category, we
consider dynamically optimized attacks Min-Sum [39] and Min-Max [39].
In Table VI, we demonstrate the final test accuracy results for CIFAR-10
dataset with IID-distribution and ResNet-20 architecture. For the sake of
presentation, we highlight the best Attack performance for those who utilize
fixed policy in addition to individual performances. For our proposed Hybrid
sparse attack we consider all four variations of the random sparsity masks
under four different sparsity regime, i.e., $\delta=\left\\{5\times
10^{-3},5\times 10^{-2},2\times 10^{-1},5\times 10^{-1}\right\\}$ as well as
the mask generated by FORCE for $\delta=5\times 10^{-3}$. The corresponding
test accuracy results are illustrated in Table VI. One of our immediate
observations is that both variations of the GAS defence mechanism are highly
effective against fixed policy attacks; we observe minimum test accuracy of
$\%85$ except for our proposed hybrid sparse attack. Regarding the performance
of the proposed hybrid sparse attack, we observe that, in the case of randomly
generated sparsity mask, the best performance is obtained with
$\Pi_{lr}^{+}(\delta)$ which induces a layer-wise sparsity constraint and
identifies the critical layers and often it performs best for moderate
sparsity regime $\delta=2\times 10^{-1}$ and the strength of the hybrid sparse
attack weakens with decreasing the sparsity ratio. Also the performance
comparison of $\Pi_{lr}(\delta)$ and $\Pi_{lr}^{+}(\delta)$ clearly highlights
the importance of targeting critical layers.
To test the advantage of the side information regarding synaptic connectivity
on the performance of the attack, we compare the performance of the attack
generated according to $\Pi_{lr}^{+}(\delta)$ and the one generated according
to pruning strategy $\Pi_{F}^{+}(\delta)$, under the same sparsity constraint
$\delta=5\times 10^{-3}$. Contrary to our initial argument that attacking
specific locations that exhibit certain vital connectivity patterns enhances
the strength of the attack, on average, we do not observe a visible
improvement in the impact of the hybrid sparse attack. However, the
interesting observation is that in the case of CC, which re-scales the
adversarial perturbation without changing the direction, the mask obtained
through pruning provides a significant observation, as illustrated in Table
VI. At this point, we recall that NN pruning strategies often overemphasize
certain layers; hence, while taking the topology-based connectivity and
sensitivity patterns into account for targeting enhances the impact of attack,
which is visible in the case of CC, accumulation of attack at certain layers
may make attack to relinquish its imperceptibility.
To verify the argument above, we follow a strategy where we again utilize the
FORCE algorithm to perform network pruning but induce an additional constraint
on the FC layer, i.e., $\delta_{FC}^{max}=2\times 10^{-1}$, in order to
achieve a sparsity mask where non-sparse positions are distributed more
uniformly compared to the previous case, where most of them accumulated at FC
layer. The results back our argument, and visible improvements are observed in
the performance, especially against the GAS defence mechanism, which divides
the vector into smaller sub-vectors and performs aggregation for each sub-
vector separately. For the completeness of our analysis regarding the relation
between the sparsity pattern and the attack performance, we further utilize
the ERK-base sparsity masks [13], which divides the sparsity budget among the
layers according to the number of incoming and outgoing connections between
the hidden layers. [Rewrite] However, similar to the FORCE, which
overemphasizes the initial conv layer and particularly the penultimate FC
layer, the ERK method overemphasizes the initial conv layer, where all the
weights are preserved as illustrated in 1 with the green colour. Hence,
similar to vanilla FORCE, the overemphasized initial conv layer makes the ERK
method less desirable against certain defence mechanisms, especially on GAS.

Figure 1: Distribution of the non-sparse locations (remaining weights) in
ResNet-20 architecture after pruning with ERK, Vanilla Force and Force with
sparsity constraint on FC layer, $\delta^{max}_{FC}=0.25$, for $\delta=0.005$.
The x-axis denotes the NN layers, where Cw$i$ denotes the weights of the $i$th
convolutional layer, and Fc1 denotes the weights of the fully connected layer
at the end.

Overall, from the discussion above, we conclude that there is a certain trade-
off regarding the formation of the attack; the underlying sparsity mask should
be designed in parallel to the topology-based connectivity and sensitivity
patterns identified through network pruning, but also the position of the non-
sparse weights, where the attack targets more aggressively, should be not be
concentrated at particular layers. As we already discuss above, such
unintended concentration can be mitigated by enforcing a layer-wise sparsity
constraint. Through extensive simulations, we have analyzed various
alternatives for regularizing the layer-wise sparsity, and the best yet simple
solution is to have a constraint for the FC layer with $\delta_{FC}^{max}$
Next, we repeat the same simulation for the non-iid data distribution
scenario, and the results are illustrated in Table VII. In this more practical
scenario, we have shown that the proposed hybrid sparse attack surpasses all
the known defence mechanisms, including GAS, which is effective against all
fixed policy Byzantine attacks, almost achieves Byzantine-free accuracy, and
makes a model diverge. One of the key aspects here is that the performance of
$\Pi_{FC}^{+}(\delta)$, compared to its baseline form ALIE, demonstrates how
attacking only a very limited number of carefully selected weights enhances
the strength of the attack significantly. Consequently, the simulation results
obtained for ALIE, Min-Sum, and hybrid sparse attack, where all three
strategies utilize the same base attack formation, demonstrate that the level
of aggressiveness and the carefully chosen target locations significantly
increase the strength of the attack, which particularly visible in the case of
GAS defence.
Finally, to better illustrate the impact of various Byzantine strategies
against the convergence of the training process orchestrated by different
robust aggregators, we plot the convergence dymanics in Fig. 2, for the both
IID and Non-IID scenarios.

Figure 2: Test accuracy results of training ResNet-20 architecture with
Cifar-10 dataset distributed IID and non-IID over $k=25$ client $k_{m}=5$ of
them being malicious. The training and testing are performed over 100 epochs
and repeated for 9 different aggregation mechanisms and under 8 different
Byzantine attack strategies. The reported results are obtained by averaging 3
number of independent trials. TABLE VII: Final test accuracy results of
training ResNet-20 architecture with Cifar-10 dataset distributed non-IID over
$k=25$ client $k_{m}=5$ of them being malicious. The training is performed
over 100 epochs and repeated for 8 different aggregation mechanisms and under
8 different Byzantine attack strategies. The reported results are obtained by
averaging 3 number of independent trials. We use  blue,  red and  green to
highlight the best results for fixed policy attacks, ALIE, Bit Flip, IPM,
Label Flip, ROP dynamically optimized attacks, Min-Sum, Min-Max and proposed
sparse hybrid attack variations with different sparsity constraint $\delta$,
respectively, and we underline the best attack result against each defence
mechanism.

Method / AGGR | Bulyan | CC | CM | M-Krum | RFA | TM | GAS-KRUM | GAS-Bulyan
---|---|---|---|---|---|---|---|---
No ATK | 86.04 $\pm$ 0.47 | 86.87 $\pm$ 0.3 | 79.94 $\pm$ 0.43 | 88.1 $\pm$ 0.4 | 86.44 $\pm$ 0.72 | 86.66 $\pm$ 0.53 | 86.66 $\pm$ 0.49 | 86.31 $\pm$ 0.49
ALIE | 34.73 $\pm$ 0.69 | 59.72 $\pm$ 3.42 | 34.48 $\pm$ 1.89 | 49.66 $\pm$ 9.25 | 32.85 $\pm$ 3.35 | 42.07 $\pm$ 7.2 | 57.0 $\pm$ 11.37 | 63.62 $\pm$ 5.28
Bit Flip | 79.76 $\pm$ 1.04 | 81.79 $\pm$ 0.49 | 73.54 $\pm$ 1.15 | 85.25 $\pm$ 0.52 | 80.94 $\pm$ 0.64 | 76.99 $\pm$ 1.29 | 84.22 $\pm$ 0.31 | 84.48 $\pm$ 0.24
IPM | 46.26 $\pm$ 0.88 | 83.26 $\pm$ 0.7 | 61.87 $\pm$ 4.58 | 73.18 $\pm$ 3.9 | 67.88 $\pm$ 1.91 | 77.88 $\pm$ 1.9 | 83.31 $\pm$ 0.61 | 82.35 $\pm$ 0.74
Label Flip | 80.88 $\pm$ 1.03 | 85.23 $\pm$ 0.39 | 72.44 $\pm$ 0.24 | 87.31 $\pm$ 0.32 | 84.63 $\pm$ 0.03 | 72.95 $\pm$ 1.5 | 85.95 $\pm$ 0.55 | 85.55 $\pm$ 0.34
ROP | 22.46 $\pm$ 5.42 | 28.93 $\pm$ 1.11 | 31.19 $\pm$ 2.41 | 87.3 $\pm$ 0.73 | 34.39 $\pm$ 1.99 | 55.6 $\pm$ 1.59 | 64.63 $\pm$ 1.74 | 62.52 $\pm$ 1.19
Best Attack * | 22.46 $\pm$ 5.42 | 28.93 $\pm$ 1.11 | 31.19 $\pm$ 2.41 | 49.66 $\pm$ 9.25 | 34.39 $\pm$ 1.99 | 42.07 $\pm$ 7.2 | 57.0 $\pm$ 11.37 | 62.52 $\pm$ 1.19
Min-Sum | 21.59 $\pm$ 4.12 | 38.08 $\pm$ 2.63 | 26.08 $\pm$ 2.48 | 27.19 $\pm$ 5.87 | 26.83 $\pm$ 2.2 | 32.48 $\pm$ 1.94 | 45.77 $\pm$ 4.67 | 45.15 $\pm$ 5.78
Min-Max | 82.21 $\pm$ 0.12 | 34.04 $\pm$ 1.4 | 25.98 $\pm$ 2.98 | 86.89 $\pm$ 0.51 | 30.19 $\pm$ 2.21 | 18.03 $\pm$ 1.54 | 30.05 $\pm$ 16.57 | 85.89 $\pm$ 0.42
$\Pi_{r}(\delta=5\times 10^{-3})$ | 22.74 $\pm$ 2.4 | 47.4 $\pm$ 2.25 | 26.96 $\pm$ 1.7 | 27.37 $\pm$ 4.12 | 33.84 $\pm$ 10.12 | 36.56 $\pm$ 7.23 | 59.96 $\pm$ 4.97 | 48.11 $\pm$ 13.35
$\Pi_{r}(\delta=5\times 10^{-2})$ | 18.95 $\pm$ 2.18 | 25.22 $\pm$ 3.37 | 28.13 $\pm$ 1.66 | 13.57 $\pm$ 3.53 | 28.27 $\pm$ 5.97 | 37.8 $\pm$ 8.69 | 59.37 $\pm$ 3.24 | 43.02 $\pm$ 7.57
$\Pi_{r}(\delta=2\times 10^{-1})$ | 11.71 $\pm$ 1.43 | 19.57 $\pm$ 2.39 | 27.3 $\pm$ 1.09 | 10.04 $\pm$ 0.06 | 39.79 $\pm$ 6.29 | 23.92 $\pm$ 8.62 | 55.63 $\pm$ 8.78 | 23.36 $\pm$ 7.91
$\Pi_{r}(\delta=5\times 10^{-1})$ | 25.48 $\pm$ 4.24 | missing15.41 $\pm$ 3.46 | 20.88 $\pm$ 2.88 | 9.9 $\pm$ 0.56 | 28.87 $\pm$ 8.89 | missing10.0 $\pm$ 0.0 | 57.33 $\pm$ 4.99 | 53.34 $\pm$ 8.65
$\Pi_{r}^{+}(\delta=5\times 10^{-3})$ | 9.73 $\pm$ 0.59 | 20.09 $\pm$ 1.61 | 22.51 $\pm$ 4.25 | 11.86 $\pm$ 1.59 | 11.3 $\pm$ 1.84 | 33.97 $\pm$ 7.44 | 36.48 $\pm$ 10.87 | 31.07 $\pm$ 6.62
$\Pi_{r}^{+}(\delta=5\times 10^{-2})$ | 12.18 $\pm$ 2.22 | 17.34 $\pm$ 4.88 | 22.14 $\pm$ 1.37 | 11.96 $\pm$ 2.52 | 10.0 $\pm$ 0.0 | 25.69 $\pm$ 7.31 | 33.21 $\pm$ 13.93 | 10.0 $\pm$ 0.0
$\Pi_{r}^{+}(\delta=2\times 10^{-1})$ | 17.96 $\pm$ 6.48 | 18.27 $\pm$ 1.32 | 27.27 $\pm$ 1.03 | 11.09 $\pm$ 1.57 | 10.0 $\pm$ 0.0 | 10.01 $\pm$ 0.01 | 41.3 $\pm$ 6.21 | 15.61 $\pm$ 8.46
$\Pi_{r}^{+}(\delta=5\times 10^{-1})$ | 16.42 $\pm$ 2.35 | 16.57 $\pm$ 3.2 | 23.25 $\pm$ 1.08 | 9.71 $\pm$ 0.27 | 14.51 $\pm$ 3.32 | missing10.0 $\pm$ 0.0 | 19.18 $\pm$ 3.44 | 60.35 $\pm$ 5.04
$\Pi_{lr}(\delta=5\times 10^{-3})$ | 24.06 $\pm$ 1.46 | 56.06 $\pm$ 1.81 | 32.75 $\pm$ 2.96 | 45.4 $\pm$ 5.98 | 28.89 $\pm$ 7.26 | 46.65 $\pm$ 1.43 | 45.33 $\pm$ 3.25 | 43.87 $\pm$ 11.0
$\Pi_{lr}(\delta=5\times 10^{-2})$ | 22.84 $\pm$ 4.02 | 25.7 $\pm$ 4.03 | 25.95 $\pm$ 5.18 | 9.41 $\pm$ 1.53 | 44.18 $\pm$ 7.48 | 43.24 $\pm$ 8.61 | 61.14 $\pm$ 4.62 | 57.16 $\pm$ 6.89
$\Pi_{lr}(\delta=2\times 10^{-1})$ | 16.96 $\pm$ 6.18 | 20.73 $\pm$ 2.51 | 27.63 $\pm$ 0.73 | 22.02 $\pm$ 7.64 | 27.98 $\pm$ 8.25 | 32.66 $\pm$ 1.65 | 51.07 $\pm$ 12.81 | 38.03 $\pm$ 5.0
$\Pi_{lr}(\delta=5\times 10^{-1})$ | 19.39 $\pm$ 4.05 | 15.85 $\pm$ 2.0 | 20.98 $\pm$ 4.08 | 10.78 $\pm$ 1.18 | 25.24 $\pm$ 2.47 | missing10.0 $\pm$ 0.0 | 54.67 $\pm$ 4.5 | 48.17 $\pm$ 11.51
$\Pi_{lr}^{+}(\delta=5\times 10^{-3})$ | 10.11 $\pm$ 0.03 | 25.11 $\pm$ 1.05 | 27.99 $\pm$ 2.51 | 14.02 $\pm$ 2.14 | 10.0 $\pm$ 0.0 | 23.3 $\pm$ 4.98 | 34.71 $\pm$ 3.65 | 21.67 $\pm$ 4.19
$\Pi_{lr}^{+}(\delta=5\times 10^{-2})$ | 10.31 $\pm$ 1.27 | 16.68 $\pm$ 3.68 | 23.33 $\pm$ 3.51 | 10.3 $\pm$ 0.41 | 10.0 $\pm$ 0.0 | 27.39 $\pm$ 9.94 | 22.2 $\pm$ 8.98 | 11.29 $\pm$ 1.82
$\Pi_{lr}^{+}(\delta=2\times 10^{-1})$ | 12.63 $\pm$ 1.71 | 18.8 $\pm$ 0.44 | 24.73 $\pm$ 2.48 | 10.25 $\pm$ 0.35 | 12.08 $\pm$ 2.93 | 11.68 $\pm$ 1.2 | 34.92 $\pm$ 16.93 | 22.29 $\pm$ 13.13
$\Pi_{lr}^{+}(\delta=5\times 10^{-1})$ | 22.23 $\pm$ 4.22 | 17.79 $\pm$ 3.21 | 21.73 $\pm$ 1.41 | 9.7 $\pm$ 0.23 | 29.02 $\pm$ 7.81 | 10.0 $\pm$ 0.0 | 11.8 $\pm$ 2.36 | 51.62 $\pm$ 10.66
$\Pi_{ERK}^{+}(\delta=5\times 10^{-3})$ | 15.49 $\pm$ 5.99 | 15.82 $\pm$ 3.36 | 22.13 $\pm$ 1.38 | 10.03 $\pm$ 0.7 | missing9.41 $\pm$ 0.83 | 10.0 $\pm$ 0.0 | 10.0 $\pm$ 0.0 | 10.0 $\pm$ 0.0
$\Pi_{ERK}^{+}(\delta=5\times 10^{-2})$ | 64.01 $\pm$ 8.4 | 61.59 $\pm$ 2.58 | 27.93 $\pm$ 3.56 | 48.92 $\pm$ 8.47 | 54.25 $\pm$ 4.71 | 41.33 $\pm$ 3.68 | 47.92 $\pm$ 10.8 | 71.43 $\pm$ 4.58
$\Pi_{ERK}^{+}(\delta=2\times 10^{-1})$ | 70.02 $\pm$ 3.23 | 52.71 $\pm$ 4.45 | 27.81 $\pm$ 2.11 | 46.76 $\pm$ 9.56 | 49.88 $\pm$ 4.37 | 26.18 $\pm$ 2.8 | 32.47 $\pm$ 2.22 | 71.08 $\pm$ 2.33
$\Pi_{ERK}^{+}(\delta=5\times 10^{-1})$ | 78.71 $\pm$ 3.46 | 41.61 $\pm$ 1.3 | 23.24 $\pm$ 2.41 | 47.92 $\pm$ 2.2 | 26.6 $\pm$ 6.46 | 20.67 $\pm$ 1.52 | 22.34 $\pm$ 3.55 | 72.27 $\pm$ 19.0
$\Pi_{F}^{+}(\delta=5\times 10^{-3})$ | 26.31 $\pm$ 3.41 | 24.28 $\pm$ 2.96 | 26.41 $\pm$ 2.33 | 10.93 $\pm$ 1.31 | 39.57 $\pm$ 5.77 | 10.0 $\pm$ 0.0 | missing10.0 $\pm$ 0.0 | 64.26 $\pm$ 7.74
$\Pi_{F}^{+}(\delta=5\times 10^{-3})$ ($\delta^{max}_{FC}$ = 0.25) | missing9.93 $\pm$ 0.3 | 17.61 $\pm$ 4.04 | 22.89 $\pm$ 2.13 | 9.84 $\pm$ 0.14 | 10.0 $\pm$ 0.0 | missing10.0 $\pm$ 0.0 | 10.01 $\pm$ 0.01 | missing9.24 $\pm$ 1.08
$\Pi_{F}^{+}(\delta=1\times 10^{-2})$ ($\delta^{max}_{FC}$ = 0.25) | 10.29 $\pm$ 0.42 | 16.23 $\pm$ 3.5 | missing19.89 $\pm$ 1.54 | missing8.79 $\pm$ 2.3 | 10.0 $\pm$ 0.0 | missing10.0 $\pm$ 0.0 | 10.0 $\pm$ 0.0 | 10.0 $\pm$ 0.0

##  V Conclusions

In this work, we introduce a novel Byzantine attack framework that has been
designed to be effective against different types of defence mechanisms
simultaneously. The proposed framework generates an attack by combining two
components: sparse but more aggressive and dense but more stealthy, forming a
stronger but imperceptible attack. Further, by utilizing a network pruning
framework, we identify the sensitive locations, those later targeted by the
sparse component of the Byzantine attack, to increase the strength of the
attack. Finally, through extensive simulations over different NN networks,
datasets and defence mechanisms, we have shown the effectiveness of our
approach. Overall, in this work, we have initiated the discussion of how the
existing form of the Byzantines can be enhanced by exploring the nature of the
network topology, and we believe our extensive analysis will draw further
attention and possibly lead to a new research direction in the robust
federated learning context. In the scope of this work, we have limited our
focus to extracting certain topology based side information (identifying
certain important/sensitive weights) to scale the base adversarial
perturbation accordingly, as a future research direction, we are planning to
design a framework where the direction of the perturbations is also determined
with respect to the network topology.

## References

  * [1] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to backdoor federated learning,” in _AISTATS_ , 2020.
  * [2] G. Baruch, M. Baruch, and Y. Goldberg, “A little is enough: Circumventing defenses for distributed learning,” in _NeurIPS_ , 2019.
  * [3] J. Bernstein, J. Zhao, K. Azizzadenesheli, and A. Anandkumar, “signsgd with majority vote is communication efficient and fault tolerant,” in _ICLR_ , 2019.
  * [4] A. N. Bhagoji, S. Chakraborty, P. Mittal, and S. Calo, “Analyzing federated learning through an adversarial lens,” in _ICML_ , 2019.
  * [5] B. Biggio, B. Nelson, and P. Laskov, “Poisoning attacks against support vector machines,” in _ICML_ , 2012.
  * [6] P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer, “Machine learning with adversaries: Byzantine tolerant gradient descent,” in _NIPS_ , 2017.
  * [7] X. Cao, M. Fang, J. Liu, and N. Z. Gong, “Fltrust: Byzantine-robust federated learning via trust bootstrapping,” _arXiv preprint arXiv:2012.13995_ , 2020.
  * [8] X. Chen, T. Chen, H. Sun, S. Z. Wu, and M. Hong, “Distributed training with heterogeneous data: Bridging median-and mean-based algorithms,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 21 616–21 626, 2020.
  * [9] P. de Jorge, A. Sanyal, H. Behl, P. Torr, G. Rogez, and P. K. Dokania, “Progressive skeletonization: Trimming more fat from a network at initialization,” in _International Conference on Learning Representations_ , 2021.
  * [10] E.-M. El-Mhamdi, R. Guerraoui, A. Guirguis, L. N. Hoang, and S. Rouault, “Genuinely distributed byzantine machine learning,” ser. PODC ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 355–364. [Online]. Available: https://doi.org/10.1145/3382734.3405695
  * [11] E.-M. El-Mhamdi, R. Guerraoui, and S. Rouault, “Distributed momentum for byzantine-resilient stochastic gradient descent,” in _ICLR_ , 2021.
  * [12] E. M. El Mhamdi, R. Guerraoui, and S. Rouault, “The hidden vulnerability of distributed learning in byzantium,” in _ICML_ , 2018.
  * [13] U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen, “Rigging the lottery: Making all tickets winners,” in _Proceedings of the 37th International Conference on Machine Learning_ , 2020.
  * [14] M. Fang, X. Cao, J. Jia, and N. Z. Gong, “Local model poisoning attacks to byzantine-robust federated learning,” in _USENIX Conference on Security Symposium_ , 2020.
  * [15] S. Farhadkhani, R. Guerraoui, N. Gupta, R. Pinot, and J. Stephan, “Byzantine machine learning made easy by resilient averaging of momentums,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 6246–6283.
  * [16] J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” in _International Conference on Learning Representations_ , 2019. [Online]. Available: https://openreview.net/forum?id=rJl-b3RcF7
  * [17] C. Fung, C. J. Yoon, and I. Beschastnikh, “The limitations of federated learning in sybil settings.” in _Usenix RAID_ , 2020.
  * [18] E. Gorbunov, S. Horváth, P. Richtárik, and G. Gidel, “Variance reduction is an antidote to byzantines: Better rates, weaker assumptions and communication compression as a cherry on the top,” 2022.
  * [19] R. Guerraoui, N. Gupta, R. Pinot, S. Rouault, and J. Stephan, “Differential privacy and byzantine resilience in sgd: Do they add up?” ser. PODC’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 391–401. [Online]. Available: https://doi.org/10.1145/3465084.3467919
  * [20] A. Gupta, T. Luo, M. V. Ngo, and S. K. Das, “Long-short history of gradients is all you need: Detecting malicious and unreliable clients in federated learning.” Berlin, Heidelberg: Springer-Verlag, 2022, p. 445–465. [Online]. Available: https://doi.org/10.1007/978-3-031-17143-7_22
  * [21] N. Gupta and N. H. Vaidya, “Fault-tolerance in distributed optimization: The case of redundancy,” in _Proceedings of the 39th Symposium on Principles of Distributed Computing_ , ser. PODC ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 365–374. [Online]. Available: https://doi.org/10.1145/3382734.3405748
  * [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016.
  * [23] R. Jin, Y. Huang, X. He, H. Dai, and T. Wu, “Stochastic-sign sgd for federated learning with theoretical guarantees,” _arXiv preprint arXiv:2002.10940_ , 2020.
  * [24] S. P. Karimireddy, L. He, and M. Jaggi, “Learning from history for byzantine robust optimization,” in _ICML_ , 2021.
  * [25] ——, “Byzantine-robust learning on heterogeneous datasets via bucketing,” in _ICLR_ , 2022.
  * [26] J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” _NIPS workshop on Private Multiparty Machine Learning_ , 2016.
  * [27] A. Krizhevsky, V. Nair, and G. Hinton, “Cifar-10 (canadian institute for advanced research).”
  * [28] Y. LeCun and C. Cortes, “MNIST handwritten digit database,” 2010. [Online]. Available: http://yann.lecun.com/exdb/mnist/
  * [29] N. Lee, T. Ajanthan, and P. Torr, “SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY,” in _International Conference on Learning Representations_ , 2019. [Online]. Available: https://openreview.net/forum?id=B1VZqjAcYX
  * [30] Y. Liu, C. Chen, L. Lyu, F. Wu, S. Wu, and G. Chen, “Byzantine-robust learning on heterogeneous data via gradient splitting,” in _Proceedings of the 40th International Conference on Machine Learning_ , ser. ICML’23. JMLR.org, 2023.
  * [31] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _AISTATS_ , 2017.
  * [32] T. Minka, “Estimating a dirichlet distribution,” 2000.
  * [33] L. Muñoz-González, K. T. Co, and E. C. Lupu, “Byzantine-robust federated machine learning through adaptive model averaging,” _arXiv preprint arXiv:1909.05125_ , 2019.
  * [34] J. Peng, W. Li, and Q. Ling, “Variance reduction-boosted byzantine robustness in decentralized stochastic optimization,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 4283–4287.
  * [35] J. Peng, Z. Wu, and Q. Ling, “Byzantine-robust variance-reduced federated learning over distributed non-i.i.d. data,” _CoRR_ , vol. abs/2009.08161, 2020. [Online]. Available: https://arxiv.org/abs/2009.08161
  * [36] K. Pillutla, S. M. Kakade, and Z. Harchaoui, “Robust aggregation for federated learning,” _IEEE Transactions on Signal Processing_ , 2022.
  * [37] M. Raynal, D. Pasquini, and C. Troncoso, “Can decentralized learning be more robust than federated learning?” _arXiv preprint arXiv:2303.03829_ , 2023.
  * [38] M. Raynal and C. Troncoso, “On the conflict of robustness and learning in collaborative machine learning,” _arXiv preprint arXiv:2402.13700_ , 2024.
  * [39] V. Shejwalkar and A. Houmansadr, “Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning,” in _NDSS_ , 2021.
  * [40] V. Shejwalkar, A. Houmansadr, P. Kairouz, and D. Ramage, “Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning,” in _IEEE Symposium on Security and Privacy_ , 2022.
  * [41] Z. Sun, P. Kairouz, A. T. Suresh, and H. B. McMahan, “Can you really backdoor federated learning?” _arXiv preprint arXiv:1911.07963_ , 2019.
  * [42] H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli, “Pruning neural networks without any data by iteratively conserving synaptic flow,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 6377–6389.
  * [43] K. Varma, Y. Zhou, N. Baracaldo, and A. Anwar, “Legato: A layerwise gradient aggregation algorithm for mitigating byzantine attacks in federated learning,” in _2021 IEEE 14th international conference on cloud computing (CLOUD)_. IEEE, 2021, pp. 272–277.
  * [44] C. P. Wan and Q. Chen, “Robust federated learning with attack-adaptive aggregation,” _arXiv preprint arXiv:2102.05257_ , 2021.
  * [45] Z. Wu, Q. Ling, T. Chen, and G. B. Giannakis, “Federated variance-reduced stochastic gradient descent with robustness to byzantine attacks,” _IEEE Transactions on Signal Processing_ , vol. 68, pp. 4583–4596, 2020.
  * [46] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” _ArXiv_ , 2017.
  * [47] C. Xie, K. Huang, P.-Y. Chen, and B. Li, “Dba: Distributed backdoor attacks against federated learning,” in _ICLR_ , 2020.
  * [48] C. Xie, O. Koyejo, and I. Gupta, “Fall of empires: Breaking byzantine-tolerant sgd by inner product manipulation,” in _Uncertainty in Artificial Intelligence_ , 2020.
  * [49] C. Xie, S. Koyejo, and I. Gupta, “Zeno: Distributed stochastic gradient descent with suspicion-based fault-tolerance,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 6893–6901.
  * [50] W. Xie, T. Pethick, A. Ramezani-Kebrya, and V. Cevher, “Mixed nash for robust federated learning,” _Transactions on Machine Learning Research_ , 2023.
  * [51] C. Yin and Q. Zeng, “Defending against data poisoning attack in federated learning with non-iid data,” _IEEE Transactions on Computational Social Systems_ , pp. 1–13, 2023.
  * [52] D. Yin, Y. Chen, R. Kannan, and P. Bartlett, “Byzantine-robust distributed learning: Towards optimal statistical rates,” in _ICML_ , 2018.
  * [53] H. Zhu and Q. Ling, “Byzantine-robust aggregation with gradient difference compression and stochastic variance reduction for federated learning,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 4278–4282.
  * [54] K. Özfatura, E. Özfatura, A. Küpçü, and D. Gunduz, “Byzantines can also learn from history: Fall of centered clipping in federated learning,” _IEEE Transactions on Information Forensics and Security_ , vol. 19, pp. 2010–2022, 2024.

###  -A Additional Numerical Results

Since smaller NN architectures are more vulnerable to Byzantine attacks, we
mainly focus on moderately large NN architectures, i.e., ResNet-20, and the
classification task on CIFAR-10. For the sake of completeness, in this
subsection, we further conduct the same experiments with different NN networks
and datasets, to be more precise, we have analyzed the byzantine performance
and compare the existing attack and defence frameworks for the classification
task on the FMNIST dataset with a 2-layer CNN architecture and classification
task on MNIST dataset with 2-layer MLP architecture. For our hybrid sparse
attack, we consider the configuration of $\Pi_{F}^{+}(\delta=5\times 10^{-3},$
$\delta^{max}_{FC}=0.25$ which often provides the best results in the case of
CIFAR-10 dataset except the MNIST where we do not induce any limitation since
MLP network only consist of fully connected layers as such we utilize
$\Pi_{F}^{+}(\delta=1\times 10^{-2},$ $\delta^{max}$ for MNIST classification
task. The experimental results and the convergence behaviour under different
robust aggregators and Byzantine attacks are illustrated in Fig. 3 and Fig. 4,
respectively.

Figure 3: Test accuracy results of training 2-Layer CNN architecture with
FMNIST dataset distributed IID and non-IID over $k=25$ client $k_{m}=5$ of
them being malicious. The training and testing are performed over 100 epochs
and repeated for 9 different aggregation mechanisms and under 8 different
Byzantine attack strategies. The reported results are obtained by averaging 3
number of independent trials. Figure 4: Test accuracy results of training
2-Layer MLP architecture with MNIST dataset distributed IID and non-IID over
$k=25$ client $k_{m}=5$ of them being malicious. The training and testing are
performed over 100 epochs and repeated for 9 different aggregation mechanisms
and under 8 different Byzantine attack strategies. The reported results are
obtained by averaging 3 number of independent trials.

###  -B Network Prunning

In this section, we further provide the details regarding the implementation
of the FORCE network prunning framework.

####  -B1 Sparsity Scheduler

Similar to [9], we consider exponential decay sparsity scheduling that is over
$T$ prunning steps the number of non-sparse positions decreased from
$\kappa_{0}=d$ to $\kappa_{T}=\kappa$ gradually in the following manner:

$\kappa_{t}=\lfloor e^{(\frac{t}{T}\log\kappa+(1-\frac{t}{T})\log d)}\rfloor.$

####  -B2 Prunning Algorithm

The overall iterative prunning algorithm, executed over $T$ steps with respect
to the sparsity scheduling in eqn (-B1), is illustrated in Algorithm 4, where
at each iteration $t$, each byzantine node first computes the saliency
$\mathbf{s}_{i}(\bm{\theta},\mathbf{c}_{t-1})$, with respect to initial
network weights $\bm{\theta}$ and the precious binary sparsity mask
$\mathbf{c}_{t-1}$ according to eqn. (-B1), then seek a consensus on the
saliency measure. Then based on the saliency score, the non-sparse locations
for the given scheduled sparsity constraint $\kappa_{t}$, simply through a
$Top_{k}$ operation.

Algorithm 4 Iterative Network Prunning

1:Inputs: Target sparsity $\kappa$, Initialized weights
$\bm{\theta}\in\mathbb{R}^{d}$, Time steps $T$, Local dataset
$\left\\{D_{i}\right\\}_{i\in\mathcal{K}_{m}}$

2:Outputs: $\mathbf{c}$

3:Initialize $\mathbf{c}_{0}\leftarrow{}\bm{1}_{d}$

4:Scheduled Sparsity $\left\\{\kappa_{1},\ldots,\kappa_{T}\right\\}$ acc. eqn.
(-B1)

5:for $t=1,\ldots,T$ do,

6: for $i\in\mathcal{K}_{m}$ do

7: Sample mini-batch: $\zeta\in D_{i}$

8: Compute Saliency:
$\mathbf{s}_{i}(\bm{\theta},\mathbf{c}_{t-1})=|\bm{\theta}\odot\nabla
F(\mathbf{c}_{t-1}\odot\bm{\theta};\zeta)|$

9: Consensus Saliency:
$\mathbf{s}(\bm{\theta},\mathbf{c}_{t-1})\leftarrow\frac{1}{k_{m}}\sum_{i\in\mathcal{K}_{m}}\mathbf{s}_{i}(\bm{\theta},\mathbf{c}_{t-1})$

10: Indices with maxim saliency measure
$\mathcal{I}=Top_{\kappa_{t}}(\mathbf{s})$

11: Update binary mask:
$\mathbf{c}_{t}\leftarrow\mathbf{c}:\mathbf{c}[i]=1,~{}\text{if}~{}i\in\mathcal{I}$