# Towards risk-informed PBSHM: Populations as hierarchical systems

A.J. Hughes, P. Gardner & K. Worden

Dynamics Research Group, Department of Mechanical Engineering, University of
Sheffield,
Mappin Street, Sheffield S1 3JD, UK

## Abstract

The prospect of informed and optimal decision-making regarding the operation
and maintenance (O&M) of structures provides impetus to the development of
structural health monitoring (SHM) systems. A probabilistic risk-based
framework for decision-making has already been proposed. The framework
comprises four key sub-models: the utility model, the failure-modes model, the
statistical classifier, and the transition model. The cost model consists of
utility functions that specify the costs of actions and structural failures.
The failure-modes model defines the failure modes of a structure as
combinations of component and substructure failures via fault trees. The
statistical classifier and transition model are models that predict the
current and future health-states of a structure, respectively. Within the
data-driven statistical pattern recognition (SPR) approach to SHM, these
predictive models are determined using machine learning techniques. However,
in order to learn these models, measured data from the structure of interest
are required. Unfortunately, these data are seldom available across the range
of environmental and operational conditions necessary to ensure good
generalisation of the model.

Recently, technologies have been developed that overcome this challenge, by
extending SHM to populations of structures, such that valuable knowledge may
be transferred between instances of structures that are sufficiently similar.
This new approach is termed population-based structural heath monitoring
(PBSHM).

The current paper presents a formal representation of populations of
structures, such that risk-based decision processes may be specified within
them. The population-based representation is an extension to the hierarchical
representation of a structure used within the probabilistic risk-based
decision framework to define fault trees. The result is a series, consisting
of systems of systems ranging from the individual component level up to an
inventory of heterogeneous populations. The current paper considers an
inventory of wind farms as a motivating example and highlights the inferences
and decisions that can be made within the hierarchical representation.

Keywords: population-based structural health monitoring; risk; decision-
making; value of information

##  1 Introduction

Structural health monitoring (SHM) is a technology that aims to detect damage
within mechanical, civil and aerospace structures and infrastructure [1]. By
inferring information about the health of a structure from discriminative
features extracted from data acquired throughout a monitoring campaign, these
systems can facilitate informed predictions relating to one or more of the
following problems regarding the health of a structure, as summarised in
Rytter’s hierarchy [2]:

  * •

The presence of damage in a structure (detection).

  * •

The location of damage within a structure (localisation).

  * •

The type of damage present in a structure (classification).

  * •

The extent of damage in a structure (severity).

  * •

The remaining safe/useful life of a structure (prognosis).

By informing predictions with data from a monitoring system, one can also
inform decision-making regarding the operation and maintenance of structures
and this can yield benefits such as improved safety, reduced operation costs
and operational lifetime extension.

Recent works have explicitly framed structural health monitoring in the
context of decision-making [3, 4, 5]. The approach to decision-making for SHM
presented in [5], adopts a probabilistic risk-based perspective. In this
approach, probability distributions over structural health states are inferred
from data via statistical classifiers. The distributions are then forecast via
a transition model and mapped to probabilities of failure for specific failure
modes of interest via Bayesian network representations of fault trees. Optimal
decisions are found by maximising the expected utility when considering both
the risk of structural failure and the cost of maintenance actions. Several
submodels have been identified as elements that are required to sufficiently
define SHM decision processes; these submodels include statistical classifiers
for inferring health-states and health-state transition models. In order to
achieve robust decision-making, these submodels require labelled data for
learning and/or validation.

A critical challenge associated with the development of SHM systems is the
scarcity of the data necessary for the learning and validation of models.
Prior to the implementation of a monitoring system, there is often a lack of
comprehensive labelled data across the health-states of interest for a given
structure as obtaining data corresponding to damage states tends to be
prohibitively expensive or otherwise infeasible. One approach to circumvent
this issue in the development of classification models is to utilise online
active learning algorithms to preferentially obtain labelled data via
structural inspections after a monitoring system is installed [6, 7, 8]. In
[9], a methodology for determining a transition model using qualitative data
from historical inspections is demonstrated.

Population-based structural health monitoring (PBSHM), provides a holistic
framework for overcoming data scarcity in the development of predictive models
for SHM [10, 11, 12, 13]. The core principal of PBSHM is that predictions
about individual structures can be improved with the use of information
transferred from other similar structures.

The current paper aims to further the core principal of PBSHM, such that
_decisions_ about the operation of both individual structures and populations
of structures can be improved via the transfer of information. This is
achieved by extending the hierarchical representation of structures, used to
develop fault trees in the risk-based approach to decision-making for
traditional SHM, to hierarchical representations of populations of structures.
Throughout the current paper, an inventory of offshore wind farms are
referenced as a motivating example.

The layout of the current paper is as follows. Background theory is provided
for both PBSHM and risk-based SHM in Sections 2 and 3, respectively.
Subsequently, a hierarchical representation of individual structures is
presented in Section 4. This representation is extended to populations of
structures in Section 5. Inferences and decisions within the population
hierarchy are defined and discussed in Section 6. Finally, conclusions are
provided in Section 7.

##  2 Population-based SHM

The foundations of PBSHM have been presented in a series of journal papers,
each detailing the fundamental concepts of the approach; homogeneous
populations [10], heterogeneous populations [11], mapping and transfer [12],
and the geometric spaces in which structures exist [13]. By adopting a
population-based approach to SHM, such that knowledge and information can be
transferred between similar structures, there is the potential for improved
diagnostic and prognostic capabilities [14].

In the most general sense, a population can be considered to simply be a set
of structures. Given the broad nature of this definition, in order to achieve
useful transfer of knowledge and information between structures, it is
discerning to consider specific classes of populations based upon the
similarity of the constitutive structures. Thus, the notions of homogeneous
and heterogeneous populations are introduced in [10, 11, 12].

###  2.1 Homogeneous and heterogeneous Populations

Within a population, structures may share common characteristics such as
geometries, topologies, materials, and boundary conditions. Consider a
population of wind turbines in an offshore wind farm and suppose these
turbines are of the same model; developed to the same ISO standards and
possessing common components, materials, aerodynamic design and so on.
Qualitatively, these structures can be regarded as nominally identical.
Populations comprised exclusively of nominally-identical structures are termed
homogeneous populations. Specific instances of structures in a homogeneous
population can be considered to be perturbations of a population form [10].
For further discussions on population forms, the reader is directed to [10].
Other examples of homogeneous populations include a fleet of Airbus A380s, an
array of small modular nuclear reactors, and the Global Positioning System
(GPS) satellite constellation.

Variation between structures in homogeneous populations may arise because of
factors such as environmental conditions and manufacturing defects. Returning
to the example of an offshore wind farm, one could imagine that two turbines
at differing locations in the farm may experience different geotechnical
conditions – perhaps as a result of varying geological composition in the
seabed. Variability in such conditions could affect the boundary conditions of
the monopile turbine towers and therefore modify the behaviours and data
exhibited by these otherwise nominally-identical structures.

In essence, heterogeneous populations form the complement of the set of
homogeneous populations [14]; that is, heterogeneous populations are not
exclusively comprised of structures that are nominally identical.
Heterogeneous populations represent more general sets of structures and allow
for differing designs, large variability in boundary conditions, and even
multiple types of structure. While there may be stark differences between
individual structures in a heterogeneous population, there may nonetheless be
similarities that can be exploited to achieve useful knowledge and information
transfer.

Consider again the offshore wind farm example and suppose that the population
is comprised of wind turbines each with three blades. Suppose also that the
operating company manage an additional wind farm in a distinct location,
comprised of four-blade turbines. Useful inferences could be achieved by
considering these wind farms as two homogeneous populations, however, further
insights could also be gained by considering them as a single heterogeneous
population. For example, similarities may be present in the tower design
between both types of wind turbine; hence, by considering a larger population
from which to make observations, improved predictive models can be developed
for this specific substructure. Other types of heterogeneous populations that
may be useful to consider include inventories of aircraft comprised of a
variety of models, and multiple suspension bridges with differing designs
(e.g. single-span, multi-span).

Thus far, similarities between structures have been described somewhat
qualitatively, however, to better indicate where information transfer may
work, it is useful to quantify this similarity.

###  2.2 Similarity between structures

Graph theory provides a rigorous and rich framework for representing and
comparing discrete structured objects and has proved to be an invaluable
modelling tool in fields such as chemistry and proteomics.

In [11], the notion of the irreducible element (IE) model for structures is
introduced as a representation of structures with relatively low-dimension
when compared to alternatives such as finite element, or CAD models. The IE
representation involves abstracting a structure into discrete elements having
geometrical and material information (e.g. beams, plates, shells) and
relationships (e.g. joints) so as to sufficiently capture the nature of a
structure. Here, the ‘nature’ one wishes to capture pertains to health
monitoring problems associated with a structure.

Once an IE representation of a structure has been obtained, the information
can be encoded into an attributed graph (AG). Whereas the purpose of the IE
model is to present key characteristics of a structure in a human-readable
format, the purpose of the AG is to embed a structure space, so as to
facilitate the efficient pair-wise comparison of structures. With structures
embedded into a metric space via AGs, one can utilise graph-matching
algorithms to find common subgraphs between sets of structures. These
subgraphs indicate substructures that are common within sets of structures and
can be used to inform where transfer may be applicable. Furthermore, measures
of closeness within the space of AGs (or common subgraphs) can be used to
quantify similarity; in [11] the Jaccard index is used and in [15] a variety
of graph kernels are demonstrated.

In summary, structures can be mapped into a graphical domain to facilitate
comparison, identify common substructures and quantify similarity. By
conducting this similarity assessment for structures within a population, one
can determine where it is likely that information and knowledge can be
successfully transferred between individual structures.

###  2.3 Mapping and Transfer

As mentioned previously, the primary benefit in taking a population-based
approach to SHM is gaining the ability to transfer knowledge and information
between sufficiently-similar individual structures; thereby overcoming issues
associated with data scarcity.

The sharing of knowledge and information between individual structures can be
achieved via a number of methodologies. One manner in which this can be
achieved is by having a statistical representation of the aforementioned
population form, as demonstrated in [10]. Another approach, presented in [16],
shares datasets in joint hierarchical statistical models of a population.
Methodologies founded upon transfer learning have also been successfully
demonstrated [12]. The principal of transfer learning is closely aligned with
the goals of PBSHM; specifically, a branch of transfer learning known as
domain adaptation. In domain adaptation, datasets are adapted in a manner that
allows a model constructed for a source domain to generalise to a target
domain.

For knowledge/information transfer to be successful, it is imperative that
these source and target domains are comparable. This constraint can be adhered
to by employing the similarity assessment outlined in the previous section.

Thus far, PBSHM has been considered with respect to predictions and
inferences. Before incorporating decisions into the PBSHM framework,
background on the risk-based approach to decision-making for traditional SHM
is provided.

##  3 Probabilistic risk-based SHM

The probabilistic risk-based approach to SHM is founded on the notion that
monitoring systems should be designed and developed with consideration for the
specific decision-support applications motivating their implementation.

In the SHM paradigm detailed in [1], monitoring campaigns begin with a process
termed operational evaluation. This stage in the development of an SHM system
is concerned with specifying the context for an SHM system, dealing with
aspects such as the safety/economic justification and the environmental and
operational conditions. In [5], it is proposed that the decision-making
processes associated with an SHM campaign should be considered from the outset
of a monitoring campaign as part of the operational evaluation.

To begin defining the decision processes that one may wish to inform with a
monitoring system, one must identify a set of failure modes or conditions for
a structure that one may wish to prevent, in addition to a set of actions that
can be executed to aid in the mitigation of failures. Furthermore, as part of
the economic justification of a monitoring system, costs or utilities must be
assigned to these failures and actions. The prediction of specific failure
events and the informed selection of optimal mitigating actions should provide
the basis for the development of monitoring systems guiding, choices for
aspects of the monitoring system such as: sensors and their placement on the
structure; data processing; and the discriminative features and models used to
classify structural health states.

Once a monitoring system developed with respect to decision-making is
implemented, optimal strategies can be found by maximising expected utility
with consideration for the risk of failure and the cost of mitigating actions.
This can be achieved by representing the decision processes as a probabilistic
graphical model (PGM). The following two subsections provide background on
PGMs and the modelling of SHM decision processes as PGMs respectively.

###  3.1 Probabilistic graphical models

Probabilistic graphical models (PGMs) are graphical representations of
factorisations of joint probability distributions, and are a powerful tool for
reasoning and decision-making under uncertainty. For this reason, they are apt
for representing and solving decision problems in the context of SHM, where
there is uncertainty in the health states of structures. While there exist
multiple forms of probabilistic graphical model, the key types utilised for
representing SHM decision processes are Bayesian networks (BNs) and influence
diagrams (IDs) [17].

Bayesian networks are directed acyclic graphs (DAGs) comprised of nodes and
edges. Nodes represent random variables, and edges connecting nodes represent
conditional dependencies between variables. In the case where the random
variables in a BN are discrete, the model is defined by a set of conditional
probability tables (CPTs). For continuous random variables, the model is
defined by a set of conditional probability density functions (CPDFs).

$X$$Y$$Z$ Figure 1: An example Bayesian network.

Figure 1 shows a simple Bayesian network comprised of three random variables
$X$, $Y$ and $Z$. $Y$ is conditionally dependent on $X$ and is said to be a
child of $X$, while $X$ is said to be a parent of $Y$. $Z$ is conditionally
dependent on $Y$ and can be said to be a child of $Y$ and a descendant of $X$,
while $X$ is said to be an ancestor of $Z$. The factorisation described by the
Bayesian network shown in Figure 1 is given by $P(X,Y,Z)=P(X)\cdot P(Y|X)\cdot
P(Z|Y)$. Given observations on a subset of nodes in a BN, inference algorithms
can be applied to compute posterior distributions over the remaining
unobserved variables. Observations of random variables are denoted in a BN via
grey shading of the corresponding nodes, as is demonstrated for $X$ in Figure
1.

$W_{c}$$W_{f}$$D$$U$ Figure 2: An example influence diagram representing the
decision of whether to go outside or stay in under uncertainty in the future
weather condition given an observed forecast.

Bayesian networks may be adapted into influence diagrams to model decision
problems. This augmentation involves the introduction of two additional types
of node, as shown in Figure 2: decision nodes, denoted as squares, and utility
nodes, denoted as rhombi. For influence diagrams, edges connecting random
variables to utility nodes denote that the utility function is dependent on
the states of the random variables. Similarly, edges connecting decisions
nodes to utility nodes denote that the utility function is dependent on the
decided actions. Edges from decision nodes to random variable nodes indicate
that the random variables are conditionally dependent on the decided actions.
Edges from random variable or decision nodes to other decision nodes do not
imply a functional dependence but rather order, i.e. that the
observations/decisions must be made prior to the next decision being made.

To gain further understanding of IDs, one can consider Figure 2. Figure 2
shows the ID for a simple binary decision; stay home and watch TV or go out
for a walk, i.e. $\text{dom}(D)=\\{\text{TV},\text{ walk}\\}$. Here, the agent
tasked with making the decision has access to a weather forecast $W_{f}$ which
is conditionally dependent on the future weather condition $W_{c}$. The
weather forecast and future condition share the same possible states
$\text{dom}(W_{f})=\text{dom}(W_{c})=\\{\text{bad},\text{ good}\\}$. The
utility achieved $U$, is then dependent on both the future weather condition
and the decided action. For example, one might expect high utility gain if the
agent decides to go for a walk and the weather condition is good.

In general, a policy $\delta$ is a mapping from all possible observations to
possible actions. The problem of inference in influence diagrams is to
determine an optimal strategy
$\bm{\Delta}^{\ast}=\\{\delta^{\ast}_{1},\ldots,\delta^{\ast}_{n}\\}$ given a
set of observations on random variables, where $\delta^{\ast}_{i}$ is the
$i^{th}$ decision to be made in a strategy $\bm{\Delta}^{\ast}$ that yields
the maximum expected utility (MEU). For further details on the computation of
the MEU for influence diagrams, the reader is directed to [18]. Defined as a
product of probability and utility, the expected utility can be considered as
a quantity corresponding to risk.

###  3.2 Decision framework

A probabilistic graphical model for a general SHM decision problem across a
single time-slice is shown in Figure 3. Here, a maintenance decision $d$ is
shown for a simple fictitious structure $\bm{S}$, comprised of two
substructures $\bm{s}_{1}$ and $\bm{s}_{2}$, each of which are comprised of
two components; $c_{1,2}$ and $c_{3,4}$, respectively.

$U_{F_{t}}$$F_{\bm{S}}$$h\bm{s}_{2}$$h\bm{s}_{1}$$hc_{3}$$hc_{4}$$hc_{1}$$hc_{2}$$U_{F_{t+1}}$$F^{\prime}_{t+1}$$\bm{H}_{t}$$\bm{H}_{t+1}$$\bm{\nu}_{t}$$d_{t}$$U_{d_{t}}$
Figure 3: An influence diagram representing a partially-observable Markov
decision process over one time-slice for determining the utility-optimal
maintenance strategy for a simple structure comprised of four components. The
fault-tree failure-mode model for time $t+1$ has been represented as the node
$F^{\prime}_{t+1}$ for compactness.

The overall decision process model shown in Figure 3 is based upon a
combination of three sub-models; a statistical classifier, a failure-mode
model, and a transition model.

The failure condition of the structure $F_{\bm{S}}$ is represented as a random
variable within the PGM. This failure condition $F_{\bm{S}}$ can be expressed
as a failure mode of the global structure that can be specified by a fault
tree; a combination of local failures at a component, joint and substructure
level related by Boolean logic gates. In order to fit into the PGM framework,
fault trees must be mapped into Bayesian networks. Fortunately, there is a
well-established mapping for this presented in [19, 20]. Essentially,
components, joints and substructures receive random variables in the PGM
corresponding to their respective local health states. The conditional
probability tables defining the relationship between these random variables
correspond to the Boolean truth tables for each of the logic gates in the
fault tree defining the failure mode $F_{\bm{S}}$.

Figure 3 considers a failure mode dependent on two substructures
$\bm{s}_{1-2}$, which in turn are each dependent on two components $c_{1,2}$
and $c_{3,4}$. For the decision process shown in Figure 3, component health
states are denoted by $hc_{1-4}$, and substructure health states are denoted
by $h\bm{s}_{1,2}$. An advantage of considering specific failure modes and
their representations as fault trees is that doing so yields the health states
that must be targeted by the monitoring system; the local health states of the
components can be summarised in a global health-state vector. For the example
shown in Figure 3, this health-state vector is given by
$\bm{H}=\\{hc_{1},hc_{2},hc_{3},hc_{4}\\}$. In essence, the purpose of this
failure model is to map from a distribution over the global health states to a
probability of structural failure. Finally, the failure states associated with
the variable $F_{\bm{S}}$ are given utilities via the function represented by
the node $U_{F}$. As it is necessary to consider the future risk of failure in
the decision process, this failure-mode model and utility function are
repeated for each time-step.

As previously mentioned, a random variable denoted $\bm{H}_{t}$ is used to
represent the latent global health state of the structure at time $t$. Within
the decision process, the function of the statistical classifier is to provide
a posterior probability distribution over the latent health state
$\bm{H}_{t}$, inferred via observations on a set of discriminative features
$\bm{\nu}_{t}$. This probability distribution over health-states may be
obtained via a generative model $P(\bm{\nu}|\bm{H})$ as shown in Figure 3, or
obtained more directly via a discriminative classifier which yields
$P(\bm{H}|\bm{\nu})$. Here, the use of a probabilistic classifier is vital to
ensure decisions made are robust to uncertainty in the health state of the
structure.

Finally, a transition model is used to forecast the future health states,
given the current health state and a decided action, i.e.
$P(\bm{H}_{t+1}|\bm{H}_{t},d_{t})$. The transition model considers the
degradation of the structure under the various operational and environmental
conditions a structure may experience, while accounting for uncertainties in
each.

By employing decision-process models such as the one presented here, one can
obtain optimal strategies regarding the operation and maintenance of
individual structures by maximising the expected utility.

##  4 Structures as hierarchies

A key assumption implicit in the development of the fault-tree failure models
within the risk-based SHM decision framework, is that structures can be
represented as a hierarchy, or, in other terms, as a system of systems of
systems. As it is outside the scope of the current paper, a comprehensive and
consistent notation for referencing specific elements of a structure is not
established here. Rather, the constituent levels and elements within the
hierarchical representation are presented, in addition to the process by which
one arrives at them.

Consider a structure of interest $S$. To obtain a hierarchical representation
for $S$, one must first decompose $S$ into a discrete number of constituent
elements, which are referred to as substructures. Substructures are considered
to be entities which may, in principle, be assembled remotely or available for
independent testing prior to incorporation into the full-scale structure.
Within the hierarchical representation, some substructures may be further
decomposed up until the stage at which it would no longer be meaningful or
useful to do so. Substructures at this stage are referred to as components. As
such, components are considered to be substructures which cannot (or need not)
be decomposed further; these are the smallest element of a structure one might
reasonably monitor. A notable sub-class of component is the joint. Joints are
considered to be the physical mechanisms by which substructures are joined
together.

A diagram illustrating the hierarchical representation of a structure is shown
in Figure 4. The levels in the hierarchy that specifies the system of systems
of systems shown are denoted as $\mathcal{S}^{1}$, $\mathcal{S}^{2}$, and
$\mathcal{S}^{3}$ – corresponding to the component, substructure and
substructure levels, respectively. Within each level of the hierarchy,
elements can be listed.

$\mathcal{S}^{3}:$ Structure, $S$$\mathcal{S}^{2}$: Substructure,
$s_{1}$$s_{2}$$s_{3}$$\cdots$ $\mathcal{S}^{1}$: Component, $c_{1}$
$\cdots$$c_{2}$$c_{1}$$\cdots$$c_{2}$$c_{1}$$\cdots$$\cdots$ Figure 4: A
structure as systems of systems.

Returning to the example of a wind farm, it would be perfectly reasonable to
consider a single turbine as an individual structure, representing the
$\mathcal{S}^{3}$ level in a hierarchy. In the $\mathcal{S}^{2}$ level one may
consider substructures such as the drive train, blades, or tower. Finally, in
the $\mathcal{S}^{1}$ level one may have components such as the gearbox or
bearings comprising the drive train, or the web and shells comprising the
blades.

The hierarchical representation of structures facilitates the specification of
the decision process that motivate the development and implementation of SHM
technologies. This facilitation is achieved by decomposing structures into
constituent substructures and components which can then be used to define
failure modes of the structure. Given a finite set of failure modes of
interest, one can then specify critical components, and therefore health
states, to be targeted by a monitoring system.

##  5 Populations of structures as hierarchies

A natural method for incorporating decision-making into PBSHM, is to extend
the hierarchical representation of structures to hierarchical representations
of populations. The number of levels required in a hierarchy is of course
dependent on context. However, it is deemed that an additional three levels
provide sufficient generality for most PBSHM applications, and indeed the
discussions in the current paper.

The additional levels necessary to extend the hierarchical representation to
populations of structures can be summarised as follows:

  * •

$\mathcal{S}^{4}$ – Type/Model Inventory: This level of the hierarchy
corresponds to the lowest population level and represents an organisational
grouping in which all individual structures in the population are of the same
type/model and can be considered to be nominally identical. Thus, populations
at this level in the hierarchy are homogeneous.

  * •

$\mathcal{S}^{5}$ – Group Inventory: This next population level corresponds to
a set of $\mathcal{S}^{4}$ inventories for which it is necessary or convenient
to consider as a group for operational reasons such as asset management. As a
group inventory may be formed of disparate type/model inventories, in general,
group inventories are heterogeneous populations.

  * •

$\mathcal{S}^{6}$ – Inventory: This level of the hierarchy corresponds to the
total set of structural assets operated or owned by an organisation or
company. Again, this level will generally represent a heterogeneous
population.

Figure 5 depicts the continuation of the hierarchical representation from
$\mathcal{S}^{3}$ to $\mathcal{S}^{6}$. In Figure 5, an inventory $I$ is
considered as a system of systems of systems of systems. Once again, a list
can be formed of the constituent elements for each level in the hierarchy.

$\mathcal{S}^{6}$: Inventory, $I$ (Het. Pop.)$\mathcal{S}^{5}$: Group
Inventory, $G_{1}$ (Het. Pop.)$G_{2}$$\cdots$$T_{2}$ $\mathcal{S}^{4}$: Type
Inventory, $T_{1}$ (Hom. Pop.) $\cdots$$T_{1}$$\cdots$$\cdots$$S_{1,2}$
$\mathcal{S}^{3}$: Structure, $S_{1,1}$
$\cdots$$S_{2,2}$$S_{2,1}$$\cdots$$\cdots$$S_{1,1}$$\cdots$$\cdots$$\cdots$$\vdots$$\vdots$$\vdots$$\vdots$$\vdots$
Figure 5: An inventory as a system of systems of systems of systems.

To further elucidate this extension of the hierarchy, once again, consider the
example of an organisation operating offshore wind farms. As previously
indicated, a wind farm comprised exclusively of turbines of a single type or
model can form a homogeneous population; this corresponds to $\mathcal{S}^{4}$
in the hierarchy. In the case that the organisation is responsible for
multiple wind farms, or a single farm with a mixture of turbine types, one may
wish to organise these type/model inventories into group inventories. For
example, these group inventories may be formed from type inventories according
to the geographical jurisdiction of sub-divisions within the organisation, or
even formed from a collection of type inventories that are overseen by a
single maintenance crew. Should these populations each be comprised of a
different model of wind turbine, the group inventories formed would be
heterogeneous populations and correspond to $\mathcal{S}^{5}$ in the
hierarchy. Alternatively, if all the wind farms consist of a single type of
turbine, $\mathcal{S}^{4}$ and $\mathcal{S}^{5}$ can be merged and the group
inventories are instead homogeneous populations. Finally, the group
inventories owned by the wind farm organisation can be aggregated as an
inventory in the $\mathcal{S}^{6}$ level of the hierarchy. This level would
represent the organisation’s total structural assets and could amount to, for
example, multiple wind farms spread across the globe, maritime vessels, and
aircraft that may be used for inspection, maintenance or other operational
activities.

As is the case for traditional SHM, the hierarchical representation of
structures and populations of structures can help facilitate decision-making
for PBSHM in several ways. These decision processes are discussed further in
the following section.

##  6 Risk-informed PBSHM

Numerous decisions must be made throughout the life cycle of a PBSHM system.
Most obvious are the operation and maintenance decisions an organisation may
have to make, following the installation of a monitoring system, such as
inspections and repairs. Equally important, however, are the decisions that
must be made prior to implementation such as those made in the operational
evaluation stage of PBSHM.

###  6.1 Operational evaluation

One significant way in which adopting a hierarchical risk-based approach to
PBSHM facilitates decision-making occurs very early on, in the operational
evaluation stage. By considering specific failure modes and constructing fault
trees for individual structures, one can decide the key elements of a
structure which should be modelled in IEs and AGs. In other words, the
specification of failure modes as combinations of component and substructure
failures can be used to inform the granularity at which IEs and AGs are
constructed. A further benefit of the population-based approach is that, as
structures are considered nominally identical, large proportions of the fault
trees may be mapped across a homogeneous population, with the exception of
perhaps environment-specific failure modes.

The extension of the hierarchy to represent populations of structures via the
inclusion of levels $\mathcal{S}^{4}$ to $\mathcal{S}^{6}$ prompts one to
consider how failures may be defined at the population level. One possible way
to approach the failure of a population would be to consider the critical
missions for the operating organisation. Depending on the nature of the
organisation – whether they are non-commercial or commercial – these missions
may be related to performance measures such as availability and/or
profitability. Consider the wind farm example. Suppose that the operating
organisation are required to supply energy from the wind farm to an electrical
grid while maintaining a total population availability of 99%. This population
can then be considered to have failed if the population structural
availability falls below 99%. This population failure may be specified then by
extending the fault tree; defining the population failures as a combination of
individual failures. In addition, the organisation may wish to specify a
failure condition based upon profitability, perhaps based upon a performance
criterion related to a moving-average of the total power output. Again, this
failure could be represented as a combination of individual structure failures
and environmental conditions. This distinct failure mode is likely to be
highly correlated with the availability failure mode; fortunately, the
probabilistic graphical models employed in the risk-based approach can account
for these ‘common-cause’ failures. This approach to defining population
failures can be applied at any of the population levels within the
hierarchical representation by considering combinations of failures in the
levels below.

Defining failures at the population level within the hierarchy allows one to
assign costs during the operational evaluation stage. Following on from this,
population-scale actions can be also be defined.

###  6.2 Inferences and decisions

A fundamental process of decision-making for PBSHM is reasoning under
uncertainty. This is typically achieved via inferences. Within the
hierarchical framework for PBSHM, different types of inferences can be
defined:

  * •

I-inference: This type of inference corresponds to those usually made in
traditional SHM, and occur within the individual structure levels
$\mathcal{S}^{3}$ to $\mathcal{S}^{1}$. An example of an I-inference is the
process of determining a probability distribution over the health states of an
individual structure using data acquired from that structure.

  * •

L-inference: This type of inference occurs between levels in the hierarchical
representation of structures. These may also be types of I-inference, for
example determining the probability of failure for a (sub)structure given
local component health states. Other L-inferences may include those relating
to the validation and verification of predictive models (V&V). For example,
one may be able to validate a predictive model for a structure at the
$\mathcal{S}^{3}$ level with data measured from substructures or components at
the $\mathcal{S}^{2}$ and $\mathcal{S}^{1}$ levels, respectively.

  * •

P-inference: This type of inference occurs across populations. If the
inference is across a type inventory in $\mathcal{S}^{4}$, i.e. a homogeneous
population, they can be denoted as HomP-inferences. These inferences across
populations may utilise technologies such as forms [10]. An example of a HomP-
inference is inferring the health state of a member in a population using data
aggregated across all members in the population. On the other hand, if a
P-inference is between populations containing different types of structure,
such as within a group inventory in $\mathcal{S}^{5}$, then the inferences can
be referred to as HetP-inferences. HetP-inferences may involve using transfer
learning techniques such as domain adaptation [12]. An example of a HetP-
inference is transferring the degradation (transition) model for a blade from
a population of four-blade wind turbines to a population of three-blade wind
turbines.

These inferences within the hierarchical representation of populations,
facilitate reasoning under uncertainty using PBSHM systems; this can naturally
be extended to decision-making under uncertainty, by considering the following
types of decision:

  * •

I-decision: This type of decision is made at the individual structure levels
in the hierarchy, $\mathcal{S}^{1}$ to $\mathcal{S}^{3}$. Again, this type of
decision corresponds to decisions one may make with a traditional SHM system.
An example of an I-decision is selecting a maintenance strategy for an
individual structure, substructure, or component for repair. Unlike in
traditional SHM, in the risk-informed PBSHM approach, I-decisions can be
informed by I-, L- and P-inferences alike.

  * •

L-decision: The actions selected via this type of decision operate between
levels of the hierarchical representation. As with L-inferences these
decisions may pertain to the V&V of predictive models. For example, deciding
whether can one proceed with using a structural model validated on
substructures. Another example of this type of decision relates to resource
allocation. Suppose one has a limited budget to carry out some structural
testing to acquire data for mode updating. Under these circumstances, one
should aim to decide on a set of tests, and the levels at which these tests
are carried out, such that the largest improvement in model performance is
obtained for the given budget.

  * •

P-decision: This type of decision is made at the population levels in the
hierarchy, $S^{4}$ to $S^{6}$. These actions may pertain to resource
management. For example, one may decide to send a team of engineers to perform
inspections on a type inventory based on the probability of failure for a
population rather than the probability of failure of an individual structure.
Scheduling inspections in this manner could save both time and expenditure.
Again, these decisions may be informed via I-, L- and P-inferences.

To summarise, the hierarchical representation of populations of structures
facilitates both making inferences and making decisions for PBSHM, by allowing
for the definition of specific types of inferences and decisions.

###  6.3 Value of information transfer

Value of information (VoI) is a concept in decision theory defined to be the
amount of money/resource a decision-maker should be willing to pay in order to
gain access to information prior to making a decision. The concept of VoI has
seen some application to traditional SHM in recent works [Hughes2021b, 21].

Extending the risk-based approach to decision-making from traditional SHM to
PBSHM opens up the possibility of value of information transfer, i.e. the
price a decision-maker should be willing to pay in order to gain information
via transfer, prior to making a decision. This value arises as a result of
change in maximum expected utility that can be achieved should a change in
optimal policy occur as a result of the additional information made available
via transfer. This notion of value of information transfer yields the thought-
provoking implication that, in some contexts, it may be an optimal decision to
allow a (sub)structure to fail, since the data obtained throughout the failure
process may improve the management of the other individuals in a population.

##  7 Conclusions

To conclude, PBSHM provides a general framework for overcoming issues of data
scarcity associated with developing predictive models for detecting and
forecasting damage within structures. This advantage is achieved via
technologies that allow for the transfer of information between individual
structures within a population. Adopting a probabilistic risk-based approach
to SHM, allows inferences made about the health-states of individual
structures to inform operation and maintenance decisions via the use of
hierarchical representations of structures and fault trees. The current paper
extends this hierarchical representation of structures to representations of
populations, such that decision process can be defined over populations. Other
advantages can be gained by adopting a risk-based approach to PBSHM; for
example, the identification of critical components and substructures can be
used to inform the development of irreducible element models and the
associated attributed graphs.

## Acknowledgements

The authors would like to gratefully acknowledge the support of the UK
Engineering and Physical Sciences Research Council (EPSRC) via grant
references EP/W005816/1 and EP/R006768/1. For the purpose of open access, the
authors has applied a Creative Commons Attribution (CC BY) licence to any
Author Accepted Manuscript version arising. KW would also like to acknowledge
support via the EPSRC Established Career Fellowship EP/R003625/1.

## References

  * [1] C.R. Farrar and K. Worden.  Structural Health Monitoring: A Machine Learning Perspective.  John Wiley & Sons, Ltd, United Kingdom, 2013.
  * [2] A. Rytter.  Vibration Based Inspection of Civil Engineering Structures.  Ph.D. Thesis, Aalborg University, 1993.
  * [3] R. Schöbi and E.N. Chatzi.  Maintenance planning using continuous-state partially observable Markov decision processes and non-linear action models processes and non-linear action models.  Structure and Infrastructure Engineering, 12(8):977–994, 2016.
  * [4] M.A. Vega and M.D. Todd.  A variational Bayesian neural network for structural health monitoring and cost-informed decision-making in miter gates.  Structural Health Monitoring, page 1475921720904543, 2020.
  * [5] A.J. Hughes, R.J. Barthorpe, N. Dervilis, C.R. Farrar, and K. Worden.  A probabilistic risk-based decision framework for structural health monitoring.  Mechanical Systems and Signal Processing, 150:107339, 2021.
  * [6] L.A. Bull, T.J. Rogers, C. Wickramarachchi, E.J. Cross, K. Worden, and N. Dervilis.  Probabilistic active learning: An online framework for structural health monitoring.  Mechanical Systems and Signal Processing, 134:106294, 2019.
  * [7] A.J. Hughes, L.A. Bull, P. Gardner, R.J. Barthorpe, N. Dervilis, and K. Worden.  On risk-based active learning for structural health monitoring.  Mechanical Systems and Signal Processing, 167:108569, 2022.
  * [8] A.J. Hughes, L.A. Bull, P. Gardner, R.J. Barthorpe, N. Dervilis, and K. Worden.  On robust risk-based active-learning algorithms for enhanced decision support.  Mechanical Systems and Signal Processing, 181:109502, 2022.
  * [9] M.A. Vega, Z. Hu, and M.D. Todd.  Optimal maintenance decisions for deteriorating quoin blocks in miter gates subject to uncertainty in the condition rating protocol.  Reliability Engineering and System Safety, 204:107147, 2020.
  * [10] L.A. Bull, P. Gardner, J. Gosliga, T.J. Rogers, N. Dervilis, E.J. Cross, E. Papatheou, A.E. Maguire, C. Campos, and K. Worden.  Foundations of population-based SHM, Part I: Homogeneous populations and forms.  Mechanical Systems and Signal Processing, 148:107141, 2021.
  * [11] J. Gosliga, P.A. Gardner, L.A. Bull, N. Dervilis, and K. Worden.  Foundations of Population-based SHM, Part II: Heterogeneous populations – Graphs, networks, and communities.  Mechanical Systems and Signal Processing, 148:107144, 2021.
  * [12] P. Gardner, L.A. Bull, J. Gosliga, N. Dervilis, and K. Worden.  Foundations of population-based SHM, Part III: Heterogeneous populations – Mapping and transfer.  Mechanical Systems and Signal Processing, 148:107142, 2021.
  * [13] G. Tsialiamanis, C. Mylonas, E. Chatzi, N. Dervilis, D.J. Wagg, and K. Worden.  Foundations of population-based SHM, Part IV: The geometry of spaces of structures and their feature spaces.  Mechanical Systems and Signal Processing, 157:107692, 2021.
  * [14] K. Worden, L.A. Bull, P. Gardner, J. Gosliga, T.J. Rogers, E.J. Cross, E. Papatheou, W. Lin, and N. Dervilis.  A brief introduction to recent developments in population-based structural health monitoring.  Frontiers in Built Environment, 6:146, 2020.
  * [15] C.T. Wickramarachchi, J. Gosliga, E.J. Cross, and K. Worden.  On the use of graph kernels for assessing similarity of structures in population-based structrual health monitoring.  In Proceedings of the Thirtheenth International Workshop on Structural Health Monitoring, 2022.
  * [16] M. Dhada, M. Girolami, and A.K. Parlikad.  Anomaly detection in a fleet of industrial assets with hierarchical statistical modeling.  Data-Centric Engineering, 1:e21, 2020.
  * [17] L.E. Sucar.  Probabilistic Graphical Models: Principles and Applications.  Springer, London, 2015.
  * [18] U.B. Kjaerulff and A.L. Madsen.  Bayesian Networks and Influence Diagrams: A Guide to Construction and Analysis.  Springer, New York, 2008.
  * [19] A. Bobbio, L. Portinale, M. Minichino, and E. Ciancamerla.  Improving the analysis of dependable systems by mapping Fault Trees into Bayesian Networks.  Reliability Engineering and System Safety, 71(3):249–260, 2001\.
  * [20] S. Mahadevan, R. Zhang, and N. Smith.  Bayesian networks for system reliability reassessment.  Structural Safety, 23(3):231–251, 2001.
  * [21] A. Kamariotis, E. Chatzi, and D. Straub.  Value of information from vibration-based structural health monitoring extracted via Bayesian model updating.  Mechanical Systems and Signal Processing, 166:108465, 2022.


11institutetext: Shanghai Key Lab of Intelligent Information Processing, and
School of
Computer Science, Fudan University, Shanghai 200438, China
11email: {yxmi20<EMAIL_ADDRESS>
22institutetext: Department of Computer Science and Technology, Tongji
University,
Shanghai 201804, China
22email<EMAIL_ADDRESS>
# ARIBA: Towards Accurate and Robust Identification of Backdoor Attacks
in Federated Learning

Yuxi Mi 11 Jihong Guan 22 Shuigeng Zhou 11

###### Abstract

The distributed nature and privacy-preserving characteristics of federated
learning make it prone to the threat of poisoning attacks, especially backdoor
attacks, where the adversary implants backdoors to misguide the model on
certain attacker-chosen sub-tasks. In this paper, we present a novel method
ARIBA to accurately and robustly identify backdoor attacks in federated
learning. By empirical study, we observe that backdoor attacks are discernible
by the filters of CNN layers. Based on this finding, we employ unsupervised
anomaly detection to evaluate the pre-processed filters and calculate an
anomaly score for each client. We then identify the most suspicious clients
according to their anomaly scores. Extensive experiments are conducted, which
show that our method ARIBA can effectively and robustly defend against
multiple state-of-the-art attacks without degrading model performance.

###### Keywords:

Federated learning Backdoor attack Anomaly detection.

##  1 Introduction

Federated learning (FL) [15, 22] has emerged as a leading deep learning
framework for massively distributed training of models among a multitude of
clients under the orchestration of a central server. FL is designated to take
the advantage of the non-i.i.d. training data from thousands or even millions
of clients while preserving their privacy, as the private data and the local
training processes are not visible to any other parties.

However, the distributed architectural design and the lack of transparency in
client updates make federated learning generically vulnerable to poisoning
attacks [10, 18, 21, 31]. An adversary introducing poisoning attacks tries to
corrupt the global model at training time to degrade the model’s performance
on some subtasks or as a whole. Concretely, to achieve the attack goal, an
attacker may alter the private training data [3] of some compromised clients,
or rather, manipulate the model updates sent to the server [14, 20].

This paper focuses on the defense of a branch of poisoning attacks called
backdoor attacks, where the corrupted model’s performance on the attacker-
chosen backdoor subtask(s) is degraded while retaining high performance on the
primary task. For instance, a backdoored model of image classification may
misclassify images that trigger certain backdoors into an attacker-desired
class, while classifies the others correctly. Recent works [1, 2, 11, 24]
successfully implant backdoors of different types, including attacker-chosen
images, single-pixel or pattern triggers, and semantic features.

Since backdoor attacks do not affect the performance of the primary task, it
is more difficult to detect them than those attacks that degrade the entire
model. Besides, the privacy-preserving design of federated learning prevents
the server from auditing the integrity and honestness of clients, which makes
backdoor attack detection a particularly challenging task.

The main branches of existing defenses include dedicated techniques [9, 19,
27], Byzantine-resilient aggregations [4, 6, 30], and those exploiting model
traits such as accuracy and l2-norm [2]. Though these methods push the field a
big leap forward, there is still much room to be improved. They have obvious
limitations: requiring a priori knowledge (e.g. clean dataset) or empirical
thresholds, relying on the assumption of data i.i.d.-ness, and easily being
bypassed by an advanced attacker. More importantly, many of these methods
depend on the overall characteristics of the model, which are too coarse for
precise detection.

Figure 1: Comparison between malicious and benign filters, before and after
preprocessing. Filters directly sampled from clients’ updates are similar.
Yet, pre-processed filters show discernible differences, as a sign of backdoor
attacks taking place. ARIBA identifies suspicious clients by exploiting these
differences.

In this paper, we turn our attention to more delicate model characteristics.
Specifically, by empirical study we observe that pre-processed backdoored and
benign updates show discernible patterns, which can be exploited for attack
detection. Fig. 1 shows some filters of CNN layers of malicious and benign
updates before and after pre-processing. We can see the obvious difference in
pre-processed filter patterns between malicious and benign clients. Based on
this finding, we propose a new method ARIBA to accurately and robustly
identify backdoor attacks. Our method first pre-processes the filters to
obtain zero-centered gradients, then feeds them into an unsupervised anomaly
detection algorithm, where their anomaly scores are evaluated. Following that,
anomaly scores of clients are calculated, with which the most suspicious
clients are identified finally. Our extensive experiments validate its
effectiveness and robustness.

The main contributions of this paper are as follows: 1) We propose a novel
idea of identifying backdoor attacks via the filter patterns of model updates;
2) We develop an effective and robust method ARIBA to implement the above idea
by applying unsupervised anomaly detection over the filters, then evaluating
their anomaly scores to detect the most suspicious clients. 3) We conduct
extensive experiments to evaluate ARIBA, which shows that it can effectively
and robustly defend the state-of-the-art backdoor attacks.

##  2 Related Work

###  2.1 Federated Learning (FL)

Federated learning [15, 22] trains a deep neural network collaboratively by
iteratively merging local updates to the global model. A federated learning
framework consists of a central server and a group of clients, each with
exclusive access to its private local data $\mathcal{D}_{i}$. At each round
$t$, $n$ clients are randomly chosen by the server to share the current global
model weight $w_{G}^{t}$. Each chosen client updates $w_{G}^{t}$ for several
local epochs on its private data $\mathcal{D}_{i}$ to reach a new local weight
$w_{i}^{t+1}$ that is sent to the server. The server synchronizes these
updates to obtain a renewed global weight $w_{G}^{t+1}$ for the next
iteration, by performing aggregation such as the standard FedAvg [22]:

$w_{G}^{t+1}=w_{G}^{t}+\frac{\eta}{n}\sum_{i=1}^{n}{(w_{i}^{t+1}-w_{G}^{t})},$
(1)

where $\eta$ is a task-specific global learning rate set by the server.

###  2.2 Poisoning Attacks

Poisoning attacks alter the training pipeline of a deep learning system to
degrade model performance, which can take place in both centralized and
federated settings. State-of-the-art attacks include data poisoning that
manipulates the model’s behavior by compromising some of the training data
[12, 23], and model poisoning by inserting malicious operations or components
into the training process [13, 32].

Backdoor attacks make up a more dedicated branch of poisoning attacks, where
the attacker makes the trained model misbehave only on certain backdoors.
Centralized backdoor attacks [5, 11, 21] have been well explored. These
methods are not applicable for federated learning, since the malicious model
updates, once produced, are quickly diluted in the aggregation with a massive
multitude of benign models. In recently proposed federated backdoor attacks
[1, 2, 24, 26, 29], the attacker survives the aggregation by boosting its
update gradients. Baseline attackers carry out the attack directly while
advanced attackers also take countermeasures to conceal themselves from
possible detections.

###  2.3 Defenses

Existing defenses to backdoor attacks in FL roughly fall into three types:
examining model traits, Byzantine-resilient aggregations, and dedicated
defenses.

####  2.3.1 Examining model traits.

The most intuitive defense in FL is to examine the overall statistical traits
of candidate local updates. The server may check their test accuracy, or
compare the l2-norm of their gradients [2]. These defenses are sometimes
effective against a baseline attacker but can be easily bypassed by advanced
ones. Moreover, these defenses usually require prior knowledge on thresholds,
which is hardly available in real applications.

####  2.3.2 Byzantine-resilient aggregations.

Aggregations such as Krum, coordinate-wise median, and GeoMed [4, 6, 30] are
designed to eliminate local updates with skewed distributions. Most of them
rely on the i.i.d.-ness of training data and the convexity of loss function.
As recent research [1] shows, these defenses can hardly counteract backdoor
attacks if proper attack strategies are chosen.

####  2.3.3 Dedicated defenses.

Gu et al. [11] showed that a backdoor attack can activate certain compromised
neurons. Based on this discovery, recent work [27] proposes a pruning defense
to identify and remove suspicious neurons based on the ranking of their
averaged activation values. This defense works only at inference time on
pixel-pattern backdoors. The reported activation value also reveals the
client’s local information. FoolsGold [9] shares some common ground with our
method in that both are based on the diversity of client updates. Yet, their
method identifies attacks more coarsely by evaluating the cosine similarity
between gradients while ours looks into the patterns of model filters.
Recently, Li et al. [19] proposed a spectral anomaly detection framework that
detects the malicious model updates in their low-dimensional embeddings.
However, their framework requires extra public datasets and a centralized
training process to provide unbiased model updates, which are hardly available
in real-world federated scenarios.

##  3 Threat Model

####  3.0.1 The attacker’s capability.

The attacker has all the basic abilities of a standard model poisoning
adversary. It takes full control of one or several (say $l$) compromised
_malicious clients_ $\\{mal_{1},\cdots,mal_{l}\\}$. It can access and alter
the local training data of all malicious clients, and manipulate the local
training of malicious clients by changing objective functions or adjusting
hyperparameters such as the learning rate. Once the local training is
finished, the attacker can modify the trained weights before updating them to
the server. However, the attacker has not access to or control of any benign
client. In addition, we extend the attacker’s ability by imposing three strong
assumptions: (1) According to Shannon’s maxim [25], the attacker is assumed to
be aware of all possible defenses performed at the server-side. (2) Though FL
by design chooses a fraction of clients in each iteration randomly, we assume
all the malicious clients can be selected anytime at the attacker’s will. (3)
If the attacker has control of multiple malicious clients, these clients can
share the attacking workload by collusion.

####  3.0.2 The attacker’s goal.

The attacker desires to replace the benign global model with an adversarial
model, which retains high accuracy on the primary task, yet produces false
outputs on chosen backdoors.

To this end, the attacker first train each compromised client locally on its
malicious objective
$\operatorname*{arg\,min}_{w_{{mal}_{j}}^{t+1}}{\mathcal{L}_{mal}}$ to obtain
a trained weight $w_{{mal}_{j}}^{t+1}$. Note that
$\\{w_{{mal}_{j}}^{t+1}\\}_{j\in[l]}$ are all very close to an attacker-
desired $w_{mal}^{t+1}$ since they share the same backdoor and malicious
objective function. The attacker may also try to evade possible detection by
adding anomalous terms to the loss. Assume that FedAvg is performed, the
attacker then takes a “scaling” strategy to survive the averaging. Especially,
we generalize the strategy to a group of colluded malicious clients. As the
global model converges, the update gradients of ($n-l$) benign clients will be
cancelled out as $\sum_{i\in[n\backslash l]}{(w_{i}^{t+1}-w_{G}^{t})}\approx
0$. Therefore, the attacker can calculate a scaling factor
$\alpha=\frac{\tilde{n}}{\eta}$, where $\tilde{n}$ is an estimation of the
number of participated clients, and assigns part of the attacking workload to
each malicious client: $\alpha=\sum_{j\in[l]}{\alpha_{j}}$. The latter then
uploads
$\tilde{w}_{{mal}_{j}}^{t+1}=w_{G}^{t}+\alpha_{j}(w_{{mal}_{j}}^{t+1}-w_{G}^{t})$.
Therefore, the global model $w_{G}^{t+1}$ will be replaced by the desired
$w_{mal}^{t+1}$ as

$\displaystyle w_{G}^{t+1}$
$\displaystyle=w_{G}^{t}+\frac{\eta}{n}\sum_{i=1}^{n}{(w_{i}^{t+1}-w_{G}^{t})}$
(2)
$\displaystyle=w_{G}^{t}+\frac{\eta}{n}\left(\sum_{j\in[l]}{(\tilde{w}_{{mal}_{j}}^{t+1}-w_{G}^{t})}+\sum_{i\in[n\backslash
l]}{(w_{i}^{t+1}-w_{G}^{t})}\right)$ $\displaystyle\approx
w_{G}^{t}+\frac{\eta}{n}\sum_{j\in[l]}{(\tilde{w}_{{mal}_{j}}^{t+1}-w_{G}^{t})}$
$\displaystyle\approx
w_{G}^{t}+\frac{\tilde{n}}{n}(w_{mal}^{t+1}-w_{G}^{t})\approx w_{mal}^{t+1}.$

Figure 2: Examples of three types of backdoors. Here, triggers are (a) one
chosen “sandal” image falsely labeled as “sneaker”, (b) a fraction of “5”s,
each overlayed with a group of bright pixels on one corner, and (c) some
“car”s painted in green.

Figure 3: An illustrative explanation of our idea. (a) The benign (blue) and
malicious (orange) gradients differ in magnitude and direction. (b) The
attacker may change its gradient to imitate the benign ones. Yet, a delicate
algorithm can still tell the difference.

####  3.0.3 The attack scenarios.

We consider three types of state-of-the-art backdoors in image classification:
(1) _Targeted backdoors_. The attacker holds a target dataset of one or
several images with false labels and performs training on the dataset. The
trained global model is expected to misclassify the same set of images if they
appear again at inference time. (2) _Pixel-pattern backdoors_. The attacker
picks some training images from a certain class and overlays them with a
pattern composed of a single or a group of bright pixel(s). Images with the
same pattern should be misclassified to the attacker-desired class. (3)
_Semantic backdoors_. Instead of adding artificial patterns, the attacker
chooses a naturally occurring feature (e.g. cars painted in green) as the
backdoor. This allows the attacker to trigger it without accessing or
modifying any inference-time image. Fig. 3 shows examples of three types of
backdoors.

##  4 Method

###  4.1 Basic Idea and Rationale

Gu et al. [11] showed that pixel-pattern attacks can introduce compromised
neurons that fire only in the presence of backdoors. However, such a neuron-
wise pattern is so delicate that is discernible only when the attacker sticks
to a very specific, fixed backdoor. Furthermore, to detect these neurons, a
reference dataset must be exploited to account for the activation, which is
not available in federated settings.

We conducted empirical studies on the behaviors of both malicious and benign
clients in three different attack scenarios mentioned above. We found that
backdoor attacks can introduce discernible patterns in model weights. Such
patterns can serve as much more discernible clues for detecting malicious
clients than the model’s overall traits such as test accuracy and distance. As
the examples in Fig. 1 show, for a baseline attacker, the difference between
pre-processed filters can be easily recognized by human eyes. More
sophisticated malicious clients may conceal their intention by taking covert
measures or colluding with each other (i.e., engaging Sybil attacks), making
the patterns less discernible. Yet, a properly designed algorithm
(unsupervised anomaly detection algorithm in this paper) can still detect
these patterns with high confidence.

However, is this finding reasonable? We answer this by looking into the
principle of backdoor attacks. As illustrated in Fig. 3, when an attack takes
place, malicious local models approach the backdoor objective with the
accumulation of their gradients, while the others head for different benign
objectives. The malicious models will thus produce gradients of distinct
magnitude and direction. Certainly, the attacker may conceal its intention by
adding constraints to malicious gradients, making them less notable. However,
the difference always exists as long as the attack takes place. Such
difference will inevitably incur the difference in model weights, wlog., in
the filters of CNN layers.

Figure 4: The ARIBA Framework. ARIBA works as follows: the server (1) collects
local updates from clients, (2) pre-processes model weights to obtain
discernible filters, (3) detects all anomalous filters via an unsupervised
anomaly detection algorithm, (4) calculates the anomaly score of each client,
and identifies those with the highest and lowest scores as attackers, and
removes them, finally (5) uses the remaining local updates for aggregation.
Here, blue/organge indicates benign/malicious weights or filters. Algorithm 1
ARIBA

Input: Clients’ updates $\\{w_{i}^{t+1}\\}_{i\in[n]}$ of round $t$
Output: Cleaned updates

1: Choose a CNN layer.

2: for all $i\in[n]$ do

3: Obtain zero-centered gradients: $\tilde{\delta}_{i}^{t+1}$ $\leftarrow$
$w_{i}^{t+1}-\frac{1}{n}\sum_{i\in[n]}{w_{i}^{t+1}}$.

4: end for

5: Obtain filters
$\\{f_{i\in[n],k\in[m]}\\}\leftarrow\\{\tilde{\delta}_{i}^{t+1}\\}$.

6: Evaluate filters’ anomaly scores: $\\{s_{i,k}\\}$$\leftarrow$${\tt
FixAndPred}(\\{f_{i,k}\\})$.

7: for all $i\in[n]$ do

8: Calculate clients’ anomaly scores: $S_{i}$ $\leftarrow$
$\sum_{k\in[m]}{s_{i,k}}$.

9: end for

10: Mark clients with the highest and lowest anomaly scores as attackers, and
remove their ids $\\{c\\}$.

11: return $\\{w_{i}^{t+1}\\}_{i\in[n]\backslash\\{c\\}}$

###  4.2 Framework and Algorithm

Based on the idea above, we propose a novel method called ARIBA to support _A_
ccurate and _R_ obust _I_ dentification of _B_ ackdoor _A_ ttacks. Fig. 4
illustrates the ARIBA framework.

ARIBA can serve as a plug-in block in most federated learning systems. It
works in a three-step way: first, we pre-process model weights to obtain
discernible patterns of filters in a CNN layer. Then, we collect the filters
and feed them into an unsupervised anomaly detection algorithm to identify
suspicious filters. After that, we calculate an anomaly score for each client
based on the number of identified anomalous filters. Finally, we detect the
most suspicious clients as attackers according to clients’ anomaly scores. The
entire process is described in Alg. 1.

###  4.3 Techniques

####  4.3.1 Pre-processing.

Instead of identifying certain backdoor neurons, we focus on filter-wise
difference, which is more robust to noise. We start with pre-processing the
model weights. In each training round, the server collects from clients their
local model weights $\\{w_{i}^{t+1}\\}_{i\in[n]}$, which are accumulated by
rounds of gradients. Here, we consider only the latest gradient
$\delta_{i}^{t+1}$. Since the server always has access to the global model,
$\delta_{i}^{t+1}$ can be evaluated by:

$\delta_{i}^{t+1}=w_{i}^{t+1}-w_{G}^{t}.$ (3)

Note that we omit the global learning rate $\eta$ for simplicity, which does
not impact our derivation. For better differentiating the gradients of benign
and malicious clients, we do mean-subtraction for all gradients as follows:

$\tilde{\delta}_{i}^{t+1}=w_{i}^{t+1}-\frac{1}{n}\sum_{i\in[n]}{w_{i}^{t+1}}$
(4)

With the zero-centered gradients, we take a CNN layer (wlog., the first CNN
layer) and extract all of its filters. As an example, we use principal
component analysis to visualize the filters’ distribution. Fig. 6 shows that
these filters do expose discernible patterns, depending on the attacker’s
capability. Based on the patterns, later we identify the malicious filters by
exploiting a distance-based anomaly detection algorithm.

Algorithm 2 FixAndPred (Mahalanobis Distance)

Input: Pre-processed filters $\\{f_{i,k}\\}$
Parameter: An estimated fraction of anomalous filters $\gamma$
Output: Prediction $\\{s_{i,k}\\}$

1: Calculate the convariance matrix $\Sigma\leftarrow cov(\\{f_{i,k}\\})$.

2: Calculate the average of filters $\bar{f}\leftarrow mean(\\{f_{i,k}\\})$.

3: for all $i,k$ do

4: Calculate the Mahalanobis distance of $f_{i,k}$:
$d_{i,k}\leftarrow\sqrt{(f_{i,k}-\bar{f})^{T}\Sigma^{-1}(f_{i,k}-\bar{f})}$.

5: end for

6: Set $s_{i,k}\leftarrow 1$ for the $\gamma$-largest $d_{i,k}$

7: Set $s_{i,k}\leftarrow 0$ for the others.

8: return $\\{s_{i,k}\\}$

####  4.3.2 Detecting anomalous filters.

Here, we address how to identify anomalous filters. Actually, many
unsupervised anomaly detection algorithms can do this work. But which one
should we use? We emphasize two observations:

1) _The decisions on individual filters do not matter much_. In most cases, we
do not expect an anomaly detection algorithm to judge single filters with very
high precision. After all, it is not easy to completely see through a ‘smart’
backdoor attacker. Instead, we treat each client’s filters as a whole and pay
more attention to the overall state by counting the total numbers of normal
and anomalous filters, i.e., a client may be malicious if it has significantly
more (or less, as discussed later) anomalous filters than the normal ones. For
instance, each of the three clients in Fig. 6(b) has a number of hard-to-judge
filters (or outliers/anomalies). Yet, by putting these filters together, we
can see that the client in orange is more anomalous because it is less
overlapped with the other two clients. In summary, it is acceptable that an
anomaly detection algorithm incorrectly reports some filters, as long as it
can accurately detect the overall difference between benign and malicious
filters.

2) _Algorithm efficiency does matter_. In real-world scenarios, thousands of
clients may participate, each has thousands of filters. The detection can
therefore be very expensive. Although the workload can be reduced by filter
sampling, the number of filters is still very large since by design the
defender should examine all the clients. Hence, the chosen algorithm should be
of low time complexity.

Based on the above observations and after empirical evaluations, we choose a
Mahalanobis-distance-based detection algorithm [7], as described in Alg. 2.
Note our aim is to identify backdoor attacks, developing anomaly detection
algorithm is not our focus. The purpose that we choose the Mahalanobis-
distance-based algorithm is to validate our method. Surely there are many
other sophisticated algorithms that can be used, and any defender is free to
choose its task-specific algorithm. Assume each of the $n$ clients has $m$
filters, denoted as $f_{i,k}\ (i\in[n],k\in[m])$, its anomaly score $s_{i,k}$
is evaluated by:

$\displaystyle\\{s_{i,k}\\}={\tt FixAndPred}(\\{f_{i,k}\\}),$ (5)
$\displaystyle where\ s_{i,k}=\begin{cases}0&if\ f_{i,k}\ is\ normal\\\ 1&if\
f_{i,k}\ is\ anomalous.\end{cases}$

####  4.3.3 Identifying suspicious clients.

Figure 5: Possible statistical distributions of filters. The malicious filters
may appear more (a) or less (c) scattered than the benign ones. Advanced
attacks may produce hard-to-judge points. Yet, the malicious client can still
be identified by looking into the overall trend (b).

Figure 6: Examples of two possible anomaly score distributions. The malicious
client will obtain a significantly (a) higher or lower (b) anomaly score than
other benign clients. The difference is distinguishable for ARIBA.

We identify the most suspicious clients as attackers by summing up
$\\{s_{i,k}\\}$ for each client $i$ to obtain its anomaly score $S_{i}$:

$S_{i}=\sum_{k\in[m]}{s_{i,k}}.$ (6)

Fig. 6 shows two examples of client anomaly score distribution. The scores of
most benign clients are expected to fall within a narrow range. A malicious
client, if participated, should has a significantly higher or lower score. It
is quite natural to regard a client with a higher score (more anomalous) as a
malicious one. Yet, as shown in Fig. 6(b), a malicious client can also has a
lower score. Empirically, this usually happens to advanced attackers
attempting to conceal their intention, who train on both malicious and primary
objectives in parallel. The gradients of the two objectives, if heading for
opposite directions, may cancel each other out. Put it in anomaly detection,
its filters may appear less scattered than what benign filters should look
like, as shown in Fig. 6(c). Ergo, the detection algorithm may regard the
malicious filters as normal ones, consequently marks the corresponding clients
as benign ones instead. However, in either way, the difference between scores
can still provide us significant evidence to recognize the suspicious clients.

After identifying the suspicious clients, the server performs aggregation on
the model weights of the rest as usual.

##  5 Performance Evaluation

###  5.1 Experimental Setup

We apply our method to the three attack scenarios described in Sec. 3, i.e.,
targeted attack, pixel-pattern attack, and semantic attack. We consider three
commonly used image classification datasets: Fashion-MNIST [28] of grayscale
cloths, MNIST [17] of handwritten digits, and CIFAR-10 [16], respectively. For
Fashion-MNIST and MNIST, we apply a CNN with two convolutional layers and two
fully connected layers. This is a standard architecture for such tasks. As for
CIFAR-10, we use a real-world ResNet18. We implement the method using PyTorch
1.8.1 and CUDA 11.2. Our experiments are done on two Nvidia Titan Xp GPUs each
with 12GB RAM, and Ubuntu 18.04 LTS OS.

By default, a group of 10 clients is selected in each experiment, one of them
is malicious. For simplicity, the global learning rate is set to $\eta=1$,
i.e., the updated global model replaces the previous one completely. We first
train each model via standard federated learning till it reaches a pre-
specified test accuracy: 0.9 for Fashion-MNIST, 0.99 for MNIST, and 0.74 for
CIFAR-10.

Then, we reproduce the attacks. We set the backdoors following Fig. 3. We
apply a local learning rate $\eta_{local}$=1e-3, 1e-4, 1e-4 and train
$E_{local}=5,10,2$ epochs for three attacks, respectively. We assume that the
targeted attacker participates every round, the pixel-wise and semantic
attackers are chosen every 3 rounds since the latter two are designed to be
one-shot attack. Before aggregation, the malicious client boosts its update by
$\alpha$=10 to survive FedAvg. All attacks succeed as they implant the
backdoors with high confidence, while incurring very slight impact on the
accuracy of the primary task. The results are illustrated in Fig. 7.

Figure 7: ARIBA’s primary defending results. Red: Without defense, all three
attackers succeed with high confidence. Green: Yet, after implementing our
ARIBA method, 100% of the attacks are successfully identified and blocked, and
the model’s performance is maintained.

###  5.2 Primary Defense Results

We then plug in ARIBA before the aggregation and rerun each attack. Fig. 7
shows that our method successfully blocks all three attacks by identifying all
the malicious attempts, without degrading the global model.

Note the result defending a single malicious client is discrete: either
success (100%) or failure (0%). To quantitively evaluate our defense, we
introduce defense confidence $C$, to measure how confident the server is about
the result:

$C=\frac{1}{\log_{2}{m}}\times\max{(\log_{2}{\frac{S_{mal}}{\max{(\\{S_{i\in[n]\backslash
mal}\\})}}},\log_{2}{\frac{\min{(\\{S_{i\in[n]\backslash
mal}\\})}}{S_{mal}}})}.$ (7)

Here, $S_{i},S_{mal}$ is the anomaly score of client $i$ and the malicious
client, respectively. $m$ is the number of filters. This metric is actually
quite straightforward. The defense succeeds if and only if $S_{m}$ is the
largest or smallest, in both cases we have $C$$>$0\. $C$ gradually increases
as the difference between malicious and benign filters becomes more
discernible. For the three attacks, our ARIBA method achieves high confidence
$C$ = 0.8, 1.0 and 1.0, respectively.

###  5.3 Comparison with Existing Methods

Table 1: Comparison with existing defenses. Here, “✓”: success; “✗”: failure;
“-”: no experimental results reported.

Defense | T. | P. | S.
---|---|---|---
Test accuracy [2] | ✓ | ✗ | ✗
L2-norm [2] | ✓ | ✓ | ✓
Krum [4] | ✗ | ✗ | ✗
Coo-med [30] | ✗ | ✗ | ✗
Wu et al. [27] | - | ✓ | -
FoolsGold [9] | - | ✓ | -
Li et al. [19] | - | - | ✓
ARIBA | ✓ | ✓ | ✓

Figure 8: ARIBA can perform effective defense against advanced attackers,
including (a) a norm bounding attacker, and (b) multiple colluding clients,
each with a smaller scaling factor $\alpha_{j}$.

Here we compare our method with major existing defenses described in Sec. 2.
Results are summarized in Tab. 8. Here T./P./S. stand for targeted, pixel-
pattern and semantic attacks, respectively. Our method successfully defend all
three attacks, while there are shortcomings in each of the existing ones.

We first check prior defenses that examine model traits, such as the model’s
test accuracy and l2-norm. Note, due to the non-i.i.d.-ness of data, federated
learning is designed to accept updates that appear slightly different (i.e.,
updates under a threshold). For test accuracy, we assume a threshold of 10%,
the same as [2]. Most malicious updates fall in the range, nullifying the
defense. Norm bounding defense is effective against all baseline attacks. Yet,
as we point out below, an advanced attacker can conceal itself from the norm
detection.

To compare with the results of Byzantine-resilient aggregations, we replace
FedAvg with Krum [4] and Coo-med [30]. Results show that the attacker can
bypass them by choosing a smaller scaling factor $\alpha$=1. Note it is within
the attacker’s capability to choose $\alpha$ according to server-side defense
mechanisms.

For dedicated defenses, we quote the results of [9, 19, 27]. Those defenses
are attack-specific and are not likely to be generalized against other
defenses, since they all expect the attacker to meet certain conditions. For
instance, the attacker in [19] is assumed to possess i.i.d. images as the
benign ones apart from the overlayed pattern, which does not hold for semantic
scenario. The attacker in [27] should massively revise the dataset to produce
enough differences in vector space, which is not true for targeted attacks. In
[9], gradient updates of attackers should have high cosine similarity.

###  5.4 Defending Against Advanced Attacks

As mentioned in Sec. 1, advanced attackers can take covert measures to evade
detection. We here show ARIBA is still effective against these attackers.

####  5.4.1 Anomalous terms in objective function.

Recent works [1, 2] describe a stealthy attacker that conceal its intention by
adding an anomalous term $\mathcal{L}_{ano}$ to its training objective:
$\operatorname*{arg\,min}_{w_{mal}^{t+1}}{\mathcal{L}_{mal}+\rho\mathcal{L}_{ano}}$,
where $\rho$ is a hyperparameter. Specifically, we choose
$\mathcal{L}_{ano}=\|\delta_{mal}^{t+1}-\delta_{G}^{t}\|_{2}$ and append it to
all three attacks. The advanced attack nullifies the norm bounding defense in
Sec. 5.3. Our defense results are shown in Fig. 8(a). The decrease in $C$
shows the anomalous term does play its role. Yet, ARIBA still manages to
identify the attacker as $C\geq$0.35.

####  5.4.2 Multiple colluding clients.

An advanced attacker may also perform Sybil attacks by controlling multiple
malicious clients. By assigning to each client a smaller scaling factor
$\alpha_{j}$, individual malicious updates become less notable. We change
$\alpha_{j}$ to simulate multiple colluding clients and rerun the defense.
Fig. 8(b) shows that our method is effective ($C$$>$0) against different
$\alpha_{j}$. Note $\alpha_{j}$$<$2 is actually beyond our scope of discussion
since an attacker manipulating a majority of clients already takes full
control of the FL system.

###  5.5 Ablation Study

Figure 9: Performance comparison of different anomaly detection algorithms on
three attacks.

Table 2: Computation time (sec.) vs. #filters ($\times 10^{3}$).

Algorithm | 0.5 | 1 | 2 | 4
---|---|---|---|---
iForest | 0.48 | 0.88 | 1.47 | 2.59
LOF | 0.23 | 0.6 | 1.67 | 5.03
ARIBA | 0.04 | 0.09 | 0.23 | 0.51
PCA | 0.38 | 1.16 | 1.49 | 2.82
KPCA | 18.75 | 190.22 |  $>$300 |  $>$300
RPCC | 0.4 | 0.88 | 1.57 | 2.77

####  5.5.1 Comparison with different anomaly detection techniques.

Recall in Sec. 4 we choose the anomaly detection algorithm based on its
robustness and efficiency. Here, we compare our choice with 5 other commonly
used ones: Isolation Forest, LOF, PCA, KPCA, and RPCC. We test their
performance in all three attack scenarios. Results of F1-score in Fig. 9 show
that LOF and RPCC perform less stably than the others.

Then, we check how their computation costs change with the number of filters.
We run each experiment three times and measure the cost by average time.
Results in Tab. 9 indicate that our chosen algorithm is the most cost-
efficient among all the compared algorithms. Note we choose the fastest and
relatively robust algorithm for the sole purpose to validate our method. The
defender is free to replace it with more sophisticated, task-specific
algorithms.

Figure 10: Performance on (a) different numbers of filters sampled ($\times
10^{2}$) and (b) different CNN layers where filters are selected from.

Figure 11: Performance of ARIBA on noise-tolerant settings where (a) data is
non-i.i.d., and (b) Gaussian noise is added for differential privacy.

####  5.5.2 Filter sampling.

The computation cost can be further reduced by filter sampling, i.e., choosing
a fraction of filters from each client. To illustrate that, we sample the
filters by different proportions and compare their detection confidences.
Results in Fig. 11(a) show that filter sampling incurs almost no performance
loss while saving lots of computation cost.

####  5.5.3 Performance on different CNN layers.

By default, we sample filters from the first CNN layer. Now we generalize this
setting. We enumerate each CNN layer for targeted attack, as illustrated in
Fig. 11(b). Results show confidence is always satisfying ($C$$>$0.5)
regardless of the layer we choose. On the other hand, layers with larger size
(layers 1b–3b) seem to provide better performance.

###  5.6 Robustness to Noise

Noise in training may jeopardize the detection. It adds random disturbance to
the filters and interferes with the difference between them, consequently
degrading the anomaly detection performance. In FL, noise can come from
natural and manmade sources. To test if ARIBA is robust to noise, we consider
two settings:

####  5.6.1 Noise from non-i.i.d. data.

In the real world, the non-i.i.d.-ness of local training data can produce
natural noise. We produce unbalanced data following [2]. For each client, we
randomly select a fraction of classes and assign data only from the selected
classes. We test ARIBA for different numbers of classes sampled. Fig. 11(a)
shows that ARIBA performs well ($C$$>$0.5) under unbalanced data.

####  5.6.2 Noise from differential privacy.

Differential privacy (DP) [8] is the de facto standard privacy-preserving
technique. It deliberately introduces noise to protect clients from inference
attacks, while as a side effect, degrading the model. We simulate local DP by
adding Gaussian noise to the updates of all benign clients, according to
different privacy budgets $\epsilon$. Fig. 11(b) shows that ARIBA can tolerate
Gaussian noise as long as the global model is usable ($\epsilon$$<$0.2).

##  6 Conclusion

In this paper, we propose a novel method to identify backdoored updates by
exploiting unsupervised anomaly detection. It is based on our finding that
backdoor attacks expose discernible patterns in model weights. Concretely, we
first pre-process the filters to obtain zero-centered gradients, then feed
them into an unsupervised anomaly detection algorithm, where the filters are
evaluated to calculate an anomaly score for each client. Finally, we identify
the most suspicious clients according to their anomaly scores. Extensive
experiments show that though our method is simple, it is efficient, accurate
and robust, as well as widely applicable.

## References

  * [1] Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., Shmatikov, V.: How to backdoor federated learning. In: International Conference on Artificial Intelligence and Statistics. pp. 2938–2948. PMLR (2020)
  * [2] Bhagoji, A.N., Chakraborty, S., Mittal, P., Calo, S.: Analyzing federated learning through an adversarial lens. In: International Conference on Machine Learning. pp. 634–643. PMLR (2019)
  * [3] Biggio, B., Nelson, B., Laskov, P.: Poisoning attacks against support vector machines. arXiv preprint arXiv:1206.6389 (2012)
  * [4] Blanchard, P., El Mhamdi, E.M., Guerraoui, R., Stainer, J.: Machine learning with adversaries: Byzantine tolerant gradient descent. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. pp. 118–128 (2017)
  * [5] Chen, X., Liu, C., Li, B., Lu, K., Song, D.: Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 (2017)
  * [6] Chen, Y., Su, L., Xu, J.: Distributed statistical machine learning in adversarial settings: Byzantine gradient descent. Proceedings of the ACM on Measurement and Analysis of Computing Systems 1(2), 1–25 (2017)
  * [7] De Maesschalck, R., Jouan-Rimbaud, D., Massart, D.L.: The mahalanobis distance. Chemometrics and intelligent laboratory systems 50(1), 1–18 (2000)
  * [8] Dwork, C.: Differential privacy: A survey of results. In: International conference on theory and applications of models of computation. pp. 1–19. Springer (2008)
  * [9] Fung, C., Yoon, C.J., Beschastnikh, I.: Mitigating sybils in federated learning poisoning. arXiv preprint arXiv:1808.04866 (2018)
  * [10] Goldblum, M., Tsipras, D., Xie, C., Chen, X., Schwarzschild, A., Song, D., Madry, A., Li, B., Goldstein, T.: Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. arXiv preprint arXiv:2012.10544 (2020)
  * [11] Gu, T., Dolan-Gavitt, B., Garg, S.: Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733 (2017)
  * [12] Huang, L., Joseph, A.D., Nelson, B., Rubinstein, B.I., Tygar, J.D.: Adversarial machine learning. In: Proceedings of the 4th ACM workshop on Security and artificial intelligence. pp. 43–58 (2011)
  * [13] Ji, Y., Zhang, X., Ji, S., Luo, X., Wang, T.: Model-reuse attacks on deep learning systems. In: Proceedings of the 2018 ACM SIGSAC conference on computer and communications security. pp. 349–363 (2018)
  * [14] Kairouz, P., McMahan, H.B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A.N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al.: Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 (2019)
  * [15] Konečnỳ, J., McMahan, H.B., Yu, F.X., Richtárik, P., Suresh, A.T., Bacon, D.: Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016)
  * [16] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
  * [17] LeCun, Y., Cortes, C., Burges, C.: Mnist handwritten digit database (2010)
  * [18] Li, Q., Wen, Z., Wu, Z., Hu, S., Wang, N., Li, Y., Liu, X., He, B.: A survey on federated learning systems: vision, hype and reality for data privacy and protection. arXiv preprint arXiv:1907.09693 (2019)
  * [19] Li, S., Cheng, Y., Wang, W., Liu, Y., Chen, T.: Learning to detect malicious clients for robust federated learning. arXiv preprint arXiv:2002.00211 (2020)
  * [20] Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine 37(3), 50–60 (2020)
  * [21] Liu, Y., Ma, S., Aafer, Y., Lee, W.C., Zhai, J., Wang, W., Zhang, X.: Trojaning attack on neural networks. NDSS (2018)
  * [22] McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. pp. 1273–1282. PMLR (2017)
  * [23] Steinhardt, J., Koh, P.W., Liang, P.: Certified defenses for data poisoning attacks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. pp. 3520–3532 (2017)
  * [24] Sun, Z., Kairouz, P., Suresh, A.T., McMahan, H.B.: Can you really backdoor federated learning? arXiv preprint arXiv:1911.07963 (2019)
  * [25] van Tilborg, H.C.A., Jajodia, S. (eds.): Shannon’s Maxim, pp. 1194–1194. Springer US, Boston, MA (2011)
  * [26] Wang, H., Sreenivasan, K., Rajput, S., Vishwakarma, H., Agarwal, S., Sohn, J.y., Lee, K., Papailiopoulos, D.: Attack of the tails: Yes, you really can backdoor federated learning. arXiv preprint arXiv:2007.05084 (2020)
  * [27] Wu, C., Yang, X., Zhu, S., Mitra, P.: Mitigating backdoor attacks in federated learning. arXiv preprint arXiv:2011.01767 (2020)
  * [28] Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
  * [29] Xie, C., Huang, K., Chen, P.Y., Li, B.: Dba: Distributed backdoor attacks against federated learning. In: International Conference on Learning Representations (2019)
  * [30] Yin, D., Chen, Y., Kannan, R., Bartlett, P.: Byzantine-robust distributed learning: Towards optimal statistical rates. In: International Conference on Machine Learning. pp. 5650–5659. PMLR (2018)
  * [31] Zhang, J., Chen, J., Wu, D., Chen, B., Yu, S.: Poisoning attack in federated learning using generative adversarial nets. In: 2019 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE). pp. 374–380. IEEE (2019)
  * [32] Zou, M., Shi, Y., Wang, C., Li, F., Song, W., Wang, Y.: Potrojan: powerful neural-level trojan designs in deep learning models. arXiv preprint arXiv:1802.03043 (2018)