#  Delay and Overhead Efficient Transmission Scheduling for Federated Learning
in UAV Swarms ††thanks: The authors are with INRS, University of Québec,
Montréal, QC H5A 1K6, Canada. Emails: {duc.hoang, tuan.vu.truong, duy.hung.le,
<EMAIL_ADDRESS>
Duc N. M. Hoang, Vu Tuan Truong, Hung Duy Le, and Long Bao Le

###### Abstract

This paper studies the wireless scheduling design to coordinate the
transmissions of (local) model parameters of federated learning (FL) for a
swarm of unmanned aerial vehicles (UAVs). The overall goal of the proposed
design is to realize the FL training and aggregation processes with a central
aggregator exploiting the sensory data collected by the UAVs but it considers
the multi-hop wireless network formed by the UAVs. Such transmissions of model
parameters over the UAV-based wireless network potentially cause large
transmission delays and overhead. Our proposed framework smartly aggregates
local model parameters trained by the UAVs while efficiently transmitting the
underlying parameters to the central aggregator in each FL global round. We
theoretically show that the proposed scheme achieves minimal delay and
communication overhead. Extensive numerical experiments demonstrate the
superiority of the proposed scheme compared to other baselines.

###### Index Terms:

Communication efficiency, delay efficiency, federated learning, model
aggregation, unmanned aerial vehicles

##  I Introduction

Unmanned aerial vehicles (UAVs) have emerged as a transformative technology,
particularly for crowdsensing tasks in remote areas. The UAVs can
revolutionize data collection by using onboard sensors and cameras to gather a
diverse range of information in regions where traditional methods face
challenges due to geographical constraints or limited resources. What enhances
their utility is the integration of federated learning (FL) [1], a distributed
learning approach that harnesses the computational power of UAVs for in-the-
air collaborative learning. A swarm of UAVs can be rapidly deployed for FL-
empowered crowdsensing missions [2], where they collect data and conduct model
training thanks to their onboard computational resources and advanced sensors.
With the aid of FL, the machine learning (ML) model trained by the swarm can
leverage the comprehensive dataset collected by all individual UAVs. Through
this amalgamation of UAVs and FL, crowdsensing can be readily deployed in
remote areas without requiring communications and computing infrastructure.

To implement FL for a UAV swarm, one can require the UAVs to transmit their
local model updates to a ground station or server for global model aggregation
in each global round. However, in challenging scenarios like natural
disasters, the existing stations can be compromised and thus the aggregation
step in the FL training process can be affected. In addition, even the
wireless transmission of model updates between the UAVs and the ground station
in mission-critical applications (e.g., military surveillance operations) can
pose risks of privacy leakage and eavesdropping [3] to sensitive information.
In these infrastructure-less settings, the UAVs should collaboratively
undertake data acquisition, model training, and model aggregation among
themselves, eliminating the necessity for an external central station. To this
end, an efficient transmission scheduling scheme for the model parameters to
coordinate the FL training and aggregation among the UAVs is needed, which is
the focus of this paper.

Designing such a transmission scheduler for collaborative learning in an
infrastructure-less UAV swarm, however, faces several challenges. Firstly,
traditional FL typically requires a predetermined central aggregator to
collect and aggregate all the model updates submitted by local nodes in each
global round. For UAV swarms deployed in isolated and resource-deficient
areas, there is a need for self-coordination among the UAVs to enable model
aggregation without the help of an aggregator on the ground. Secondly,
traditional FL usually assumes that direct communications between local nodes
and the aggregator are possible. However, UAVs have a limited wireless
communication range; therefore, the distance between two UAVs can be larger
than the communication range, and multi-hop communications are thus necessary.
Supposing that a particular UAV is chosen to be the aggregator, multi-hop
communications for exchanging the local model updates between it and other
UAVs can introduce significant latency and transmission overhead. Thus, the
development of an effective scheduling scheme supporting the model data
exchanges and aggregation for FL in infrastructure-less UAV swarms is worth
investigating.

There are several studies on in-the-air FL [4, 2, 5]. However, most of them
assumed a centralized server for the deployed UAVs, which is not always
available in scenarios without dedicated infrastructure. Qu et al. [6]
proposed an FL framework in which each UAV broadcasts its local models to
immediate neighbors, and likewise, each UAV aggregates the model from its
immediate neighbors. However, there was no detailed study of communication
overhead and delay, and only limited numerical results were presented. Xiao et
al. [7] proposed a learning scheme for FL in UAV swarms considering the
varying locations of UAVs over time. While the proposed scheme guarantees
convergence and communication efficiency over dynamic scenarios, latency
reduction is not investigated. Qu et al. [8] proposed a hierarchical FL
framework to minimize the energy consumption during the training process in a
UAV swarm, whose operation depends on a leading UAV. However, communication
overhead was not thoroughly studied. Feng et al. [9] devised a blockchain-
empowered FL scheme for collaborative learning between the UAVs. However, the
authors did not design a transmission scheduling scheme with low communication
overhead and delay.

Considering the aforementioned research gaps, this paper aims to develop an
efficient transmission scheduling scheme for FL coordination in UAV swarm. In
detail, we first propose a transmission scheduler that realizes the
conventional FL scenario by choosing a suitable aggregator UAV and scheduling
the UAVs to efficiently transmit the model updates to the chosen aggregator,
based on which we obtain a global model in each training round. Next, by
modeling the swarm network as a graph and partitioning time into discrete
transmission time slots, we prove that the proposed scheme achieves minimum
delay and communication overhead. Extensive numerical experiments are
conducted to show that the proposed scheme attains desired convergence and
outperforms other baselines in terms of communication delay and overhead.

##  II System Model

We consider a UAV swarm deployed over an area where they collect sensory data
and collaboratively train a machine learning (ML) model for a given task
(e.g., classification), as depicted in Fig. 1. Note that the UAVs are
distributed in a way that they are dense enough to sense different parts of
the region of interest. We denote the set of UAVs as
$\mathcal{V}=\\{1,\dots,V\\}$, assuming that the UAVs hover at the same fixed
altitude and maintain their 2-D positions during the mission time. Note that
the UAVs are deployed in an infrastructure-less setting, meaning that there is
no ground station in charge of maneuvering them. Because of this, each UAV is
supposed to carry a battery that provides enough energy for the entire mission
so that they do not have to return and recharge.

Figure 1: Example of UAV swarm deployment.

In terms of wireless communication, we assume that all UAVs have a maximum
communication range $D^{\mathsf{com}}_{\max}$ to transmit and receive data
with their neighbors. All the UAVs are also supposed to be crowded enough so
that every pair is able to communicate with each other either through direct
or multi-hop communication. We assume that the available communication
bandwidth is divided into subbands (e.g., using orthogonal frequency division
multiple access or OFDMA), which are allocated to the UAVs so that concurrent
transmissions between the UAVs are possible with negligible interference.

Recall that the mission of the UAVs is to collect the necessary data to train
an ML model. Because each UAV is dedicated to a sensing area, the model on
each UAV would be limited to learning a handful of training information if it
can only make use of its own available data. Consequently, this could result
in, for example, the poor predictive performance of the trained model at the
end of the mission. Therefore, it is evident that a collaborative online FL
training process among the UAVs is necessary. The following describes how the
online FL training and aggregation procedure is conducted within the UAV
swarm.

###  II-A Federated Learning in UAV Swarms

We consider an online FL framework for a UAV swarm as follows. Each UAV $v$
continuously collects its sensory data during the flight time. The local
dataset on UAV $v$, denoted as $\mathcal{D}_{v}$ with
$\left|\mathcal{D}_{v}\right|$ data samples, is used to train a model that
aims to minimize an objective function $F_{v}(\mathbf{w})$, where $\mathbf{w}$
is the model parameter vector. Then the global optimization problem in the
whole swarm is formally defined as

$\min_{\mathbf{w}}F(\mathbf{w})\triangleq\sum_{v\in\mathcal{V}}\lambda_{v}F_{v}(\mathbf{w}),$
(1)

where
$\lambda_{v}=\left|\mathcal{D}_{v}\right|/\sum_{u\in\mathcal{V}}\left|\mathcal{D}_{u}\right|$
is the weight coefficient corresponding to UAV $v$. To solve (1), UAV $v$
performs an iterative stochastic gradient descent (SGD) update using its
dataset $\mathcal{D}_{v}$ to obtain its local update $\mathbf{w}^{(r)}_{v}$ at
global round $r$. Then the global model update is calculated as

$\mathbf{w}^{(r)}=\sum_{v\in\mathcal{V}}\lambda_{v}\mathbf{w}^{(r)}_{v},$ (2)

which is then used as the starting training parameters for the next round
$r+1$ on each UAV. It is seen that the global update (2) depends on the
contributions of many participating UAVs during the training process.
Therefore, in the following, we go through the entire data collection and
collaborative training process of the UAV swarm.

###  II-B Collaborative Sensing and Training Procedure

In this part, we adopt the FL design including one aggregator and multiple
training clients from the seminal FL design, which can take full advantage of
distributed datasets [10]. Subsequently, the following steps describe the
activities performed by the ordinary UAVs and a dedicated root UAV, denoted as
$v_{c}$, deployed for a crowdsensing task. The root UAV is responsible for
aggregating all local models into a global model in each training round.

  1. 1

Initialization. Each UAV transmits its beacon message notifying its
surrounding neighbors of its presence and exchanging necessary network control
information. At the end of this step, the UAVs know the information of the
entire network and the root UAV $v_{c}$.

  2. 2

Data collection. The UAVs use their on-board sensors to acquire sensory data
for a fixed period of time.

  3. 3

Local training. The UAVs perform local model updating based on their own
dataset using the SGD method with a fixed number of iterations to obtain their
$\mathbf{w}^{(r)}_{v}$ at round $r$.

  4. 4

Model aggregation. The root UAV $v_{c}$ performs the global model aggregation
process by computing $\mathbf{w}^{(r)}$ using (2) based on the local updates
obtained from other UAVs.

  5. 5

Model dissemination. The root UAV transmits $\mathbf{w}^{(r)}$ to other UAVs.
The whole UAV swarm is then ready for the next round $r+1$ starting from step
2.

Note that step 1 is executed once as the UAVs are deployed. Steps 2– 5 are
conducted within one global round and repeated until the training algorithm
converges or the maximum number of global rounds is reached. Next, we will
provide in-depth analyses of delay and message transmission overhead during
these steps.

###  II-C Delay and Overhead Analyses

In step 1, since the network discovery process is initiated only a single time
and the network remains static within the context of our study, we can afford
to overlook the delay and overhead associated with this step. Next, in step 2,
all the UAVs collect the data in a fixed period of time, which is a predefined
parameter. Therefore, we can consider this to last a fixed duration with no
transmission, thus leaving little room for delay reduction. A similar
explanation can also be used for step 3 as the number of SGD iterations is a
selective parameter. Suppose that all the UAVs have the same processing unit
and conduct training in parallel, the training time spent by one UAV is also
the amount for the whole network in this step.

In step 4, the UAVs have finished local training and their local updates have
to be accumulated at $v_{c}$ to realize the global model aggregation process.
It is challenging to design an efficient transmission scheme for the delivery
of local updates of individual UAVs to $v_{c}$. This is because the large-
scale UAV network requires multi-hop communications for exchanges of model
parameters among UAVs, causing potentially large delays and transmission
overhead. We denote the time needed for $v_{c}$ to receive the local updates
from the swarm as $T_{\mathsf{trans}}$ and the number of message transmissions
among UAVs as $N_{\mathsf{trans}}$, which will be carefully revisited later.
Then after $T_{\mathsf{trans}}$, $v_{c}$ computes the global model, which is
assumed to consume a negligible amount of time. Finally, the model
dissemination step 5 requires $v_{c}$ to send back the aggregated global model
to other UAVs. This is a well-studied wireless broadcast problem and a known
heuristic broadcast algorithm, e.g., [11], can be employed to reduce both
delays and transmission overhead.

###  II-D Scheduling Problem

After the analyses, we are left with the delay $T_{\mathsf{trans}}$ and
overhead $N_{\mathsf{trans}}$ in step 4 to be improved. In fact, these two
parameters are very much dependent on a scheduling scheme that manages the
transmissions in the network. Consider a toy example of transmission
activities that may happen without a scheduler in this step as illustrated in
Fig. 2(a). In this example, each UAV $v_{i}$ individually transmits its local
update $\mathbf{w}_{v_{i}}$ to the root UAV $v_{c}$ for aggregation. For
simplicity, let us assume $\lambda_{v_{i}}=1$ in (2). One can see that
$\mathbf{w}_{v_{5}}$ can traverse the path $(v_{5},v_{6},v_{c})$ (i.e., the
shortest path) instead of $(v_{5},v_{6},v_{4},v_{c})$ to save a transmission.
It can also be observed that we can reduce transmission overhead (or the
required number of transmissions) if individual UAVs aggregate/accumulate the
model parameters before forwarding them toward the root UAV. In particular, if
$v_{1}$ can add $\mathbf{w}_{v_{1}}$, $\mathbf{w}_{v_{2}}$ and
$\mathbf{w}_{v_{3}}$ together and send the sum in one message as depicted in
Fig. 2(b), we can further reduce $N_{\mathsf{trans}}$.

(a)

(b)

Figure 2: Illustrations of transmission schemes: (a) Independent
transmissions, (b) Combined aggregation and transmissions.

Based on the above intuitions and observations, in the next section, we design
a scheme that aims to minimize both delay and overhead by efficiently
scheduling transmissions during step 4 in a general case given any swarm
topology. Since the UAV network is static during the entire flight time, the
schedule can be applied to other global rounds until the termination
conditions are met.

##  III Proposed Scheduling Scheme

Our proposed scheduling scheme can be designed based on the following
observations. If a node can perform “local aggregation” based on its own local
update and that of other nodes, it can transmit this locally aggregated model
as just one message instead of sending its own one and relaying others
separately. However, each node must determine which local updates it should
aggregate and thus wait to receive them. Therefore, a scheduling scheme is
required for such coordination of model aggregation in the UAV network, which
is the focus of our design. In this section, based on the swarm deployment
assumptions from Section II, we will formally address this through a graph-
based approach.

We consider the swarm as an undirected and unweighted connected graph
$\mathcal{G}=(\mathcal{V},\mathcal{E})$. The set of UAVs $\mathcal{V}$ is
reused as the set of vertices or nodes111Hereinafter, “UAV” and “node” are
used interchangeably., and $\mathcal{E}$ is the set of edges formed as two
nearby UAVs are within the direct communication range of each other. We assume
that $T_{\mathsf{trans}}=\eta\Delta t$ is divided into $\eta$ time slots and
each time slot lasts a fixed duration $\Delta t$. $\Delta t$ is also
considered as the transmission time for a message between two adjacent UAVs.
Let $d(\cdot,\cdot)$ represent the number of hops along the shortest path for
two UAVs. Next, we aim to address the first crucial part in the process of
designing the scheduling scheme, which is the choice of the root node.

###  III-A Root Node Selection and Associated Delay

Because most of the transmission activities in the network are for the
delivery of local updates to the root UAV $v_{c}$, here we specify how to
choose an efficient one. First, we define the node eccentricity of a node $v$
as $\epsilon(v)\triangleq\max_{u\in\mathcal{V}}d(v,u)$, which is the longest
distance in hop count from $v$ to any other node $u$ in the graph. Then, a UAV
$v_{c}\in\mathcal{V}$ is chosen as the root node if
$v_{c}=\operatorname*{arg\,min}_{v\in\mathcal{V}}\\{\max_{v}\\{d(v,u)\\}_{u\in\mathcal{V}}\\}$.
In other words, $v_{c}$ has the minimum eccentricity or the shortest longest
distance from itself to other nodes. We also define this value as the minimum
graph eccentricity, denoted as
$\epsilon(\mathcal{G})\triangleq\min\\{\max_{v}\\{d(u,v)\\}_{u\in\mathcal{V}_{\mathcal{G}}}\\}$.

The proposed choice of root node is based on the following observations.
First, in terms of delay minimization, given any root node $u$, $u$ must wait
for the furthest node to receive its local update. Formally, this amount of
time can be calculated as
$T_{\mathsf{trans}}=\max_{w\in\mathcal{V}}d(u,w)\Delta t=\epsilon(u)\Delta t$.
Then it is clear that to minimize $T_{\mathsf{trans}}$, we should choose the
root node so that it has the shortest longest distance to any other node,
which is exactly the definition of minimum graph eccentricity
$\epsilon(\mathcal{G})$. Then we can formally state that, by the definition of
$v_{c}$, $\min T_{\mathsf{trans}}=\epsilon(\mathcal{G})\Delta
t=\epsilon(v_{c})\Delta t\triangleq T^{*}$, i.e.,
$\min\eta=\epsilon(\mathcal{G})$. This result aids in selecting the root node
$v_{c}$ to obtain the minimum $T_{\mathsf{trans}}$ given an arbitrary swarm
topology. Next, we aim to analyze the communication overhead
$N_{\mathsf{trans}}$.

###  III-B Communication Overhead Analysis

Recall that in step 4, each node has to transmit its local update to a root
node $v_{c}$ for global model aggregation. Since an edge $e\in\mathcal{E}$ is
formed based on the direct communication of two nodes, we assume that each
time a message traverses an edge, it can be counted as a transmission. Then
the number of messages to be transmitted is minimized when each node, except
the root node, only transmits once to a direct neighbor. Formally, we can
state that $\min N_{\mathsf{trans}}=|\mathcal{V}|-1\triangleq N^{*}$. $N^{*}$
can be easily achieved in the setting where all nodes directly transmit their
local update to an aggregator in single-hop networks. However, the swarm forms
a multi-hop network and thus, it requires multi-hop transmission for the model
update delivery. Also as previously discussed, if each node can deliver its
own local update as well as that of the others in one transmission, the
overhead can be reduced. This intuition guides us to the next step, where we
design a transmission scheduling scheme that aims to achieve both $T^{*}$ and
$N^{*}$.

###  III-C Proposed Scheduling Scheme

Our proposed scheduling scheme contains two main parts. The first part, named
Scheduler, is executed once by the root UAV $v_{c}$ at the end of step 1 after
the UAV swarm has identified the root UAV $v_{c}$ chosen based on Section
III-A. Then $v_{c}$ starts to construct a subgraph
$\mathcal{I}=(\mathcal{V},\mathcal{E}_{\mathcal{I}})$ as a breadth-first tree
based on $\mathcal{G}$, and a set $\mathcal{A}$ of
$\eta=\epsilon(\mathcal{I})$ transmission matrices $\mathbf{A}$. In detail,
for every $t$ we define a zero-initialized transmission matrix
$\mathbf{A}^{(t)}$ of size $|\mathcal{V}|\times|\mathcal{V}|$. We also define
the tier $k$ of a node $v$ to be its distance (in hop count) $d(v,v_{c})$ to
the root $v_{c}$. Subsequently, the set of all nodes at the same tier $k$ is
denoted as $\mathcal{V}_{k}$. Then, a tier-$k$ node $i$ is scheduled to
transmit to $j$ if $j$ belongs to the neighbor set
$\mathcal{N}_{\mathcal{I}}(i)$ of $i$ in $\mathcal{I}$ and is one-hop closer
to $v_{c}$, i.e., $j\in\mathcal{V}_{k-1}$, whose scheduled transmission is
indicated by $\mathbf{A}^{(t)}_{i,j}=1$. The constructed $\mathcal{A}$ is then
sent to all other nodes.

The second part is a function NodeAction that is executed in parallel by all
$v\in\mathcal{V}$ at every $t$ during 4. At $t$, $v$ checks to receive other
messages from its higher-tier neighbors (i.e., ones with larger $k$) in a set
$\mathcal{R}_{v}$, as scheduled in $\mathbf{A}^{(t-1)}$ at time $t-1$. As
previously discussed, in our design, a node can combine its local update with
that of others to transmit just one message denoted as
$\mathbf{w}^{\mathsf{aggr}}_{v}$. By combining these updates from
$u\in\mathcal{R}_{v}$ and its own local update $\mathbf{w}_{v}$, node $v$
computes its $\mathbf{w}^{\mathsf{aggr}}_{v}$. Finally, node $v$ sends
$\mathbf{w}^{\mathsf{aggr}}_{v}$ to a lower-tier neighbor based on
$\mathbf{A}^{(t)}$. The complete transmission scheduling algorithm is
presented in Algorithm 1.

Algorithm 1 Proposed Transmission Scheduler

1:$\triangleright$ Scheduler is executed once by $v_{c}$ in step 1.

2:function Scheduler($\mathcal{G}$, $v_{c}$)

3: Construct $\mathcal{I}$ based on $\mathcal{G}$ and $v_{c}$

4: $t$, $\eta$, $\mathcal{A}$ $\leftarrow$ $1$, $\epsilon(\mathcal{I})$,
$\\{\mathbf{A}^{(1)},\dots,\mathbf{A}^{(t)},\dots,\mathbf{A}^{(\eta)}\\}$

5: while $t\leq\eta$ do

6: for each $v$ in $\mathcal{V}_{\eta-t+1}$ do

7: for each $u$ in $\mathcal{N}_{\mathcal{I}}(v)$ do

8: if $u$ is in $\mathcal{V}_{\eta-t}$ then

9: $\mathbf{A}^{(t)}_{v,u}\leftarrow 1$

10: end if

11: end for

12: end for

13: $t$ $\leftarrow$ $t+1$

14: end while

15: return $\mathcal{A}$ to all other nodes

16:end function

17:$\triangleright$ NodeAction is executed by all nodes $v\in\mathcal{V}$ in
parallel at every $t$ during step 4.

18:function NodeAction($t$, $\mathcal{A}$, $\mathcal{G}$)

19: if $t\neq\epsilon(\mathcal{G})-d(v,v_{c})+1$ then return

20: end if

21: $\mathcal{R}_{v}\leftarrow\\{\\}$

22: for row $u$ in $\mathbf{A}^{(t-1)}\in\mathcal{A}$ do

23: if $\mathbf{A}^{(t-1)}_{u,v}==1$ then

24: Add $u$ to $\mathcal{R}_{v}$

25: end if

26: end for

27: Wait to receive all updates from nodes in $\mathcal{R}_{v}$

28: Perform aggregation:
$\mathbf{w}_{v}^{\mathsf{aggr}}\leftarrow\lambda_{v}\mathbf{w}_{v}+\sum_{u\in\mathcal{R}_{v}}\mathbf{w}_{u}^{\mathsf{aggr}}$

29: for column $u$ in $\mathbf{A}^{(t)}\in\mathcal{A}$ do

30: if $\mathbf{A}^{(t)}_{v,u}==1$ then

31: Send $\mathbf{w}_{v}^{\mathsf{aggr}}$ to $u$

32: return

33: end if

34: end for

35:end function

For the proposed algorithm, we have the following results.

######  Proposition 1.

At the end of global round $r$, the proposed transmission scheduling scheme in
Algorithm 1 achieves:

  1. R1

A globally aggregated model equal to (2).

  2. R2

The minimum transmission time $T^{*}$.

  3. R3

The minimum number of transmitted messages $N^{*}$.

###### Proof.

The proof is deferred to Appendix A. ∎

Fig. 3 illustrates how the proposed scheduling scheme works for the swarm
previously given in Fig. 2 In this example, the red edges belong to the
subgraph $\mathcal{I}$. At $t=1$, tier-$2$ nodes in the set
$\mathcal{V}_{2}=\\{v_{2},v_{3},v_{5}\\}$ begin to transmit their updates to
their respective neighbors in $\mathcal{V}_{1}=\\{v_{1},v_{4},v_{6},v_{7}\\}$.
At this point, they do not have any higher-tier neighbors so
$\mathbf{w}_{v_{i}}^{\mathsf{aggr}}=\mathbf{w}_{v_{i}}$, $\forall
v_{i}\in\mathcal{V}_{1}$. At $t=2$, as the nodes in $\mathcal{V}_{1}$ receive
the transmitted messages, they compute and transmit their
$\mathbf{w}_{v_{j}}^{\mathsf{aggr}}$, $v_{j}\in\mathcal{V}_{1}$ based on the
received messages and their own local updates $\mathbf{w}_{v_{j}}$. Finally,
the global model is
$\lambda_{\mathbf{w}_{v_{c}}}\mathbf{w}_{v_{c}}+\sum_{v_{j}\in\mathcal{V}_{1}}\mathbf{w}_{v_{j}}^{\mathsf{aggr}}$
as computed by $v_{c}$. In this example, the delay $T_{\mathsf{trans}}$ is
$\epsilon(\mathcal{I})\Delta t=2\Delta t$ and the overhead
$N_{\mathsf{trans}}$ is $|\mathcal{V}|-1=7$ messages.

Figure 3: Example of proposed transmission scheduler.

###  III-D Complexity Analysis

First, Scheduler begins with the construction of $\mathcal{I}$ with a
complexity of $\mathcal{O}(|\mathcal{V}|+|\mathcal{E}_{\mathcal{G}}|)$. The
computation of $\epsilon(\mathcal{I})$ is based on the breadth-first search
algorithm and thus is
$\mathcal{O}(|\mathcal{V}|^{2}+|\mathcal{V}|\times|\mathcal{E}_{\mathcal{I}}|)$.
Next, the initialization of $\mathcal{A}$ can take
$\mathcal{O}(\eta\times|\mathcal{V}|^{2})$ and because $\eta$ can be bounded
by $|\mathcal{V}|$, this step may take $\mathcal{O}(|\mathcal{V}|^{3})$.
Finally the nested loops also have a complexity of
$\mathcal{O}(|\mathcal{V}|^{3})$, leading to the complexity of the whole
function being
$\mathcal{O}(|\mathcal{V}|^{3}+|\mathcal{V}|\times|\mathcal{E}_{\mathcal{I}}|+|\mathcal{E}_{\mathcal{G}}|)$.
Regarding NodeAction, its initial condition and initialization steps both take
$\mathcal{O}(1)$. The first loop, the aggregation step, and the second step
all take $\mathcal{O}(|\mathcal{V}|)$ time, resulting in the complexity of
$\mathcal{O}(|\mathcal{V}|)$ for the whole function.

##  IV Numerical Results

###  IV-A Experiment Settings

We randomly place different numbers of UAVs over an area of 1000 m $\times$
1000 m so that the UAVs form a connected graph for given values of
$D^{\mathsf{com}}_{\max}=150$ m and a safety distance of 5 m. Note that all
the UAVs participate in the training process of every global round. We
evaluate our proposed scheme using two standard benchmark datasets, namely
MNIST and CIFAR-10, for two classification tasks, whose training process aims
to minimize a negative log-likelihood loss function. We also develop two
simple convolutional neural networks mainly containing several convolutional
and fully connected layers for the two tasks. Note that all the results in
Section IV-C are obtained by running each simulation as a global FL round five
times and reporting the mean values.

###  IV-B Convergence Results

We show the convergence results of our proposed scheme and an FL approach for
UAV swarms introduced by Qu et al. [6] in terms of training loss and accuracy,
as depicted in Figs. 4 and 5. In this scenario, we deploy 100 UAVs in total,
and the batch size and learning rate are set to 10 and 0.01, respectively. For
both independent and identically distributed (IID) and non-IID settings, we
report the training results until the global round where we observe the
convergence. It can be seen that our proposed scheme can take advantage of the
full participation of all UAVs similar to centralized FL, therefore achieving
better convergence speed overall.

Figure 4: Loss and accuracy with MNIST dataset.

Figure 5: Loss and accuracy with CIFAR-10 dataset.

###  IV-C Delay and Overhead Results

In Fig. 6, we compare the delay performance of different strategies of root
node choice, which directly impacts the delay that the swarm needs to complete
the model transmission process. To be specific, “Random” indicates the random
selection of a UAV, “Location-based” is the choice of the node that is closest
to central 2-D locations of the whole network, and “Proposed” indicates the
selection as in Algorithm 1. It is seen that the proposed scheme outperforms
the two baseline schemes regardless of the size of the UAV swarm, especially
at 60 and 80 UAVs. This is because we are always able to select the optimal
root node given arbitrary swarm topologies and sizes.

Figure 6: Number of time slots versus swarm size.

We further investigate the transmission overhead of model messages by
different schemes against the swarm size in Fig. 7. In detail, “Shortest Path
First” (SPF) denotes the shortest-path unicast of a local update by a node to
the root node chosen in Algorithm 1. Compared to the conventional SPF method,
our proposed scheme can effectively save the number of transmissions for a
wide range of swarm sizes. Meanwhile, it is seen that the amount of overhead
by Qu et al. [6] grows exponentially as the number of UAVs increases.
Especially at 400 UAVs, the proposed scheme can save up to 71% and 96% of the
traffic compared to SPF and [6], respectively.

Figure 7: Amount of overhead versus swarm size.

##  V Conclusion

We propose a transmission scheduling algorithm for FL conducted within UAV
swarms. To be specific, we first study the delay and communication overhead
problems in the FL training process and then propose a transmission scheduler
that achieves both minimum delay and overhead. The simulation results conclude
that the proposed scheme outperforms other baseline schemes in terms of
convergence speed, delay, and overhead in the network. Future work may include
the study of how topology variation affects the FL framework.

##  Appendix A Proof of Proposition 1

First, we introduce the following supplemental lemma.

######  Lemma 1.

$\mathcal{I}$ is a minimum spanning tree, i.e.,
$|\mathcal{E}_{\mathcal{I}}|=|\mathcal{V}|-1$, and its minimum graph
eccentricity is the same as $\mathcal{G}$, i.e.,
$\epsilon(\mathcal{I})=\epsilon(\mathcal{G})$.

###### Proof.

First, we prove that $\mathcal{I}$ is a spanning tree of $\mathcal{G}$ and, by
recalling that $\mathcal{G}$ is undirected and unweighted, any spanning tree
is a minimum spanning tree. To be a spanning tree, $\mathcal{I}$ must contain
all the vertices and have no cycle. By [12, Lemma 20.6], applying a breadth-
first search to an undirected and unweighted graph yields a breadth-first tree
$\mathcal{I}$ which includes a unique path from $v_{c}$ to any node in
$\mathcal{G}$ without cycles. Because of this, $\mathcal{I}$ is also a
spanning tree. Next, by applying [12, Theorem 20.5] inductively to
$\mathcal{I}$, every unique path between node $v$ and $v_{c}$ in $\mathcal{I}$
is a shortest path between them in $\mathcal{G}$. It follows that
$\epsilon(\mathcal{G})=\min\\{\max_{v}\\{d(u,v)\\}_{u\in\mathcal{G}}\\}=\max_{v_{c}}\\{d(u,v_{c})\\}_{u\in\mathcal{V}_{\mathcal{G}}}=\max_{v_{c}}\\{d(u,v_{c})\\}_{u\in\mathcal{V}_{\mathcal{I}}}=\epsilon(\mathcal{I})$
since $\mathcal{V}_{\mathcal{G}}=\mathcal{V}_{\mathcal{I}}=\mathcal{V}$ and
$d(u,v_{c})$ in $\mathcal{G}$ is equal to $d(u,v_{c})$ in $\mathcal{I}$ for
all $u\in\mathcal{V}$.

By Lemma 1, it follows that the transmission matrix $\mathbf{A}^{(t)}$ at
iteration $t$ in function Scheduler has exactly $|\mathcal{V}_{\eta-t+1}|$
($\eta=\epsilon(\mathcal{I})$) one-valued elements (or scheduled
transmissions) because there is only one parent at tier $(\eta-t)$ for a
tier-$(\eta-t+1)$ node. Then, we prove R1 as follows. At the end of step 4,
node $v_{c}$ performs aggregation using
$\mathbf{w}^{(r)}=\lambda_{v_{c}}\mathbf{w}_{v_{c}}+\sum_{v\in\mathcal{V}_{1}}\mathbf{w}_{v}^{\mathsf{aggr}}$.
It is noted that for each $v\in\mathcal{V}_{1}$,
$\mathbf{w}_{v}^{\mathsf{aggr}}=\lambda_{v}\mathbf{w}_{v}+\sum_{u\in\mathcal{V}_{2}\cap\mathcal{N}_{\mathcal{I}}(v)}\mathbf{w}_{u}^{\mathsf{aggr}}$.
By replacing each $\mathbf{w}_{v}^{\mathsf{aggr}}$ recursively into
$\mathbf{w}^{(r)}$ and noting that
$\mathcal{V}=\cup_{k=1}^{\epsilon(\mathcal{G})}\mathcal{V}_{k}\cup\\{v_{c}\\}$,
we obtain $\mathbf{w}^{(r)}$ in (2). Next, from Lemma 1, Algorithm 1 yields
$T_{\mathsf{trans}}=\eta\Delta t=\epsilon(\mathcal{I})\Delta
t=\epsilon(\mathcal{G})\Delta t=T^{*}$ by definition, which completes the
proof for R2. Finally, the proof for R3 is given as follows. Given $[\cdot]$
being the Iverson bracket notation, the number of scheduled transmissions
$N_{\mathsf{trans}}=\sum_{t=1}^{\epsilon(\mathcal{G})}\sum_{i,j\in\mathcal{V}}[\mathbf{A}^{(t)}_{i,j}=1]=\sum_{t=1}^{\epsilon(\mathcal{G})}|\mathcal{V}_{t}|=|\mathcal{E}_{\mathcal{I}}|=|\mathcal{V}|-1=N^{*}$
also by Lemma 1 and definition. ∎

## References

  * McMahan et al. [2017] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Proc. Artif. Intell. Stat. (AISTATS)_ , Apr. 2017, pp. 1273–1282.
  * Wang et al. [2021] Y. Wang, Z. Su, N. Zhang, and A. Benslimane, “Learning in the air: Secure federated learning for UAV-assisted crowdsensing,” _IEEE Trans. Netw. Sci. Eng._ , vol. 8, no. 2, pp. 1055–1069, Apr. 2021.
  * Lu et al. [2020] Y. Lu, X. Huang, Y. Dai, S. Maharjan, and Y. Zhang, “Differentially private asynchronous federated learning for mobile edge computing in urban informatics,” _IEEE Trans. Ind. Informat._ , vol. 16, no. 3, pp. 2134–2143, Mar. 2020.
  * Liu et al. [2021] Y. Liu, J. Nie, X. Li, S. H. Ahmed, W. Y. B. Lim, and C. Miao, “Federated learning in the sky: Aerial-ground air quality sensing framework with UAV swarms,” _IEEE Internet Things J._ , vol. 8, no. 12, pp. 9827–9837, Jun. 2021.
  * He et al. [2023] X. He, Q. Chen, L. Tang, W. Wang, and T. Liu, “CGAN-based collaborative intrusion detection for UAV networks: A blockchain-empowered distributed federated learning approach,” _IEEE Internet Things J._ , vol. 10, no. 1, pp. 120–132, Jan. 2023.
  * Qu et al. [2021a] Y. Qu, H. Dai, Y. Zhuang, J. Chen, C. Dong, F. Wu, and S. Guo, “Decentralized federated learning for UAV networks: Architecture, challenges, and opportunities,” _IEEE Netw._ , vol. 35, no. 6, pp. 156–162, Nov. 2021.
  * Xiao et al. [2021] Y. Xiao, Y. Ye, S. Huang, L. Hao, Z. Ma, M. Xiao, S. Mumtaz, and O. A. Dobre, “Fully decentralized federated learning-based on-board mission for UAV swarm system,” _IEEE Commun. Lett._ , vol. 25, no. 10, pp. 3296–3300, Oct. 2021.
  * Qu et al. [2021b] Y. Qu, C. Dong, T. Wu, Y. Zhuang, H. Dai, and F. Wu, “Efficient edge intelligence under clustering for UAV swarm networks,” in _Proc. IEEE Int. Conf. Space-Air-Ground Comput. (SAGC)_ , Oct. 2021, pp. 112–117.
  * Feng et al. [2022] C. Feng, B. Liu, K. Yu, S. K. Goudos, and S. Wan, “Blockchain-empowered decentralized horizontal federated learning for 5G-enabled UAVs,” _IEEE Trans. Ind. Informat._ , vol. 18, no. 5, pp. 3582–3592, May 2022.
  * Li et al. [2020] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of FedAvg on non-IID data,” in _Proc. Intl. Conf. Learn. Representations (ICLR)_ , 2020.
  * Pires et al. [2019] R. M. Pires, A. S. R. Pinto, and K. R. L. J. C. Branco, “The broadcast storm problem in FANETs and the dynamic neighborhood-based algorithm as a countermeasure,” _IEEE Access_ , vol. 7, pp. 59 737–59 757, 2019.
  * Cormen et al. [2022] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, _Introduction to Algorithms_. MIT Press, Apr. 2022.


# Adversarial Perturbation Intensity Achieving Chosen Intra-Technique
Transferability Level for Logistic Regression

Martin Gubri

###### Abstract

Machine Learning models have been shown to be vulnerable to adversarial
examples, ie. the manipulation of data by a attacker to defeat a defender’s
classifier at test time. We present a novel probabilistic definition of
adversarial examples in perfect or limited knowledge setting using prior
probability distributions on the defender’s classifier. Using the asymptotic
properties of the logistic regression, we derive a closed-form expression of
the intensity of any adversarial perturbation, in order to achieve a given
expected misclassification rate. This technique is relevant in a threat model
of known model specifications and unknown training data. To our knowledge,
this is the first method that allows an attacker to directly choose the
probability of attack success. We evaluate our approach on two real-world
datasets.

Machine Learning, Adversarial Learning, Adversarial Examples, Logistic
Regression, Asymptotic

##  1 Introduction

Adversarial examples theory is the study of the strategies of a defender and
an attacker in the following threat model: an adversary has the ability of
modifying an input, noted $x_{0}$ here, with the goal of crafting a new input
$x_{adv}$ that will be misclassified by the defender’s classification model.
The perturbation is noted $\delta_{0}$. Note that the attack happens at test
time. The attacker doesn’t have the ability to alter the integrity of the
model estimation.

$x_{adv}=x_{0}+\delta_{0}$

With $k$ known, untargeted adversarial examples crafting is usually defined by
the following optimization problem:

$\delta_{0}=\arg\min_{\delta}\lVert\delta\rVert_{k}\;\quad\text{s.t.}\quad
f_{d}(x_{0}+\delta;\hat{\theta}_{d})\neq y_{0}$

where $f_{d}\colon\mathcal{X}\times\Theta\to\mathcal{Y},\quad
x\times\theta\mapsto y$ is the classifier (or prediction function) of the
defender’s model, $\hat{\theta}_{d}$ is the defender’s estimate of the model
parameters, $\mathcal{X}$ the input space, $\mathcal{Y}$ the output space, and
$\Theta$ the parameters space. Note that some authors prefer another
definition for adversarial examples (Biggio & Roli, 2017).

Some authors define the optimization problem by using
$f_{d}(x_{0};\hat{\theta}_{d})$ instead of $y_{0}$. We prefer to use $y_{0}$,
because if the original input is already misclassified by the model, ie.
$f_{d}(x_{0};\hat{\theta}_{d})\neq y_{0}$, then $\delta_{0}=0_{p}$, with $p$
the number of features, if $f_{d}$ and $\hat{\theta}_{d}$ are known. This
point has consequences in limited knowledge settings developed in part 4.

To this definition, the attacker may add application specific constraints on
$\delta$. Various have been used in the literature:

  * •

$x_{0}+\delta\in\mathcal{X}$, where $\mathcal{X}$ may be $\mathbb{N}$

  * •

$\lVert\delta\rVert_{k}<d_{\text{max}}$ (Biggio et al., 2013)

  * •

$\forall i,\delta_{i}\geq 0$ (Grosse et al., 2017)

Similar conditions can be derived only on a subset of features. For simplicity
reasons, we will not use any of these conditions in the following. We assume
that $\mathcal{X}=\mathbb{R}^{p}$.

The attacker doesn’t necessary have the same knowledge than the defender.
Knowing the defender’s training data and model specifications, the attacker
can train an exact copy of $f_{d}(\cdot,\hat{\theta}_{d})$. With partial
knowledge of $f_{d}(\cdot,\hat{\theta}_{d})$, the attacker can train a
substitute model to craft adversarial examples from it (Biggio et al., 2013;
Papernot et al., 2016a, 2017). Papernot et al. (2016b) build a typology of
attacks depending on the knowledge that the attacker have on the defender’s
model and on the goal of the attacker. Biggio et al. (2013) detail the
components of the attacker’s knowledge:

  * •

the defender’s training data (completely or only a subset)

  * •

feature representation used by the defender

  * •

the type of learning algorithm and the decision function, that we called model
specification

  * •

the defender’s estimate of the model parameters.

The capability of the adversary may provide extra knowledge on the defender’s
model. The typical example is the case of the attacker having feedback from
the defender’s model (Papernot et al., 2017).

The property of transferability of adversarial examples, defined by the fact
that some adversarial examples designed to fool a specific model also fool
other models, was observed by Goodfellow, Shlens, and Szegedy (2014), Papernot
et al. (2016a), Papernot et al. (2017), among others. Papernot et al. (2016a)
identify two types of transferability: intra-technique transferability and
cross-technique transferability.

The optimal L2-adversarial example for a logistic regression (with perfect
knowledge) is the orthogonal projection of the example onto the decision
hyperplane (Moosavi-Dezfooli et al., 2016). In part 4, we use this technique
to compute adversarial example, but our method can be applied to any
adversarial example crafting technique.

The intuition guiding our work our work is that an optimal adversarial example
for the attacker surrogate model, given the limited knowledge of the attacker,
may not achieve satisfactory intra-technique transferability. If the
adversarial example is very close to the decision hyperplane, a very small
difference between $\hat{\theta}_{d}$ and $\hat{\theta}_{a}$ can lead to
failed attacks.

In part 2, we provide a probabilistic definition of an adversarial example. In
part 3, we recall the asymptotic properties of the logistic regression. In
part 4, we develop a closed-form approximate of an adversarial example having
a chosen expected successful attack rate, in the threat model of known model
specification but unknown training data for binary classification by a
logistic regression. In part 5, we apply our method on 2 datasets.

### Contributions

  * •

We introduce a new probability-based definition of adversarial example having
an arbitrary expected misclassification rate using prior distributions to
formalize the attacker knowledge on the defender’s classifier.

  * •

We make use of the asymptotic distribution of logistic regression parameters
to derive a closed form method to craft adversarial examples having a chosen
expected success attack rate, in a limited knowledge threat model. To our
knowledge, this is the first method to allow an attacker to directly tune the
probability of attack success.

  * •

We show that multiplying by the same scalar all adversarial perturbations of
the test samples computed on the attacker’s surrogate model may not be
effective to improve intra-technique transferability.

  * •

We observe the importance of knowing the estimation method used by the
defender, even for logistic regression.

  * •

We notice that in our setting the choice of the L2-regularization
hyperparameter can be beneficial to the attacker by reducing the variance of
parameters estimates.

##  2 Probabilistic definition of adversarial example ($\alpha$-adversarial
example)

We define an $\alpha$-adversarial example as an adversarial example with an
expected rate of successful attacks of $\alpha$ in an perfect or imperfect
knowledge setting:

$\min_{\delta}\lVert\delta\rVert_{k}\quad\text{s.t.}\\\
\mathbb{P}\left(F_{d}(x_{0}+\delta;\hat{\theta}_{d})\neq
y_{0}\Big{|}F_{d}\sim\mathscr{D}_{F_{d}}\cap\hat{\theta}_{d}\sim\mathscr{D}_{\hat{\theta}_{d}}(F_{d})\right)\geq\alpha$
(1)

where $\alpha\in[0,1]$ is chosen by the attacker.

$F_{d}$ is a random function drawn from the sample space
$\mathcal{F}\subset\mathcal{Y}^{\mathcal{X}}$ of the set of prediction
functions that the defender can use. $\mathscr{D}_{F_{d}}$ is the prior
knowledge of the attacker on feature representation, model type, its
structural specifications (for example, the architecture of a Neural Network),
used by the defender. We consider $\mathcal{X}$ as the space of _raw_ data,
and we include the data preprocessing step into $F_{d}$.
$\mathscr{D}_{\hat{\theta}_{d}}(F_{d})$ captures the prior knowledge on the
defender’s estimates of parameters, training data, estimation methods,
regularization, hyperparameters of the model and of the feature
representation. Then, $\mathscr{D}_{\hat{\theta}_{d}}(F_{d})$ is the joined
prior and hyperprior of the parameters of $F_{d}$. The Data Generating Process
(DGP) $\mathcal{P}_{\theta}(X,Y)$ is parametrized by $\theta$, the vector of
true model parameters and hyperparameters, and of feature representation
hyperparameters. The distribution of $\hat{\theta}_{d}$ is conditioned by
$F_{d}$, because the model (hyper)parameters may vary across model types.

We formalize the knowledge of the attacker using the joined probability
distribution of $F_{d}$ and $\hat{\theta}_{d}$. If the attacker knows
perfectly the true defender’s decision function $f_{d}$, then
$\mathscr{D}_{F_{d}}$ is a deterministic distribution and $\forall
f\in\mathcal{F},\ \mathbb{P}\left(F_{d}=f\right)=1_{f_{d}}(f)$. Instead the
attacker may have only partial knowledge on the attacker model. In practice,
the attacker may know the state-of-the-art models or the industry practices on
a given task. Then, the adversary may be able to draw a probability
distribution on a set of models used by the defender. The attacker may also
draw probability distributions of hyperparameters depending on the method used
by the defender (random search, grid search, etc.).

The attacker might want to estimate $\mathscr{D}_{F_{d}}$ using $\hat{f}_{a}$
his/her estimate of $f_{d}$ and estimate
$\mathscr{D}_{\hat{\theta}_{d}}(F_{d})$ using
$(\hat{f}_{a},\hat{\theta}_{a})$. This remark makes particularly sense if the
attacker has an oracle access to the defender model.

##  3 Recalls of the asymptotic properties of the logistic regression

The logistic regression can be seen as a Generalized Linear Model (GLM) with
Binomial distribution and a logit link111The reader not familiar with the GLM
theory can read McCullagh and Nelder (1989), which is the main book of
reference on GLM but somewhat difficult, or Chapter 15 of Fox (2016) available
there..

A GLM is defined by 3 components (Fox, 2016, p. 379):

  1. 1.

A conditional distribution of the response variable $Y_{i}$ given $X_{i}$,
member of the exponential family distribution. $Y_{i}$ are independent.

  2. 2.

A linear predictor,
$\eta_{i}=\beta_{0}+\beta_{1}X_{i,1}+\beta_{2}X_{i,2}+\cdots+\beta_{p}X_{i,p}$.

  3. 3.

A smooth and invertible link function $g(\cdot)$, $\eta_{i}=g(\mu_{i})$.

Note that the point 1. implies that a GLM is not only a transformation of the
classical linear model using a link function. GLM doesn’t have the hypothesis
of normality of the residuals.

Then, the logistic regression is a special case of GLM with
$Y_{i}\sim\mathcal{B}(m_{i},\pi_{i})$ 222The logistic regression can also be
defined with $Y_{i}$ following a Bernoulli distribution. Then, $\forall
i\in[\\![1;n]\\!],m_{i}=1$. and the logit function as link (which is the
canonical link of the Binomial distribution). Note that $m_{i}$ is known, so
it isn’t a parameter of the model.

The Maximum Likelihood Estimator $\hat{\beta}_{\text{MLE}}$ is asymptotically
normally distributed (Ferguson, 1996, p. 121). It is asymptotically unbiased
with an asymptotic variance-covariance matrix equals to the inverse of the
Fisher information matrix

$\left(X^{\intercal}WX\right)^{-1},$

with $W$ a diagonal matrix of weights defined by
$W:=\text{diag}\left\\{m_{i}\pi_{i}(1-\pi_{i})\right\\}$ (McCullagh & Nelder,
1989, p. 119). $W$ can be estimated by
$\widehat{W}=\text{diag}\left\\{m_{i}\hat{\pi}_{i}(1-\hat{\pi}_{i})\right\\}$.
Note that the same asymptotic property holds when $n$ is fixed and
$m\to\infty$.

The ridge estimator in logistic regression, noted
$\hat{\beta}_{\text{L2},\lambda_{\text{L2}}}$, is a maximum _a posteriori_
(MAP) estimator. Therefore it is asymptotically normal (Ferguson, 1996, p.
140), asymptotically biased and the asymptotic variance-covariance matrix is
given by (Le Cessie & van Houwelingen, 1992):

$\left(X^{\intercal}WX+2\lambda_{\text{L2}}I_{p}\right)^{-1}X^{\intercal}WX\left(X^{\intercal}WX+2\lambda_{\text{L2}}I_{p}\right)^{-1}$

Then, if $n$ is large, we can use the following approximations to estimate the
variance-covariance matrices:

$\begin{split}\widehat{\mathrm{Var}}\left(\hat{\beta}_{\text{MLE}}\right)=&\left(X^{\intercal}\widehat{W}X\right)^{-1}\\\
\widehat{\mathrm{Var}}\left(\hat{\beta}_{\text{L2},\lambda_{\text{L2}}}\right)=&\left(X^{\intercal}\widehat{W}X+2\lambda_{\text{L2}}I_{p}\right)^{-1}X^{\intercal}\widehat{W}X\\\
&\left(X^{\intercal}\widehat{W}X+2\lambda_{\text{L2}}I_{p}\right)^{-1}\end{split}$

##  4 Approximation of $\alpha$-adversarial examples for the logistic
regression

For convenience, in this section we define
$\tilde{x}_{0}:=~{}\begin{pmatrix}1\\\
x_{0}\end{pmatrix}~{}\in~{}\mathbb{R}^{p+1}$ and
$\tilde{\delta}_{0}:=\begin{pmatrix}0\\\
\delta_{0}\end{pmatrix}\in\mathbb{R}^{p+1}$.

We will consider the following threat model:

Perfect knowledge of $f_{d}$

  * •

The defender is using a logistic regression to perform a binary classification
task,

$\forall(x_{0},\beta)\in\mathbb{R}^{p}\times\mathbb{R}^{p+1},\
f_{d}(x_{0},\beta)=\begin{cases}1&\text{if
}\tilde{x}_{0}^{\intercal}\beta>0,\\\ 0&\text{otherwise}.\end{cases}$

  * •

The attacker knows perfectly the defender’s feature representation.

Limited knowledge of $\hat{\theta}_{d}$

  * •

The attacker doesn’t have access to the defender’s training data.

  * •

The attacker has access to some surrogate training data generated by the same
Data Generating Process (DGP) parametrized by $\beta\in\mathbb{R}^{p+1}$.

  * •

The attacker knows the specifications of the logistic regression
(regularization method and hyperparameters, estimation method).

Therefore the defender’s parameters estimation $\hat{\beta}_{d}$ is fixed but
unknown by the attacker. The attacker can compute $\hat{\beta}_{a}$ using
his/her own data.

The goal of the attacker is to find $\delta^{*}$ solving problem 2.

$\delta^{*}=\arg\min_{\delta\in\mathbb{R}^{p}}\lVert\delta\rVert_{k}\;\quad\text{s.t.}\quad\mathbb{P}\left[f_{d}(x_{0}+\delta;\hat{\beta}_{d})\neq
y_{0}\right]\geq\alpha$ (2)

where $k\in\\{1,2\\}$ and $\alpha\in[0,1]$ are chosen by the attacker.

For simplicity purposes, we consider the following suboptimal problem.

First step: The attacker compute an adversarial example for _his/her own
model_. Any adversarial example crafting technique can be used.

$\delta_{0}=\arg\min_{\delta\in\mathbb{R}^{p}}\lVert\delta\rVert_{k}\;\quad\text{s.t.}\quad
f_{d}(x_{0}+\delta;\hat{\beta}_{a})\neq y_{0}$

In the following, we consider the L2-optimal adversarial example ($k=2$) which
is the orthogonal projection of $x_{0}$ on the decision hyperplane
$\mathcal{H}_{a}=\\{x\in\mathbb{R}^{p}\;|\;f_{d}(x;\hat{\beta}_{a})=0\\}$
(Moosavi-Dezfooli et al., 2016). The associated perturbation can be computed
by

$\delta_{0}=-\frac{\tilde{x}_{0}^{\intercal}\hat{\beta}_{a}}{\lVert\hat{\beta}_{\text{a;-0}}\rVert^{2}_{2}}\hat{\beta}_{\text{a;-0}}$

where
$\hat{\beta}_{\text{a;-0}}=\left(\hat{\beta}_{a;1},\hat{\beta}_{a;2},\cdots,\hat{\beta}_{a;p}\right)^{\intercal}$.

Second step: The attacker searches an optimal scalar $\lambda^{*}$, the
intensity of the adversarial perturbation $\delta_{0}$, needed to achieve an
expected misclassification rate on the defender’s model of at least $\alpha$:

$\lambda^{*}=\arg\min_{\lambda\in\mathbb{R}}\lVert\lambda\delta_{0}\rVert_{k^{\prime}}\;\
\text{s.t.}\
\mathbb{P}\left[f_{d}(x_{0}+\lambda\delta_{0};\hat{\beta}_{d})\neq
y_{0}\right]\geq\alpha$

with $k^{\prime}\in\\{1,2\\}$. It can be simplified as

$\lambda^{*}=\arg\min_{\lambda\in\mathbb{R}}\lambda^{2}\;\quad\text{s.t.}\quad\mathbb{P}\left[f_{d}(x_{0}+\lambda\delta_{0};\hat{\beta}_{d})\neq
y_{0}\right]\geq\alpha$ (3)

We denote the $\alpha$-adversarial example:
$x_{adv}^{*}:=x_{0}+\lambda^{*}\delta_{0}$.

Problem 3 can be rewritten as

$\min_{\lambda\in\mathbb{R}}g(\lambda)\enspace\text{s.t.}\enspace
h(\lambda)\geq 0$

with $g\colon\mathbb{R}\to\mathbb{R}^{+},\quad\lambda\mapsto\lambda^{2}$ and
$h\colon\mathbb{R}\to[-1,1],\quad$
$\lambda\mapsto\mathbb{P}\left[f_{d}(x_{0}+\lambda\delta_{0};\hat{\beta}_{d})\neq
y_{0}\right]-\alpha$.

$g$ and $h$ are of class $C^{1}$. Then using the complementary slackness of
the Karush–Kuhn–Tucker conditions, if $\lambda^{*}$ is a local optimum,
$h(\lambda^{*})g^{\prime}(\lambda^{*})=0$ . If $h(\lambda^{*})=0$ the
constraint is said to be saturated, and if $g^{\prime}(\lambda^{*})=0$ it is
not saturated.

###  4.1 Case 1: Constraint saturated, ie.
$\mathbb{P}\left[f_{d}(x_{0}+\lambda\delta_{0};\hat{\beta}_{d})\neq
y_{0}\right]=\alpha$

For convenience, we define the random variable $Z$ as follow:

$\forall\lambda\in\mathbb{R},\quad
Z:=(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\hat{\beta}_{d}$

For large samples,
$\hat{\beta}_{d}\rightsquigarrow\mathcal{N}\left(\mathbb{E}(\hat{\beta}_{d}),\mathrm{Var}(\hat{\beta}_{d})\right)$.

Then, $Z\rightsquigarrow\mathcal{N}\left(\mathbb{E}(Z),\mathrm{Var}(Z)\right)$
with

$\displaystyle\mathbb{E}(Z)$
$\displaystyle=(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\mathbb{E}(\hat{\beta}_{d}),$
$\displaystyle\mathrm{Var}(Z)$
$\displaystyle=(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\mathrm{Var}(\hat{\beta}_{d})(\tilde{x}_{0}+\lambda\tilde{\delta}_{0}).$

The attacker estimates $\mathbb{E}(\hat{\beta}_{d})$ by $\hat{\beta}_{a}$, and
$\mathrm{Var}(\hat{\beta}_{d})$ by $\widehat{\mathrm{Var}}(\hat{\beta}_{a})$
which is computed as explained in part 3.

####  4.1.1 Subcase a: $y_{0}=1$

Using the quantile function of the normal distribution, if $y_{0}=1$:

$\displaystyle\mathbb{P}\left[f_{d}(x_{0}+\lambda\delta_{0};\hat{\beta}_{d})\neq
y_{0}\right]=\alpha$ $\displaystyle\Leftrightarrow\mathbb{P}\left(Z\leq
0\right)=\alpha$
$\displaystyle\Leftrightarrow(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\hat{\beta}_{d}$
$\displaystyle\quad+\sqrt{(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\mathrm{Var}(\hat{\beta}_{d})(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})}\sqrt{2}\operatorname{erf}^{-1}(2\alpha-1)$
$\displaystyle\quad=0$ (4)

Equation 4 is estimated by the attacker by :

$\displaystyle(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\hat{\beta}_{a}$
$\displaystyle\quad+\sqrt{(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\widehat{\mathrm{Var}}(\hat{\beta}_{a})(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})}\sqrt{2}\operatorname{erf}^{-1}(2\alpha-1)$
$\displaystyle\quad=0$
$\displaystyle\Rightarrow\left[(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\hat{\beta}_{a}\right]^{2}$
$\displaystyle\quad-(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\widehat{\mathrm{Var}}(\hat{\beta}_{a})(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})2\left[\operatorname{erf}^{-1}(2\alpha-1)\right]^{2}$
$\displaystyle\quad=0$
$\displaystyle\Rightarrow(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\hat{\beta}_{a}\hat{\beta}_{a}^{\intercal}(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})$
$\displaystyle\quad-(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\widehat{\mathrm{Var}}(\hat{\beta}_{a})(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})2\left[\operatorname{erf}^{-1}(2\alpha-1)\right]^{2}$
$\displaystyle\quad=0$
$\displaystyle\Rightarrow(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})^{\intercal}\left[\hat{\beta}_{a}\hat{\beta}_{a}^{\intercal}-2\left(\operatorname{erf}^{-1}(2\alpha-1)\right)^{2}\widehat{\mathrm{Var}}(\hat{\beta}_{a})\right]$
$\displaystyle\quad(\tilde{x}_{0}+\lambda\tilde{\delta}_{0})=0$
$\displaystyle\Rightarrow\tilde{x}_{0}^{\intercal}A\tilde{x}_{0}+\lambda(\tilde{x}_{0}^{\intercal}A\tilde{\delta}_{0}+\tilde{\delta}_{0}^{\intercal}A\tilde{x}_{0})+\lambda^{2}\tilde{\delta}_{0}^{\intercal}A\tilde{\delta}_{0}=0$

with
$A:=\hat{\beta}_{a}\hat{\beta}_{a}^{\intercal}-2\left(\operatorname{erf}^{-1}(2\alpha-1)\right)^{2}\widehat{\mathrm{Var}}(\hat{\beta}_{a})$.

Then, $\lambda\in\mathbb{R}$ can be computed by solving a second degree
equation. If there are two solutions, we choose the one that satisfy Equation
4. We denote the solution of the second degree equation $\lambda_{\alpha}$.

####  4.1.2 Subcase b: $y_{0}=0$

$\displaystyle\mathbb{P}\left[f_{d}(x_{0}+\lambda\delta_{0};\hat{\beta}_{d})\neq
y_{0}\right]=\alpha$ $\displaystyle\Leftrightarrow\mathbb{P}\left(Z\leq
0\right)=1-\alpha$

$\lambda_{\alpha}$ can be derived similarly to subcase a, replacing $\alpha$
by $1-\alpha$.

###  4.2 Case 2: Constraint not saturated, ie.
$\mathbb{P}\left[f_{d}(x_{0}+\lambda\delta_{0};\hat{\beta}_{d})\neq
y_{0}\right]>\alpha$

In this case, $g^{\prime}(\lambda^{*})=0$. It immediately follows that
$\lambda^{*}=0$.

If $\mathbb{P}\left[f_{d}(x_{0};\hat{\beta}_{d})\neq y_{0}\right]>\alpha$,
then 0 is the global minimum. Otherwise and if $\lambda_{\alpha}$ exists, then
$\lambda_{\alpha}$ is the global minimum, because it is the unique point that
saturates the constraint.

To sum up, the problem 3 can be solved by:

$\lambda^{*}=\begin{cases}0&\text{if
}\mathbb{P}\left[f_{d}(x_{0};\hat{\beta}_{d})\neq y_{0}\right]>\alpha,\\\
\lambda_{\alpha}&\text{otherwise}.\end{cases}$

##  5 Applications

We applied our analysis on 2 datasets: the UCI spambase set and the dogs vs
cats image set. These two datasets are binary classification problems. The
code is available for reproducibility purpose on GitHub and Framagit.

Table 1: Classification accuracies for the 3 estimation methods studied on the
spam dataset.
Estimation method |  Accuracy
in-sample |  Accuracy
out-of-sample
---|---|---
IRLS |  93.11% |  92.61%
Unregularized liblinear |  93.11% |  92.75%
L2-regularized liblinear |  93.14% |  92.75%

###  5.1 Spambase Data Set

The UCI spam dataset is small enough to estimate the logistic regression using
the Iteratively Reweighted Least Squares (IRLS) estimation method, generally
used for GLM, provided by the statsmodels Python module. We also trained an
unregularized and a L2-regularized logistic regression using the Scikit-learn
implementation and the liblinear solver.

The accuracies are pretty similar between estimation methods (Table 1). But
the estimated variance-covariance matrices are very different from the IRLS
estimation and the liblinear ones. It leads to very different intensities to
achieve the same misclassification level for some examples. In Table 2, we can
observe very different values of $\lambda^{*}$ across the 3 estimation methods
studied for a arbitrary example $x_{0}$ in the test set. This is mainly due to
the high difference in the estimations of $\beta_{41}$, because
$\hat{\beta}_{41;\text{IRLS}}=-48.08$, $\hat{\beta}_{41;\text{URLB}}=-3.07$
and
$\widehat{\mathrm{Var}}\left(\hat{\beta}_{41;\text{IRLS}}\right)-\widehat{\mathrm{Var}}\left(\hat{\beta}_{41;\text{URLB}}\right)=1332.74$
whereas the second biggest element-wise difference between the two covariance
matrices in absolute value is $12.56$. It emphasis the importance of knowing
the estimation method used by the defender.

Table 2: Predicted probabilities of being a spam by the attacker model of an
arbitrary test examples $x_{0}$, its original perturbation $x_{0}+\delta_{0}$,
and its intensified perturbation $x_{0}+\lambda^{*}\delta_{0}$, and values of
the intensities $\lambda^{*}$, for the 3 estimation methods studied on the
spam dataset. Note that $y_{0}=1$ and $\alpha=0.95$.
Predicted probabilities and Intensities |  IRLS |  Unregularized liblinear |  L2-regularized liblinear
---|---|---|---
$\mathbb{P}(Y=1|X=x_{0},\hat{\beta}_{a})$ |  99.999986% |  99.999892% |  99.999919%
$\mathbb{P}(Y=1|X=x_{0}+\delta_{0},\hat{\beta}_{a})$ |  49.999606% |  49.999656% |  49.999649%
$\mathbb{P}(Y=1|X=x_{0}+\lambda^{*}\delta_{0},\hat{\beta}_{a})$ |  $1.687831e^{-74}\%$ |  1.004755% |  2.040357%
$\lambda^{*}$ |  12.059536 |  1.333934 |  1.275921

Figure 1: Violin plot of the intensities of perturbations in the test set for
$\alpha=0.9$. The figure is zoomed in for better visualization.

Figure 2: Intensities of perturbations versus misclassification levels for the
same example $x_{0}$ than Table 2.

Table 2 also reports the estimated probabilities of being a spam by the
attacker model for the intensified perturbations of $x_{0}$ across estimation
methods. It insists on the fact that we do not have to confound the estimated
probability of an adversarial example to be in a specific class by the
attacker’s surrogate model, and the probability $\alpha$ of being classified
in a specific class by the defender’s model.

Figure 2 represents the intensity of the adversarial orthogonal perturbation
against misclassification levels from 0 to 1 for the same arbitrary example
$x_{0}$. The intensity associated to the IRLS estimation is higher than the
other two for all values of $\alpha$ higher than 0.5, and it explodes sooner
when $\alpha$ tends towards 1. Notice that the the intensities have different
scale across examples. It confirm our intuition that multiplying all examples
by the same scalar in not the best way to improve intra-technique
transferability.

Figure 1 shows the box plots and the kernel density estimations of the
perturbation intensity in the test set for a fixed misclassification level of
$0.90$ across the estimation methods. The median of intensities computed on
the unregularized model is sightly greater than the one on the L2-regularized
model.

Figure 3: Quantiles of the empirical distributions of the intensities
$\lambda^{*}$ in the test set and out-of-sample accuracy, versus values of the
L2-regularization hyperparameter $\lambda_{\text{L2}}$ (in log-scale). The red
lower bound represents the first decile, the red upper one the last decile,
and the red line the median.

Figure 3 represents the evolution of the intensities of the adversarial
perturbations with respect to the L2-regularization hyperparameter. The
relation is not straightforward. A very strong regularization is beneficial to
the attacker, because it leads to very small parameter variance, which at the
end implies smaller perturbations to achieve the same misclassification level.
A very small values of regularization leads to instability in the intensities,
which is good for the defender. Interestingly, when the regularization is
strong enough to lower the accuracy, it increases the intensity until
saturation of the constraint ($\lambda^{*}=0$). With this exception in mind,
we can globally said that if regularization leads to better estimates in terms
of MSE (Mansson & Shukur, 2011), it is also beneficial to the attacker. Then,
we make the hypothesis that there is a trade-off in the defender’s choice of
L2-regularization hyperparameter between performance and security.

###  5.2 Dogs vs Cats Images

We also applied our results to the Dogs versus Cats images dataset, available
on Kaggle, which is composed of 25000 labeled images of cats and dogs.

We preprocess the images by normalizing the luminance and resizing them to a
squared shape of 64 by 64 pixels. The low resolution is necessary for us,
because the computation of the variance-covariance matrix needs the inversion
of a $p\times p$ matrix. We preserve the aspect ratio by adding gray bars as
necessary to make them square. Even using 64 by 64 pixels images, the
resulting 12288 features are too large for the GLM estimation using IRLS. We
only used the Scikit-learn implementation of logistic regression. We trained a
L2-regularized logistic regression, using the SAG solver, where the
regularization hyperparameter is chosen by grid search of 100 values on 3-Fold
Cross Validation.

Table 3: Classification accuracies the logistic regression trained on the dogs
vs cats images.
| Accuracy
---|---
In-sample | 73.37%
Out-of-sample | 58.07%

The accuracy of our model is poor (Table 3), because the data are not linearly
separable. It clearly overfits our training data.

We choose to perturb 2 squared images from the test set, which are represented
in Figure 5. Original and adversarial images cannot be distinguish by the
human eye. Image 1 is correctly classified as a cat. The intensity of the
perturbation of this image is a increasing function of the misclassification
level (Figure 6(a)): as $\alpha$ increases, the associated adversarial
examples is further away from the decision hyperplane. Image 2 is not
correctly classified as a cat. Then, the perturbation intensity is negative
and is a decreasing function of the misclassification level: a stronger
misclassification implies to be further away from the decision boundary in the
same half-space where is the original example. As seen in Figure 5(b) and
6(b), the value of $\lambda^{*}$ associated to Image 2 for $\alpha=0.75$ is
$0$, because the probability that the original example is misclassified is
$0.88$.

Figure 4: Violin plot of the intensities of perturbations in the test set for
different levels of misclassification. The figure is zoomed in for better
visualization.

(a) Image 1

(b) Image 2

Figure 5: Original image, attacker’s L2-optimal adversarial image, adversarial
images achieving misclassification levels of 0.75, 0.90 and 0.95, and their
associated perturbations, for 2 images from the test set. Image 1 is correctly
classified by our logistic regression and Image 2 is not. Perturbations are
represented in absolute values and multiplied by 100.

(a) Test Image 1

(b) Test Image 2

Figure 6: Intensities of perturbations versus misclassification levels.

We computed the intensities associated to the misclassification levels of
$0.75$, $0.9$ and $0.95$, for each test example. The empirical distributions
of $\lambda^{*}$ grouped by $\alpha$ are represented as violin plots in Figure
4. The intensities are scattered, because of the differences in scales of the
initial perturbations $\delta_{0}$, and the fact that the variance-covariance
matrix of $\beta$ is higher in some directions of $\delta_{0}$ than others.
Moreover, increasing the misclassification level seems to lead to higher
empirical variance of the intensities.

###  5.3 Conclusions, limitations and future work

In this paper, we show a simple way to craft an adversarial example that
achieves an expected misclassification rate in the case of limited knowledge,
in which the attacker knows that the defender uses a logistic regression, but
doesn’t know the defender’s training data. We defined an adversarial example
having an expected misclassification rate of $\alpha$ by the defender, as an
$\alpha$-adversarial example. Using 2 real-world datasets, we show the
importance to compute the intensity of adversarial perturbations at the
individual level: computing an adversarial perturbation on the attacker
surrogate model and applying the same intensity across all perturbations is a
suboptimal strategy to achieve satisfactory intra-technique transferability.

Our method can be used on any adversarial perturbation technique that only
uses the surrogate attacker model without considering the defender’s model.
But it is based on the assumptions that _(i)_ the attacker has a very large
number of training examples, _(ii)_ the defender has a very large number of
training data generated by the same DGP than the defender’s data, and _(iii)_
the specifications (optimization method, regularization, hyperparameters,
etc.) are known. Moreover, to be computationally feasible, the number of
features $p$ cannot be very large, because the computation of the variance-
covariance matrix of the parameters needs the inversion of a $p\times p$
matrix.

Future research may be to:

  * •

Extend our results to multinomial logistic regressions

  * •

Add other penalization methods

  * •

Use finite sample distributions to have a better estimate of the variance-
covariance matrix of the parameters when the number of training data is not
large

  * •

Solve optimally the optimization problem 2 to compute $\delta^{*}$ instead of
the suboptimal solution $\lambda^{*}$

  * •

Handle the additional constraints listed in section 1, like
$x_{0}+\delta\in\mathcal{X}$

  * •

Extend our method to other models that have known asymptotic or finite-sample
parameters distributions

  * •

Evaluate the cross-technique transferability of $\alpha$-adversarial examples

  * •

Extend the method of $\alpha$-adversarial examples to the case of unknown
model, unknown model specification or unknown hyperparameters, but known
distributions of these elements.

## References

  * Biggio & Roli (2017) Biggio, Battista and Roli, Fabio.  Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning.  _arXiv:1712.03141 [cs]_ , December 2017.
  * Biggio et al. (2013) Biggio, Battista, Corona, Igino, Maiorca, Davide, Nelson, Blaine, Šrndić, Nedim, Laskov, Pavel, Giacinto, Giorgio, and Roli, Fabio.  Evasion Attacks against Machine Learning at Test Time.  In _Machine Learning and Knowledge Discovery in Databases_ , Lecture Notes in Computer Science, pp. 387–402. Springer, Berlin, Heidelberg, September 2013.  doi: 10.1007/978-3-642-40994-3˙25.
  * Ferguson (1996) Ferguson, Thomas S.  _A Course in Large Sample Theory_.  Taylor & Francis, July 1996.  ISBN 978-0-412-04371-0.
  * Fox (2016) Fox, John.  _Applied Regression Analysis and Generalized Linear Models_.  SAGE Publications, 2016.  ISBN 978-0-7619-3042-6.
  * Goodfellow et al. (2014) Goodfellow, Ian J., Shlens, Jonathon, and Szegedy, Christian.  Explaining and Harnessing Adversarial Examples.  _arXiv:1412.6572 [cs, stat]_ , December 2014.
  * Grosse et al. (2017) Grosse, Kathrin, Papernot, Nicolas, Manoharan, Praveen, Backes, Michael, and McDaniel, Patrick.  Adversarial Examples for Malware Detection.  In _Computer Security – ESORICS 2017_ , Lecture Notes in Computer Science, pp. 62–79. Springer, Cham, September 2017.  doi: 10.1007/978-3-319-66399-9˙4.
  * (7) Kaggle.  Dogs vs. Cats.  URL https://www.kaggle.com/c/dogs-vs-cats.
  * Le Cessie & van Houwelingen (1992) Le Cessie, S and van Houwelingen, Johannes Hans.  Ridge Estimators in Logistic Regression.  _Applied Statistics_ , 41:191–201, 1992.
  * Lichman (2013) Lichman, M.  _UCI Machine Learning Repository_.  University of California, Irvine, School of Information and Computer Sciences, 2013.  URL https://archive.ics.uci.edu/ml/datasets/spambase.
  * Mansson & Shukur (2011) Mansson, Kristofer and Shukur, Ghazi.  On Ridge Parameters in Logistic Regression.  _Communications in Statistics - Theory and Methods_ , 40(18):3366–3381, September 2011.  doi: 10.1080/03610926.2010.500111.
  * McCullagh & Nelder (1989) McCullagh, P. and Nelder, John A.  _Generalized Linear Models, Second Edition_.  CRC Press, August 1989.  ISBN 978-0-412-31760-6.
  * Moosavi-Dezfooli et al. (2016) Moosavi-Dezfooli, Seyed-Mohsen, Fawzi, Alhussein, and Frossard, Pascal.  DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks.  pp. 2574–2582, 2016.
  * Papernot et al. (2016a) Papernot, Nicolas, McDaniel, Patrick, and Goodfellow, Ian.  Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples.  _arXiv:1605.07277 [cs]_ , May 2016a.
  * Papernot et al. (2016b) Papernot, Nicolas, McDaniel, Patrick, Jha, S., Fredrikson, M., Celik, Z. B., and Swami, A.  The Limitations of Deep Learning in Adversarial Settings.  In _2016 IEEE European Symposium on Security and Privacy (EuroS P)_ , pp. 372–387, March 2016b.  doi: 10.1109/EuroSP.2016.36.
  * Papernot et al. (2017) Papernot, Nicolas, McDaniel, Patrick, Goodfellow, Ian, Jha, Somesh, Celik, Z. Berkay, and Swami, Ananthram.  Practical Black-Box Attacks Against Machine Learning.  In _Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security_ , ASIA CCS ’17, pp. 506–519, New York, NY, USA, 2017. ACM.  doi: 10.1145/3052973.3053009.
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Seabold & Perktold (2010) Seabold, Skipper and Perktold, Josef.  Statsmodels: Econometric and statistical modeling with python.  In _9th Python in Science Conference_ , 2010.