* [41] Z. Li, X. Zhu, Y. Lian, and Q. Sun, “Constructing secure content-dependent watermarking scheme using homomorphic encryption,” in _2007 IEEE International Conference on Multimedia and Expo_. IEEE, 2007, pp. 627–630.
  * [42] A. Acar, H. Aksu, A. S. Uluagac, and M. Conti, “A survey on homomorphic encryption schemes,” _ACM Computing Surveys_ , vol. 51, no. 4, pp. 1–35, Sep 2018.
  * [43] R. L. Rivest, L. Adleman, M. L. Dertouzos _et al._ , “On data banks and privacy homomorphisms,” _Foundations of secure computation_ , vol. 4, no. 11, pp. 169–180, 1978.
  * [44] C. Gentry, “Fully homomorphic encryption using ideal lattices,” in _Proceedings of the forty-first annual ACM symposium on Theory of computing_ , 2009, pp. 169–178.
  * [45] S. Halevi and V. Shoup, “Algorithms in helib,” in _Annual Cryptology Conference_. Springer, 2014, pp. 554–571.
  * [46] K. Laine and R. Player, “Simple encrypted arithmetic library-seal (v2. 0),” _Technical report, Technical report_ , 2016.
  * [47] J. Daemen and V. Rijmen, “Reijndael: The advanced encryption standard.” _Dr. Dobb’s Journal: Software Tools for the Professional Programmer_ , vol. 26, no. 3, pp. 137–139, 2001.
  * [48] K. Lauter, W. Dai, and K. Laine, “Protecting privacy through homomorphic encryption,” 2021.
  * [49] A. Pham, M. Samragh, S. Wagh, and E. Wenger, “Private movie recommendations for children,” in _Protecting Privacy through Homomorphic Encryption_. Springer, 2021, pp. 163–167.
  * [50] T. ElGamal, “A public key cryptosystem and a signature scheme based on discrete logarithms,” _IEEE transactions on information theory_ , vol. 31, no. 4, pp. 469–472, 1985.
  * [51] D. Boneh, E.-J. Goh, and K. Nissim, “Evaluating 2-dnf formulas on ciphertexts,” in _Theory of cryptography conference_. Springer, 2005, pp. 325–341.
  * [52] M. v. Dijk, C. Gentry, S. Halevi, and V. Vaikuntanathan, “Fully homomorphic encryption over the integers,” in _Annual international conference on the theory and applications of cryptographic techniques_. Springer, 2010, pp. 24–43.
  * [53] “Microsoft SEAL (release 4.0),” https://github.com/Microsoft/SEAL, Mar. 2022, microsoft Research, Redmond, WA.
  * [54] L. Sweeney, “k-anonymity: A model for protecting privacy,” _International journal of uncertainty, fuzziness and knowledge-based systems_ , vol. 10, no. 05, pp. 557–570, 2002.
  * [55] R. Agrawal and R. Srikant, “Privacy-preserving data mining,” in _Proceedings of the 2000 ACM SIGMOD international conference on Management of data_ , 2000, pp. 439–450.
  * [56] R. E. Walpole, R. H. Myers, S. L. Myers, and K. Ye, _Probability and statistics for engineers and scientists_. Macmillan New York, 1993, vol. 5.
  * [57] T. Anttal, “Smart home dataset with weather information,” 2019, April, version 1. Retrieved July 05, 2022. [Online]. Available: https://www.kaggle.com/datasets/taranvee/smart-home-dataset-with-weather-information


# Graph-based methods coupled with specific distributional distances for
adversarial attack detection

Dwight Nwaigwe  Univ. Grenoble Alpes, Inria, CNRS, LJK. Grenoble. France
Univ. Grenoble Alpes, Inserm, U1216, Grenoble Institut Neurosciences.
Grenoble. France  Lucrezia Carboni  Univ. Grenoble Alpes, Inria, CNRS, LJK.
Grenoble. France  Univ. Grenoble Alpes, Inserm, U1216, Grenoble Institut
Neurosciences. Grenoble. France  Martial Mermillod  Univ. Grenoble Alpes,
Univ. Savoie Mont Blanc, CNRS, LPNC. Grenoble. France  Sophie Achard  Univ.
Grenoble Alpes, Inria, CNRS, LJK. Grenoble. France  Michel Dojat  Univ.
Grenoble Alpes, Inserm, U1216, Grenoble Institut Neurosciences. Grenoble.
France

###### Abstract

Artificial neural networks are prone to being fooled by carefully perturbed
inputs which cause an egregious misclassification. These adversarial attacks
have been the focus of extensive research. Likewise, there has been an
abundance of research in ways to detect and defend against them. We introduce
a novel approach of detection and interpretation of adversarial attacks from a
graph perspective. For an input image, we compute an associated sparse graph
using the layer-wise relevance propagation algorithm [Bach et al., 2015].
Specifically, we only keep edges of the neural network with the highest
relevance values. Three quantities are then computed from the graph which are
then compared against those computed from the training set. The result of the
comparison is a classification of the image as benign or adversarial. To make
the comparison, two classification methods are introduced: 1) an explicit
formula based on Wasserstein distance applied to the degree of node and 2) a
logistic regression. Both classification methods produce strong results which
lead us to believe that a graph-based interpretation of adversarial attacks is
valuable.

##  1 Introduction

Artificial neural networks (ANN) are known to be prone to misclassifying
carefully perturbed inputs [Goodfellow et al., 2015]. These perturbed inputs,
called adversarial, have been at the forefront of research in the machine
learning community for the past decade. There is a lot of interest in creating
new adversarial detection and defense methods, especially as this has
consequence for a variety of real-world domains that rely on ANN for
classification [Cireşan et al., 2012], [Finlayson et al., 2019], [Xu et al.,
2020].

But among the known methods it is apparent that few of them, as diverse as
they are, study adversarial attacks from a graph theory perspective. The
objective of this paper is the exploration of adversarial attacks using graph-
based methods. Indeed, the ANN structure can be described by a graph. In the
most basic example, if one considers a standard feedforward ANN, then in a
graphical representation, the neurons are associated to vertices/nodes and the
weights between them are associated to edges. One may take this representation
as inspiration for studying ANN from a graph perspective, although we stress
that there is more than one way to obtain a graph from an ANN.

In [Hassabis et al., 2017], the authors provide a survey of the history of
interactions between neuroscience and artificial intelligence and they note
how much of the modern success in artificial intelligence can be traced to the
understanding of or inspiration by biological systems. There is a line of
research in neuroscience that studies the brain using elements of graph theory
[Bullmore and Sporns, 2009], and this provides some motivation for the use of
graph-theoretic approaches to studying ANN. This can be seen as part of a
wider effort in research that seeks to improve artificial intelligence using
bio-inspired ideas called neuromorphic computing. More examples from
neuromorphic computing are [Yang et al., 2023a], [Yang et al., 2022a], [Yang
et al., 2023b], [Yang et al., 2022b]. Bio-inspired methods seek to bring the
benefits of low-power consumption and more efficient performance or
reliability by comparison and inspiration from the human brain.

In this document, we study the detection of adversarial examples using graphs.
Given an input to the neural network, we compute an associated sparse graph.
From this graph, we then use a combination of three indices: the most relevant
outermost edges, the degree of nodes, and an importance measure. These indices
are then used to predict whether the input is adversarial. For this purpose,
we test two very different approaches- a logistic regression and a statistical
test based on the Wasserstein distance applied to the degree of nodes. Lastly,
we interpret the relative strength of attacks through our graph-based
approach. An advantage of the detection methods introduced is that they
include a thresholding step which is non-differentiable, thereby precluding
gradient masking [Papernot et al., 2017] and making it difficult to make
adaptive attacks. As part of our studies we also provide benchmarks.

##  2 Background and related work

There have been some efforts in interpreting ANN in graph-theoretic ways. The
authors of [Zambra et al., 2020] study the existence and properties of motifs,
clusters of neurons in ANN which appear often. In [La Malfa et al., 2022], the
authors interpret ANN as a graph and study how MNIST and CIFAR datasets
exhibit different distributions under defined quantities (e.g. node input
strength, neuron strength). In [Corneanu et al., 2019], a topological study of
ANN is made via its functional graph, a graph obtained from correlations of
neurons. Other work [Mocanu et al., 2016],[Liu et al., 2020],[Naitzat et al.,
2022] apply a similar topological view to studying ANN. Despite relating
graphs to ANN, none of these articles demonstrate how to use graphs to detect
adversarial examples, nor do they provide statistics on detection. An
interesting use of graphs occurs in [Cheng et al., 2021] where they are used
to evaluate the robustness of an ANN, as opposed to adversarial detection. In
[Ma et al., 2018] (“LID”), [Lee et al., 2018], [Harder et al., 2021], and
[Feinman et al., 2017], logistic regression is used to classify an input as
benign or adversarial based on certain features, none of which are graph
related. The Local Intrinsic Dimensionality (LID) of a sample is introduced in
[Michael, 2017] and is motivated by the growth of the volume of a ball as a
function of radius and dimension. In [Michael, 2017], an analogy is made by
replacing volume with a cumulative probability distribution. The premise of
LID is that adversarial examples exhibit significantly greater LID scores than
do benign examples. Given a sample input to a pretrained neural network, the
LID method begins by computing the neural network’s activations. For each
activation, an approximation to its LID score is made, and the result is given
to a logistic classifier to determine if the sample input is benign or
adversarial. Statistical approaches to adversarial detection can be found in
[Drenkow et al., 2022] (“RSA”) and [Roth et al., 2019], also neither of which
use graph methods. In Random Subspace Analysis (RSA) [Drenkow et al., 2022],
projections are used to lower the dimensionality of the space of the
pretrained neural network’s activations before classifying an input as benign
or adversarial. For each layer of the pretrained neural network, a set of
approximate projection matrices are created to project the dimension of each
layer onto a much smaller space. Then, class protoypes are defined for each
class by obtaining the average of each neuron’s activations. For a given input
to the neural network that one wishes to classify as benign or adversarial,
the following is done: for a given layer and for each projection matrix,
compute the distance in projected space between the class prototype and the
sample input’s activations; record the class that minimizes this difference
and add it to a set. After applying this procedure for each layer, one obtains
a set of sets where the first set is indexed by layer and the second is
indexed by the projection matrix. From this set of sets, a type of voting
procedure is done and if the result is less than a threshold, the sample input
is labeled adversarial. Our methods extend and complement the previous methods
by showing the power of graph theory perspectives applied to studying
adversarial attacks, from either a logistic regression or a pure statistics
perspective. We also compare our methods with LID and RSA.

##  3 Graph generation and quantities of interest

To compute the associated directed graph $\mathcal{G}$ for a neural network
and input pair, we use layer-wise relevance propagation [Bach et al., 2015],
[Montavon et al., 2019]. This algorithm produces a saliency map, but its
intermediate steps allows one to assign quantities to neurons which can be
interpreted as an indicator of the influence that a neuron has on the output.
We begin by providing a brief overview of layer-wise relevance propagation.
Following the notation in [Montavon et al., 2019] for the LRP-$\alpha\beta$
rule, signals are propagated from the output layer towards the input layers.
For a neuron $k$ in layer $\ell+1$ that is connected to neuron $i$ in layer
$\ell$, the propagated signal from $k$ to $i$ is defined to be

$R_{i,k}^{\ell,\ell+1}=R_{k}^{\ell+1}\left(\alpha\frac{a_{i}\max(w_{ik},0)}{\epsilon+\sum_{h}a_{h}\max(w_{hk},0)}-\beta\frac{a_{i}\min(w_{ik},0)}{\epsilon+\sum_{h}a_{h}\min(w_{hk},0)}\right)$
(1)

where $R_{k}^{\ell}$ is the relevance of neuron $k$, $a_{i}$ is the activation
of neuron $i$ in layer $\ell$; $w_{hk}$ is the weight between neurons $h,k$;
$\epsilon$ is a small parameter; and $\alpha-\beta=1$. The relevance of a
neuron $k$ in layer $l$ is given by

$R_{k}^{\ell}=\sum_{i}R_{k,i}^{\ell,\ell+1}.$ (2)

Equations (1) and (2) are used together recursively. To start the algorithm,
one assigns the relevance of the output neurons of the neural network to be
equal to the neural network output. Upon completion of the algorithm, we rank
the edge relevance scores $\\{R_{i,k}^{\ell,\ell+1}\\}$ in descending order
and keep the top 1%. Our thresholding is inspired by [Bullmore and Sporns,
2009], [Sporns, 2013]. In these works, the authors only keep the strongest
connections. The edges that are kept become the edges in our induced graph
$\mathcal{G}$. One can compute various quantities from $\mathcal{G}$. One such
quantity is given by

$I(v_{i})=\sum_{j\neq i}\frac{1}{2^{d(v_{i},v_{j})}}$ (3)

where $\\{v_{i}\\}_{i}$ is the set of nodes and $d(v_{i},v_{j})$ is the
distance between vertices $v_{i},v_{j}$. We note that the distance between
adjacent nodes is given by (1) and the distance between any pair of nodes is
given by the path that corresponds to the shortest sum of distances of
adjacent nodes. An intuitive meaning of (3) is that more importance is given
to a vertex that has many neighbors and short distances to them. This equation
is inspired by closeness centrality [Bavelas, 1950] which is given by

$C(v_{i})=\frac{1}{\sum_{j\neq i}d(v_{i},v_{j})}.$ (4)

A difference between (3) and (4) is that the former is monotone in the
cardinality of $\\{v_{i}\\}_{i}$. For bipartite graphs, or “stacks” of
bipartite graphs (one can think of multi-layer perceptrons) a measure of
closeness centrality tends to be not useful, hence the motivation for (3).

Another quantity of interest is the degree of a vertex, which is defined to be
the difference between out degree and in degree:

$\deg(v)=\deg_{out}(v)-\deg_{in}(v).$ (5)

Our last quantity of interest are the values of certain edges of
$\mathcal{G}$. This allows us to incorporate some of $\mathcal{G}$’s topology.
The edges we use are those that correspond to the last two layers of the
original neural network. We only use these edges because using all edges would
require a data structure of size $O(n_{1}n_{2},...n_{l})$, where $n_{i}$ is
the number of nodes in layer $i$. Clearly, this requires an extensive amount
of memory when $n_{i}$ is large. One can see that in general, when using graph
data, it is preferable, at least from a memory standpoint, to use a quantity
whose size is much smaller than $O(n_{1}n_{2},...n_{l})$, for instance a
dataset whose size is $O(|V|)$, where $V$ is the set of nodes. In fact, our
use of degree and node importance as computed for each node meets this
constraint.

In [Guo et al., 2022], the neurons just before the softmax layer are studied,
which has a similarity with our study of outermost edge relevance. In that
article, the authors use the said neurons to compare robustness of non-human
primate and human vision with regards to adversarial images. This lends a
further (biological) motivation to our use of outermost edge relevance. Since
we apply a threshold to the edges of $\mathcal{G}$, there are nodes of
$\mathcal{G}$ which are not adjacent to an edge. More generally, the edges
among the set $\\{\mathcal{G}_{i}\\}_{i}$ need not be the same, where
$\\{\mathcal{G}_{i}\\}_{i}$ represents a set of graphs induced from the same
architecture. To enforce consistency of representation for the outermost edge
relevances we create a weighted adjacency matrix of the same dimension as the
adjacency matrix for nodes in the last two layers. The edge relevance values
that are above the threshold are recorded as is and those below this
percentile are set to 0. The matrix is then flattened into a vector. This
flattened vector is our third quantity of interest, and its nonzero components
are given by (1), assuming the component is greater than the threshold.

Lastly, we note that it would be very difficult to create an adaptive attack
to counter the methodology proposed here since our detection methods involve
graph thresholding, a non-differentiable operation.

Table 1: Summary of relevant graph statistics.

formula | name
---|---
$R_{i,k}^{\ell,\ell+1}=\text{threshold}\left(R_{k}^{\ell+1}\left(\alpha\frac{a_{i}\max(w_{ik},0)}{\epsilon+\sum_{h}a_{h}\max(w_{hk},0)}-\beta\frac{a_{i}\min(w_{ik},0)}{\epsilon+\sum_{h}a_{h}\min(w_{hk},0)}\right)\right)$ | outermost edge relevance
$I(v_{i})=\sum_{j\neq i}\frac{1}{2^{d(v_{i},v_{j})}}$ | node importance
$\deg=\deg_{out}(v)-\deg_{in}(v)$ | degree

##  4 A statistical test based on Wasserstein distances

The Wasserstein-1 distance between two probability distributions $p$ and $q$
defined on a measurable metric space $\mathcal{X}$ is given by

$\mathcal{W}(p,q)=\min_{\pi(x,y)\in\Pi}\int\left\lVert
x-y\right\rVert_{1}d\pi(x,y)$ (6)

where $\Pi$ is the set of all measures on $(\mathcal{X},\mathcal{X})$ whose
marginal distributions are given by $p$ and $q$. In the case when $p$ consists
of one sample $x$ and $q$ consists of discrete samples $(y_{i})_{i=1}^{N}$,
then

$\mathcal{W}(\delta_{x},q)=\frac{1}{N}\sum_{i}^{N}\|x-y_{i}\|_{1}.$ (7)

where $\delta_{x}$ is the distribution with support at $x$. Wasserstein
distances have been applied to machine learning in several ways. In [Cherian
and Aeron, 2020], Wasserstein distances are used to compress data into a small
dimensional subspace while maintaining a large distance from adversarial
distributions. Other work [Wong et al., 2019] uses Wasserstein distances to
create adversarial attacks.

Our goal in using Wasserstein distances is different than those in the
examples mentioned. Our goal is to apply Wasserstein differences for benign
and adversarial graph statistics in order to classify an input as benign or
adversarial. The statistic we are concerned with is degree.

Let $\hat{\mathcal{B}}_{i}$ denote the empirical distribution of degree in the
case when benign inputs are correctly classified as belonging to class $i$.
Similarly, let $\hat{\mathcal{A}}_{i}$ denote the empirical distribution that
corresponds to perturbed inputs which the model incorrectly classifies as
belonging to class $i$, and whose unperturbed image is correctly classified.
Note that since we are concerned with degree, the domain of the distribution
function $\hat{\mathcal{B}}_{i}$ is a vector whose dimension is equal to the
number of nodes in the induced graphs. If for some input, the model outputs
class $i$, we would like to know if the output was generated by a random
variable with distribution $\mathcal{B}_{i}$ or with distribution
$\mathcal{A}_{i}$ where the lack of a hat denotes the true distribution.As
before, we first construct the graph $\mathcal{G}$ for the sample and compute
a sample degree vector, which we denote by the random variable $\mathbf{Z}$.
For a yet to be defined subset of nodes $\mathcal{S}$, we define the following
Wasserstein Sums Ratio (WSR) quantity:

$\text{WSR}(\mathcal{S},\hat{\mathcal{A}}_{i},\hat{\mathcal{B}}_{i},\mathbf{Z},i)=\frac{\sum_{j\in\mathcal{S}}\mathcal{W}(\delta_{\mathbf{Z}_{j}},\hat{\mathcal{B}}_{i}^{j})}{\sum_{j\in\mathcal{S}}\mathcal{W}(\delta_{\mathbf{Z}_{j}},\hat{\mathcal{A}}_{i}^{j})}$
(8)

where the $j$ in $\hat{\mathcal{A}}_{i}^{j}$ refers to the empirical
distribution for node $j$, and similarly for $\hat{\mathcal{B}}_{i}^{j}$.
Equation (8) says that for each node that belongs to $\mathcal{S}$, we compute
Wasserstein-1 distances node-wise from the sample to the empirical
distributions and we sum over the node indices and compute the ratio. If the
ratio is less than some threshold, we classify the input as benign, otherwise
as adversarial. It may occur that the denominator of (8) is equal to 0, thus,
in this case, a small term is added to the numerator and denominator. This can
happen if the empirical distributions
$\\{\hat{\mathcal{A}}_{i}^{j}\\}_{j\in\mathcal{S}}$ only have support at a
point. Lastly, we note that we could have also computed the Wasserstein
distance in $\mathbb{R}^{N}$, where $N$ is the number of nodes in
$\mathcal{G}$. However, that is a more involved procedure. Using (7), we can
write (8) as

$\text{WSR}(\mathcal{S},\hat{\mathcal{A}},\hat{\mathcal{B}},\mathbf{Z},i)=\frac{\frac{1}{N_{\hat{\mathcal{B}}_{i}^{j}}}\sum_{j\in\mathcal{S}}\sum_{k=1}^{N_{\hat{\mathcal{B}}_{i}^{j}}}\|\mathbf{Z}_{j}-y_{i}^{j}(k)\|_{1}}{\frac{1}{N_{\hat{\mathcal{A}}_{i}^{j}}}\sum_{j\in\mathcal{S}}\sum_{k=1}^{N_{\hat{\mathcal{A}}_{i}^{j}}}\|\mathbf{Z}_{j}-x_{i}^{j}(k)\|_{1}}$
(9)

where $y_{i}^{j}(k)$ is a sample from $\hat{\mathcal{B}}_{i}^{j}$ and
$x_{i}^{j}(k)$ is a sample from $\hat{\mathcal{A}}_{i}^{j}$, and
$N_{\hat{\mathcal{B}}_{i}^{j}}$ is the number of samples in
$\hat{\mathcal{B}}_{i}^{j}$, respectively for $\hat{\mathcal{A}}_{i}^{j}$.
Lastly, we make the set $\mathcal{S}$ as follows: we calculate

$\Delta_{i}^{j}:=\mathbb{E}X_{i}^{j}-\mathbb{E}Y_{i}^{j}$ (10)

where $X_{i}^{j}$ has distribution $\hat{\mathcal{A}}_{i}^{j}$ and $Y_{i}^{j}$
has distribution $\hat{\mathcal{B}}_{i}^{j}$ and $\mathbb{E}$ is expected
value. We then create the set

$\mathcal{S}:=\\{j:\Delta_{i}^{j}<0\ \text{for all}\ i\\}.$ (11)

The set $\mathcal{S}$ identifies nodes where the mean of the benign
distribution is greater than the adversarial distribution for all classes.
Should it happen that $\hat{\mathcal{A}}_{i}^{j}$ is empty for some $j$ (we
have experienced this only for one combination of model and attack), one may
create a placeholder version of it by setting each entry to a very large
negative value (the large negative value has the effect of removing the index
$j$ from consideration when making the set $\mathcal{S}$). Algorithm 1 shows
adversarial detection using WSR.

Algorithm 1 Adversarial detection using WSR (variant 1)

Input: neural network $\mathcal{NN}$, $\text{image}\ I$; ${\tau}$,
$\mathcal{S}$ , $\hat{\mathcal{A}}_{i}^{j}$; $\hat{\mathcal{B}}_{i}^{j}$ for
all $i$ and $j$

$i\leftarrow\mathcal{NN}(I)$

compute $\mathcal{G}$ from $I$ and $\mathcal{NN}$

compute node degree $\mathbf{z}$ from $\mathcal{G}$

$val\leftarrow\text{WSR}(\mathcal{S},\hat{\mathcal{A}},\hat{\mathcal{B}},\mathbf{z},i)$

if $val$ <$\tau$ then classify $I$ as benign, otherwise classify $I$ as
adversarial.

The way we construct $\mathcal{S}$ has the tendency to pick nodes that
generalize well across all classes at the expense of nodes that specialize. In
an alternative algorithm, we propose to use the specialized nodes. For a given
output that is classified as class $i$, we use
$\mathcal{S}_{i}=\\{j:\Delta_{i}^{j}<0\\}$. This can result in a more accurate
test using our approach, but at the expense of a little longer computation
since there are more nodes to use for computations. The algorithm is shown in
Algorithm 2.

Algorithm 2 Adversarial detection using WSR (variant 2)

Input: neural network $\mathcal{NN}$, $\text{image}\ I$; ${\tau_{i}}$,
$\mathcal{S}_{i}$ , $\hat{\mathcal{A}}_{i}^{j}$; $\hat{\mathcal{B}}_{i}^{j}$
for all $i$ and $j$

$i\leftarrow\mathcal{NN}(I)$

compute $\mathcal{G}$ from $I$ and $\mathcal{NN}$

compute node degree $\mathbf{z}$ from $\mathcal{G}$

$val\leftarrow\text{WSR}(\mathcal{S}_{i},\hat{\mathcal{A}},\hat{\mathcal{B}},\mathbf{z},i)$

if $val$ <$\tau_{i}$ then classify $I$ as benign, otherwise classify $I$ as
adversarial.

##  5 Consistency

We would like to analyze under what conditions (8) is a faithful predictor. We
treat the case of a finite-width ANN with sufficiently many neurons. A finite-
width ANN has the property that the degree distribution has compact support,
which implies that the Wasserstein distance between an empirical degree
distribution and true distribution is bounded, and the Wasserstein distance is
continuous with respect to $\|\cdot\|_{\infty}$. We begin our proof of
consistency by showing that given a real-valued random variable $X$; an
empirical distribution $\hat{F}_{n}$ of some other real-valued random variable
with true distribution $F$; a function $G$ (whose arguments are a random
variable and a distribution) that is uniformly continuous in the second
argument with respect to $\|\cdot\|_{\infty}$; and bounded, that

$\mathbb{E}_{X}G(X,\hat{F}_{n})\xrightarrow{a.s.}\mathbb{E}_{X}G(X,F)$ (12)

as $n\to\infty$. To prove (12), it is sufficient to show that

$G(X,\hat{F}_{n})\xrightarrow{a.s.}G(X,F)\ \forall{x}.$ (13)

Under identical and independently distributed (iid) assumptions, the Glivenko-
Cantelli lemma states that
$\|\hat{F}_{n}-\hat{F}\|_{\infty}\xrightarrow{a.s.}0$. This combined with the
uniform continuity of $G$ in the second argument with respect to
$\|\cdot\|_{\infty}$ proves (13). To prove (12), we let
$h_{n}(x)=G(x,\hat{F}_{n})$ and $h(x)=G(x,F)$. From (13) we have
$h_{n}(x)\xrightarrow{a.s.}h(x)$ for all $x$ as $n\to\infty$. We may combine
this with the boundedness assumption to use the Lebesgue dominated convergence
theorem, resulting in
$\lim_{n\to\infty}\mathbb{E}_{X}h_{n}(X)=\mathbb{E}_{X}\lim_{n\to\infty}h_{n}(X)=\mathbb{E}_{X}h(X)$
almost surely.

We now begin to analyze (8), and we start by supposing that our random
variable $\mathbf{Z}$ corresponds to the benign case. Let

$\displaystyle\begin{split}U_{j,i}^{b}&=\mathcal{W}(\delta_{\mathbf{Z}_{j}},\hat{\mathcal{B}}_{i}^{j})\\\
U_{j,i}^{a}&=\mathcal{W}(\delta_{\mathbf{Z}_{j}},\hat{\mathcal{A}}_{i}^{j}).\end{split}$
(14)

For additional simplicity, let us assume that quantities defined in (14) are
iid over the index $j$. The iid assumption implies that

$\mathbf{E}_{\mathbf{Z}_{j}}\mathcal{W}(\delta_{\mathbf{Z}_{j}},\hat{\mathcal{B}}_{i}^{j})=:\mathbf{E}U_{i}^{b}$

and

$\mathbf{E}_{\mathbf{Z}_{j}}\mathcal{W}(\delta_{\mathbf{Z}_{j}},\hat{\mathcal{A}}_{i}^{j})=:\mathbf{E}U_{i}^{a}$

for all $i$. By equation (12), the results we obtain going forward will hold
for the population distribution in high probability assuming our empirical
distributions have enough samples. By the weak law of large numbers,

$\left|\frac{\sum_{j=1}^{|\mathcal{S}|}\mathcal{W}(\delta_{\mathbf{Z}_{j}},\hat{\mathcal{B}}_{i}^{j})}{|\mathcal{S}|}-\mathbf{E}U_{i}^{b}\right|<\epsilon_{1}\
\text{as}\ |\mathcal{S}|\to\infty$

Similarly,

$\left|\frac{\sum_{j=1}^{|\mathcal{S}|}\mathcal{W}(\delta_{\mathbf{Z}_{j}},\hat{\mathcal{A}}_{i}^{j})}{|\mathcal{S}|}-\mathbf{E}U_{i}^{a}\right|<\epsilon_{2}\
\text{as}\ |\mathcal{S}|\to\infty.$

Then (8) is equal to

$\displaystyle\frac{\sum_{j=1}^{|\mathcal{S}|}U_{j,i}^{b}}{\sum_{j=1}^{|\mathcal{S}|}U_{j,i}^{a}}$
$\displaystyle=\frac{|\mathcal{S}|\mathbf{E}U_{i}^{b}+|\mathcal{S}|\epsilon_{1}}{|\mathcal{S}|\mathbf{E}U_{i}^{a}+|\mathcal{S}|\epsilon_{2}}$
$\displaystyle=\frac{\mathbf{E}U_{i}^{b}+\epsilon_{1}}{\mathbf{E}U_{i}^{a}+\epsilon_{2}}$
$\displaystyle\to\frac{\mathbf{E}U_{i}^{b}}{\mathbf{E}U_{i}^{a}}\ \text{as }\
|\mathcal{S}|\to\infty$ (15)

where $\epsilon_{1}$ and $\epsilon_{2}$ are $o(|\mathcal{S}|)$. If we consider
the case when $\mathbf{Z}$ is adversarial, we get a similar limit as in (5).
Thus for consistency, we need the two limits to not be equal, thus we write

$\frac{\mathbf{E}U_{i}^{b}}{\mathbf{E}U_{i}^{a}}<\frac{\mathbf{E}V_{i}^{b}}{\mathbf{E}V_{i}^{a}}$
(16)

where we use $V$ to denote adversarial quantities. This is equivalent to
$\mathbf{E}U_{i}^{b}\ \mathbf{E}V_{i}^{a}<\mathbf{E}U_{i}^{a}\
\mathbf{E}V_{i}^{b}$. This is a realistic assumption for distributions with
different means. A classification threshold, $\tau$, is then picked such that

$\frac{\mathbf{E}U_{i}^{b}}{\mathbf{E}U_{i}^{a}}<\tau<\frac{\mathbf{E}V_{i}^{b}}{\mathbf{E}V_{i}^{a}}.$
(17)

An interesting example of (16) is the case in which
$\mathbf{E}U_{i}^{b}=\mathbf{E}V_{i}^{a}$ and
$\mathbf{E}U_{i}^{a}=\mathbf{E}V_{i}^{b}$ and where all terms do not equal 1.
In this instance, (8) in the benign case will be the inverse of that in the
adversarial case. Furthermore, neither ratio will equal 1. This happens when
adversarial distributions are simply shifts of benign distributions.

##  6 Experimental details

###  6.1 Datasets

We train our models on MNIST, CIFAR-10, and SVHN datasets. For each model we
create adversarial examples using the Adversarial Robustness Toolbox [Nicolae
et al., 2018]. For CIFAR-10 and SVHN, all images were enlarged to (224, 224,
3). Images are preprocessed using built-in Keras layers that handle input
preprocessing.

###  6.2 Architectures

We experiment with five models, two of which are detailed in Tables 2-3 while
the other three are VGG-19, InceptionResnetV2 and MobileNet. The last layers
of VGG-19, InceptionResnetV2, and MobileNet are preloaded from Keras, and
their last layers are replaced with three custom, fully-connected layers, with
output sizes 4096, 1000, and 10, respectively, and trained with ImageNet
weights. With respect to these three models, we only compute graph-based
quantities from these layers to keep the run-time reasonable. For models 1 and
2, we use all layers.

###  6.3 Hyperparameters

The values of $\epsilon$ and $\alpha$ in our implementation of
LRP-$\alpha\beta$ are 2 and $10^{-7}$, respectively. In our implementation of
RSA we use $M=8,K=16$, and the layer used is the third from the output layer.
For creating noisy samples in the algorithm in LID, we use Gaussian noise of
zero mean and variance 0.05. Also in our implementation of LID, we only use
the last 10 layers for computational ease.

###  6.4 Attacks

We consider the fast gradient sign method, [Goodfellow et al., 2015],
projected gradient descent [Madry et al., 2018], untargeted Carlini-Wagner L2
[Carlini and Wagner, 2017], DeepFool [Moosavi-Dezfooli et al., 2016], Square
[Andriushchenko et al., 2020], and Auto [Croce and Hein, 2020] attacks. Fast
gradient sign method attacks are clipped when perturbations are outside a ball
of radius 10% in the $\ell^{\infty}$ norm. Projected gradient descent attacks
are crafted using the same norm but with up to a 5% perturbation; the number
of iterations was 40 except for InceptionResnetV2, MobileNet, and VGG19, in
which 10 were used. Square and Auto attacks have the same norm and
perturbation as projected gradient descent attacks. Optimization is done using
ADAM with learning rate 0.01. For each attack we generate 10,000 adversarially
perturbed images from 10,000 original (test data) images. In creating training
data for the detection methods we introduce, approximately 14,000 samples were
used, and the methods are compared on approximately 6,000 samples. For RSA the
numbers are approximately the same. For LID, we use approximately 6,000
training and test samples each, with the exception of models 1 and 2 in which
we use approximately 7,000 training and 3,000 test samples as these models
were simpler and thus enabled faster computations.

Table 2: Architecture of Model 1

layer type | output size | activation function
---|---|---
fully connected | 300 | reLu
fully connected | 200 | reLu
fully connected | 150 | reLu
fully connected | 150 | reLu
fully connected | 100 | sigmoid
fully connected | 10 | softmax

Table 3: Architecture of Model 2

layer type | output size | activation function
---|---|---
conv | 3 filters, kernel size (4,4) | identity
maxpool | pool size=(2,2), strides=(2,2) | reLu
conv | 3 filters, kernel size (4,4) | identity
maxpool | pool size=(2,2), strides=(2,2) | reLu
fully connected | 100 | reLu
fully connected | 10 | softmax

##  7 Results and Discussion

###  7.1 Comparison of logistic regression approaches

In Tables LABEL:table:benchmarking-logreg-cifar10, LABEL:table:benchmarking-
logreg-svhn,and LABEL:table:benchmarking-logreg-mnist we report the
specificity (percentage benign samples that are correctly detected) and
sensitivity (percentage adversarial samples that are correctly detected). One
can see that the various graph statistics considered here can be strong,
sensitive, and specific predictors of adversarial attacks in the case of using
logistic regression. Among Mobilenet, InceptionResnetV2 and VGG19, the three
graph-based statistics perform more or less the same in their classification
of samples as benign or adversarial and they generally perform well. For
instance, when MobileNet is given SVHN data that is attacked by Square, the
specificity and sensitivity are 99.66% and 99.04%, respectively. From the
tables, we see that the worst performance tends to occur under Carlini-Wagner
and Deepfool attacks. These two attacks are believed to be among the most
difficult to detect, so our results are consistent with this assumption. In
particular, when VGG19 is given SVHN data that is attacked by Carlini-Wagner,
the results show that our logistic regression models are good at detecting
benign samples, but mostly fail to catch the adversarial ones. To be more
concrete, when considering degree, the specificity and sensitivity are 97.67%
and 29.68%, respectively. For node importance, the values are 98.84% and
9.68%, respectively. When using outermost edge relevance, the corresponding
values are 100% and 0%, meaning that our detector erroneously classifies all
input as benign; apparently in this case, the differences between outermost
edge relevances in the benign and adversarial cases are too similar.

Among Model 1 and 2, degree is a significantly better predictor than node
importance and outermost edge relevance, while outermost edge relevance for
Model 2 is a poor predictor across all attacks, being unable to detect
adversarial images. This is because the outermost edge relevance for benign
and adversarial samples are thresholded to 0. The largest relevances for Model
2 are found in layers closer to the input layer.

In comparison to LID, our results are superior across almost all model/attack
combinations. The tables show that for LID, the accuracy averaged between
benign and adversarial detection rates is around 50% which is much less than
than what we achieve using our graph-based statistics. We note that [Drenkow
et al., 2022] also reports similar numbers for implementation of LID, thereby
concurring with our results.

Lastly, in Figure 1, we include sample data for the three graph statistics
introduced in Section 3. Note that because there are thousands of nodes, we
only choose one node to show.

Table 4: Comparison between logistic regression methods. First and second
quantities in each entry are benign and adversarial detection rate,
respectively. FGSM, PGD, CW2, DF, Sq, and AA represent fast gradient sign
method, projected gradient descent, Carlini-Wagner L2, Square, and Auto
attacks, respectively. Values are percentages. Outermost edge relevance for
Model 2 is a poor predictor because the outermost edge relevance for benign
and adversarial samples are identical (0). During the thresholding process,
the relevance for the edges corresponding to the output layer are set to 0
because they are relatively small.

model |  |  | Attack
---|---|---|---
|  | FGSM | PGD | CW2 | DF | Sq | AA
MobileNet | degree | 99.64/98.10 | 99.63/99.23 | 82.21/90.74 | 80.23/91.34 | 94.08/93.22 | 100/99.63
node importance | 99.52/99.35 | 99.46/100 | 64.50/99.43 | 66.78/93.91 | 91.86/93.39 | 99.89/100
outermost edge relevance | 100/99.98 | 99.71/99.61 | 85.27/90.06 | 87.75/89.24 | 100/99.85 | 100/99.86
LID | 23.49/85.93 | 12.01/97.90 | 22.30/79.84 | 35.59/68.10 | 24.47/98.60 | 7.02/87.35
InceptionResNetV2 | degree | 100/99.95 | 100/99.92 | 84.05/94.31 | 73.88/86.22 | 96.18/98.50 | 99.66/100
node importance | 100/100 | 100/100 | 65.76/88.13 | 44.12/96.08 | 89.30/99.36 | 100/99.96
| outermost edge relevance | 99.96/99.70 | 100/99.70 | 80.36/78.76 | 80.96/74.43 | 99.21/96.59 | 100/99.78
| LID | 40.13/66.23 | 76.58/44.09 | 21.08/73.91 | 72.97/28.78 | 67.79/41.59 | 31.60/75.93
| VGG19 | degree | 100/99.86 | 99.92/99.73 | 99.77/04.20 | 97.55/99.08 | 98.98/98.21 | 99.16/100
| node importance | 99.96/100 | 99.96/100 | 100/0.70 | 94.88/99.35 | 98.24/98.28 | 97.97/100
| outermost edge relevance | 99.73/99.72 | 99.96/99.96 | 100/0 | 99.66/96.82 | 99.66/97.21 | 100/99.92
| LID | 79.66/6.57 | 99.39/42.71 | 41.77/57.55 | 99.73/0.88 | 98.78/0.11 | 22.14/76.08

(a)

model |  |  | Attack
---|---|---|---
|  | FGSM | PGD | CW2 | DF | Sq | AA
MobileNet | degree | 100/100 | 100/99.77 | 77.25/91.87 | 55.44/92.96 | 99.66/99.04 | 100/100
node importance | 100/100 | 99.66/100 | 55.61/90.53 | 54.39/88.69 | 89.30/99.36 | 100/99.96
outermost edge relevance | 100/99.79 | 100/99.85 | 99.59/76.24 | 79.85/74.90 | 99.21/96.96 | 100/99.81
LID | 57.12/40.29 | 69.78/37.14 | 77.47/23.03 | 92.97/7.03 | 88.45/7.71 | 85.23/37.52
InceptionResNetV2 | degree | 99.88/99.84 | 98.63/100 | 76.00/91.52 | 75.29/92.63 | 87.35/97.11 | 99.41/100
node importance | 100/100 | 100/99.96 | 45.68/93.81 | 89.52/94.36 | 89.52/94.36 | 100/100
| outermost edge relevance | 99.96/99.68 | 100/99.60 | 74.06/76.25 | 92.95/90.58 | 92.95/90.58 | 100/99.92
| LID | 92.14/4.10 | 69.78/37.14 | 77.17/23.02 | 53.32/48.55 | 29.87/59.27 | 10.75/61.80
| VGG19 | degree | 100/99.93 | 100/100 | 97.67/29.68 | 98.80/99.18 | 96.97/99.57 | 99.96/100
| node importance | 100/99.94 | 100/99.96 | 98.84/9.68 | 99.34/99.39 | 98.64/99.73 | 100/100
| outermost edge relevance | 100/99.88 | 100/99.81 | 100/0 | 99.96/98.37 | 100/97.83 | 100/99.92
| LID | 67.79/40.38 | 2.26/85.83 | 42.24/47.81 | 4.14/92.37 | 7.01/98.72 | 6.47/29.00

(b)

model |  |  | Attack
---|---|---|---
|  | FGSM | PGD | CW2 | DF | Sq | AA
Model 2 | degree | 99.00/95.45 | 99.90/95.45 | 98.05/99.19 | 97.26/99.20 | 99.86/65.44 | 99.97/95.34
node importance | 84.80/24.54 | 92.79/12.46 | 52.75/62.37 | 34.22/73.53 | 97.71/5.53 | 94.50/8.04
outermost edge relevance | 100/0 | 100/0 | 100/0 | 100/0 | 100/0 | 100/0
LID | 81.94/13.43 | 88.19/10.41 | 74.90/21.72 | 71.79/22.27 | 52.77/50.96 | 16.10/87.53
Model 1 | degree | 95.96/98.91 | 94.76/84.19 | 6.58/88.76 | 95.51/96.71 | 98.63/71.97 | 96.02/84.48
node importance | 81.67/96.19 | 96.09/54.83 | 0.14/99.70 | 88.14/96.57 | 100/2.09 | 93.87/60.39
| outermost edge relevance | 90.34/86.50 | 89.31/85.25 | 49.55/42.19 | 95.68/93.58 | 100/0 | 89.47/88.57
| LID | 50.60/49.75 | 72.07/25.38 | 47.20/55.53 | 71.79/22.27 | 52.77/50.96 | 26.71/72.12

(c)

(a) Top subfigure is a histogram of degree for benign images, computed for
node number 426. Bottom subfigure shows the histogram computed for adversarial
images. (b) Top subfigure is a histogram of node importance for benign images,
computed for node number 426. Bottom subfigure shows the histogram computed
for adversarial images. (c) Top subfigure is a histogram of outermost edge
relevance for benign images, computed for a certain edge. Bottom subfigure
shows the histogram computed for adversarial images. Figure 1: Data shown is
for InceptionResNetV2, Square attack.

###  7.2  Comparison of statistical approaches

Tables LABEL:table:benchmarking-wsr-cifar10, LABEL:table:benchmarking-wsr-
svhn, and LABEL:table:benchmarking-wsr-mnist show results in terms of AUROC
(area under receiver operating characteristic curve) for various detection
methods. In almost all cases, WSR2 provides more accurate predictions than
WSR1 as predicted in Section 5. Further, both WSR variations outperform RSA.
Model 1, in comparison to the other models, performs somewhat poorly under
WSR1. This seems to be due to Model 1 having the least number of neurons,
making the corresponding $|\mathcal{S}|$ relatively small which implies slower
convergence of the statistical test. More generally, we can also see from the
tables that model/attack pairs with small $|\mathcal{S}|$ tend to have worse
results under WSR. This is particularly noticeable in the case of Carlini-
Wagner and Deepfool attacks under WSR1; this lower performance was also noted
in our results using logistic regression, and provides evidence in support of
the belief that Carlini-Wagner and Deepfool attacks are among the strongest.
In Figure 2, one can see histograms of the distributions
$\hat{\mathcal{B}}_{i}^{j}$ and $\hat{\mathcal{A}}_{i}^{j}$ for $j=426$,
$i=3$. Because there are thousands of nodes, we only choose one value of $j$
to show.

Table 5: Comparison of AUROC for statistical detection methods. FGSM, PGD,
CW2, DF, Sq, and AA represent fast gradient sign method, projected gradient
descent, Carlini-Wagner L2, Deepfool, Square, and Auto attacks, respectively.
WSR1 and WSR2 are WSR variants 1 and 2 respectively. Values are percentages.

model |  |  | Attack
---|---|---|---
|  | FGSM | PGD | CW2 | DF | Sq | AA
MobileNet | WSR1 | 99.98 | 100 | 79.78 | 73.56 | 92.98 | 100
WSR2 | 100 | 94.47 | 98.00 | 92.11 | 99.25 | 100
RSA | 51.31 | 68.56 | 84.94 | 48.79 | 97.44 | 85.74
InceptionResNetV2 | WSR1 | 99.64 | 99.00 | 81.46 | 64.23 | 84.88 | 100
WSR2 | 97.26 | 99.05 | 95.69 | 88.54 | 89.93 | 99.98
RSA | 94.67 | 98.89 | 99.06 | 98.59 | 98.72 | 95.07
| VGG19 | WSR1 | 100 | 99.98 | 74.83 | 98.78 | 97.08 | 99.98
| WSR2 | 99.87 | 99.97 | 97.36 | 99.32 | 99.70 | 100
| RSA | 69.08 | 54.28 | 71.70 | 73.77 | 76.58 | 63.78

(d)

model |  |  | Attack
---|---|---|---
|  | FGSM | PGD | CW2 | DF | Sq | AA
MobileNet | WSR1 | 100 | 100 | 81.62 | 78.95 | 99.29 | 100
WSR2 | 100 | 100 | 95.20 | 90.26 | 99.80 | 100
RSA | 80.12 | 63.12 | 87.69 | 82.83 | 76.48 | 82.27
InceptionResNetV2 | WSR1 | 100 | 100 | 78.35 | 80.59 | 92.12 | 100
WSR2 | 99.86 | 99.91 | 93.84 | 94.29 | 96.81 | 100
RSA | 51.77 | 61.81 | 56.35 | 60.25 | 58.44 | 58.19
| VGG19 | WSR1 | 100 | 100 | 76.33 | 99.65 | 99.54 | 100
| WSR2 | 100 | 100 | 94.57 | 99.68 | 99.87 | 100
| RSA | 80.69 | 57.27 | 76.79 | 79.63 | 81.77 | 58.23

(e)

model |  |  | Attack
---|---|---|---
|  | FGSM | PGD | CW2 | DF | Sq | AA
Model 2 | WSR1 | 95.16 | 95.37 | 96.48 | 95.44 | 91.77 | 95.54
WSR2 | 96.75 | 96.23 | 96.73 | 96.08 | 91.74 | 96.68
RSA | 66.81 | 62.81 | 58.54 | 55.95 | 68.33 | 63.12
Model 1 | WSR1 | 95.53 | 81.06 | 41.25 | 82.48 | 89.30 | 83.60
WSR2 | 96.36 | 94.40 | 39.65 | 97.81 | 99.48 | 94.36
RSA | 71.80 | 72.10 | 51.64 | 89.60 | 96.34 | 71.85

(f)

Figure 2: Data shown is for Inceptionresnetv2, Square attack. Top subfigure
shows histogram of $\hat{\mathcal{B}}_{i}^{j}$ for $j=426$, $i=3$. Bottom
subfigure shows histogram of $\hat{\mathcal{A}}_{i}^{j}$ for $j=426$, $i=3$.

(a) WSR1 for Model 2

(b) WSR2 for InceptionResnetV2.  Figure 3: Empirical distributions of WSR1 for
Model 2 and WSR2 for InceptionResnetV2. The top subplot of a figure shows WSR
computed for adversarial examples, and the bottom subplot shows the
computation for benign examples. FGSM, PGD, CW2, DF, Sq, and AA represent fast
gradient sign method, projected gradient descent, Carlini-Wagner L2, Deepfool,
Square, and Auto attacks, respectively. For Model 2 and CW2, values above 200
are set to 200 for ease of display. Note that the benign and adversarial plots
for Model 2 tend to agree with the remark made in Section 1 about inverses.

We can use WSR and logistic regression in a complementary way. For instance,
graph-based quantities generated from VGG19 and Carlini-Wagner attacks tend to
be poorly classified with logistic regression. In contrast, WSR2 performs well
in this case, and it can be used in place of logistic regression.

As an aside, in [Roth et al., 2019], the authors claim that adding noise to
images affects the logits in such a way that adversarial inputs can be
detected. We considered using equation 5 from [Roth et al., 2019] as a
baseline in place of RSA, but chose not to because of the extremely large time
needed for the source code to run and secondly, our initial results suggested
that this method gives poor accuracy, near 50%, which is much lower than the
performance reports in the literature. In our effort to increase accuracy we
experimented with different hyperparameters, including noise, but to no avail.
This calls into question the usefulness and robustness of using equation 5 in
[Roth et al., 2019].

###  7.3 Nodal analysis

The distribution of node quantities is highly dependent on the model and
attack. It can be seen (Table 5) that AUROC for WSR decreases as the strength
of the attack increases (we consider a partial order of in increasing attack
strength to be: Fast Gradient Sign Method, Projected Gradient Descent and
Carlini-Wagner L2). We can relate this observation to how the cardinality of
$|\mathcal{S}|$ varies with model/attack. The cardinality of $|\mathcal{S}|$
can be seen in Tables LABEL:table:S-cifar10, LABEL:table:S-svhn, and
LABEL:table:S-mnist. For CIFAR-10 and SVHN datasets, we observe that the
cardinality tends to be a lot smaller for Carlini-Wagner L2 and DeepFool
attacks, and it seems to explain the lower accuracy achieved by WSR on these
attacks. We recall that from Section 5, the accuracy of WSR increases with
$|\mathcal{S}|$.

We also note that in some cases the benign distribution of WSR and the
adversarial distribution of WSR are centered at points which are close to
inverses. This seems to be the case for Model 2, as shown in Figure 3. This is
in agreement with an earlier remark in Section 5 about equation (8) having
inverse values under benign and adversarial examples.

Table 6: Cardinality of $\mathcal{S}$ by model and attack. FFGSM, PGD, CW2,
DF, Sq, and AA represent fast gradient sign method, projected gradient
descent, Carlini-Wagner L2, Deepfool, Square, and Auto attacks, respectively.

model |  |  | Attack
---|---|---|---
|  | FGSM | PGD | CW2 | DF | Sq | AA
MobileNet | WSR1 | 877 | 734 | 51 | 68 | 405 | 79
InceptionResnetV2 | WSR1 | 378 | 484 | 109 | 180 | 240 | 120
VGG19 | WSR1 | 180 | 776 | 34 | 925 | 692 | 578

(c)

model |  |  | Attack
---|---|---|---
|  | FGSM | PGD | CW2 | DF | Sq | AA
MobileNet | WSR1 | 422 | 636 | 78 | 97 | 551 | 287
InceptionResnetV2 | WSR1 | 754 | 1269 | 109 | 160 | 922 | 496
VGG19 | WSR1 | 945 | 1253 | 63 | 1379 | 929 | 843

(d)

model |  |  | Attack
---|---|---|---
|  | FGSM | PGD | CW2 | DF | Sq | AA
Model 2 | WSR1 | 129 | 105 | 50 | 34 | 52 | 115
Model 1 | WSR1 | 33 | 10 | 7 | 5 | 4 | 10

(e)

##  8 Conclusion

We demonstrate that neural network architectures can be interpreted in a graph
context from which we can use the statistics of graph-based quantities to
detect adversarial attacks. We introduce three statistics that we applied to
our graphs and used them as predictors of adversarial attack. We show that
this approach can produce high detection performances with logistic
regression. We also study the distributions of node degree using a statistical
test based on Wasserstein distances. One limitation of our method is that
layer-wise relevance propagation is designed for ANNs with ReLU activations.
Thus, the use of layer-wise relevance propagation may produce less
explainability for neural networks with non-ReLu activations. However, ReLU is
arguably the most popular type. It would be interesting to expand the class of
neural networks in this study to include those with more non-ReLu activations,
despite the possible gap in explainability. We note that Model 1 includes one
non-ReLu activation (sigmoid) and it can be considered a small step in this
direction. Another direction of our work may include the use of other saliency
map methods such as [Springenberg et al., 2015] or [Simonyan et al., 2014]
instead of layer-wise relevance propagation. Further, an ablation study can be
done by removing specific and sensitive nodes from a neural network. In our
context, these nodes can be from the set $\mathcal{S}$ that is used in our
statistical tests. It would be interesting to see the relationship between the
neural network’s classification accuracy and the effectiveness of the
statistical tests. We find it intriguing that a sparse graph encodes
sufficient information about inputs to a neural network. We hope that the
perspective introduced here will provide a different perspective of
understanding adversarial attacks.

##  9 Acknowledgments

L. Carboni and D. Nwaigwe are the recipients of a grant from MIAI@Grenoble
Alpes (ANR 19-P3IA-003).

## References

  * [Andriushchenko et al., 2020] Andriushchenko, M., Croce, F., Flammarion, N., and Hein, M. (2020).  Square attack: A query-efficient black-box adversarial attack via random search.  In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII, volume 12368 of Lecture Notes in Computer Science, pages 484–501. Springer.
  * [Bach et al., 2015] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. (2015).  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PLOS ONE, 10(7):e0130140.
  * [Bavelas, 1950] Bavelas, A. (1950).  Communication patterns in task-oriented groups.  The Journal of the Acoustical Society of America, 22(6):725–730.
  * [Bullmore and Sporns, 2009] Bullmore, E. and Sporns, O. (2009).  Complex brain networks: graph theoretical analysis of structural and functional systems.  Nature Reviews Neuroscience, 10(3):186–198.
  * [Carlini and Wagner, 2017] Carlini, N. and Wagner, D. (2017).  Towards evaluating the robustness of neural networks.  In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57, San Jose, CA, USA. IEEE.
  * [Cheng et al., 2021] Cheng, W., Deng, C., Zhao, Z., Cai, Y., Zhang, Z., and Feng, Z. (2021).  Spade: A spectral method for black-box adversarial robustness evaluation.  In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1814–1824. PMLR.
  * [Cherian and Aeron, 2020] Cherian, A. and Aeron, S. (2020).  Representation learning via adversarially-contrastive optimal transport.  In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR.
  * [Cireşan et al., 2012] Cireşan, D., Meier, U., Masci, J., and Schmidhuber, J. (2012).  Multi-column deep neural network for traffic sign classification.  Neural Networks, 32:333–338.
  * [Corneanu et al., 2019] Corneanu, C. A., Madadi, M., Escalera, S., and Martinez, A. M. (2019).  What does it mean to learn in deep networks? And, how does one detect adversarial attacks?  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4752–4761, Long Beach, CA, USA. IEEE.
  * [Croce and Hein, 2020] Croce, F. and Hein, M. (2020).  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.  In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 2206–2216. PMLR.
  * [Drenkow et al., 2022] Drenkow, N., Fendley, N., and Burlina, P. (2022).  Attack agnostic detection of adversarial examples via random subspace analysis.  In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2815–2825, Waikoloa, HI, USA. IEEE.
  * [Feinman et al., 2017] Feinman, R., Curtin, R. R., Shintre, S., and Gardner, A. B. (2017).  Detecting adversarial samples from artifacts.  arXiv preprint ARXIV.1703.00410.
  * [Finlayson et al., 2019] Finlayson, S. G., Bowers, J. D., Ito, J., Zittrain, J. L., Beam, A. L., and Kohane, I. S. (2019).  Adversarial attacks on medical machine learning.  Science, 363(6433):1287–1289.
  * [Goodfellow et al., 2015] Goodfellow, I. J., Shlens, J., and Szegedy, C. (2015).  Explaining and harnessing adversarial examples.  In 3rd International Conference on Learning Representations.
  * [Guo et al., 2022] Guo, C., Lee, M., Leclerc, G., Dapello, J., Rao, Y., Madry, A., and Dicarlo, J. (2022).  Adversarially trained neural representations are already as robust as biological neural representations.  In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8072–8081. PMLR.
  * [Harder et al., 2021] Harder, P., Pfreundt, F.-J., Keuper, M., and Keuper, J. (2021).  Spectraldefense: detecting adversarial attacks on cnns in the fourier domain.  In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8, Shenzhen, China. IEEE.
  * [Hassabis et al., 2017] Hassabis, D., Kumaran, D., Summerfield, C., and Botvinick, M. (2017).  Neuroscience-inspired artificial intelligence.  Neuron, 95(2):245–258.
  * [La Malfa et al., 2022] La Malfa, E., La Malfa, G., Caprioli, C., Nicosia, G., and Latora, V. (2022).  Deep neural networks as complex networks.  arXiv preprint ARXIV.2209.05488.
  * [Lee et al., 2018] Lee, K., Lee, K., Lee, H., and Shin, J. (2018).  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, pages 7167–7177.
  * [Liu et al., 2020] Liu, S., van der Lee, T., Yaman, A., Atashgahi, Z., Ferraro, D., Sokar, G., Pechenizkiy, M., and Mocanu, D. C. (2020).  Topological insights into sparse neural networks.  In Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part III, volume 12459 of Lecture Notes in Computer Science, pages 279–294. Springer.
  * [Ma et al., 2018] Ma, X., Li, B., Wang, Y., Erfani, S. M., Wijewickrema, S. N. R., Schoenebeck, G., Song, D., Houle, M. E., and Bailey, J. (2018).  Characterizing adversarial subspaces using local intrinsic dimensionality.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
  * [Madry et al., 2018] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2018).  Towards deep learning models resistant to adversarial attacks.  In International Conference on Learning Representations.
  * [Michael, 2017] Michael, H. (2017).  Local Intrinsic Dimensionality I: An Extreme-Value-Theoretic Foundation for Similarity Applications.  In Similarity search and applications, SISAP, pages 64–79. Springer Berlin Heidelberg, New York, NY.
  * [Mocanu et al., 2016] Mocanu, D. C., Mocanu, E., Nguyen, P. H., Gibescu, M., and Liotta, A. (2016).  A topological insight into restricted Boltzmann machines.  Machine Learning, 104(2-3):243–270.
  * [Montavon et al., 2019] Montavon, G., Binder, A., Lapuschkin, S., Samek, W., and Müller, K.-R. (2019).  Layer-Wise Relevance Propagation: An Overview, pages 193–209.  Springer International Publishing, Cham.
  * [Moosavi-Dezfooli et al., 2016] Moosavi-Dezfooli, S.-M., Fawzi, A., and Frossard, P. (2016).  Deepfool: a simple and accurate method to fool deep neural networks.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2574–2582, Las Vegas, NV, USA. IEEE.
  * [Naitzat et al., 2022] Naitzat, G., Zhitnikov, A., and Lim, L.-H. (2022).  Topology of deep neural networks.  Journal of Machine Learning Research, 21(1).
  * [Nicolae et al., 2018] Nicolae, M.-I., Sinn, M., Tran, M. N., Buesser, B., Rawat, A., Wistuba, M., Zantedeschi, V., Baracaldo, N., Chen, B., Ludwig, H., Molloy, I., and Edwards, B. (2018).  Adversarial robustness toolbox v1.2.0.  CoRR, 1807.01069.
  * [Papernot et al., 2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., and Swami, A. (2017).  Practical black-box attacks against machine learning.  In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pages 506–519, Abu Dhabi United Arab Emirates. ACM.
  * [Roth et al., 2019] Roth, K., Kilcher, Y., and Hofmann, T. (2019).  The odds are odd: A statistical test for detecting adversarial examples.  In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5498–5507. PMLR.
  * [Simonyan et al., 2014] Simonyan, K., Vedaldi, A., and Zisserman, A. (2014).  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In Bengio, Y. and LeCun, Y., editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings.
  * [Sporns, 2013] Sporns, O. (2013).  Structure and function of complex brain networks.  Dialogues in Clinical Neuroscience, 15(3):247–262.
  * [Springenberg et al., 2015] Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. A. (2015).  Striving for simplicity: The all convolutional net.  In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings.
  * [Wong et al., 2019] Wong, E., Schmidt, F. R., and Kolter, J. Z. (2019).  Wasserstein adversarial examples via projected sinkhorn iterations.  In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6808–6817. PMLR.
  * [Xu et al., 2020] Xu, H., Ma, Y., Liu, H.-C., Deb, D., Liu, H., Tang, J.-L., and Jain, A. K. (2020).  Adversarial attacks and defenses in images, graphs and text: a review.  International Journal of Automation and Computing, 17(2):151–178.
  * [Yang et al., 2023a] Yang, S., Pang, Y., Wang, H., Lei, T., Pan, J., Wang, J., and Jin, Y. (2023a).  Spike-driven multi-scale learning with hybrid mechanisms of spiking dendrites.  Neurocomputing, 542:126240.
  * [Yang et al., 2022a] Yang, S., Tan, J., and Chen, B. (2022a).  Robust spike-based continual meta-learning improved by restricted minimum error entropy criterion.  Entropy, 24(4):455.
  * [Yang et al., 2023b] Yang, S., Tan, J., Lei, T., and Linares-Barranco, B. (2023b).  Smart traffic navigation system for fault-tolerant edge computing of internet of vehicle in intelligent transportation gateway.  IEEE Transactions on Intelligent Transportation Systems, pages 1–12.
  * [Yang et al., 2022b] Yang, S., Wang, J., Deng, B., Azghadi, M. R., and Linares-Barranco, B. (2022b).  Neuromorphic context-dependent learning framework with fault-tolerant spike routing.  IEEE Transactions on Neural Networks and Learning Systems, 33(12):7126–7140.
  * [Zambra et al., 2020] Zambra, M., Maritan, A., and Testolin, A. (2020).  Emergence of network motifs in deep neural networks.  Entropy, 22(2):204.


# Tracking Turbulence Through Financial News During COVID-19

Philip Hossu  Natalie Parde
University of Illinois at Chicago
Department of Computer Science
{phossu2<EMAIL_ADDRESS>
###### Abstract

Grave human toll notwithstanding, the COVID-19 pandemic created uniquely
unstable conditions in financial markets. In this work we uncover and discuss
relationships involving sentiment in financial publications during the 2020
pandemic-motivated U.S. financial crash. First, we introduce a set of expert
annotations of financial sentiment for articles from major American financial
news publishers. After an exploratory data analysis, we then describe a CNN-
based architecture to address the task of predicting financial sentiment in
this anomalous, tumultuous setting. Our best performing model achieves a
maximum weighted F1 score of 0.746, establishing a strong performance
benchmark. Using predictions from our top performing model, we close by
conducting a statistical correlation study with real stock market data,
finding interesting and strong relationships between financial news and the
S&P 500 index, trading volume, market volatility, and different single-factor
ETFs.

##  1 Introduction

Recent advancements in fundamental natural language understanding problems
Zhang et al. (2019); Raffel et al. (2020); Clark et al. (2020) have generated
renewed interest in their downstream deployment in many applications; one such
example is extrapolating meaning from large quantities of financial
publications. The year 2020 and its accompanying financial unpredictability
presents an interesting case study and test bed for financial sentiment models
that rely on complex inferences. In this work we tackle that problem, and
uncover numerous interesting and important relationships involving sentiment
in financial publications during the 2020 pandemic-related U.S. financial
crash. This work is unique as we focus not on lexicon scoring, nor developing
a trading strategy during normal market conditions, but instead on analyzing
the relationship between sentiment and market movement during a financial
crisis — a subject seldom studied due to the infrequency of financial crashes
and pandemics. Our key contributions are as follows:

  * •

We collect a set of expert annotations for 800 financial article titles and
descriptions with COVID-19-specific vocabulary, labeled for negative, neutral,
or positive sentiment.

  * •

We introduce CNN-based models to distinguish between sentiment categories in
this challenging dataset, achieving a maximum weighted F1 score of 0.746.

  * •

We perform correlation studies between our top model’s predictions and popular
market measures: S&P 500 Average, S&P 500 Trading Volume, Cboe Volatility
Index, Single-Factor ETFs, and FAANG+ stocks.

Our experiments yield intriguing findings, including a strong relationship
between predicted market sentiment and the S&P 500 Average and a notably lower
correlation with value stocks compared to other factors.

##  2 Related Work

Sentiment analysis is a popular NLP task Ribeiro et al. (2016) that has been
employed in a variety of practical fields, ranging from product review
analyses for marketing purposes Hu and Liu (2004), to tracking online voter
sentiment Tumasjan et al. (2011), as well as to financial markets. Current
methods for extracting sentiment from bodies of text fall primarily into two
categories: lexicon-based and machine learning methods. Lexicon-based
approaches ultimately produce dictionaries containing key/value pairs for
scoring text. Although simplistic, they have the added benefit of being
extremely efficient, requiring user programs to do little more than load a
dictionary into memory and perform linear searches.

However, lexicon-based methods tend to fall short compared to modern deep
learning techniques, especially in domains with specialized vocabularies.
Machine learning methods often outperform lexicons in financial text
applications Man et al. (2019). Financial texts not only contain highly
specialized language referring to specific actions and outcomes (eg.,
[Company] prices senior notes announces a type of bond offering), but also
exhibit language that can have very different meaning compared to general
settings (eg., “bull” may typically have a neutral or negative connotation,
but is overwhelmingly positive in a financial context).

A recent surge of work in financial sentiment analysis emerged during SEMEVAL
2017’s Task 5 Cortis et al. (2017), in which participants were asked to create
systems capable of predicting article sentiment on a continuous scale from -1
(bearish/negative/price-down) to 1 (bullish/positive/price-up), given a
dataset of 1647 annotated financial statements and headlines. The top two
performing systems Mansar et al. (2017); Kar et al. (2017) performed within
0.001 points of each other, yet used vastly different techniques. Mansar et
al. (2017) utilized three primary input features: pre-trained GloVe embeddings
Pennington et al. (2014), DepecheMood scores Staiano and Guerini (2014), and
VADER scores Hutto and Gilbert (2014). They paired these with a CNN-based
architecture, leveraging the following layers in order: convolution, max
pooling, dropout layer, fully connected layer, dropout layer, and fully
connected layer.

Conversely, Kar et al. (2017) used pre-trained Word2Vec embeddings Mikolov et
al. (2013) as input separately to a CNN and a bidirectional gated recurrent
unit (BiGRU), and also generated a set of hand-crafted features which were fed
to multiple fully-connected layers. All three of these model outputs were
concatenated, before being input to another fully-connected layer which
ultimately issued a prediction. We take inspiration from the former paper
Mansar et al. (2017), and experiment with a similar CNN-based approach.

##  3 Data Sources

We collected financial articles using the NewsFilter.IO
API.111http://newsfilter.io This service provides straightforward and high
quality data access for online articles from most major American publishers of
financial news. Articles are provided in a JSON dictionary format with fields
corresponding to title, description, publisher name, published time, and stock
tickers mentioned, among others. We refer to this dataset from here onward as
Fin-Cov-News.

To collect market and share prices, we downloaded publicly available data from
Yahoo Finance.222https://finance.yahoo.com/ Yahoo Finance allows users to
download daily price data given inputs of a specific stock or instrument
symbol and date range.

###  3.1 Basic Statistics

We collected data from January 1 to December 31, 2020. This yielded 192,154
articles from the following sources: Wall Street Journal, Bloomberg,
SeekingAlpha, PR Newswire, CNBC, and Reuters. Considering only article titles,
the dataset contains 2,069,244 words, or 1,660,842 words without common
stopwords Bird et al. (2009), with a vocabulary size of 118,719 unique words.

Word | Occurrences | Word | Occurrences
---|---|---|---
coronavirus | 12427 | results | 5101
u.s. | 12198 | reports | 4966
new | 11620 | billion | 4913
says | 10839 | china | 4872
market | 7808 | stocks | 4835
announces | 7468 | dividend | 4715
2020 | 7282 | virus | 4232
covid-19 | 6914 | oil | 4041
global | 6659 | declares | 3948
Table 1: Most common words appearing in titles in Fin-Cov-News.

The twenty most common words in titles (minus punctuation) are shown in Table
1. The (mean, min, max) set for titles is (8.64, 1, 54), and for article
descriptions is (27.35, 1, 1309). As expected, a significant number of COVID
words appear in the dataset (e.g., “coronavirus,” “covid-19,” and “virus”),
alongside traditional finance words like “dividend.”

###  3.2 Further Data Analysis

We aggregated articles by daily frequency and computed the occurrences over
time for different sets of query words, producing the proportion of all posts
published in a given day which contain the query words in the post title or
description. Our first plot, shown in Figure 1, uses common “Earnings” query
words: {revenue, earnings, dividend, miss, beat, estimates}. These words peak
notably near the beginnings and ends of fiscal quarters. The blue line denotes
the daily percent of posts mentioning one (or multiple) query terms, and the
grey bars denote the daily post volume.

Figure 1: Percentage of posts by day corresponding to “Earnings” terms.

We additionally consider a query of “COVID-19-related” words: {covid, covid19,
coronavirus, virus, corona, covid-19, 2019-ncov, 2019ncov, sars-cov-2,
sarscov2}. Interestingly, we note a spike of COVID-related keywords before the
actual spike of cases in the U.S., and a steady decline between April and June
despite the increasing ground truth number of cases and ongoing economic
impact. One may also observe that the peak percent of posts mentioning the
“COVID-19-related” keywords is around 60% in late-March of 2020, and the
lowest percent is in September with around 10%. Figure 2 shows these
percentages, again with the blue line denoting the daily percent of posts
mentioning one (or multiple) query terms, and the grey bars denoting the daily
post volume.

Figure 2: Percentage of posts by day corresponding to “COVID-19-related”
terms.

When comparing to the actual COVID-19 statistics as aggregated by the U.S.
Centers for Disease Control (CDC), made publicly available via their
website,333https://covid.cdc.gov/covid-data-tracker/ we observe a strong
initial financial press reaction to the virus, despite the low infection rate.
The press decreased its focus on COVID-19 over time, despite its continuing
impact on society and the economy.

###  3.3 Additional External Datasets

While labeling the Fin-Cov-News data was essential for learning COVID-19
related trends, we additionally utilized samples of two external datasets
created for similar tasks. For the first (SemEval), from SEMEVAL-2017 Cortis
et al. (2017), we sample 750 random articles and use cutoff points at -0.33
and 0.33 to threshold the continuous sentiment labels to the categorical
negative, neutral, and positive. This yielded 156 negative articles, 413
neutral, and 181 positive.

We additionally sample 750 random phrases from v1.0 of the Financial Phrase
Bank (FinPhraseBank, Malo et al., 2014). This dataset similarly contains news
phrases from financial data sources with categorical negative, neutral, and
positive labels. We sampled exclusively from the “AllAgree" subset (containing
only samples exhibiting a unanimous label decision from the annotators) to
ensure the highest possible quality in our training data.

##  4 Annotation

To train our model to identify language patterns during the 2020 COVID-19
financial crisis, we first acquired labels for a sample from Fin-Cov-News.

###  4.1 Annotator Information

Annotations were solicited from four annotators (one of our authors and three
external experts). Our external annotators (referred to as B, C, and D) were
recruited from the quantitative strategy research group of a top Investment
Management/Exchange Traded Fund provider, where annotator A (author) worked
for a Summer. All three external annotators are full CFA
charter444https://www.cfainstitute.org/en/programs/cfa holders and have worked
in the investment space for at least three years. The experts completed the
labeling task on a volunteer basis, and were given approximately two weeks to
complete their labels.

###  4.2 Labeling Scheme

We selected a basic categorical (negative/neutral/positive) labeling scheme,
at the article level. Annotators were provided with XLSX files with each row
containing an article title, a truncated article description ($\sim$25 words),
and a label column to be completed. Labeling guidelines are shown in Table 2.

Label |  Description |  Examples
---|---|---
Negative |  Article reports news expected to negatively impact the price of a specific company or industry, or market health as a whole, either now or in the future. |  COVID-19 cases spread in a country (and it is unclear that specific company or industry stands to gain), a company announces layoffs or bankruptcy, or a news blurb indicates that the price of a share is down.
Positive |  Article reports news expected to positively impact the price of a specific company or industry, or market health as a whole, either now or in the future. |  A company is in talks for a significant new business partnership, COVID-19 vaccine progress is positive, or a news blurb indicates that the price of a share is up.
Neutral |  Article cannot be placed with any notable degree of certainty in positive or negative categories. |  Article mentions a company or event which has no influence on U.S. markets, article mentions multiple companies with multiple sentiments, or annotator is uncertain if the company will benefit from or be hurt by the news.
Table 2: Labeling guidelines for Fin-Cov-News.

###  4.3 Inter-Annotator Agreement

We allowed annotators to volunteer for a round number of articles they felt
comfortable labeling, resulting in the following amount of data: (A, 300), (B,
200), (C, 300), (D, 300). Between each set of articles, 50 were shared among
all annotators to compute inter-annotator agreement. The distributions of all
annotations, separated by annotator, are shown in Figure 3, with -1, 0, and 1
corresponding to negative, neutral, and positive labels. Annotators C and D
both leaned heavily towards assigning neutral labels and being selective with
good news or bad news, whereas Annotator B had a fairly uniform distribution
of labels across their data subset.

Figure 3: Annotator label distributions.

Pairwise Cohen’s Kappa Cohen (1960) scores are shown in Table 3. Annotator B
was an outlier, exhibiting poor agreement with the other two experts, C and D,
likely due to the annotator’s increased willingness to assign
positive/negative labels. Experts C and D agreed well with each other at
$\kappa=0.685$. Following convention Landis and Koch (1977), annotator pairs
(A,D) and (C,D) exhibit “substantial” agreement, and annotator pairs (A,B) and
(A,C) exhibit high “moderate” agreement. However, Annotator B exhibits only
“fair” agreement with Annotator C and barely reaches the bottom threshold of
“moderate” with Annotator D.

Based on this analysis, we removed Annotator B from the dataset, leaving 800
labeled articles in Fin-Cov-News. We highlight the inherent uncertainty during
the labeling of these financial documents — upon manual inspection, it is not
uncommon for “close-misses,” where some annotators thought an article was
positive and others thought it was neutral (or some thought it was negative,
others neutral). Even when considering exclusively domain experts, it is
difficult to obtain complete agreement.

|  A |  B |  C |  D
---|---|---|---|---
A |  |  0.574 |  0.574 |  0.709
B |  |  |  0.305 |  0.430
C |  |  |  |  0.685
D |  |  |  |
Table 3: Pairwise inter-annotator agreement among annotators A, B, C, and D.

##  5 Methods and Evaluation

We experiment with various models to predict financial sentiment, varying
input data and features as well as model structure. To evaluate our models, we
report both macro and weighted F1 scores (harmonic mean of precision and
recall) as computed by the Scikit-Learn Python library Pedregosa et al.
(2011). We include both because although weighted F1 may more effectively
reflect performance in real-world settings, it may also skew positive since we
observe a slight class imbalance in our training data – roughly 60% of labels
are neutral, with the remaining 40% split between positive and negative.

We assessed performance relative to two baseline models, AllNeutral and
VADERMax, to establish dataset learnability above chance or naive heuristics.
We report performance metrics for these baselines in Table 4. AllNeutral is a
majority-class baseline that predicts “neutral” for each instance, and
VADERMax is a rule-based heuristic that predicts labels based on the maximum
VADER score for any word in the instance. AllNeutral achieves a poor macro F1
score, but a weighted F1 score around 0.5 due to the label imbalance. VADERMax
performs comparably, giving evidence that lexicon-based heuristics are
insufficient in a financial crisis/recovery context.

Method | Macro F1 | Weighted F1
---|---|---
AllNeutral | 0.264 | 0.519
VADERMax | 0.300 | 0.536
Table 4: Performance of baseline approaches.

###  5.1 Model Design

To ensure competitive performance, we experiment with different word
embeddings, text preprocessing steps, and model inputs and architectures. All
experiments were performed in Python 3, using Keras Chollet et al. (2015).

For the model architecture, we adapt a similar Convolutional Neural Network
(CNN) design to that of Mansar et al. (2017), shown in Figure 4. We also
experimented with a BiGRU-based model, as they have demonstrated competitive
performance relative even to Bidirectional LSTMs in comparable tasks Sinha and
Khandait (2020). However, despite the $\sim$30x increase in training time,
upon evaluation the BiGRU performed similarly to the CNN. For this reason, we
selected to use the CNN for our model input experiments.555Future work may be
able to achieve higher performance using more complex Transformer-based
models, such as those by Devlin et al. (2019) and Araci (2019). Since our
focus here was on establishing proof of concept in this challenging setting,
substantial finetuning experiments remained out of scope but offer intriguing
future possibilities. For all model results, maximum F1 scores are reported on
the testing data over 100 epochs of training.

Figure 4: CNN model architecture.

We experimented with a variety of input features. All models are trained using
a concatenation of article title text and a truncated version ($\sim$25 words)
of the article description. In this subset of experiments, we consider only
the 800 labeled samples of Fin-Cov-News in a simple 85/15 train/test split.
Our input features and preprocessing steps were as follows:

  * •

COVID-19 Embeddings: We created custom embeddings for the missing words
{coronavirus, covid19, 2019ncov, covid, sarscov2}. These embeddings are arrays
of length 300, consisting of floating point numbers which would not otherwise
have words assigned to them (e.g., coronavirus = [-5.99, -5.99, …]).

  * •

Percent Replacements: We separated commonly seen percentages in the form of:
+5% $\rightarrow$ {plus, 5, percent}.

  * •

Dash Removal: We removed hyphens or dash characters (e.g., -, –, or —) from
the text.

  * •

Stopword Removal: We removed stopwords using the NLTK stopwords list Bird et
al. (2009).

  * •

Punctuation Removal: We removed all punctuation characters in the set {!, ",
#, $, %, &, ’, (, ), *, +, \, -, ., $/$, :, ;, $<$, =, $>$, ?, @, [, ], ^, _,
‘, {, $|$, }, $\sim$} from the text.

The pre-processing steps for each model in Table 5 are defined as follows: M1
- {None}, M2 - {Dash Removal, COVID-19 Embeddings}, M3 - {Dash Removal,
COVID-19 Embeddings, Stopword Removal}, M4 - {Dash Removal, COVID-19
Embeddings, Stopword Removal, Percent Replacements}, M5 - {Dash Removal,
COVID-19 Embeddings, Stopword Removal, Percent Replacements, Punctuation
Removal}. We find that the optimal level of preprocessing occurs with M4.
Although the regular expressions incorporated into M5 reduce the number of
missing words significantly, the model performance does not improve, and we
omit the extra processing step in our final model.

Method |  Words Missing |  Max Macro F1 |  Max Weighted F1
---|---|---|---
M1 |  27.01% |  0.445 |  0.571
M2 |  24.62% |  0.505 |  0.619
M3 |  23.86% |  0.552 |  0.670
M4 |  22.78% |  0.584 |  0.668
M5 |  9.90% |  0.538 |  0.645
Table 5: Performance outcomes from text preprocessing experiments, displaying
maximum macro and weighted F1 scores.

We compare conditions leveraging pretrained GloVe embeddings, pretrained
Word2Vec embeddings, and the inclusion of VADER scores in Table 6, using a
85/15 split of Fin-Cov-News. The model inputs are defined as follows: N1:
{300-length GloVe 6B, VADER Scores}, N2: {300-length Google News Word2Vec,
VADER Scores}, N3: {300-length Google News Word2Vec, No VADER Scores}. We find
that Word2Vec embeddings consistently result in higher performance than GloVe
embeddings. This is likely because the GloVe pretraining data is more general-
purpose, whereas the Word2Vec model was pretrained on news articles, likely
capturing more rich relationships for our use case.

Method |  Words Missing |  (Max, Avg.) Macro F1 |  (Max, Avg.) Weighted F1
---|---|---|---
N1 |  17.18% |  0.502, 0.473 |  0.602, 0.600
N2 |  22.78% |  0.565, 0.552 |  0.667, 0.641
N3 |  22.78% |  0.504, 0.483 |  0.599, 0.598
Table 6: Performance outcomes for experiments considering word embeddings and
VADER features, dislaying maximum and averaged macro and weighted F1 scores.

###  5.2 Expansion to External Data

For the best performing model, we expanded training to a full SemEval \+
FinPhraseBank \+ Fin-Cov-News dataset. The macro and weighted evaluation
scores are shown in Table 7. Of the 345 test samples, our best performing CNN
model predicted 256 articles correctly ($\sim$74.2%), 77 off-by-one
($\sim$22.3%), and 12 clear mispredictions ($\sim$3.5%).

Method | Precision | Recall | F1
---|---|---|---
Macro | 0.666 | 0.679 | 0.670
Weighted | 0.753 | 0.742 | 0.746
Table 7: Performance of the best-performing model on an expanded SemEval \+
FinPhraseBank \+ Fin-Cov-News dataset.

While still an error, off-by-one errors signify that the model predicted only
one label away from the true value. We further decompose these incorrect
predictions to gain a better understanding of what the model was missing. We
deem errors when the model erred on the side of caution (e.g., predicting
neutral when the label was positive or negative) to be acceptable errors, as
these errors would likely cause minimal detrimental effects in downstream
systems. We report our findings below:

  * •

Predict neutral, true negative [acceptable error] = $14/77$ off-by-ones

  * •

Predict neutral, true positive [acceptable error] = $20/77$ off-by-ones

  * •

Predict negative, true neutral [unacceptable error] = $18/77$ off-by-ones

  * •

Predict positive, true neutral [unacceptable error] = $25/77$ off-by-ones

Shifting perspective to the twelve instances for which the model made clear
mispredictions, there were ten instances for which the model predicted
positive and the true label was negative, and two instances of the opposite.
Upon manual inspection of these errors, we found that the articles in question
were indeed fairly ambiguous. For example, when the model saw: one best years
bull market erased s&p 500 rose almost 29 percent course 2019, good second
best year bull market. took little four weeks fall apart., the algorithm
thought this was good news – not able to catch on the subtlety of “little four
weeks fall apart.”

##  6 Market Correlations and Findings

We extend the predictions of our best performing model to the entire year
(January 1 – December 31, 2020) of article titles + truncated short
descriptions to uncover the relationships between financial articles and
actual market movement. We remove weekends (when the market is closed) from
our dataset to avoid times when share prices are relatively stable. We compute
Pearson correlation coefficients in all cases using the Stats module of Scipy
Virtanen et al. (2020).

###  6.1 S& P500 Averages

We first compare our model predictions and the S&P 500
average,666https://finance.yahoo.com/quote/%5EGSPC?p=%5EGSPC shown in the top
two rows of Table 8. Averaged Daily Model Sentiment refers to a simple daily
aggregate of the article sentiment. Smoothed Daily Model Sentiment is a 10-day
rolling average of the article sentiment. We find that the rolling average has
a strong correlation with the S&P 500, at $r=0.886$. This information is also
presented visually in the first three plots of Figure 5.

Item | $r$
---|---
S&P 500 Close vs. Averaged Daily Model Sentiment | 0.770
S&P 500 Close vs. Smoothed Daily Model Sentiment | 0.886
S&P 500 Volume vs. Smoothed Daily Model Sentiment | -0.563
VIX Close vs. Smoothed Daily Model Sentiment | -0.716
S&P 500 Volume vs. VIX Close | 0.713
Table 8: Pearson correlation scores ($r$) between S&P 500 closing averages,
S&P 500 trading volume, averaged daily model sentiment, smoothed daily model
sentiment, and the Cboe Volatility Index (VIX). Figure 5: Stacked timeseries
visualizing the S&P 500 closing averages, averaged daily model sentiment,
smoothed daily model sentiment, S&P 500 trading volume, and Cboe Volatility
Index from January 1 – December 31, 2020.

###  6.2 Trading Volume and the Cboe Volatility Index

We additionally hypothesized that trading volume and market volatility would
correlate with predicted financial news sentiment. To assess this, we
considered the simple trading volume related to the S&P500, as well as the
Cboe Volatility Index (VIX) — a well-regarded way to measure market volatility
based on options contracts. These correlations are shown in rows three through
five of Table 8. Interestingly, while our smoothed sentiment model outputs
exhibit non-zero correlation with both the S&P500 trading volume and the VIX,
the magnitude of this correlation is notably higher with the VIX.

###  6.3 Factor Analysis

Next, we investigated which single factor-based strategies aligned most
closely with predicted financial article sentiment. Factor investing is a
classical investing technique that seeks to identify certain attributes of
securities that are mathematically linked with higher returns. The body of
academic work on this topic is extensive (we direct interested readers to Fama
and French (1993, 2014)).

For strategy consistency, we selected four single factor exchange traded funds
(ETFs) from iShares by
BlackRock,777https://www.ishares.com/us/strategies/smart-beta-investing by far
the largest ETF provider as of March 16,
2021.888https://www.etf.com/sections/etf-league-tables/etf-league-
table-2021-03-16 iShares defines their four factor-based ETFs as follows:

  * •

Value ($VLUE): Stocks discounted relative to fundamentals.

  * •

Quality ($QUAL): Financially healthy companies.

  * •

Momentum ($MTUM): Stocks with an upward price trend.

  * •

Size ($SIZE): Smaller, more nimble companies.

While these are over-simplified definitions of the factors, they provide a
basic intuition regarding which companies were pooled into which of the
single-factor ETFs. The statistical correlations between these tickers and our
model results are shown on Table 9. We note that Value is the clear outlier
when compared to the other factors, with a significantly lower correlation
($r=0.625$).

Item | $r$
---|---
$VLUE Close vs. Smoothed Daily Model Sentiment | 0.625
$QUAL Close vs. Smoothed Daily Model Sentiment | 0.889
$MTUM Close vs. Smoothed Daily Model Sentiment | 0.862
$SIZE Close vs. Smoothed Daily Model Sentiment | 0.850
Table 9: Pearson correlation scores ($r$) between the $VLUE, $QUAL, $MTUM, and
$SIZE ETFs and smoothed daily model sentiment.

###  6.4 FAANG+

Finally, during the COVID-19 financial crisis and recovery, tech-focused
stocks bounced back quickly and even reached all-time highs within 2020. For
this reason, we specifically consider how well predicted financial news
sentiment correlated with this class of stocks. For our financial data, we
used the MicroSectors (BMO Harris) FAANG+ ETN
($FNGS).999https://www.microsectors.com/fang This ETN tracks the performance
of top tech companies, weighting Facebook, Apple, Amazon, Netflix, Alphabet,
Alibaba, Baidu, NVIDIA, Tesla and Twitter equally. It displayed significantly
higher returns than the overall market during our time period of interest. In
Fin-Cov-News, 15,701 articles were tagged with one of these FAANG+ companies,
accounting for around 8% of overall articles.

We found a correlation of $r=0.441$ (or $r=0.669$ with smoothed model
sentiment) between the news for these specific companies versus the $FNGS ETN;
this was notably lower than the overall market correlations that we found
earlier. Figure 6 shows that sentiment tracks market performance fairly well
for the initial crash, but while the tech stocks recover quickly and add to
their gains, sentiment continues to oscillate around a neutral point. This
behavior highlights one of the potential shortcomings of a purely sentiment-
driven trading strategy — when market behavior is fairly stable and trending
upward, the news sentiment does not always progressively grow to match this
trend.

Figure 6: Stacked timeseries visualizing the $FNGS ETN and smoothed daily
model sentiment from January 1 – December 31, 2020.

##  7 Conclusion

In this work, we comprehensively investigated the relationship between
financial news sentiment and stock market performance during the 2020
pandemic-induced financial crisis. While comparable works on this subject
Zammarchi et al. (2021) focus primarily on the use of social media data in
conjunction with lexicons or standard machine learning models, we leveraged a
task-specific CNN model to predict financial article sentiment over time,
achieving a maximum F1 of 0.746. This enabled us to subsequently perform a
correlation study that considered not only the stock market as a whole, but
also different factor-based strategies and industry subsets. As part of this
work we collected expert annotations of financial sentiment for 800 news
articles (a subset of Fin-Cov-News), with strong inter-annotator agreement
(averaged pairwise $\kappa=0.656$). We make these annotations available for
other interested parties by request.

With a solid deep learning-based sentiment model and an abundance of stock
data, there is a high ceiling for where future work could lead. One obvious
avenue would be continued improvement of the financial sentiment prediction
model, although significant improvements may require more labeled data. Using
the current model predictions, more fine-grained analysis could also be
performed on other subsets of companies or industries of interest. While it is
clear that news sentiment could not completely localize the market
crash/bottom, an intriguing extension of our work would be to consider
sentiment prediction on a more fine-grained time interval, and investigate
whether it could be used to inform higher frequency trading strategies that
could weather highly turbulent market conditions.

## References

  * Araci (2019) Dogu Araci. 2019.  FinBERT: Financial Sentiment Analysis with Pre-trained Language Models.  _arXiv:1908.10063 [cs]_.  ArXiv: 1908.10063.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  " O’Reilly Media, Inc.".
  * Chollet et al. (2015) François Chollet et al. 2015.  Keras.  https://github.com/fchollet/keras.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * Cohen (1960) Jacob Cohen. 1960.  A Coefficient of Agreement for Nominal Scales.  _Educational and Psychological Measurement_ , 20(1):37–46.
  * Cortis et al. (2017) Keith Cortis, André Freitas, Tobias Daudert, Manuela Huerlimann, Manel Zarrouk, Siegfried Handschuh, and Brian Davis. 2017.  SemEval-2017 task 5: Fine-grained sentiment analysis on financial microblogs and news.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 519–535, Vancouver, Canada. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv:1810.04805 [cs]_.  ArXiv: 1810.04805.
  * Fama and French (1993) Eugene F. Fama and Kenneth R. French. 1993.  Common risk factors in the returns on stocks and bonds.  _Journal of Financial Economics_ , 33(1):3–56.
  * Fama and French (2014) Eugene F. Fama and Kenneth R. French. 2014.  A Five-Factor Asset Pricing Model.  SSRN Scholarly Paper ID 2287202, Social Science Research Network, Rochester, NY.
  * Hu and Liu (2004) Minqing Hu and Bing Liu. 2004.  Mining and summarizing customer reviews.  In _Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining_ , KDD ’04, pages 168–177, Seattle, WA, USA. Association for Computing Machinery.
  * Hutto and Gilbert (2014) Clayton Hutto and Eric Gilbert. 2014.  Vader: A parsimonious rule-based model for sentiment analysis of social media text.  In _Proceedings of the International AAAI Conference on Web and Social Media_.
  * Kar et al. (2017) Sudipta Kar, Suraj Maharjan, and Thamar Solorio. 2017.  RiTUAL-UH at SemEval-2017 Task 5: Sentiment Analysis on Financial Data Using Neural Networks.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 877–882, Vancouver, Canada. Association for Computational Linguistics.
  * Landis and Koch (1977) J. Richard Landis and Gary G. Koch. 1977.  The Measurement of Observer Agreement for Categorical Data.  _Biometrics_ , 33(1):159–174.
  * Malo et al. (2014) Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. 2014\.  Good debt or bad debt: Detecting semantic orientations in economic texts.  _Journal of the Association for Information Science and Technology_ , 65(4):782–796.
  * Man et al. (2019) X. Man, T. Luo, and J. Lin. 2019.  Financial Sentiment Analysis(FSA): A Survey.  In _2019 IEEE International Conference on Industrial Cyber Physical Systems (ICPS)_ , pages 617–622.
  * Mansar et al. (2017) Youness Mansar, Lorenzo Gatti, Sira Ferradans, Marco Guerini, and Jacopo Staiano. 2017.  Fortia-FBK at SemEval-2017 Task 5: Bullish or Bearish? Inferring Sentiment towards Brands from Financial News Headlines.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 817–822, Vancouver, Canada. Association for Computational Linguistics.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient Estimation of Word Representations in Vector Space.  _arXiv:1301.3781 [cs]_.  ArXiv: 1301.3781.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global Vectors for Word Representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Ribeiro et al. (2016) Filipe N. Ribeiro, Matheus Araújo, Pollyanna Gonçalves, Marcos André Gonçalves, and Fabrício Benevenuto. 2016.  SentiBench \- a benchmark comparison of state-of-the-practice sentiment analysis methods.  _EPJ Data Science_ , 5(1):1–29.
  * Sinha and Khandait (2020) Ankur Sinha and Tanmay Khandait. 2020.  Impact of News on the Commodity Market: Dataset and Results.  _arXiv:2009.04202 [cs]_.  ArXiv: 2009.04202.
  * Staiano and Guerini (2014) Jacopo Staiano and Marco Guerini. 2014.  DepecheMood: a Lexicon for Emotion Analysis from Crowd-Annotated News.  _arXiv:1405.1605 [cs]_.  ArXiv: 1405.1605.
  * Tumasjan et al. (2011) Andranik Tumasjan, Timm O. Sprenger, Philipp G. Sandner, and Isabell M. Welpe. 2011\.  Election Forecasts With Twitter: How 140 Characters Reflect the Political Landscape.  _Social Science Computer Review_ , 29(4):402–418.
  * Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2020.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 17:261–272.
  * Zammarchi et al. (2021) Gianpaolo Zammarchi, Francesco Mola, and Claudio Conversano. 2021.  Impact of the COVID-19 outbreak on Italy’s country reputation and stock market performance: a sentiment analysis approach.  _arXiv:2103.13871 [cs]_.  ArXiv: 2103.13871.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  ERNIE: Enhanced language representation with informative entities.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1441–1451, Florence, Italy. Association for Computational Linguistics.