Büchel

# Adversarial Attacks on Spiking Convolutional Neural Networks for Event-based
Vision

Julian Büchel  IBM Research, Zürich, Switzerland  Gregor Lenz  SynSense,
Zürich, Switzerland  Yalun Hu  SynSense, Chengdu, China  Sadique Sheik
SynSense, Zürich, Switzerland  Martino Sorbaro  AI Center, ETH Zürich and
Institute of Neuroinformatics, University of Zürich and ETH Zürich,
Switzerland

###### Abstract

Event-based dynamic vision sensors provide very sparse output in the form of
spikes, which makes them suitable for low-power applications. Convolutional
spiking neural networks model such event-based data and develop their full
energy-saving potential when deployed on asynchronous neuromorphic hardware.
Event-based vision being a nascent field, the sensitivity of spiking neural
networks to potentially malicious adversarial attacks has received little
attention so far. We show how white-box adversarial attack algorithms can be
adapted to the discrete and sparse nature of event-based visual data, and
demonstrate smaller perturbation magnitudes at higher success rates than the
current state-of-the-art algorithms. For the first time, we also verify the
effectiveness of these perturbations directly on neuromorphic hardware.
Finally, we discuss the properties of the resulting perturbations, the effect
of adversarial training as a defense strategy, and future directions.

###### keywords:

Spiking Convolutional Neural Networks, Adversarial Examples, Neuromorphic
Engineering, Robust AI, Dynamic Vision Sensors

<EMAIL_ADDRESS><EMAIL_ADDRESS>
## Introduction

Compared to the neural networks commonly used in deep learning, Spiking Neural
Networks (SNNs) resemble the animal brain more closely in at least two main
aspects: the way their neurons communicate through spikes, and their dynamics,
which evolve in continuous time. Aside from offering more biologically
plausible neuron models for computational neuroscience, research in the
applications of SNNs is currently blooming because of the rise of neuromorphic
technology. Neuromorphic hardware is directly compatible with SNNs and enables
the design of low-power models for use in battery-operated, always-on devices.

Adversarial examples are an “intriguing property of neural networks” (Szegedy
et al., 2013) by which the network is easily fooled into misclassifying an
input which has been altered in an almost imperceptible way by the attacker.
This property is usually undesirable in applications: it was proven, for
example, that an adversarial attack may pose a threat to self-driving cars
(Eykholt et al., 2018). Because of their relevance to real-world applications,
a large amount of work has been published on this subject, typically following
a pattern where new attacks are discovered, followed by new defense
strategies, in turn followed by proof of other strategies that can still break
through them (see Akhtar and Mian 2018 for a review).

With the advent of real-world applications of spiking networks in neuromorphic
devices, it is essential to make sure they work securely and reliably in a
variety of contexts. In particular, there is a significant need for research
on the possibility of adversarial attacks on neuromorphic hardware used for
computer vision tasks. In this paper, we make an attempt at modifying event-
based data, by adding and removing events, to generate adversarial examples
that fool a spiking network deployed on a convolutional neuromorphic chip.
This offers important insight into the reliability and security of
neuromorphic vision devices, with important implications for commercial
applications.

###  0.1 What is event-based sensing?

Event-based Dynamic Vision Sensors (DVSs) share characteristics with the
mammalian retina and have several advantages over conventional, frame-based
cameras:

  * •

Camera output in the form of events and thus power consumption are directly
driven by changes in the visual scene, omitting output completely in the case
of a static scene.

  * •

Pixels fire independently of each other which results in a stream of events at
microsecond resolution instead of frames at fixed intervals. This enables very
low latency and high dynamic range.

The sparse, asynchronous DVS output does not suit current high-throughput,
synchronous accelerators such as GPUs. To process event-based data
efficiently, neuromorphic hardware is being developed, where neurons are only
updated whenever they receive an event. Spiking neuromorphic systems include
large-scale simulation of neuronal networks for neuroscience research (Furber
et al., 2012) and low-power real-world deployments of machine learning
algorithms. Spiking Convolutional Neural Networks (SCNNs) as well as
conventional Convolutional Neural Networks (CNNs) have been run on
neuromorphic chips such as IBM’s TrueNorth and HERMES (Esser et al., 2016;
Khaddam-Aljameh et al., 2021), Intel’s Loihi (Davies et al., 2018) and
SynSense’s Speck and Dynap-CNN (Liu et al., 2019) for low-power inference. The
full pipeline of event-based sensors, stateful spiking neural networks, and
asynchronous hardware – which is present in SynSense’s Speck – allows for
large gains in power efficiency compared to conventional systems.

###  0.2 Adversarial attacks on discrete data

The history of attack strategies against various kinds of machine learning
algorithms pre-dates the advent of deep learning (Biggio and Roli, 2018), but
the phenomenon received widespread interest when adversarial examples were
first found for deep convolutional networks (Szegedy et al., 2013). In
general, given a neural network classifier $C$ and an input $x$ which is
correctly classified, finding an adversarial perturbation means finding the
smallest $\delta$ such that $C(x+\delta)\neq C(x)$. Here, “smallest” refers to
minimizing $\|\delta\|$, where the norm is chosen arbitrarily depending on the
requirements of the experiment. For example, using the $L^{\infty}$ norm
(maximum norm) will generally make the perturbation less noticeable to a human
eye, while the use of the $L^{1}$ norm will encourage sparsity, i.e. a smaller
number of perturbed pixels.

There are two main challenges in transferring existing adversarial algorithms
to event-based vision:

  * •

The presence of a continuous time dimension, as opposed to frames taken at
fixed intervals;

  * •

The binary discretization of input data and SNN activations, as opposed to
traditional image data (at least 8 bit) and floating point network
activations.

Figure 1: Schematic of the attack procedure on DVS data.

Event-based sensors encode information in the form of events that have a
timestamp, location $(x,y)$ and polarity (lighting increased or decreased).
Because at any point in time an event can either be triggered or not, one can
simply view event-based inputs as binary data by discretizing time (Figure 1).
In this view, the network’s input is a three-dimensional array whose entries
describe the number of events at a location $(x,y)$ and in time bin $t$; an
additional dimension, of size 2, is added due to the polarity of events. If
the time discretization is sufficiently precise, and no more than one event
appears in each voxel, the data can be treated as binary.

In this work, we present new algorithms that adapt the adversarial attacks
SparseFool (Modas et al., 2018), and adversarial patches (Brown et al., 2017),
to work with the time dynamics of spiking neural networks, and with the
discrete nature of event-based data. We focus on the case of white box
attacks, where the attacker has full access to the network and can
backpropagate gradients through it. We test our attacks on the Neuromorphic
MNIST (N-MNIST) (Orchard et al., 2015) and IBM Gestures (Amir et al., 2017)
datasets, which are the most common benchmark datasets within the neuromorphic
community. Importantly, for the first time, we also test the validity of our
methods by deploying the attacks on neuromorphic hardware.

Our contributions can be summarized as follows:

  * •

We contribute algorithms that adapt several adversarial attacks strategies to
event-based data and SNNs, with detailed results to quantify their
effectiveness and scalability.

  * •

We show that these adapted algorithms outperform state-of-the-art algorithms
in the domain of SCNNs.

  * •

We show targeted universal attacks on event-based data in the form of
adversarial patches, which do not require prior knowledge of the input.

  * •

We validate the resulting adversarial examples on an SNN deployed on a
convolutional neuromorphic chip. To the best of our knowledge, this is the
first time the effectiveness of adversarial examples is demonstrated directly
on neuromorphic hardware.

## Related Work

Despite the growing number of SNNs deployed on digital (Davies et al., 2018)
and analog (Moradi et al., 2018) neuromorphic hardware, robustness to
adversarial perturbations has received comparatively little attention by the
research community. Some methods proposed for attacking binary inputs have
focused on brute-force searches with heuristics to reduce the search space
(Bagheri et al., 2018; Balkanski et al., 2020). Algorithms of this family do
not scale well to large input sizes, as the number of queries made to the
network grows exponentially. In particular, this becomes a serious problem
when the time dimension is added, greatly increasing the dimensionality of the
input.

In Marchisio et al. (2021), the authors demonstrate various algorithms for
attacking DVS data that serves as input to an SNN. While some attacks are
brute force ("Dash Attack", "Corner Attack") and therefore do not scale,
"Frame Attack" simply adds events in hard-coded locations and produces
perturbations that are orders of magnitudes larger than all of the
perturbations of the algorithms presented here. The only algorithm that
exploits gradient information is "Sparse Attack", against which we
compare111We followed the implementation at
https://github.com/albertomarchisio/DVS-Attacks.. Unfortunately, the authors
do not report quantitative results on the magnitudes of their perturbations
(i.e. how many spikes are added or removed), which is an important metric for
demonstrating efficiency of the attack.

Sharmin et al. (2020) demonstrate interesting properties of the inherent
robustness of SNNs to adversarial attacks, with variations depending on the
training method. Their work, however, only uses static image data with
continuous pixel values converted to Poisson spike input frequencies, which is
rather different from working with actual event-based camera input.

As Liang et al. (2020) note, white-box attacks can exploit “surrogate”
gradients in order to calculate gradients of a loss function with respect to
the input in SNNs. Not only do these gradients reflect the true dependency of
the output to the input, but they also capture temporal information via
Backpropagation Through Time (BPTT). In their work, the authors use a Dirac
function for their surrogate gradient, which causes, as they report,
considerable “vanishing gradient” problems. While they show good results, they
were forced to use additional tricks such as probabilistic sampling and an
artificial construction of gradients termed “Restricted Spike Flipper” in
their manuscript. In our work we solved the vanishing gradient issue by
resorting to a more effective surrogate function (Section 0.5).

Outside of the domain of SNNs, various studies explore the adversarial
robustness of conventional neural networks deployed on digital accelerators or
analog in-memory computing based accelerators. For example, Stutz et al.
(2020) demonstrate that standard networks are susceptible to bit errors in the
SRAM array storing the quantized weights in deep learning accelerators. The
authors further show that one can mitigate this susceptibility via random bit
flipping during training. While this work performs experiments using noise
models obtained from various SRAM arrays, it still lacks a full-fledged
hardware demonstration. Cherupally et al. (2021) obtain a noise model of a
RRAM-based crossbar often found in analog in-memory computing architectures.
The noise model is then used in simulation to study the adversarial robustness
of neural networks deployed on such a crossbar.

Besides attacking neural networks via input perturbation, a large body of work
on directly attacking the physical hardware exists. Hardware fault injection
attacks (see Giraud and Thiebeauld 2004 for a survey) are mostly based on
voltage glitching, electromagnetic pulses, or rapid row activation in DRAM
(Kim et al., 2014).

In this work, we focus on input-based attacks, which we verify in simulations
and on commercially available spiking neuromorphic hardware. This step is
important to determine real-world effectiveness, as the exact simulation of
asynchronous, event-based processing with microsecond resolution is
prohibitively expensive on conventional hardware.

## Methods

###  0.3 Attack strategies

####  0.3.1 SparseFool on discrete data

To operate on event-based data efficiently, the ideal adversarial algorithm
requires two main properties: sparsity and scalability. Scalability is needed
because of the increased dimensionality given by the additional time
dimension. Sparsity ensures that the number of events added or removed is kept
to a minimum. One approach that combines the above is SparseFool (Modas et
al., 2018), which iteratively finds the closest point in $L^{2}$ on the
decision boundary of the network using the DeepFool algorithm (Moosavi-
Dezfooli et al., 2015) as a subroutine, followed by a linear solver that
enforces sparsity and boundary constraints on the perturbation. DeepFool finds
the smallest perturbation in $L^{2}$ by iteratively moving the input in the
direction orthogonal to the linearized decision boundary around the current
input. Since decision boundaries of neural networks are non-linear, this
process has to be repeated until a misclassification is triggered. Because
SNNs have discrete outputs (the number of spikes over time for each output
neuron), it is easier to suffer from vanishing gradients as the perturbation
approaches the decision boundary. This occurs because DeepFool calculates the
perturbation that just reaches the decision boundary, which is very small when
the input is already close to the decision boundary. Therefore, adding this
small perturbation to the input might not reflect in the number of emitted
spikes as the membrane potential must cross the spiking threshold. We made the
following changes to overcome these issues. Firstly, we clamped the
perturbation at every iteration of DeepFool so that it was no smaller than a
value $\eta$, in order to protect against vanishing gradients. $\eta$ was
treated as a hyperparameter that should be kept as small as possible without
incurring vanishing gradients. Without $\eta$, SparseFool yields a success
rate of 12.9% on 100 samples of the N-MNIST dataset, where the success rate is
the percentage of attacked samples that led to a misclassification out of the
samples that were originally classified correctly by the network. Secondly, to
account for the discreteness of event-based data, we rounded the output of
SparseFool to the nearest integer at each iteration. Finally, SparseFool
normally involves upper and lower bounds $l$ and $u$ on pixel values (normally
set, for images, to $l=0;u=255$). We exploit these to enforce the binary
constraint on the data ($l=0;u=1$), or, in the on-chip experiments, to fix a
maximum firing rate in each time bin, which is the same as that of the
original input ($l=0;u=\max(\text{input)}$). From now on, we will refer to
this variant as SpikeFool.

####  0.3.2 Adversarial patches

As the name suggests, adversarial patches are perturbations that are
accumulated in a certain region (patch) of the image. The idea is that these
patches are generated in a way that enables the adversary to place them
anywhere in the image. This attack is targeted to a desired label, and
universal, i.e. not input-specific. To test a more realistic scenario where an
adversary could potentially perform an attack without previous knowledge of
the input, we apply these patches to the IBM hand gesture dataset. We note
that the prediction of the CNN trained on this dataset is mostly determined by
spatial location of the input. For example, the original input of “Right Hand
Wave” is not recognized as such if it is shifted or rotated by a substantial
amount. In order to simulate effective realistic attacks, we choose to limit
both computed and random attack patches to the area of where the actual
gesture is performed. As in Brown et al. (2017), we generate the patches using
Projected Gradient Descent (PGD) on the log softmax value of the target output
neuron. PGD is performed iteratively on different images of the training set
and the position of the patch is randomized after each sample. For each item
in the training data, the algorithm updates the patch until the target label
confidence has reached a pre-defined threshold. The algorithm skips the point
if the original label equals the target label. This process is repeated for
every training sample and for multiple epochs. To measure the effectiveness of
our computed patches, we also generate random patches of the same size, and
measure the target success rates. In a random patch, every pixel has a 50%
chance of emitting a spike at each time step.

###  0.4 Datasets

N-MNIST consists of 300 ms-long recordings of MNIST digits that are captured
using a three-fold saccadic motion of a DVS sensor (Orchard et al., 2015). We
bin the events of each recording into 60 time steps, capping the maximum
number of events to 1 per pixel.

IBM Gestures dataset consists of recordings of 11 classes of human gestures,
captured under three different lighting conditions (Amir et al., 2017). For
this dataset, the model must have the ability to process temporal features to
distinguish between clockwise and counterclockwise versions of the same
gesture. We bin 200 ms slices of recordings into 20 frames each, again capping
the frames to 1 per pixel. For experiments on the chip we choose a higher
temporal resolution and bin the same 200 ms slices into 100 frames each.

###  0.5 Network models

The spiking networks used for the N-MNIST and IBM Gestures tasks are trained
using the publicly available PyTorch-based SNN library
Sinabs222http://sinabs.ai which supports the same non-leaky integrate-and-fire
neurons available on the neuromorphic chip.

Models used for N-MNIST were trained using _weight transfer_ , whereby an
equivalent CNN is trained on accumulated frames (i.e. summing the data over
the time dimension), and the CNN weights are transferred to the spiking
network (Rueckauer et al., 2017; Sorbaro et al., 2020). The model we used is a
LeNet-5 architecture with 20, 32, 128, 500 and 10 channels respectively. The
network achieves 85.05% classification test accuracy with full precision
weights and 85.2% with 8-bit quantized weights.

For the IBM Gestures task, training is done using BPTT since for this dataset
we have to learn temporal features as well. We make use of a surrogate
gradient in the backwards pass to enable learning despite the discontinuous
nature of spikes (Neftci et al., 2019). The model has a LeNet-5 architecture
plus batchnorm layers with 8, 8, 8, 64 and 11 channels respectively. This
network achieves a classification test accuracy of 84.2%. The accuracy is
retained if the network weights are quantized to 8 bits. The network used for
the on-chip experiments does not have batch-normalization layers as they
introduce biases which are not supported on the hardware.

###  0.6 Experiments on the neuromorphic chip

We used a multi-core spiking neural network processing chip prototype from
SynSense, called “Speck2b”. It uses asynchronous logic design to keep dynamic
power that is consumed whenever spikes are routed to a minimum. That means
that neurons update and spike only when an input is received, but are not
limited to time bins of clock cycles, which is very different than
conventional von Neumann hardware. The chip and its 327 000 neurons are fully
configurable, supporting convolutional, linear, as well as pooling
connectivity. Event input (from datasets such as N-MNIST and IBM gestures) is
sent to the FPGA as a block of events in $(t,p,x,y)$ format. The FPGA then
handles the feeding to the chip at the correct timestamps. This is important
as the chip itself, using fully asynchronous logic, does not understand the
notion of time steps. Every neuron will compute its input on demand and fire
as soon as it is ready. Since events have to be handled in a sequential
manner, this can sometimes lead to slight changes in the order of input
events, which then has consequences for neuron activation further downstream
(in later layers). If many events arrive in a short time span, they are
buffered depending on the throughput limitations of each core. Those
limitations together with weight quantization on the neuromorphic hardware
lead to slight differences in output when compared to simulation.

The multi-core, single-chip neuromorphic processor we used for the experiments
features an integrated DVS sensor for real-time, fully integrated vision and
is commercially available for programming as part of a development kit, which
includes USB connectivity, power management and an FPGA for communication.
Since the chip features an on-chip DVS sensor, its main application is in the
vision domain, making it even more relevant for our study.

On the chip, weight precision, number of computations per second and
throughput are reduced, as the hardware is optimized for very low power
consumption. The networks detailed in the previous sections have to be
modified in order to make them suitable for on-chip inference: their weights
are rescaled and discretized as required by the chip’s 8-bit weight precision.
This is one factor that can lead to a degradation in prediction accuracy when
compared to simulations. A second and more important factor why simulations do
not mimic our chip’s behavior exactly is the need for time discretization when
training our networks off-chip (see Figure 1).

As this work focuses on white-box attacks, we first compute the adversarial
examples using the network simulation on the computer, and then test both
original and attacked spiketrains in simulation and on the chip. However, the
simulation and the attack work in discrete time, while the chip receives
events in continuous time. In order to convert the discrete-time attacked
raster back to a list of events for the chip, we compare the original and
attacked rasters, identifying new events added by the attack and adding them
to the original list (Figure 1). We empirically found very few events were
removed by SparseFool; for example, in the N-MNIST experiment, there were 28
additions per each removal (see supplementary material for more information).
For simplicity, we therefore chose to ignore removals for on-chip experiments.

## Results

Figure 2: Examples of adversarial inputs on the, N-MNIST (top) and IBM
Gestures (bottom) datasets, as obtained by the SpikeFool method. The captions
show the original (true) label, correctly identified, and the class later
identified by the model. The data was re-framed in time for convenience of
visualization. Red indicates added spikes. See the supplementary video for
more examples and motion visualization.
Figure 3: Our method (b,d) significantly reduces the magnitude of the
perturbations introduced by the adversary. The runner-up method proposed in
Liang et al. (2020) (a,c) adds visibly more events and is thus less stealthy.

###  0.7 Adversarial attack performance

Table 1 shows the comparison of SpikeFool with current state-of-the-art attack
algorithms in the spiking domain on two widely used benchmarks. We compare
various metrics such as success rate, median elapsed time per sample, and,
most importantly, the median perturbation size. Results from Marchisio et al.
(2021) and Liang et al. (2020) are drawn from our own implementations, closely
following published code (if available) from the authors.

| Attack strategy | Hardware |  Success Rate (%) |  Median Elapsed Time (s/sample) |  Median No. Queries |  Median $L^{0}$
---|---|---|---|---|---|---
N-MNIST | SpikeFool ($\eta=0.2,\lambda=2$) | GPU | 99.53 | $24.87\pm 0.0815$ | 45 | $\mathbf{256}$
SpikeFool ($\eta=0.5,\lambda=2$) | GPU | 99.88 | $12.00\pm 0.072$ | 27 | 272
Liang et al. 2020 | GPU | $\mathbf{100.00\pm 0.00}$ | $0.69\pm 1e-4$ | $\mathbf{2}$ | $865.3\pm 6.37$
Marchisio et al. 2021 | GPU | $74.41\pm 0.33$ | $\mathbf{0.64\pm 0.001}$ | 3 | $3.23e4\pm 360.32$
IBM | SpikeFool ($\eta=0.1,\lambda=3$) | GPU | $\mathbf{100.00}$ | $2.56\pm 0.003$ | 11 | 311
SpikeFool ($\eta=0.1,\lambda=2$) | GPU | 99.87 | $2.4\pm 0.002$ | 11 | 200
SpikeFool ($\eta=0.1,\lambda=1$) | GPU | 97.61 | $2.85\pm 0.02$ | 17 | $\mathbf{116}$
SpikeFool | Speck2b | 88.05 |  |  |
Liang et al. 2020 | GPU | $99.77\pm 0.126$ | $0.53\pm 0.004$ | 9 | $1345.5\pm 11.48$
| Marchisio et al. 2021 | GPU | $92.44\pm 0.05$ | $\mathbf{0.14\pm 2e-4}$ | $\mathbf{2}$ | $227343\pm 54.56$

† Samples for which the attack was unsuccessful were considered to have
undefined $L^{0}$. Attacks were repeated 5 times on 1000 samples. We report
the mean and standard deviation for non-deterministic metrics. The median
number of queries did not show any variation over 5 repetitions.

Table 1: Result comparison between SpikeFool and related works. $\eta$
indicates the minimum step size for updates to the perturbation: higher values
of $\eta$ find less precise perturbations (larger $L^{0}$ values), but are
sometimes needed in order to prevent zero-gradient issues within the
algorithm. $\lambda$ is the sparsity parameter: lower $\lambda$ (with a
minimum of 1) yields sparser results, but gives a slightly lower success rate.
The success rate is defined as the fraction of samples that were initially
correctly classified, for which the attack algorithm converged to an
adversarial example that the network classifies incorrectly.

We find that the attack presented in Marchisio et al. (2021) largely fails to
achieve reasonable perturbation magnitudes on both benchmarks. This is likely
because of the lack of bounds and rounding in their method, which result in
large deviations. In contrast, the attack presented in Liang et al. (2020)
achieves near perfect success rates in very short time at relatively low
perturbation sizes. However, this speed comes at the cost of perturbation
size: our method yields perturbations that are up to $11\times$ smaller than
the ones generated by Liang et al. (2020). To put these numbers into
perspective, Figure 3 compares the adversarial samples generated by both
methods. One can observe the clear difference between the two methods and
could argue that attacks generated by Liang et al. (2020) are more visible.
Interestingly, we observe that SpikeFool often resorts to inserting small
patches limited to key areas. We believe this is due to the fact that the
network is inherently robust to salt-and-pepper noise and that introducing
localized patches is by far more effective. Figure 2 and the supplementary
video show additional examples of successful attacks. Further information
about how many spikes are added and removed during training can be found in
the supplementary material.

We also run a random subset of 777 DVS Gesture samples that were attacked
using SpikeFool ($\lambda=2.0,\eta=0.3$) on our neuromorphic hardware and
observe a success rate of 88.05% misclassified samples, in comparison to
93.28% in simulation. This performance is lower compared to the one reported
in Table 1 due to the much higher time resolution needed on chip, which makes
it harder to find an attack. A possible reason for this discrepancy lies in
how the chip is limited in computing capacity by weight precision and
restricted throughput per time unit, which causes some of the input events to
be dropped. Furthermore, the conversion of binned data back into lists of
spikes is necessarily lossy. In terms of attack efficiency, we observe a
median difference in number of spikes of 903 among the attacks that were
successful on chip, corresponding to a median 9.3% increase in the number of
spikes per sample.

###  0.8 Adversarial patches

Although we have demonstrated that one can achieve high success rates on
custom spiking hardware that operates with microsecond precision, the
applicability of this method is still limited, as the adversary needs to
suppress and add events at high spatial and temporal resolution, thus making
the assumption that the adversary can modify the event-stream coming from the
DVS camera. Furthermore, SpikeFool assumes knowledge of the model and requires
computing the perturbation offline, which is not feasible in a timely manner.
In a more realistic setting, the adversary is assumed to generate
perturbations by changing the input the DVS camera receives on the fly, by
e.g. adding a light-emitting device to the visual input of the DVS camera.

Using the training data from the IBM Gestures dataset, we generate an
adversarial patch for each target class with high temporal precision (event
samples of 200 ms are binned using 0.5 ms-wide bins) and evaluate the
effectiveness in triggering a targeted misclassification both in simulation
and on-chip using the test data. To simulate spatial imprecision during
deployment, each test sample is perturbed by a patch that was randomly placed
within the area of the original gesture. Table 2 summarizes our findings on
target success rates for generated and random patches. Simulated results show
high success rates, and on-chip performance shows a slight degradation, which
can be expected due to weight quantization on the tested specialized hardware.
We also find that the chip has trouble processing inputs because most of the
added patch events occur concentrated in the beginning of recordings in a
large transient peak. In one case, the targeted attack for label “Arm Roll“
mostly fails on chip as not all events are processed, which makes it harder to
differentiate from “Hand Clap”, a similar gesture that occurs in the same
central spatial location. This could be mitigated by limiting the number of
events in a patch to ensure that they could all be correctly processed on the
chip.

| Target label |  Hand clap |  RH Wave |  LH Wave |  RH Clockwise |  RH Counter Clockwise |  LH Clockwise |  LH Counter Clockwise |  Arm Roll |  Air Drum |  Air Guitar |  Other
---|---|---|---|---|---|---|---|---|---|---|---|---
| Adversarial patch | 90.3 | 99.0 | 89.8 | 87.3 | 79.7 | 49.7 | 51.5 | 63.6 | 79.1 | 92.3 | 64.7
Adv. patch (on-chip) | 94.0 | 89.0 | 94.1 | 81.3 | 65.1 | 35.9 | 43.8 | 5.0 | 82.7 | 87.3 | 66.8
Random patch | 18.8 | 80.7 | 77.0 | 0 | 0 | 3.6 | 0.6 | 0 | 0 | 12.6 | 16.6
Rand. patch (on-chip) | 43 | 76.8 | 72.2 | 0 | 0 | 9.0 | 2.4 | 0 | 0 | 0 | 17.7

Table 2: Adversarial patches for different target labels were evaluated on–
and off–chip. Shown here are the success rates in percent for each target
label. An attack is considered successful if the original label is not the
target label and the network predicts the target label when the patch is
applied.

We compare this result with a baseline of randomly generated patches, and we
observe that two labels, namely “Left“ and “Right Hand Wave” subsume all other
attacked labels in this case. This hints that randomly injecting events in
various locations is not enough to perturb network prediction to a desired
label and that our algorithm succeeds in finding a meaningful patch. We find
that generating patches with lower temporal resolution heavily degrades
performance on-chip, as the chip operates using microsecond precision. We also
perform ablation studies where we move the patch far from the position where
the gesture is expected and observe that the patch mostly triggers a different
misclassification, mostly the gesture that is expected at the new position. We
find that this observation originates from the fact that the network heavily
relies on the spatial location of the gesture and one should, if the hardware
allows, consider larger networks that are invariant to the position of the
gesture. To summarize, adversarial patches are effective in triggering a
targeted misclassification both on– and off–chip compared to randomly
generated ones. Figure 4 and the supplementary video show examples of
successful patch attacks. Importantly, these attacks are universal, meaning
that they can be applied to any input and do not need to be generated for each
sample.

Figure 4: Examples of adversarial patches successfully applied to a single
“right hand clockwise” data sample, with different target classes. See also
the supplementary video for motion visualization and more examples.

###  0.9 Defense via adversarial training

Once it is known that a model or system is sensitive to a certain type of
adversarial attack, it is natural to investigate whether there is a way to
build a network that is more resistant to these attacks. We therefore
experimented with adversarial training using the TRadeoff-inspired Adversarial
Defense via Surrogate-loss minimization (TRADES) method (Zhang et al., 2019).
The method adds a new term to the loss function, which minimizes the Kullback-
Leibler divergence between the output of the network on the original input and
the output when the adversarial example is presented:

$\mathcal{L}_{\text{rob}}=\mathcal{L}+\frac{\beta_{\text{rob}}}{B}\operatorname{D_{\text{KL}}}(f(\mathbf{x}_{\text{adv}});f(\mathbf{x}_{0})).$

Here, $B$ is the batch size, $\beta_{\text{rob}}$ is the parameter that
defines the trade-off between robustness and accuracy, $f$ is the network and
$\mathbf{x}_{\text{adv}}$ is the adversarial input. To find
$\mathbf{x}_{\text{adv}}$ at training time, we use a PGD-based attack, since
it can be batched — but we attack the resulting networks using SpikeFool at
test time.

More specifically, we use PGD in the $L^{\infty}$ domain and choose
$\epsilon=0.5$ as the maximum perturbation, with $N_{\text{pgd}}=5$ attack
steps. We use PGD in the spiking domain by accumulating the gradients in full-
precision using a straight-through estimator (Bengio et al., 2013). The
details of our spiking-adapted implementation of PGD are described in the
supplementary material, including extensions, and a full comparison with
SpikeFool.

Although we use a much simpler attack strategy at training time, we found that
it produced perturbations of reasonable sizes while being extremely efficient
and sufficiently effective: for the choices of $\beta_{\text{rob}}$ we
considered, we see that the adversarially-trained network requires stronger
and less stealthy attacks before it is fooled (Figure 5). As expected,
SparseFool’s success rate is still high, since it aims to find a solution no
matter the costs; but there is an increase in the number of added spikes
required, which is indeed a sign of robustness (Table 3). Further work is
required for a comprehensive investigation of other defense strategies.

| Network |  Test Accuracy (%) |  Success Rate (%) |  Median $L^{0}$
---|---|---|---|---
IBM |  $\beta_{\text{rob}}=0.0$ (normal) | 79.23 | 98.19 | 438.0
$\beta_{\text{rob}}=0.01$ | 81.36 | 98.78 | 511.0
$\beta_{\text{rob}}=0.05$ | 75.27 | 97.65 | 1434.0
$\beta_{\text{rob}}=0.1$ | 73.16 | 95.46 | 1316.0

Table 3: Training with a small robustness term ($\beta_{\text{rob}}=0.01$)
increases generalization and therefore improves the test accuracy of the
network. For increasing values of $\beta_{\text{rob}}$, the amount of changed
events needed to trigger a misclassification increases dramatically (more than
$3\times$), while the performance of the network decreases (hence the name
“tradeoff-inspired”). Figure 5: Adversarial training visibly increases the
number of events that need to be added/removed in order to trigger a
misclassification.

## Discussion

We studied the possibility of fooling SNNs through adversarial perturbations
to DVS data, and verified these perturbations on a spiking convolutional
neuromorphic chip. There were two main challenges to this endeavor: the
discrete nature of event-based data, and their dependence on time. DVS attacks
also have different sparsity requirements, because the magnitude of the
perturbation is measured in terms of number of events added or removed. For
this purpose, we adopted a surrogate-gradient method and backpropagation-
through-time to perform white-box attacks on spiking networks. We presented
SpikeFool, and adapted version of SparseFool, which we compared to current
state-of-the-art methods on well-known benchmarks. We find that SpikeFool
achieves near perfect success rates at lowest perturbation magnitudes on time-
discretized samples of the N-MNIST and IBM Gestures datasets. In the best
cases, the attack requires the addition of less than a hundred events over 200
ms. To the best of our knowledge, we were also the first to show that the
perturbation is effective on a network deployed on a neuromorphic chip,
implying that the method is resilient to the small but non-trivial mismatch
between simulated and deployed networks.

Additionally, since SpikeFool computes perturbations offline and not on a live
stream of DVS events, we also investigated a more realistic setting, where an
adversary can inject spurious events in the form of a patch inserted into the
visual field of the DVS camera. We demonstrated that we can generate patches
for different target labels. Although these patches require a much higher
amount of added events, they do not require prior knowledge of the input
sample and therefore offer a realistic way of fooling deployed neuromorphic
systems. A natural next step would be to understand whether it is possible to
build real-world patches that can fool the system from a variety of distances
and orientations, as Eykholt et al. (2018) did for photographs. Moreover, it
will be interesting to see how important knowledge about the architecture is
and if one can generate patches by having access to a network that differs
from the one deployed.

## Conflict of Interest Statement

Author J.B. is employed by IBM. Authors G.L., Y.H. and S.S. are employed by
SynSense. The remaining authors declare that the research was conducted in the
absence of any commercial or financial relationships that could be construed
as a potential conflict of interest.

## Author Contributions

J.B. and M.S. conceived and designed the research. All authors developed
software and simulations. J.B., M.S., G.L. and Y.H. performed experiments and
collected data. J.B., M.S., G.L. and Y.H. analyzed and interpreted the data.
J.B., M.S. and G.L. drafted the manuscript. All authors performed critical
revision of the manuscript and approved the final version for publication.

## Funding

M.S. was supported by an ETH AI Center postdoctoral fellowship during part of
this work.

## Acknowledgments

The authors would like to thank Seyed Moosavi-Dezfooli for valuable help in
understanding and applying the SparseFool and DeepFool algorithms, and the
Algorithms team at SynSense for their support, suggestions, and assistance in
using the Speck chip. We also thank the anonymous reviewers for many helpful
comments.

## Data Availability Statement

The code and links to the datasets used for running the experiments are
publicly available at https://github.com/jubueche/SPAA

## References

  * Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ , 2013.
  * Eykholt et al. (2018) Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song.  Robust physical-world attacks on deep learning visual classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1625–1634, 2018.
  * Akhtar and Mian (2018) Naveed Akhtar and Ajmal Mian.  Threat of adversarial attacks on deep learning in computer vision: A survey.  _IEEE Access_ , 6:14410–14430, 2018.  10.1109/ACCESS.2018.2807385.
  * Furber et al. (2012) Steve B Furber, David R Lester, Luis A Plana, Jim D Garside, Eustace Painkras, Steve Temple, and Andrew D Brown.  Overview of the spinnaker system architecture.  _IEEE Transactions on Computers_ , 62(12):2454–2467, 2012.
  * Esser et al. (2016) Steven K Esser, Paul A Merolla, John V Arthur, Andrew S Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J Berg, Jeffrey L McKinstry, Timothy Melano, Davis R Barch, et al.  Convolutional networks for fast, energy-efficient neuromorphic computing.  _Proceedings of the national academy of sciences_ , 113(41):11441–11446, 2016.
  * Khaddam-Aljameh et al. (2021) R. Khaddam-Aljameh, M. Stanisavljevic, J. Fornt Mas, G. Karunaratne, M. Braendli, F. Liu, A. Singh, S. M. Müller, U. Egger, A. Petropoulos, T. Antonakopoulos, K. Brew, S. Choi, I. Ok, F. L. Lie, N. Saulnier, V. Chan, I. Ahsan, V. Narayanan, S. R. Nandakumar, M. Le Gallo, P. A. Francese, A. Sebastian, and E. Eleftheriou.  Hermes core – a 14nm cmos and pcm-based in-memory compute core using an array of 300ps/lsb linearized cco-based adcs and local digital processing.  In _2021 Symposium on VLSI Technology_ , pages 1–2, 2021.
  * Davies et al. (2018) Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, Yuyun Liao, Chit-Kwan Lin, Andrew Lines, Ruokun Liu, Deepak Mathaikutty, Steven McCoy, Arnab Paul, Jonathan Tse, Guruguhanathan Venkataramanan, Yi-Hsin Weng, Andreas Wild, Yoonseok Yang, and Hong Wang.  Loihi: A neuromorphic manycore processor with on-chip learning.  _IEEE Micro_ , 38(1):82–99, 2018.  10.1109/MM.2018.112130359.
  * Liu et al. (2019) Qian Liu, Ole Richter, Carsten Nielsen, Sadique Sheik, Giacomo Indiveri, and Ning Qiao.  Live demonstration: face recognition on an ultra-low power event-driven convolutional neural network asic.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 0–0, 2019.
  * Biggio and Roli (2018) Battista Biggio and Fabio Roli.  Wild patterns: Ten years after the rise of adversarial machine learning.  _Pattern Recognition_ , 84:317–331, 2018.
  * Modas et al. (2018) Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard.  Sparsefool: a few pixels make a big difference.  _CoRR_ , abs/1811.02248, 2018.  URL http://arxiv.org/abs/1811.02248.
  * Brown et al. (2017) Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer.  Adversarial patch.  _CoRR_ , abs/1712.09665, 2017.  URL http://arxiv.org/abs/1712.09665.
  * Orchard et al. (2015) Garrick Orchard, Ajinkya Jayawant, Gregory K. Cohen, and Nitish Thakor.  Converting static image datasets to spiking neuromorphic datasets using saccades.  _Frontiers in Neuroscience_ , 9:437, 2015.  ISSN 1662-453X.  10.3389/fnins.2015.00437.  URL https://www.frontiersin.org/article/10.3389/fnins.2015.00437.
  * Amir et al. (2017) Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, and Dharmendra Modha.  A low power, fully event-based gesture recognition system.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7388–7397, 2017.  10.1109/CVPR.2017.781.
  * Moradi et al. (2018) S. Moradi, N. Qiao, F. Stefanini, and G. Indiveri.  A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (dynaps).  _IEEE Transactions on Biomedical Circuits and Systems_ , 12(1):106–122, 2018.  10.1109/TBCAS.2017.2759700.
  * Bagheri et al. (2018) Alireza Bagheri, Osvaldo Simeone, and Bipin Rajendran.  Adversarial training for probabilistic spiking neural networks, 2018.
  * Balkanski et al. (2020) Eric Balkanski, Harrison Chase, Kojin Oshiba, Alexander Rilee, Yaron Singer, and Richard Wang.  Adversarial attacks on binary image recognition systems.  _CoRR_ , abs/2010.11782, 2020.  URL https://arxiv.org/abs/2010.11782.
  * Marchisio et al. (2021) Alberto Marchisio, Giacomo Pira, Maurizio Martina, Guido Masera, and Muhammad Shafique.  Dvs-attacks: Adversarial attacks on dynamic vision sensors for spiking neural networks.  In _2021 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–9. IEEE, 2021.
  * Sharmin et al. (2020) Saima Sharmin, Nitin Rathi, Priyadarshini Panda, and Kaushik Roy.  Inherent adversarial robustness of deep spiking neural networks: Effects of discrete input encoding and non-linear activations.  In _European Conference on Computer Vision_ , pages 399–414. Springer, 2020.
  * Liang et al. (2020) Ling Liang, Xing Hu, Lei Deng, Yujie Wu, Guoqi Li, Yufei Ding, Peng Li, and Yuan Xie.  Exploring adversarial attack in spiking neural networks with spike-compatible gradient.  _CoRR_ , abs/2001.01587, 2020.  URL http://arxiv.org/abs/2001.01587.
  * Stutz et al. (2020) David Stutz, Nandhini Chandramoorthy, Matthias Hein, and Bernt Schiele.  Bit error robustness for energy-efficient dnn accelerators, 2020.  URL https://arxiv.org/abs/2006.13977.
  * Cherupally et al. (2021) Sai Kiran Cherupally, Jian Meng, Adnan Siraj Rakin, Shihui Yin, Injune Yeo, Shimeng Yu, Deliang Fan, and Jae sun Seo.  Improving the accuracy and robustness of rram-based in-memory computing against rram hardware noise and adversarial attacks.  _Semiconductor Science and Technology_ , 37, 2021.
  * Giraud and Thiebeauld (2004) Christophe Giraud and Hugues Thiebeauld.  A survey on fault attacks.  In Jean-Jacques Quisquater, Pierre Paradinas, Yves Deswarte, and Anas Abou El Kalam, editors, _Smart Card Research and Advanced Applications VI_ , pages 159–176, Boston, MA, 2004. Springer US.  ISBN 978-1-4020-8147-7.
  * Kim et al. (2014) Yoongu Kim, Ross Daly, Jeremie Kim, Chris Fallin, Ji Hye Lee, Donghyuk Lee, Chris Wilkerson, Konrad Lai, and Onur Mutlu.  Flipping bits in memory without accessing them: An experimental study of dram disturbance errors.  _SIGARCH Comput. Archit. News_ , 42(3):361–372, jun 2014.  ISSN 0163-5964.  10.1145/2678373.2665726.  URL https://doi.org/10.1145/2678373.2665726.
  * Moosavi-Dezfooli et al. (2015) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  Deepfool: a simple and accurate method to fool deep neural networks.  _CoRR_ , abs/1511.04599, 2015.  URL http://arxiv.org/abs/1511.04599.
  * Rueckauer et al. (2017) Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu.  Conversion of continuous-valued deep networks to efficient event-driven networks for image classification.  _Frontiers in neuroscience_ , 11:682, 2017.
  * Sorbaro et al. (2020) Martino Sorbaro, Qian Liu, Massimo Bortone, and Sadique Sheik.  Optimizing the energy consumption of spiking neural networks for neuromorphic applications.  _Frontiers in neuroscience_ , 14:662, 2020.
  * Neftci et al. (2019) Emre O Neftci, Hesham Mostafa, and Friedemann Zenke.  Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks.  _IEEE Signal Processing Magazine_ , 36(6):51–63, 2019.
  * Zhang et al. (2019) Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.  Theoretically principled trade-off between robustness and accuracy.  In _International Conference on Machine Learning_ , pages 7472–7482. PMLR, 2019.
  * Bengio et al. (2013) Yoshua Bengio, Nicholas Léonard, and Aaron Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation, 2013.  URL https://arxiv.org/abs/1308.3432.
  * Madry et al. (2019) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks, 2019.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax, 2017.
  * LeCun and Cortes (2010) Yann LeCun and Corinna Cortes.  MNIST handwritten digit database.  2010\.  URL http://yann.lecun.com/exdb/mnist/.
  * Kingma and Ba (2014) Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _International Conference on Learning Representations_ , 12 2014.

##  1 PGD adapted to the spiking domain

###  1.1 Continuous-discrete PGD

PGD (Madry et al., 2019) is a standard attack algorithm that iteratively
computes the gradient of a loss function w.r.t. the inputs, takes a step in
the direction of the gradient and, if needed, reduces the perturbation to
fulfill a constraint on the maximum perturbation magnitude. For this algorithm
to work in the spiking domain, some changes are necessary. The first
modification is given by rounding the adversarial input after every update.
However, doing so, updates are retained only if the gradient magnitude is
large enough, otherwise the small changes made to the input are lost due to
the subsequent discretization. Instead, we use straight-through estimation
(Bengio et al., 2013), an approach that prevents this loss of information: we
keep a continuous version of the image as a copy, but use the gradients
computed on the discretized image to update the continuous version which is
kept in memory. This lets us accumulate updates across iterations
independently of rounding.

Secondly, after convergence, to adapt PGD to the scenario where we want to
find the smallest perturbation that triggers a misclassification, we sort the
continuous values based on the difference between the original data and the
continuous perturbed version. We then iterate through the sorted list of
indices and create a final discrete data sample starting with the original
input and flipping each binary value in order, until a misclassification is
triggered. The latter process improves the perturbation size, but comes at a
high computational cost. Therefore, we chose to skip this step when using
continuous-discrete PGD for adversarial training (Section 0.9 of the main
text).

###  1.2 Probabilistic PGD

We also devised an alternative way of using PGD on discrete data, which we
call “Probabilistic PGD ". Probabilistic PGD works by assuming that the binary
input was generated by sampling from a series of independent Bernoulli random
variables. This approach aligns with how the DVS camera generates thexamplese
binary data: the probability of emitting a spike at time $t$ is proportional
to the change in light intensity, a continuous metric. For each round of PGD,
the input is sampled in a differentiable manner by the Gumbel-softmax
reparameterization trick (Jang et al., 2017):

$\mathbf{x_{\text{adv}}}=\sigma\left(\frac{\text{log}(\mathbf{r})-\text{log}(\mathbf{1}-\mathbf{r})+\text{log}(\mathbf{p}_{\text{adv}})-\text{log}(\mathbf{1}-\mathbf{p}_{\text{adv}})}{T}\right)$

where $\mathbf{r}\sim\mathcal{U}(\mathbf{0},\mathbf{1})$, $T=0.01$ is a
temperature parameter, and $\sigma$ is the sigmoid function. Note that bold-
faced variables indicate matrices in the shape of the input to the neural
network. The underlying probabilities $\mathbf{p}_{\text{adv}}$, instead of
the pixel values $\mathbf{x_{\text{adv}}}$, are updated using the gradient
obtained from the loss function that is minimized by PGD. We observed that
this generally improved performance compared to the PGD version explained
above (Table 4). Gradients are averaged over $N_{\text{mc}}=10$ samples of
$\mathbf{r}$. It should be noted that the need for a gradient sampling
procedure significantly increases the runtime.

###  1.3 Comparison to SpikeFool

Table 4 contains the results of our comparison analysis.

| Attack strategy |  Success Rate (%) |  Median Elapsed Time (s/sample) |  Median No. Queries |  Median $L^{0}$
---|---|---|---|---|---
N-MNIST | Continuous-discrete PGD | 48.63 | 72.56 | 1052 | †
Probabilistic PGD | 100.00 | 88.99 | 1091 | 839
SpikeFool ($\eta=0.2,\lambda=2$) | 99.76 | 30.22 | 45 | 254
SpikeFool ($\eta=0.5,\lambda=2$) | 99.88 | 13.08 | 26 | 268
IBM | Continuous-discrete PGD | 88.30 | 16.68 | 747 | 695
Probabilistic PGD | 99.22 | 16.80 | 555 | 303
SpikeFool ($\eta=0.1,\lambda=3$) | 100.00 | 2.78 | 11 | 310
SpikeFool ($\eta=0.1,\lambda=2$) | 99.87 | 2.57 | 11 | 200
SpikeFool ($\eta=0.1,\lambda=1$) | 97.69 | 3.02 | 17 | 116

†Samples for which the attack was unsuccessful were considered to have
undefined $L^{0}$. Because C-D PGD fails more than half of the time, the
median is undefined.

Table 4: Result comparison between PGD-based methods and SpikeFool.

##  2 The binarized MNIST dataset

| Attack |  Success Rate (%) |  Median Elapsed Time (s/sample) |  Median No. Queries |  Median $L^{0}$
---|---|---|---|---|---
B-MNIST | SCAR | 100.00 | 1.14 | 1175 | 7
Continuous-discrete PGD | 98.89 | 0.16 | 102 | 50
Probabilistic PGD | 99.70 | 0.54 | 275 | 23
SpikeFool ($\eta=0.2,\lambda=2$) | 99.90 | 0.08 | 11 | 14

Table 5: Comparison of attack strategies for B-MNIST. SCAR was implemented
according to the pseudocode in Balkanski et al. (2020). Figure 6: Example of
adversarial attacks on B-MNIST. Blue indicates removed pixels, red added
pixels. We note that in this lower-dimensional case, the effect of the attack
is semantically interpretable: for example, adding a stroke that closes the
upper left part of a “7" makes it look like a “9" not only for the network but
also for a human observer.

We tried our methods on an additional dataset. This is a binarized version of
MNIST (B-MNIST for short), which is derived from the popular MNIST Handwritten
Digits database (LeCun and Cortes, 2010), binarized so that pixel values 0 to
127 are mapped to white, and 128 to 255 are mapped to black. No other
preprocessing is applied. This is _not_ a dataset of DVS recordings: we use it
in order to compare our white box attacks against the SCAR attacks for binary
datasets (Balkanski et al., 2020). SCAR is a black-box algorithm, i.e. it only
assumes access to the output probabilities of the network. The algorithm flips
bits in areas chosen according to a specific heuristic and keeps flipped those
that cause a change in the confidence of the network.

###  2.1 Results

Table 5 compares the different algorithms on B-MNIST and shows that SpikeFool
finds successful adversarial examples with a low median $L^{0}$ (i.e. number
of perturbed pixels), while requiring a very low median execution time. Figure
6 illustrates samples of perturbations found by SpikeFool and the
corresponding label that was predicted by the network after applying the
perturbation. Because of the small sample size and the fact that there is no
time dimension, B-MNIST enables us to compare SpikeFool to a less
computationally efficient methods like SCAR. However, more realistic datasets
are needed to truly evaluate the feasibility of applying these algorithms, as
shown in the main text.

###  2.2 Network

For the B-MNIST experiments, we use a non-spiking network, similar to the one
used in (Balkanski et al., 2020): two $3\times 3$ convolutional layers (32 and
64 channels each), with ReLU activations, followed by $2\times 2$ max-pooling,
dropout, and a fully connected layer of 128 features, projecting onto the
final layer of 10 output units. The network is trained for 50 epochs at batch
size 64, using the Adam (Kingma and Ba, 2014) optimizer with learning rate
$10^{-3}$ on a cross-entropy loss function. The network reached a test
accuracy of 99.12%.

##  3 Empirical analysis of SpikeFool perturbations

With the aim of gaining more insight into the behavior of our methods, we
studied the characteristics of the perturbations resulting from SpikeFool
attacks in more detail. For this, we focused on two specific experiments: a
SpikeFool run on N-MNIST with hyperparameters $\eta=0.5$ and $\lambda=2$; and
the same IBM Gestures experiment that was run on-chip.

First, we notice that SpikeFool-based perturbations rarely involve the removal
of events. In the N-MNIST experiment considered here, an average of 7.6 events
are removed from each sample, compared to an average of 214 spikes added. This
justified our choice to ignore removed events in the course of the on-chip
experiments. As is evident from the examples in the manuscript, we also find
that SpikeFool’s adversarial perturbations tend to insert spikes at the
beginning of the sample, with only a few spikes added later in time. The top
left panel of figure 7 shows the time profile of the perturbations in detail.
We believe this is a consequence of the use of the non-leaky neuron model. In
non-leaky neurons, information can be stored indefinitely in the membrane
potential, so early spikes have a bigger chance of contributing to a spike
later in time, and are more effective compared to events added later in the
sample. This effect is also present in the IBM Gestures experiment, but looks
less prominent, possibly because networks trained with BPTT on data with
richer features in time have non-trivial dynamics. In this sense, we expect
this phenomenon to be further reduced or disappear entirely when the task is
strictly linked to the time evolution of the input signal, such as in auditory
speech recognition. The timing of adversarial events could potentially be used
for model interpretability purposes, to measure how much the model relies on
temporal features.

Figure 7: Properties of the adversarial perturbations found by SpikeFool, for
two experiments: N-MNIST (in simulation, $\eta=0.5,\lambda=2$) and IBM
Gestures (as tested on chip). Top left: Number of events in time within each
data sample. The shaded areas represent the 0.1-0.9 inter-quantile range (not
shown for the ‘original’ curve in the bottom panel). The perturbation tends to
consist of spikes added at the beginning of the sample, especially for N-MNIST
which does not rely on temporal structure for inference. Very few spikes are
removed, which justifies the choice of ignoring removed spikes in on-chip
experiments. The periodic structure of N-MNIST samples is intrinsic to the
dataset, recorded with saccades. Bottom left: Distribution of increase in
number of spikes after the attack, relative to the original number. Right:
Matrices showing the label identified by the network when presented with the
adversarial examples, given the original label, for the two experiments. Most
IBM Gestures classes are perturbed towards the ‘other’ class, while there is
no clear structure in the N-MNIST case. LH = Left Hand, RH = Right Hand, (C)CW
= (Counter) ClockWise.

Further to the median values reported in table 1 of the main text, the lower
left panel of figure 7 reports the full distributions of the number of added
or removed events. Here, we display the numbers relative to the original
number of events in the sample. We notice a minority of cases where the attack
is successful only at the cost of a very significant injection of events.

Finally, we analyzed the statistics of classes identified by the networks
after the attack. SpikeFool is used as an “untargeted” algorithm, i.e. it
attempts to change the output of the network but without requirements on what
the new class should be. Unsurprisingly, the “other gesture” class is a
natural target class for many ground truth classes, but there are some
exceptions which we find rather natural, such as “left hand wave” gestures
being most often converted to “left hand clockwise”. Conversely, we observe no
dominant target class in the N-MNIST experiment. If the target class structure
is undesirable, targeted attacks can be used instead.

##  4 Training details

###  4.1 NMNIST model

The analog CNN was trained with Adam at batch size 64 with learning rate
$10^{-3}$ for 10 epochs. We then rescaled the weights by layer-wise global
factors so that the 99th percentile of activity was the same at each layer, as
described by Rueckauer et al. (2017).


#  What is Learning ?
A primary discussion about Information and Representation

Wu Hao
email<EMAIL_ADDRESS><EMAIL_ADDRESS>
Department of Computer Science, University of York

###### Abstract

Nowadays, represented by Deep Learning techniques, the field of machine
learning is experiencing unprecedented prosperity and its influence is
demonstrated in academia, industry and civil society. “Intelligent” has become
a label which could not be neglected for most applications; celebrities and
scientists also warned that the development of full artificial intelligence
may spell the end of the human race. It seems that the answer to building a
computer system that could automatically improve with experience is right on
the next corner. While for AI or machine learning researchers, it is a
consensus that we are not anywhere near the core technique which could bring
the Terminator, Number 5 or R2D2 into real life, and there is not even a
formal definition about what is intelligence, or one of its basic properties:
Learning. Therefore, even though researchers know these concerns are not
necessary currently, there is no generalised explanation about why these
concerns are not necessary, and what properties people should take into
account that would make these concerns to be necessary. In this paper, starts
from analysing the relation between information and its representation, a
necessary condition for a model to be a learning model is proposed. This
condition and related future works could be used to verify whether a system is
able to learn or not, thus enriched our understanding of learning: one
important property of Intelligence.

###### Contents

  1. 1 Introduction
    1. 1.1 Intuitive introduction: Information
    2. 1.2 Intuitive introduction: Learning
  2. 2 Information
    1. 2.1 Information Generator and Representation
    2. 2.2 Domain and Range of a Mapping Relation
    3. 2.3 Information (generator) Type
    4. 2.4 Common constraint:Linear separable hypersurface
  3. 3 Learning
    1. 3.1 Necessary condition of Learning
    2. 3.2 AI Effect
  4. 4 Problems and Feature Works
  5. 5 Conclusion

##  1 Introduction

Although machine learning or artificial intelligence researchers know that we
are not anywhere near the technique which could enable us to build a machine
with real intelligence, almost all discussion about various properties of
Intelligence stay on a philosophical level, such as the ability of learning,
the ability of logical reasoning, self-consciousness, or even emotion. Formal
mathematical definitions of these properties are very undefined, and we do not
even have any binary criteria to verify whether a system really possesses any
of these abilities, let alone formal methods which can be used to analyse the
level of each of these abilities.

By generalising our intuitive understanding of learning ability, this paper
gives a concise necessary condition for being a learning model. And this paper
also presents further research directions which would enable us to analyse
whether a model will be a learning model or non-learning model during the
development process.

The discussion starts with intuitive introductions that prepare readers with
conceptual understanding of the viewpoint of the following formal discussion.

###  1.1 Intuitive introduction: Information

We are surrounded and processing various information constantly. And certainty
plays a critical role in representing information. Because one piece of
information must be carried by at least one certain representation.

  * •

“I’m 188cm” : in this example a certain number together with a unit (cm)
represents the information on height, the first person pronoun represents an
identity related to the height information.

  * •

“Turn left ? Right!” : in this example, the uncertainty of context brings
ambiguity.

Even when we are measuring the uncertainty of a system (entropy), we will get
a certain number or a range with certain boundary; or if we cannot get a
certain boundary, we still have the certain representation of the system we
are measuring; or even if we do not know what system we are measuring, we
still have a certain definition of entropy; and if we have no certainty about
anything, then there is no information at all. In other words, the amount of
certainties also equivalent to the amount of information. Therefore, we can
summary Feature One of information as follows:

One piece of information is equivalent to at least one certain representation,
and vice versa.

###  1.2 Intuitive introduction: Learning

We as human cannot sense vast amount of physical realities naturally, such as
the existence of almost all submicroscopic particles, the existence of
majority region of electromagnetic wave, the movement of earth, the fact that
the Earth is a sphere, actually this list could keep growing and includes
almost all knowledge we have learned since the born of modern science. And we
are all very familiar with the process of learning new knowledge based on
known knowledge, and based on new learned knowledge to learn even newer
knowledge. Firstly, this paper will give a generalised definition of the
relative relation between the known knowledge and new learned knowledge. The
introduction of this definition starts with an example:

A man was charged with murder, judge or juror has no information about the
reality of innocent or guilty. A reasonable sentence must be the consequence
of implementing proper methods on all evidences.

The evidence is local information, and the reality of innocent or guilty is
global information. So the same as the known knowledge (local information) and
new learned knowledge (global information) as introduced above. Actually, the
notion of “no observation independent reality ” has been widely accepted for
decades. The most important thing is, the global and local relation not only
exists in learning high level abstract concept, but also exists in learning
very fundamental concepts, such as object identification. Because we have no
direct access to the world[1, p.18] other than though our sensors. The
information of the existence of an object (global information) is the
consequence of learning based on its different possible appearances (local
information) received by our retina. And Feature One of information tells us
there must be a certain representation of this object (global information).

Further discussion of learning (section 3) is after a formal discussion about
information (section 2). These sections are all based on the explanation of
mapping relations from a new viewpoint.

##  2 Information

###  2.1 Information Generator and Representation

Figure 1: Mapping Relation and Generators

Suppose there is a mapping relation between a set $X$ and a set $Y$, so any
subset of $X$ can be regarded as being generated by a generator, and the
corresponding elements of subset of $Y$ are representations of this generator.
The joint of any two subsets of $X$, or any two corresponding subsets of $Y$
are not necessarily to be null.

###  2.2 Domain and Range of a Mapping Relation

When a mapping relation $F$ is defined between domain $D$ and range $O$. Then,
with respect to $F$, any subset of its domain could correspond to an unknown
generator, and the corresponding elements in the range of that domain are the
representation of that unknown generator.

Figure 2: Mapping relation between domain and range

Then, with respect to $F$ (could be a function), we can define local
information and global information as follows:

  * •

Local information: Any input $x\in D$ is a local information with respect to
$F$.

  * •

Global information: Any element of subset $Y\subset O$ and the corresponding
subset $X=\\{x|y\in Y,y=F(x)\\}$ are global information.

So the global information $y\in Y,Y_{n}\subset O$ represents an unknown
generator $G_{n}$. It is obvious that an unknown information generator $G_{n}$
could have a lot global representations: all elements of $Y_{n}$, and due to
the Feature One of information, we can say that all elements of $Y_{n}$ should
follows a certain constraint $c$. And if the corresponding elements of domain
subset $X_{n}$ do not follow the same constraint $c$, we can say that the
global representation ($y\in Y_{n}$) of the information generator $G_{n}$ is
the invariant representation with respect to its appearances ($\forall x\in
X_{n}$).

Example One:

  * •

Domain: A CCD array, or our retina all provide a very large domain. If the
value of each pixel is between 0-255 and the size of the image is 200*200
pixels, than the corresponding size of the domain is $256^{40000}$, which is a
very large number.

  * •

Local information: Every image of a dog, every possible moment captured by our
retina, these are all local information.

  * •

Global information: The existence of the dog, and all its appearances are
global information. But we can learn the invariant representation form local
information, a camera cannot.

###  2.3 Information (generator) Type

It is obvious that different mapping relation pairs different subsets between
domain and range.

Figure 3: Different type of information

Therefore, we can say that a mapping relation along with its domain define the
type of unknown generator $G_{n}$, or in other words defines the type of
information.

###  2.4 Common constraint:Linear separable hypersurface

As introduced in section 2.2, global information must follow a certain
constraint $c$, so the certainty of information can be guaranteed. There are a
large number of possible constraints, and we denote the set of all possible
constraint as $C$. Now we would like to know which constraint is preferable?
and Why? The following discussion gives the answer of this two questions.

Assumption One:

Within certain type of information, there exists a mapping relationship that
can define the differences between information generators.

Based on Assumption One and Feature One of information, we know that a mapping
relation, denoted as $F_{C}$, map different subsets generated by unknown
generators to global information that distinguishable by applying constraint
$c_{R}$. By choosing the simplest constraint $c_{R}$ that each $y\in O$
represent a distingusiable information generator, so the expression of $F_{C}$
is:

$F_{C}:D\to O(D\subset R^{N},O\subset R)$ (1)

According to this expression, the differences of information generators can be
identified as follow:

A set $X_{1}\subset D$ is the global information of an unknown information
generator $G_{1}$, then $y_{1}=F_{C}(x):x\in X_{1},y_{1}\in R$ ; another set
$X_{2}\subset D$ is the global information of an unknown information generator
$G_{2}$, then we have $y_{2}=F_{C}(x):x\in X_{2},y_{2}\in R$.

If $y_{1}\neq y_{2}$, then we can say that $G_{1}$ and $G_{2}$ are different
information generators, and the type of information is defined by $F_{C}$.

Figure 4: Representation and the Common Constraint

Furthermore, for real numbers in range $O$, they can be represent by parallel
hyperplanes defined by a vector set $V_{C}$, so we can say that points on
these hyperplanes are population representations (global information) 111The
term “population representation in figure 4 means the invariant representation
is high dimensional of different unknown information generators, and this also
explain why researchers prefer the property of linear separable so much,
because it is not only easy understanding, but also is the common constraint
that every element of constraint set $C$ can convert to.

##  3 Learning

Previous section shows a formal way of explaining local information and its
global representation. What being learned is nothing more than information, or
in other words, is nothing more than global representation, so the analysis of
learning ability should based on the analysis of the mapping relation as
introduced in previous section. And again this section starts with an
intuitive example.

Example Two:

  1. (A)

select dob from table1 where name = “wu hao”

  2. (B)

$y=2x$, when $x=2$, we have $y=4$

  3. (C)

P(is human $|$ a picture of me) = 0.9999

  4. (D)

People see picture 1, they know it is a pig. People see picture 2, they know
it is a dragon. People see picture 3, even though they may have never seen
this before and do not know the name, they still know it is different.

Figure 5: Three Pictures

These four models in example two all include one input, a mapping relation
(function), and one output. A is a database, B is a linear function, C is a
powerful probabilistic model, and D is us. By analysing example two from the
mapping relation point of view we know that:

  1. (A)

: The database will return $NULL$ for all inquiries that do not match what
have been stored previously.

  2. (B)

: Outputs will be different when the inputs are different. And for almost all
regression problems, we have assumptions about the mapping relation between
two subsets of our observations, and after modelling our hypothesis of the
mapping relation, for a new set of input we can almost always expect the
output set contains new element which we have never seen before.

  3. (C)

: When applying probabilistic model, our hypothesis is based on the belief
that there are unknown statistical laws which represent different concepts [2]
. Assume mapping relation $F_{model}$ is able to mimic the unknown statistical
law of the appearance of a cup perfectly 222This assumption helps to avoid the
discussion of convergence problem., so the mapping relation we get is as
follow:

Figure 6: Probabilistic Model (example one)

At this stage, $F_{model}$ is able to give the chance of being a cup for any
element of the domain. What if we want $F_{model}$ to have the ability to give
the chance of being other object, such as a dog. Different from what we
observed in regression problem, there is no real number naturally related to
different objects. Therefore we chose another way of constructing our
hypothesis of observation as follow:

Figure 7: Probabilistic Model two

$F_{model}:R^{N}\to R\times I(I\ is\ indicator\ set)$

But the introduction of this “Indicator Set” brings a problem. The size of
this indicator set represent the information of the amount of objects
$F_{model}$ can effectively identify. This is a global information (denoted as
$\\#I$) with respect to identities of each object, and it is given by us. For
objects which are not included in our hypothesis of $F_{model}$, the outputs
are almost all clustered around 0.

  4. (D)

We, as human with well developed vision, are always able to identify object
which have never been seen before, or in other words, the function of our
vision system are able to define new global information.

###  3.1 Necessary condition of Learning

Therefore, combining the discussion of information and its representation with
above discussion, the intuitive understanding of the ability of learning is:

Learning means the ability to define new information generators

From this point of view, B and D are learning model, A and C are non-learning
model. And by generalising the description above, we have a formal necessary
condition of a mapping relation $F_{model}$ to be a learning model as follow:

Condition S:

$\forall X_{S}\subset D,Y_{S}=F_{model}(X_{S}),\exists X_{N}\subset D\setminus
X_{S}:Y_{N}=F_{model}(X_{N}),(Y_{N}\bigtriangleup Y_{S}\neq
Y_{N})\&(Y_{N}\bigtriangleup Y_{S}\neq Y_{S})$

For any subset $X_{S}$ of the domain D, the corrsponding subset of range is
$Y_{S}$, exists a set $X_{N}$ which is a subset of the complement set of
$X_{S}$, so that we have a subset $Y_{N}=F_{model}(X_{N})$, the symmetric
differences of $Y_{N}$ and $Y_{S}$ is neither $Y_{S}$ nor $Y_{N}$.

Generally, if a mapping relation $F_{model}$ satisfies Condition S, then we
can say $F_{model}$ is able to learn its domain and it is a learning model,
otherwise it is not able to learn its domain and it is a non-learning model.
But there are actually two strategies which enable a non-learning model
$F_{model}$ to satisfy condition S, thus behaves like it is able to learn its
domain.

  1. (A)

Shrinking the scale of the domain.

  2. (B)

Iterating over the domain

Example Three:

Figure 8: Strategies A and B

  * •

As shown in figure 8, in picture 1, function $F^{*}$ is defined on domain $D$,
and it is not able to learn its domain because it does not satisfy condition
S. But by shrinking its domain as shown in picture 2, $F^{*}$ behaves like it
is able to learn most of its domain. This is strategy A

  * •

As shown in figure , in picture 5, a set of function
{$F_{1},F_{2},F_{3},F_{4}$} forms an approximation of function $F_{L}$ which
satisfies condition S over its domain, but neither of these four functions
satisfy condition S. This is strategy B.

Usually these two strategies are being applied together.

For neutralising the effect brought by strategy A and B, condition S needs to
be extended a little bit.

Condition $S^{*}$:

Suppose domain D is defined on $N$ dimensional space, extend the scale of
dimension $M$ of the domain to infinite and mapping relation $F_{model}$ still
satisfies condition S, then we can say $F_{model}$ is a learning model on
domain D, dimension M.

If $F_{model}$ satisfies condition S* on every dimension of its domain $D$,
then we can say $F_{model}$ is able to learn its domain $D$, and it is a
complete learning model. On the other hand, as shown in figure 8 picture 5,
the iteration process behaves similar with a database systems as shown in
picture 3 and 4. And a database system is a typical memory system, so we can
also say:

  * •

Not being able to satisfy condition $S^{*}$ is a sufficient condition for a
model to be a Non-Learning model.

  * •

Non-Learning model and memory system are equivalent.

###  3.2 AI Effect

$F_{model}$ contains our hypothesis about the mapping relation between
different subsets of our observation. The ideal situation is our hypothesis
could be as close to the unknown mapping relation as possible. One method of
measuring the validity of our hypothesis is as follow:

Suppose unknown mapping relation is $U:D\to O$, and our guess if H. So for any
$o\subset O$, we have $T=U(o)$, and $G=H(o)$, then $J(T,G)$ 333 $J(T,G)$ is
the Jaccard similarity coefficient of these two sets, It can also being used
to define the simplicity of a machine learning problem is the validity of our
hypothesis.

For simple problems, such as regression problem, our hypothesis works fine ,
but its only learn simple mapping relation. But when facing complex
optimisation problems, for getting a desired mapping relation $F_{model}$,
strategies A and B are usually being applied together and if researchers do
not exam the property of information being used carefully, it is possible that
information comes from outside of this structure can be introduced. Usually
the choice of introducing information from outside the structure will help to
reduce the difficulties of constructing a desired $F_{model}$ or improve its
performance. But this behaviour will eventually cause $F_{model}$ fail to
satisfy condition $S^{*}$, as shown by the example of figure 7 (the
probabilistic model).This somehow explains Rodney Brooks complain:

“ Every time we figure out a piece of it, it stops being magical; we say, ’Oh,
that’s just a computation ”

##  4 Problems and Feature Works

  1. 1.

More necessary conditions or a sufficient condition

In this paper, condition $S^{*}$ is a necessary condition for a mapping
relation to be a learning model, this does not rule out the possibility that
some mapping relation $M_{F}$ could satisfy condition $S^{*}$ but still
against our understanding of learning. Therefore, further discussion about
condition on mapping relation is necessary.

  2. 2.

Necessary condition on structural level

The discussion of previous section indicates that Condition $S^{*}$ of mapping
relation is insufficient for analysing the structural detail of a given model,
necessary condition on structural level is required to guide the construction
process of getting a desired mapping relation and presumably would also give
us a formal explanation about why introducing global information from outside
the structure could make the problem easy to be solved or improve the
performance but also causes losing the ability of learning at the same time.

  3. 3.

Verifying state-of-the-art artificial intelligence systems.

Because of the overheated expectation of an realistic AI system and the
following failure in 1970s, researchers usually avoid using the term “AI”
since then. Instead they express the similar idea by implying that their
systems are able to learn similar features as human brain could do or be able
to exhibit unexpected behaviours.

These days, some systems have demonstrated impressive performances which not
only make the discussion of AI a hot topic again but also raised the alert
over the possibility of people may losing control of Artificial Intelligence.
But no matter how powerful these systems could be, from machine learning point
of view, they are still solutions of optimisation problems which based on
different hypothesises of our observation. Therefore their ability of learning
is independent from their performance and can be verified in the same way as
the discussion of probabilistic model in section 3. The research results of
problem one and two will provide more powerful tools for verifying the
learning ability of a give model.

  4. 4.

The common learning model

As discussed in section 2.4, linear separable is the common constraint that
all constraints can convert to, therefore all learn models (include us) can
convert to the common learning model which follows this common constraint. A
prototype which demonstrates the common learning model theory is the zero to
one step for harvesting information learned by future artificial intelligence
system.

  5. 5.

Begin with object recognition

The ability of learning is one property of an intelligent system, the learning
result is not necessary to be right all the time, and the concept of “right”
is also a global information which needs to be defined in future research.

The process of learning highly abstract information is tightly related to
other part of intelligence, after all, during the past 300000 years’
development of homo sapiens, most of our learned highly abstract realities are
“Wrong” and comparing with realities we know today, it seems “overfitting” is
a common phenomenon of the highly abstract information learning process (World
Elephant, then Geocentric Theory). Therefore, further study of learning will
focus on the learning process of relatively independent and low level abstract
information, such as the object recognition problem.

  6. 6.

Dataset separation and merge problem

When explaining the object recognition problem using the theoretical framework
proposed in previous sections, two seemingly counterintuitive deductions are
dataset separation and merge problems.

     * •

Dataset separation problem: When dataset generated by one object is separated
, there could be two different set of invariant representations.

     * •

Dataset merge problem: When two dataset of different object are merged
together, there could be a new set of invariant representations.

Figure 9: Dataset separate and merge phenomenon

The description above is the common intuitive understanding of these two
problem, but there are two mistakes about this understanding:

     * •

The information generators is defined by the mapping relation, so these two
problems are equivalent.

     * •

Since there is no observation independent reality, this is not a problem, it
is a phenomenon which can be used to verify the validity of future
implementation based on this theoretical framework.

Note

It seems particularly hard for people to understand the dataset merge
phenomenon in object recognition. It indicates that if we could combine two
objects together, such as a dog and a cat, and guarantee the smoothness of the
transformation of their appearances, then we will see them as a same
thing(things). This confused conclusion exists only because we have the
information of the existence of these two distinguishable objects at first,
and there is no real world experience which bring us into this mental
experiment. But we can always almost recognise “Optimus Prime”, “Bumblebee”,
“Jazz” and other characters in transformers (the movie or the cartoon), no
matter they appear as cars or human form robots. In this scenario, the
recognition process of a transformer is no more difficult than recognise
anything else and we take it for granted. Therefore, the confusion about cat
and dog could be completely solved someday in the future which “Marvel” or
“DC” decide to bring us a new superhero whose appearance transforms between
cat and dog. And this phenomenon should be able to be verified experimentally
if there are corresponding neuroscientific approaches.

  7. 7.

Other related topics

With the progress of future works, some other related topics will help to
provide a better understanding about the essence of learning and its relation
with intelligence. These topics include: detailed analysis of non-learning
model, formal discussion of overfitting problem in the process of learning
highly abstract information, strategies that could make a learning model
behaves like a non-learning model.

##  5 Conclusion

John Connor asked T-800:

“Can you learn stuff that you haven’t been programmed with? so you could
be…you know, more human? And not such a dork all the time?”

Even though that is just a scenario of a movie, still we all want to known the
answer of this question. And for researchers, the most interesting part is how
this question can be answered.

Machine Learning theories focus on solving different optimisation problems, so
we could model hypothesises of our observation. While, the discussion of
learning should focus on the behaviour of the model. In this paper, by
analysing the relations among observable appearance(local information),
reality (global information), their representation and the generalisation of
our intuitive understanding of learning, we have a necessary condition $S^{*}$
which can be used to get the answer of John’s question.

Starting from a viewpoint which has been missed out, the analysis in this
report shows that, Learning is not an action of absorbing existed information.
In contrast, learning is an action of define information generators, or more
specifically, is the ability of always being able to define new information
generators. And preliminary verification indicates that our hypothesis of what
we observed could be a learning model which only learn trivial information,
and when facing complex problems 444Complexity as defined in section 3.2.,
lack of learning ability is a trade-off for reducing the complexity or
improving the performance through introducing global information from outside
the structure.

It is possible in the future that we could harvest the information generated
by a learning model which can learn like us but with greater learning ability.
And focusing on the direction of getting a common learning model which can
learn like us, this paper illustrates the logic dependencies of many topics
that related to understanding the essence of learning.

## References

  * [1] Sebastian Luft.  Subjectivity and lifeworld in transcendental phenomenology.  Northwestern University Press, 2011.
  * [2] Vladimir Vapnik.  The nature of statistical learning theory.  springer, 2000.