# [RE] CNN-generated images are surprisingly easy to spot…
for now

Joel Frank
Ruhr-University Bochum
&Thorsten Holz
Ruhr-University Bochum
Corresponding author<EMAIL_ADDRESS>
## Reproducibility Summary

This work evaluates the reproducibility of the paper “CNN-generated images are
surprisingly easy to spot… for now” by Wang et al. published at CVPR 2020. The
paper addresses the challenge of detecting CNN-generated imagery, which has
reached the potential to even fool humans. The authors propose two methods
which help an image classifier to generalize from being trained on one
specific CNN to detecting imagery produced by unseen architectures, training
methods, or data sets.

### Scope of Reproducibility

The paper proposes two methods to help a classifier generalize: (i) utilizing
different kinds of data augmentations and (ii) using a diverse data set. This
report focuses on assessing if these techniques indeed help the generalization
process. Furthermore, we perform additional experiments to study the
limitations of the proposed techniques.

### Methodology

We decided to implement the methods from scratch to highlight possible hurdles
for practitioners who want to adopt the results. For our experiments, we
utilized the training and test data provided by the authors, as well as,
creating our own data set to analyze how the proposed techniques perform on
different data sets. Note that overall we estimate the entire experiments to
take around 590 GPU hours.

### Results

In general, we were able to replicate the overall results reported in the
paper. However, we also found significant differences in our experiments.
After further investigations, we discovered that we implemented two data
augmentations slightly different. Additionally, we identified three test data
sets, which exhibit fluctuating results across multiple runs with the same
setup. However, the overall trends of our experiments matched the original
publication, i.e., data diversity and augmentations help generalization. We
also performed several additional experiments on other data sets, highlighting
limitations and clarifying the methods.

### What was Easy

The paper is very detailed and we could faithfully replicate the experiments
(bar the aforementioned exception). We only utilized the authors’ code
sparingly, but the repository is very well documented and provides pre-trained
models.

### What was Difficult

The naming of one of the augmentations caused confusion on our part. Note that
the description in the paper is correct, however, the naming suggest the
method works different than actually implemented. This resulted in us
implementing a variation of the proposed technique. Surprisingly, our method
actually improved on the original results.

### Communication with Original Authors

Most of our question could be resolved by comparing our implementation against
the authors’ code. We contacted the first author regarding our different
results, he was very responsive and answered all of our questions.

##  1 Introduction

High-quality implementations of state-of-the-art image synthesis methods are
available to the general public (Karras et al., 2020). This technology is
astonishing and concerning at the same time. One the one hand, such algorithms
can be used for practical use cases like image-to-image translation (Isola et
al., 2017; Liu et al., 2017; Zhu et al., 2017) or image super-resolution (Lai
et al., 2017; Ledig et al., 2017; Lim et al., 2017). On the other hand, they
can generate images which even trick human observers (Simonite, 2019).
Recently, fake news and tampered media have become practical problems which
even have the potential to influence democratic processes (Thompson and
Lapowsky, 2017; Hao, 2019).

Recognizing these problems, Wang et al. (2020) propose a “universal” detector
for CNN-generated images. While creating a detector for know architectures has
been done before (Zhang et al., 2019; Wang et al., 2019a), the authors of this
paper ask whether it is conceivable to train a classifier which generalizes to
new, unseen architectures. The authors demonstrated that with a diverse data
set and with carefully chosen data augmentation methods, a standard image
classifier can achieve this goal. More specifically, they train a ResNet50 (He
et al., 2016) classifier on a data set consisting of images generated by 20
different instances of ProGAN (Karras et al., 2018). Additionally, they
augment the training process by utilizing aggressive image filtering
techniques (i.e., Gaussian Blur and JPEG compression). Combining both
techniques allows them to create an image classifier which even generalized to
StyleGAN2 (Karras et al., 2020), which was published concurrently.

##### Scope of reproducibility:

We opted to reimplement the methods from scratch, aiming at highlighting
difficulties for practitioners who want to adapt these methods. While the
authors released a very well documented code repository, we initially
abstained from inspecting the source code to not prime ourselves. In the later
stages of our analysis, we used their repository to uncover some small
implementation differences, which lead to our implementation actually
improving on the original results.

Using our implementation, we aim at verifying the two main claims of the
paper:

  1. 1.

The proposed data augmentations (usually) improve generalization.

  2. 2.

More diverse data sets help the classifier generalize to unseen architectures.

We recreate parts of the original experiments which directly investigate these
two claims (see Section 4.2, Section 4.3, Table 2, and the Figures 2 and 3 in
the original paper (Wang et al., 2020)). Departing from their work, we perform
several additional experiments: First, we further investigate the first claim,
training multiple additional data set combinations to clarify differences
between our results and the original paper. Second, we generate a new training
set from a different CNN generator (StyleGAN2 (Karras et al., 2020)) and
evaluate if the results transfer. Finally, we investigate if the method is
exclusive to the specific image classifier used in the paper. To this end, we
train two other classifiers and compared them to the original results: A
different image classifier architecture (Simonyan and Zisserman, 2015) and a
classifier which leverages spectral analysis (Frank et al., 2020).

##  2 Methodology

In this section, we provide an overview of our methodology. We start by
presenting the different data sets used in our experiments. Then, we describe
the overall experimental setup, including the models we studied, data
augmentations performed, details on the training procedure, our evaluation
criteria, and computational requirements. All of our reproducibility efforts
are based on the arXiv version of the paper (v2 – submitted 04. April 2020)
and the corresponding GitHub repository111available at
https://peterwang512.github.io/CNNDetection/ (commit f692c13 – 26. October
2020). We also make our implementation, our pre-trained models and our data
set publicly available 222available at
https://github.com/Joool/ReproducabilityCNNEasyToSpot.

###  2.1 Data Sets

For training our networks, we used two different data sets: The first one is
provided by the original publication. The authors used 20 ProGAN (Karras et
al., 2018) models, each trained on a different LSUN category (Yu et al.,
2015), to sample 18K generated images per model. They also collected 18K
training images for each of the corresponding classes, resulting in a total
data set of $(18.000+18.000)\cdot 20=720.000$ images.

Additionally, we created a new data set to study the proposed techniques on a
different base generator. We downloaded the pre-trained models for StyleGAN2
(Karras et al., 2020). Using these models, we generated a training set of 36K
train (18K fake and 18K real) images for the LSUN _cat and horse_ data sets,
following the pre-/post-processing steps outlined in the original publication.
Since these categories are also contained in the data set by Wang et al., this
allows for a one-to-one comparison.

We exclusively used the test data set provided by the authors. The data set
consists of images from 11 different synthesis models:

  * •

GANs: The data set contains samples from six different Generative Adversarial
Networks (GANs): ProGAN (Karras et al., 2018), StyleGAN (Karras et al., 2019),
BigGAN Brock et al. (2019), GauGAN (Park et al., 2019), CycleGAN (Zhu et al.,
2017), and StarGAN (Choi et al., 2018).

  * •

Perceptual loss: The authors also include samples from generative models which
are directly trained to optimize a perceptual loss: Cascaded Refinement
Networks (CRN) (Chen and Koltun, 2017) and Implicit Maximum Likelihood
Estimation (IMLE) (Li et al., 2019).

  * •

Image manipulation models: It includes samples from Seeing In The Dark (SITD)
(Chen et al., 2018), which approximates long-exposure photography, and the
Second-Order Attention Network (SAN) (Dai et al., 2019), which generates
images at a higher resolution.

  * •

Deep fakes: Finally, the authors include samples from the FaceForensics++
(Rössler et al., 2019) data set.

###  2.2 Experiment Setup

In the following, we provide an overview of the different experiments
performed in the paper. We start by discussing the different models used,
present the data augmentations proposed by Wang et al., and, proceed by
presenting the training schedule used in all experiments. Finally, we conclude
by discussing how the evaluation and with an overview of the computational
requirements needed for reproducing this report.

##### Models used:

Following the original paper, we use a ResNet50 (He et al., 2016) model as the
image classifier for the majority of our experiments. The classifier is pre-
trained on ImageNet (Russakovsky et al., 2015). We use random crops (to
$224\times 224$ pixels) and horizontal flipping during training of all of our
classifiers. Additionally, we scale the image to the range $[0,1]$, remove the
mean, and scale them to unit variance.

For the experiments presented in Section 3.3.2, we also train a VGG-11
(Simonyan and Zisserman, 2015) model and a classifier which utilizes spectral
information (Frank et al., 2020). The VGG model is also pre-trained on
ImageNet and uses the same preprocessing as our ResNet models. The spectral
classifier also uses random crops and horizontal flipping as outlined above.
Afterwards, we follow Frank et al. and use Discrete Cosine Transfrom-II (Ahmed
et al., 1974) (DCT-II) to transform the images from the spatial to the
spectral domain. Afterwards, we use MinMax-scaling to scale the images to the
range $[0,1]$ and train a ResNet50 model from scratch on the transformed data.

##### Data augmentations:

Wang et al. proposed several data augmentations which help their classifier
generalize to unseen architectures:

  * •

_Gaussian blur (Blur):_ Before cropping, with 50% probability, the images are
blurred with $\sigma\sim\text{Uniform}\\{0,3\\}$

  * •

_JPEG-compressiong (JPEG):_ with 50% probability images are JPEG-ed by two
popular image libraries, OpenCV and PIL, with quality $\sim$
Uniform$\\{30,\dots,100\\}$.

  * •

_Blur + JPEG (Blur & JPEG (0.5)):_ images are possibly blurred and JPEG-ed,
each with 50% probability.

  * •

_Blur + JPEG (Blur & JPEG (0.1)):_ similar to the previous, but the
probability is dropped to 10%.

All augmentations are applied before cropping and flipping (to the entire
training set). Additionally, we also evaluate no augmentations (_No Aug._) as
a baseline. Note that the JPEG standard only specifies quality setting in the
range $[30,95]$. However, the authors sample in the range $[30,100]$. We
follow their sampling, but clip all values to the allowed range.

##### Training details:

At the start of the training, we randomly sample $10\%$ of the training set
for validation. We train all model using Adam (Kingma and Ba, 2015) with
$\beta_{1}=0.9$, $\beta_{2}=0.999$, a batch size of 64, and an initial
learning rate of $10^{-4}$. We drop the learning rate by $10\times$ when the
validation accuracy stagnates for 5 epochs. When the learning rate reaches
$10^{-7}$, we terminate training and select the best classifier based on the
validation accuracy.

We abstained from validating additional combinations of (Adam-)
hyperparameters, learning rate, and batch size. The selection of data
augmentation and data set variety is already computational challenging and can
be viewed as a hyperparameter in its own right. However, we note that this
might be an interesting direction for future experiments.

We use the same settings for training the VGG network. When training the DCT-
ResNet we change the initial learning rate to $10^{-3}$ since we train the
underlying model from scratch.

##### Evaluation criteria:

We follow the authors and use Average Precision (AP) as our evaluation metric.
It is a ranking-based metric which is not sensitive to the “base rate” of the
fraction of fake images and thus commonly used throughout the literature (Zhou
et al., 2018; Huh et al., 2018; Wang et al., 2019b). Additionally, we compute
the mean Average Precision (mAP) over all data sets, to compute an overall
tendency. During testing, all images are center cropped without resizing to
match the training dimension (without augmentations).

##### Computational requirements:

All our experiments are run on two desktop machines. Each machine is running
Ubuntu 18.04, with 64GB RAM, a AMD Ryzen 7 3700X 8-Core Processor, and two
GeForce RTX 2080Ti. Training a single ResNet50 model on one GPU on the entire
data set (20 classes) takes around 42 hours. Most of the experiments are
conducted on smaller data set (2–8 classes), which significantly reduces the
training time to roughly 6 hours. Overall, we estimate the entire experiments
to take around 590 GPU hours, not including initial testing runs.

Table 1:  Average Precision of training a ResNet50 classifier on the entire
data set with different augmentations. We report the average precision for
both our results (Reproduced) and the the original publication (Wang et al.).
Additionally, we report the results of a (Corrected) implementation of the
Blur + JPEG augmentation. Chance is 50%, the best possible results is 100%,
and we highlight the ProGAN results in gray since the classifier is trained on
similar data. We train on the entire data set (20 classes).

!  Name Result ProGAN StyleGAN BigGAN CycleGAN StarGAN GauGAN CRN IMLE SITD
SAN DeepFake mAP No Aug. Wang et al. 100.0 96.3 72.2 84.0 100.0 67.0 93.5 90.3
96.2 93.6 98.2 90.1 Reproduced 100.0 96.8 73.5 81.9 100.0 68.2 95.1 88.8 97.1
87.2 98.4 89.7 Blur Wang et al. 100.0 99.0 82.5 90.1 100.0 74.7 66.6 66.7 99.6
53.7 95.1 84.4 Reproduced 100.0 94.0 _71.7_ 81.9 100.0 71.0 63.5 63.1 93.8
_83.9_ 98.6 83.8 JPEG Wang et al. 100.0 99.0 87.8 93.2 91.8 97.5 99.0 99.5
88.7 78.1 88.1 93.0 Reproduced 100.0 99.4 89.4 93.3 94.5 96.4 99.1 99.3 91.1
69.7 92.5 93.2 Blur + JPEG (0.5) Wang et al. 100.0 98.5 88.2 96.8 95.4 98.1
98.9 99.5 92.7 63.9 66.3 90.8 Reproduced 100.0 99.4 88.8 94.1 96.7 96.2 98.5
99.1 92.9 72.3 _93.7_ 93.8 Corrected 100.0 99.1 88.8 94.8 94.5 97.5 99.3 99.4
84.8 _74.1_ 73.0 91.4 Blur + JPEG (0.1) Wang et al. 100.0 99.6 84.5 93.5 98.2
89.5 98.2 98.4 97.2 70.5 89.0 92.6 Reproduced 100.0 99.5 85.7 93.3 97.9 93.6
99.2 99.4 95.5 77.7 95.0 94.2 Corrected 100.0 99.6 86.5 94.7 97.1 92.5 99.1
99.4 95.3 _82.2_ 96.9 94.8

We highlight differences (w.r.t the original publication) greater 5% in bold
and greater 10% additionally in _cursive_.

##  3 Results

Overall we were able to reproduce the trends of the original results and
verify the two claims of the publication. In Section 3.1, we examine the first
claim, studying in detail the different data augmentations proposed by the
authors. In Section 3.2, we examine the second claim, evaluating if more
diverse data help generalization. Finally, in Section 3.3, we depart from the
original experiments and investigate further to gain additional insights into
the proposed methods.

###  3.1 Claim: _Data augmentations improve generalization_

First, we investigate the claim that data augmentations improve generalization
and reproduce the second half of Table 2 in the original publication.
Specifically, we train five classifier on the entire training set, with each
classifier trained on one data augmentation introduced in Section 2.2. The
results are presented in Table 1.

##### Results:

We can successfully recreate the general trend that data augmentations help
generalization. However, for specific cases our results differ heavily from
the reported results by Wang et al.. This is especially noticeable for the
blur augmentation, where our results differ in multiple data sets (StylGAN,
BigGAN, CycleGAN, CRN, SITD and SAN). The results for the SAN data set even
differ by $30.2\%$. We also seem to slightly outperform the results by Wang et
al..

We verified our implementation against the original published code and noticed
two substantial differences: First, we implemented Gaussian filtering with the
build-in methods provided by PyTorch (Paszke et al., 2019), while Wang et al.
build their own method using SciPy (Virtanen et al., 2020). Second, we apply
blurring in conjunction with JPEG compression with a probability of 50%. In
contrast, Wang et al. first apply blurring with a probability of 50% and
_then_ apply JPEG with a probability of 50%. Note that the paper describes
this correctly.

To investigate if the results are indeed connected to our implementation, we
perform an additional experiment. We implemented a corrected version of the
Blur + JPEG augmentation and rerun the experiment. The results are mixed.
While we achieve a closer result for the DeepFake data set, the changes lead
to even higher differences in the SAN and SITD data sets. We repeat the
experiment two additional times to investigate if this might be related to
chance. We obtained widely varying results on the SITD ($84.8/85.9/90.7$), the
SAN ($72.6/74.1/78.3$), and the DeepFake ($72.4/73.0/76.1$) data sets. A full
table can found in the supplementary material.

The authors already noted the SAN and DeepFake data set as exceptions. Our
model recovers better from using a combination of blurring and JPEG
compression for these sets. Thus, we hypothesize that our implementation may
apply more aggressive preprocessing (we always apply blurring and JPEG
compression), but on fewer training examples (our implementation is all or
nothing; Wang et al. can apply blurring, JPEG compression, or both). The
probabilistic nature of these data augmentations can also explain the
fluctuation in the SITD, SAN, and DeepFake results. Obtaining statistical
guarantees for our results would be ideal. However, we would need to perform
multiple runs of ours’ and Wang et al.’s implementation. With a single run
already taking 42 GPU hours, we abstain from further investigation. We also do
not investigate the hand-crafted blurring implementaion. We assume that future
practitioners will preferably use the build-in PyTorch method we utilized
(unavailable at the time of original publication). We hypothesize that our
experiments better reflect future results.

###  3.2 Claim: _More diverse data improves generalization_

Table 2:  Average Precision (AP) of training a ResNet50 classifier on
different subsets of the training data. We report the average precision over
11 different CNN generators for both, our results (Reproduced) and the results
from the original publication (Wang et al.). Chance is 50% and the best
possible results is 100%. Note that the classifier is trained on ProGAN and we
thus display the results in gray. The mean Average Precision (mAP) is obtained
by taking the mean of the individual APs. Note that we apply blurring and JPEG
compression with 50% probability during training.

!  Setting Result ProGAN StyleGAN BigGAN CycleGAN StarGAN GauGAN CRN IMLE SITD
SAN DeepFake mAP 2-class Wang et al. 98.8 78.3 66.4 88.7 87.3 87.4 94.0 97.3
85.2 52.9 58.1 81.3 Reproduced 99.3 81.0 71.8 88.4 83.5 89.7 98.8 99.5 82.0
_72.3_ 61.5 84.3 4-class Wang et al. 99.8 87.0 74.0 93.2 92.3 94.1 95.8 97.5
87.8 58.5 59.6 85.4 Reproduced 99.8 90.6 79.7 92.5 91.2 93.5 98.1 98.7 95.0
_74.6_ _75.7_ 90.0 8-class Wang et al. 99.9 94.2 78.9 94.3 91.9 95.4 98.9 99.4
91.2 58.6 63.8 87.9 Reproduced 100.0 96.7 81.8 92.7 92.6 95.0 98.2 99.3 85.7
_69.8_ _77.9_ 90.0 16-class Wang et al. 100.0 98.2 87.7 96.4 95.5 98.1 99.0
99.7 95.3 63.1 71.9 91.4 Reproduced 100.0 98.5 87.0 94.5 95.0 96.8 99.2 99.3
_84.5_ _77.9_ 76.8 91.8 20-class Wang et al. 100.0 98.5 88.2 96.8 95.4 98.1
98.9 99.5 92.7 63.9 66.3 90.8 Reproduced 100.0 99.4 88.8 94.1 96.7 96.2 98.5
99.1 92.9 72.3 _93.7_ 93.8

We highlight differences (w.r.t the original publication) greater 5% in bold
and greater 10% additionally in _cursive_.

Second, we investigate how diversity of the data set affects the results by
reproducing the rest of Table 2. Similar to the original paper, we train five
different classifiers on subsets of the original training data
($\\{2,4,8,16,20\\}$-classes) and use blurring and JPEG compression with a
probability of 50% during training. The results are displayed in Table 2.

##### Results:

Overall, we can again recreate the general trend that data diversity improves
generalization. The results mostly match, but we can again observe widely
different results for the SITD, SAN and DeepFake data sets. We attribute these
differences to the earlier observed sensitivity to the random character of
these experiments.

The original publication observed that data diversity helped with
generalization up to a specific point. The authors hypothesized that at this
point (16 to 20 classes) the training set may be diverse enough for practical
generalization. We agree that data set diversity helps with generalization. We
cannot fully agree that there exists a point of diminishing returns. Utilizing
more classes allows our classifiers to achieve significantly better results on
the DeepFake ($76.8\to 93.7$ AP) and SITD ($84.5\to 92.9$ AP) data sets.

While the diminishing returns might be related to chance, we additionally
investigate if they are connected to the aggressive data augmentations used in
the original experiments. We perform a control experiment where we again train
multiple classifiers on subsets of the training data with all augmentations
disabled. Note the original publication only assessed subsets with
augmentations enabled.

The results are presented in Figure 1, a full table can be found in the
supplementary material. Contrary to our hypothesis, the data augmentation help
the model generalize in the presence of more diverse data. However, they also
lead to varying results on the SITD, SAN, and DeepFake data set. We assume the
same probabilistic nature observed in Section 3.1 is related to the
fluctuations. For the other data sets, we assume that the augmentations act as
a regularizer, preventing the model from overfitting to the training set. We
study this phenomena further in Section 3.3.1. Overall we conclude that more
diverse data improves generalization.

###  3.3 Results beyond the original paper

We perform further experiments to get more insights into the original paper.
First, we investigate if the results are dependent on the generator used to
create the training set by sampling a second data sets from StyleGAN2 (Karras
et al., 2020). Second, we investigate if the results are dependent on the
classifier used by comparing the ResNet50 against a different image classifier
(VGG-11) and a classifier which leverages spectral analysis (Frank et al.,
2020).

[width=]content/media/dataset.pdf

[width=]content/media/datasets_no_aug.pdf

Figure 1:  Comparison of different data set sizes. When training with
augmentations we observe a steady increase in performance while increasing the
diversity of the data set. We get mixed results for training without
augmentations. Apparently, the higher data diversity can only be utilized with
the aid of data augmentations. Best viewed in color.

[width=]content/media/progan.pdf

[width=]content/media/stylgan.pdf

Figure 2:  Comparison between two training sets generated by ProGAN and
StyleGAN2. The classifier trained on data sets generated by ProGAN generalizes
significantly better to other architectures. Blurring seems to help the
classifier trained on StyleGAN2 to generalize better, the other augmentations
hurt performance. Best viewed in color.

####  3.3.1 Research Question: _Do the results depend on the generator which
creates the data set?_

We want to investigate if the generalization capabilities depend on the
generator which creates the training set. As described in Section 2.1, we
utilized pre-trained StyleGAN2 models to create a novel data set.

We then trained five ResNet50 classifier using the augmentation proposed by
Wang et al.. For comparison, we also trained five classifier on LSUN _cat and
horse_ data generated by ProGAN. In contrast to our previous experiments, we
also include a StyleGAN2 test set which was provided by Wang et al. after the
original publication. The results are displayed in Figure 2 and the
corresponding Table can be found in the supplementary material.

##### 1) Data augmentations help ProGAN generalise to StyleGAN2:

When utilizing the proposed data set augmentation, the classifiers trained on
ProGAN generalize to detect images from StyleGAN2
($87.6\to\\{86.3,87.1,89.2,97.1\\}$ AP). However, the same does not apply for
StyleGAN2. Data generated by StyleGAN2 already achieves quite a high
performance on ProGAN, thus adding data augmentations does only slightly
improve or even hurts performance.

##### 2) Data augmentations are specific to the data set:

Blurring seems to generally hurt the ProGAN classifier (CycleGAN, GauGAN, CRN,
IMLE, SAN, and overall mAP), while it allows the StyleGAN2 classifier to
generalize to the ProGAN, the StarGAN, the SITD, and, the DeepFake data set.
It also achieves the best overall performance. Thus, data augmentations seem
to be specific to the training data set used and should be evaluated
carefully.

##### 3) ProGAN serves as a better prior for generalization:

If we compare the individual performance and the overall mAP, the classifiers
trained on ProGAN seem to better generalize to other architectures. We
hypothesize that this is due to the fact that StyleGAN2 is a very recent
proposal (released in 2020), while all other generators were proposed earlier
(2016-2019). ProGAN is itself an older generator (2018), thus the images
generated by it might have more in common with older architectures.

We conclude that the results indeed depend on the generator used for creating
the data set. The data augmentations do not transfer and our results imply
that going forward, practitioners might have to continuously update their
training data sets to generalize to new architectures. Note that we only
performed experiments with two classes, adding more classes might improve the
performance of StyleGAN2 as a training set.

Table 3:  Comparison of different classifiers trained with different data
augmentations. We compare the AP and mAP across different classifiers which
are trained on the 8-class data set. The Blur + JPEG augmentations refers to
the 50% variant. Chance is 50%, the best possible results is 100%, and we
highlight the ProGAN results in gray since the classifier is trained on
similar data. For each classifier we highlight its best result per colum in
bold.

!  Classifier Augmentation ProGAN StyleGAN BigGAN CycleGAN StarGAN GauGAN CRN
IMLE SITD SAN DeepFake mAP ResNet50 No Aug. 100.0 97.9 77.0 81.9 100.0 73.5
98.0 95.4 96.8 84.9 97.6 91.2 Blur 100.0 97.1 71.6 84.7 100.0 67.0 78.2 76.4
97.6 75.5 98.0 86.0 JPEG 100.0 96.7 83.5 91.6 90.2 95.9 98.8 99.4 86.1 71.6
85.2 90.8 Blur + JPEG 100.0 96.7 81.8 92.7 92.6 95.0 98.2 99.3 85.7 69.8 77.9
90.0 VGG-11 No Aug. 100.0 96.6 76.8 78.3 100.0 60.7 56.8 56.8 99.7 91.4 94.5
82.9 Blur 100.0 94.9 79.1 81.2 100.0 56.6 51.6 51.5 99.9 82.2 92.1 80.9 JPEG
100.0 98.7 87.3 91.1 97.6 85.8 98.8 99.1 94.1 80.3 91.1 93.1 Blur + JPEG 99.8
94.5 85.3 93.3 89.3 91.2 98.4 98.0 95.0 68.2 77.5 90.1 DCT-ResNet No Aug.
100.0 98.0 87.5 63.1 97.2 96.0 65.7 75.6 43.7 52.2 57.0 76.1 Blur 100.0 97.5
82.5 71.7 99.5 96.9 71.4 86.0 49.2 47.4 59.3 78.3 JPEG 100.0 93.1 69.0 50.8
58.0 43.9 79.8 95.7 96.5 61.4 71.9 74.5 Blur + JPEG 100.0 98.2 86.8 54.4 88.5
74.9 82.0 95.7 97.9 59.3 68.7 82.4

####  3.3.2 Research Question: _Do data augmentations transfer to other
classifier?_

Finally, we want to investigate the question if the results transfer to other
classifiers. We train two other classifiers to compare our results: a VGG-11
model and a classifier which utilizes spectral information (DCT-ResNet). The
results are depicted in Table 3, additionally, we provide a plot of the
results in the supplementary material. When we report relative numbers, these
are in reference to the results without augmentations (No Aug.).

When using (No. Aug/Blur) augmentations the VGG model is incapable of
generalizing to the GauGAN ($60.7/56.6$ AP), CRN ($56.8/51.5$ AP) and IMLE
($56.8/51.5$ AP) data sets. This can be circumvented by utilizing the JPEG
augmentation, which also leads to the overall best result ($93.1$ mAP). Yet,
the augmentation does cause a drop in AP for the StarGAN, SITD, SAN and
DeepFake data sets. This performance drop can also not be offset by using both
augmentations in conjunction.

The performance of the DCT based classifier is worse overall
($76.1/78.3/74.5/82.4$ mAP). Blur seem to help generalization, allowing for
better results on CycleGAN, StarGAN, CRN and IMLE data sets. JPEG augmentation
allows the classifier to drastically improve its results on SITD ($+52.8$ AP)
and helps generalization to CRN ($+14.1$ AP), IMLE ($+20.1$ AP), SAN ($+9.2$
AP) and DeepFake ($+14.$9 AP). However, it severely hurts performance for
StarGAN ($-39.2$ AP) and GauGAN ($-52.1$ AP), and, to a lesser extend,
StyleGAN, BigGAN and CycleGAN. Combining both augmentations leads to a good
compromise, achieving the best overall performance ($82.4$ mAP) and reducing
the negative effects for StarGAN ($-8.7$ AP) and GauGAN ($-21.1$ AP).

Overall, we conclude that the proposed data augmentations transfer to other
classifiers. However, there exists no silver bullet which helps generalization
in general.

##  4 Discussion

Since we focused our efforts on reproducing the paper from scratch, we can
only partially comment on the source code. However, we later used it to
compare our implementation against. In the following, we want to give an
overview of our observations.

###  4.1 What was easy?

The overall description in the paper is very detailed. The authors discuss in
detail how they collected the data for the test set, provide a good
description of their training procedure, and summarize their results clearly
with the aid of tables and figures. Additionally, the authors include a long
appendix with further details. We did find one description in the paper
confusing, which we discuss in more detail in Section 4.2.

As discussed above, we only had a limited exposure to using the authors’ code.
The parts we looked at were of good code quality, however, sometimes could use
more comments. The README file of the repository is very detailed, it includes
clear instruction on how to set up the code, download the corresponding data,
train models from scratch, and evaluate the results. Additionally, the authors
include links to download pre-trained models. We used these models to validate
our results and found them quite intuitive to use.

###  4.2 What was difficult?

We came across very few hurdles when reproducing the code. However, we did end
up slightly miss-implementing one of the data augmentations. In the paper, the
authors describe their introduced data augmentations in Section 4.1. The
description for the _Blur+JPEG_ augmentation states: “the image is possibly
blurred and JPEG-ed, each with 50% probability”. However, the shorthand
explicitly uses a plus sign and in the remainder of the paper only the
shorthand is used. Thus, based on the frequent use of the plus sign, we
assumed the augmentation _always_ applies the blurring and JPEG compression
together. This resulted in us implementing a slightly different version of the
data augmentation, where we apply both in conjunction with 50% probability. We
want to stress that the paper states this correctly, we simply missed the last
part of the description.

###  4.3 Communication with original authors

Due to the quality of the paper and the availability of the code repository,
we could resolve every question. We did message the authors regarding our
different results for the Blur and JPEG augmentation. They agreed that it is
difficult to conclude an exact reason why and how augmentations help
generalization. For further research direction, they suggested further
research into which features the CNN learns.

##  5 Conclusion

In summary, we believe the paper to be reproducible. We successfully recreated
the original experiments from scratch. While our results are different, the
overall trend remains the same. However, our additional experiments revealed
several limitations to these claims.

More diverse data sets help the classifier generalize to unseen architectures,
but: the choice of generator is crucial and data set-specific augmentations
have to be used. When using no augmentations, more data diversity hurts the
performance of the classifier trained on ProGAN. When we instead use StyleGAN2
as the generator for the data set, we achieve significantly lower performance
for all tested configurations. This suggest the possibility that future
practitioners might have to constantly update their training sets with new
architectures.

Also, the data augmentations seem to be specific for the training data set
used. When using ProGAN generated data as the training set, the augmentations
transfer across different classifiers. However, only blurring helps StyleGAN2
generalize, while the other proposed augmentations hurt performance. Thus, we
advice practitioners to carefully investigate which augmentations help their
current situation.

## Acknowledgements

We would like to thank our our anonymous reviewers for their valuable
feedback. This work was supported by the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) under Germany’s Excellence Strategy – EXC-2092
CaSa – 390781972.

## References

  * Ahmed et al. [1974] Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao.  Discrete cosine transform.  _IEEE Transactions on Computers_ , 1974.
  * Brock et al. [2019] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large Scale GAN Training for High Fidelity Natural Image Synthesis.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Chen et al. [2018] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun.  Learning to See in the Dark.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Chen and Koltun [2017] Qifeng Chen and Vladlen Koltun.  Photographic Image Synthesis with Cascaded Refinement Networks.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2017\.
  * Choi et al. [2018] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Dai et al. [2019] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang.  Second-order attention network for single image super-resolution.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Frank et al. [2020] Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz.  Leveraging Frequency Analysis for Deep Fake Image Recognition.  In _International Conference on Machine Learning (ICML)_ , 2020.
  * Hao [2019] Karen Hao.  The Biggest Threat of Deepfakes isn’t the Deepfakes Themselves.  _MIT Technology Review_ , 2019.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Huh et al. [2018] Minyoung Huh, Andrew Liu, Andrew Owens, and Alexei A Efros.  Fighting Fake News: Image Splice Detection via Learned Self-Consistency.  In _European Conference on Computer Vision (ECCV)_ , 2018.
  * Isola et al. [2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-Image Translation with Conditional Adversarial Networks.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Karras et al. [2018] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive Growing of GANs for Improved Quality, Stability, and Variation.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A Style-Based Generator Architecture for Generative Adversarial Networks.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Karras et al. [2020] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and Improving the Image Quality of StyleGAN.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Kingma and Ba [2015] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , 2015.
  * Lai et al. [2017] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Ledig et al. [2017] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-Realistic Single Image Super-Resolution using a Generative Adversarial Network.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Li et al. [2019] Ke Li, Tianhao Zhang, and Jitendra Malik.  Diverse Image Synthesis from Semantic Layouts via conditional IMLE.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2019\.
  * Lim et al. [2017] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced Deep Residual Networks for Single Image Super-Resolution.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 2017.
  * Liu et al. [2017] Ming-Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised Image-to-Image Translation Networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Park et al. [2019] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic Image Synthesis with Spatially-Adaptive Normalization.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Rössler et al. [2019] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner.  FaceForensics++: Learning to Detect Manipulated Facial Images.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2019\.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 2015.
  * Simonite [2019] Tom Simonite.  Artificial Intelligence Is Coming for Our Faces.  _Wired_ , 2019.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  In _International Conference on Learning Representations (ICLR)_ , 2015.
  * Thompson and Lapowsky [2017] Nicholas Thompson and Issie Lapowsky.  How Russian Trolls Used Meme Warfare to Divide America.  _Wired_ , 2017.
  * Virtanen et al. [2020] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 2020.
  * Wang et al. [2019a] Run Wang, Lei Ma, Felix Juefei-Xu, Xiaofei Xie, Jian Wang, and Yang Liu.  Fakespotter: A simple baseline for spotting ai-synthesized fake faces.  _arXiv preprint arXiv:1909.06122_ , 2019a.
  * Wang et al. [2019b] Sheng-Yu Wang, Oliver Wang, Andrew Owens, Richard Zhang, and Alexei A Efros.  Detecting Photoshopped Faces by Scripting Photoshop.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2019b.
  * Wang et al. [2020] Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A. Efros.  CNN-generated images are surprisingly easy to spot…for now.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Yu et al. [2015] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao.  LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop.  _arXiv preprint arXiv:1506.03365_ , 2015.
  * Zhang et al. [2019] Xu Zhang, Svebor Karaman, and Shih-Fu Chang.  Detecting and simulating artifacts in gan fake images.  In _IEEE International Workshop on Information Forensics and Security (WIFS)_ , 2019.
  * Zhou et al. [2018] Peng Zhou, Xintong Han, Vlad I Morariu, and Larry S Davis.  Learning Rich Features for Image Manipulation Detection.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Zhu et al. [2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2017\.

## Supplementary Material.

##  Appendix A Training on different subsets of ProGAN without augmentations.

We report the average precision over 11 different CNN generators for both
training configurations. Chance is 50% and the best possible results is 100%.
Note that the classifiers are trained on ProGAN and thus we display the
results in gray. The mean Average Precision (mAP) is obtained by taking the
mean of the individual APs.

!  Setting Result ProGAN StyleGAN BigGAN CycleGAN StarGAN GauGAN CRN IMLE SITD
SAN DeepFake mAP 2-class Blur + JPEG (0.5) 99.3 81.0 71.8 88.4 83.5 89.7 98.8
99.5 82.0 72.3 61.5 84.3 No Aug. 99.9 94.4 78.9 82.1 99.9 76.7 94.9 92.2 96.0
83.9 95.5 90.4 4-class Blur + JPEG (0.5) 99.8 90.6 79.7 92.5 91.2 93.5 98.1
98.7 95.0 74.6 75.7 90.0 No Aug. 100.0 97.0 77.0 81.9 100.0 73.5 98.0 95.4
96.8 84.9 97.6 91.2 8-class Blur + JPEG (0.5) 100.0 96.7 81.8 92.7 92.6 95.0
98.2 99.3 85.7 69.8 77.9 90.0 No Aug. 99.99 96.3 75.7 89.2 100.0 75.2 96.2
94.8 96.4 91.61 97.4 92.1 16-class Blur + JPEG (0.5) 100.0 98.5 87.0 94.5 95.0
96.8 99.2 99.3 84.5 77.9 76.8 91.8 No Aug. 100.0 97.6 74.7 81.1 100.0 67.8
96.4 93.6 98.3 89.9 97.9 90.7 20-class Blur + JPEG (0.5) 100.0 99.4 88.8 94.1
96.7 96.2 98.5 99.1 92.9 72.3 93.7 93.8 No Aug. 100.0 96.8 73.5 81.9 100.0
68.2 95.1 88.8 97.1 87.2 98.4 89.7

##  Appendix B Using StyleGAN2 to generate the training data set.

We compare the AP and mAP of multiple ResNet50 classifier when trained on data
generated from both ProGAN and StyleGAN. We include a new test data set based
on multiple StyleGAN2 instances. When the score is computed over samples from
generators on which the classifier was trained on, we highlight them in gray.
Chance is 50%, best possible result is 100%.

!  Augmentations Training Set ProGAN StyleGAN StyleGAN2 BigGAN CycleGAN
StarGAN GauGAN CRN IMLE SITD SAN DeepFake mAP No Aug. ProGAN 99.1 93.4 87.6
75.3 82.8 99.8 81.8 93.7 88.2 91.2 70.3 84.6 86.4 StyleGAN2 98.5 97.1 100.0
64.9 61.8 100.0 68.5 86.8 87.9 52.4 40.9 59.1 76.2 Blur ProGAN 99.9 94.5 97.1
76.6 80.6 100.0 68.3 84.2 75.0 95.6 67.0 97.5 85.8 StyleGAN2 99.2 93.9 99.9
65.7 60.4 100.0 65.4 83.6 79.1 84.2 48.9 92.0 80.6 JPEG ProGAN 99.2 92.3 86.3
80.0 89.2 92.3 94.7 95.2 95.3 89.0 62.6 72.2 87.4 StyleGAN2 92.3 96.3 99.4
67.5 55.3 67.7 77.1 91.3 91.7 74.0 41.7 38.9 74.5 Blur + JPEG (0.5) ProGAN
99.1 92.8 87.1 79.3 89.1 91.1 93.1 95.7 97.6 87.0 60.9 78.7 87.6 StyleGAN2
91.9 95.7 99.1 67.6 57.0 64.8 77.7 84.8 81.7 88.5 46.3 40.2 74.2 Blur + JPEG
(0.1) ProGAN 99.6 93.7 89.2 79.0 88.1 92.7 91.8 95.8 94.6 93.0 67.7 76.2 88.3
StyleGAN2 96.1 97.8 99.6 65.7 57.0 73.3 70.0 92.3 92.8 68.8 42.6 44.1 74.6

##  Appendix C Multiple runs for the corrected version of the Blur + JPEG
augmentation

Comparison of multiple runs when training a classifier from scratch with the
corrected Blur + JPEG augmentation. We observe a high fluctuation in the SITD,
SAN, and, DeepFake data set.

!  Augmentations ProGAN StyleGAN2 BigGAN CycleGAN StarGAN GauGAN CRN IMLE SITD
SAN DeepFake mAP Run 1 100.0 99.1 88.8 94.8 94.5 97.5 99.3 99.4 84.8 74.1 73.0
91.4 Run 2 100.0 99.1 89.5 95.1 93.9 98.0 99.1 99.4 90.7 72.6 72.4 91.8 Run 3
100.0 99.2 88.9 94.7 95.9 97.3 99.5 99.5 85.9 78.3 76.1 92.3 Mean $\pm$ std.
100.0 $\pm$ 0.0 99.1 $\pm$ 0.1 89.1 $\pm$ 0.4 94.9 $\pm$ 0.2 94.8 $\pm$ 1.0
94.8 $\pm$ 0.4 99.3 $\pm$ 0.3 99.4 $\pm$ 0.1 87.1 $\pm$ 3.1 75.0 $\pm$ 3.0
73.8 $\pm$ 2.0 91.8 $\pm$ 0.5

##  Appendix D Comparing if data augmentations transfer across models.

We compare the AP and mAP across different classifiers which are trained on
the 8-class data set. We can observe the general trend that data augmentations
help classifier generalize. However, there are notable exceptions for both VGG
(StarGAN/SAN/DeepFake) and the DCT-ResNet
(BigGAN/CycleGAN/StarGAN/GauGAN/SAN).

[width=]content/media/resnet50.pdf

[width=]content/media/vgg.pdf

[width=]content/media/dct.pdf

Figure 3:  Different classifier trained with data augmentations. The data
augmentations seem to transfer to other classifiers and, in some instances,
are the key to generalization. These experiments were conducted on eight
classes from the training data set. Best viewed in color.


# An Analytical Estimation of Spiking Neural Networks Energy Efficiency
††thanks: This research is funded by the ANR project DeepSee, Université Côte
d’Azur, CNRS and Région Sud Provence-Alpes-Côte d’Azur.

Edgar Lemaire  LEAT, Univ. Côte d’Azur, CNRS
<EMAIL_ADDRESS>Loïc Cordone  Renault Software Factory
LEAT, Univ. Côte d’Azur, CNRS
<EMAIL_ADDRESS>Andrea Castagnetti  LEAT, Univ. Côte d’Azur,
CNRS
<EMAIL_ADDRESS>Pierre-Emmanuel Novac  LEAT, Univ. Côte
d’Azur, CNRS
<EMAIL_ADDRESS>Jonathan Courtois  LEAT, Univ. Côte
d’Azur, CNRS
<EMAIL_ADDRESS>Benoît Miramond  LEAT, Univ. Côte d’Azur,
CNRS
<EMAIL_ADDRESS>
###### Abstract

Spiking Neural Networks are a type of neural networks where neurons
communicate using only spikes. They are often presented as a low-power
alternative to classical neural networks, but few works have proven these
claims to be true. In this work, we present a metric to estimate the energy
consumption of SNNs independently of a specific hardware. We then apply this
metric on SNNs processing three different data types (static, dynamic and
event-based) representative of real-world applications. As a result, all of
our SNNs are 6 to 8 times more efficient than their FNN counterparts.

###### Index Terms:

Spiking neural networks, Energy metrics, Computational metrics, Event-based
processing, Low-power artificial intelligence

##  I Introduction

Neuromorphic computing has been studied for many years as a game changer to
address low-power embedded AI, assuming that the inspiration from the brain
will natively come with a reduction in energy consumption. Neuromorphic
computing mainly focuses on the encoding and the processing of the information
with spikes. If this property takes an obvious place in the biological
functioning, it is far from obvious that it is the only one to explain the
efficiency of the brain. It is therefore necessary to ask the question whether
considering this characteristic in isolation brings a gain compared to the
classical neural networks used in deep learning. This is the question that
this paper seeks to answer by restricting the study to standard machine
learning tasks on three different types of data: static, dynamic and event-
based data.

There already exist comparisons between Spiking Neural Networks (SNNs) and
Formal Neural Networks (FNNs, i.e. non-spiking Artificial Neural Networks) in
the literature. However, such comparisons are hardly generalizable since they
focus on specific applications or hardware targets [1], [2]. Moreover, the
considered applications are often toy examples not representative of real-
world AI tasks. Another approach consists in producing metrics in order to
evaluate the relative energy consumption between the two coding domains, based
on their respective synaptic operations and activity. We thus propose a novel
metric for energy consumption estimation taking synaptic operations, memory
accesses and element addressing into account. Moreover, our metric is mostly
independent from low-level implementation or hardware target to ensure its
generality.

The proposed metric is described and applied to three datasets representative
of the aforementioned data types: CIFAR-10 for the static case, Google Speech
Commands V2 for the dynamic case, and Prophesee NCARS for the event-based
case. Moreover, those datasets are closer to real-world applications than
usual benchmarks of the neuromorphic community. The metric is used in
conjunction with accuracy measurements to provide an in-depth evaluation of
those three application cases and their relevance for spiking acceleration. We
use the advanced Surrogate Gradient Learning technique and Direct Encoding
spike conversion method, since they offer the best trade-off between
prediction accuracy and synaptic activity [3].

Our code and trained SNN models are available upon request.

##  II State of the art

####  II-1 Data Encoding

In order to process data in SNNs, it must be encoded into spikes. Rate, Time
and Direct encoding are three methods used to convert conventional data
towards spiking domain. Rate coding [4] is the most notorious, since it
provides state-of-the-art accuracy on most AI tasks. However, it generates a
lot of spikes over a large number of timesteps, drastically impacting
computational and energy efficiency. Time coding [5] intends to cope with this
issue by encoding information in latency rather than rate, thus generating
much fewer spikes. Yet, the temporal sparsity of latency-coded spikes causes
long processing times, resulting in an energy overhead.

To cope with those limitations, we address a novel encoding scheme: Direct
Encoding [6]. It should be noted that this term is a proposition of ours. In
Direct Encoding, the first processing layer is made of hybrid neurons with
analog inputs and spiking behaviour (IF, LIF…). The weights of this layer are
learned during training, thus encoding can be tuned to reduce spiking activity
in the network [7]. In the present work, we evaluate Direct Encoding on
CIFAR-10 and GSC datasets. Additionally, we evaluate native spike encoding
using event cameras [8]. In this method, each pixel of the sensor generates a
spike whenever it detects a brightness variation, thus encoding movement into
spikes. With such sensor, the input spiking activity is very low, since only
the information of interest (i.e. moving objects) are returned by the camera.
This property helps improving the computational and energy efficiency of SNNs.
We evaluate native encoding using the Prophesee NCARS dataset.

####  II-2 Training of SNN in the literature

Spiking neural networks cannot use the classical backpropagation training
algorithm to learn their weights because its activations (spikes) are binary
and thus non-differentiable. Encoding static data such as images using rate
coding enables the conversion of an already trained FNN to a SNN. The most
common way is to replace the ReLU neurons of the FNN by IF neurons [9].
However, the prediction accuracy obtained through conversion is systematically
inferior to their FNN counterpart, while generating a lot of spikes over a
large number of timesteps. Numerous works have studied how to train SNNs
directly in spiking domain. The best results were obtained using
backpropagation-based learning rules, such as the surrogate gradient [10]. To
circumvent the non-differentiability of spikes in SNNs, the main idea of
surrogate gradient learning is to use two distinct functions in the forward
and backward passes: an Heaviside step function for the first, and a
differentiable approximation of the Heaviside in the latter, such as a sigmoid
function. Using surrogate gradient learning requires a fixed number of
timesteps. As the number of computations performed by SNNs increases with the
number of timesteps, being able to fix it beforehand and to tune it during the
training is vital to increase the computational efficiency of SNNs.

####  II-3 Comparisons based on measurements

In the literature, a few comparisons of SNNs and FNNs have been produced based
on hardware measurements. Some papers show competitive results for SNNs: in
[11], the authors highlighted the influence of the spike encoding method on
the accuracy and computational efficiency of the SNN. They compared the
spiking and formal networks through a Resnet-18 like architecture on two
classification datasets. They found that SNNs reached higher or equivalent
accuracy and energy efficiency. In [1], the authors showed that an SNN could
reach twice the power and resource efficiency of an FNN, with an MLP on MNIST
dataset targeting ASIC. However, those encouraging results are still very
specific and thus hardly generalizable. In a more holistic approach, the
authors of [4] performed a design space exploration (including encoding,
training method, level of parallelism…) and showed that the advantage of the
SNN depended on the considered case, making it difficult to draw general
rules. In [2], researchers showed that SNNs on dedicated hardware (Loihi)
demonstrated better energy efficiency than equivalent FNNs on generic hardware
(CPU and GPU) for small topologies, but observed the opposite using larger
CNNs. Once more, the conclusions depended on the studied case and could not be
generalized. Albeit encouraging, those results are not sufficient to draw
general conclusions regarding the savings offered by event-based processing,
since they depend on the selected application, network hyper-parameters and
hardware targets. Therefore, another approach consists in comparing both
coding domains through estimation metrics, taking a step back to produce more
general conclusions.

####  II-4 Comparisons based on metrics

Most energy consumption metrics are based on the number of synaptic
operations: accumulations (ACC) in the SNN and multiplication-accumulations
(MAC) in the FNN. Those models have limitations: energy consumption is
assimilated to the energy consumption of synaptic operations [12], thus other
factors (such as neuron addressing in multiplexed architecture or memory
accesses) are often neglected. Moreover, the models usually do not take into
account some specific mechanisms, like membrane potential leakage, reset and
biases integration. In [11], the authors proposed a metric based on synaptic
operations only, and found great energy consumption savings for the SNN (up to
$126\times$ more efficient than the FNN baseline). In [4] the authors
demonstrated that such simplistic metrics were not always coherent with actual
energy consumption of circuits on FPGA. When taking memory into account,
another team [13] found equivalent energy consumptions for SNNs and FNNs using
various topologies on CIFAR10. Additionally, reference [14] measured a
theoretical maximum spike rate of 1.72 to guarantee energy savings in the SNN
based on a detailed metric, accounting for synaptic operations, memory
accesses and activation broadcast. Those energy consumption models are
enlightening, but still fail to settle whether event-based processing is
sufficient to increase energy efficiency. That is mostly because those metrics
are too hardware specific, or do not take all significant sources of energy
consumption into account.

In the present work, we propose a metric intended to be independent from low-
level implementation choices, based on three main operations: neuron
addressing, synaptic operations and memory accesses.

##  III Metrics

###  III-A Operational cost

In this section we define a metric to compute the number of ACC and MAC due to
synaptic operations in SNNs and FNNs.

####  III-A1 Convolutional layers

For a convolution layer, the number of filters is defined by $C_{\text{out}}$
and their size are noted $C_{\text{in}}\times H_{\text{kernel}}\times
W_{\text{kernel}}$, where $C$, $H$ and $W$ stands for channel, height and
width. The input and output of the layer are composed of a set of feature
maps, with shapes $(C_{\text{in}}\times H_{\text{in}}\times W_{\text{in}})$
and $(C_{\text{out}}\times H_{\text{out}}\times W_{\text{out}})$ respectively.
In the following we consider the padding mode “same” and a stride $S$. The
number of timesteps is noted $T$. The equations describing the number of MAC
and ACC operations in FNNs and SNNs, for convolution layers, are summaried in
Eq. 1.

$\begin{split}\text{MAC}_{\text{Conv-l}}^{\text{FNN}}&=C_{\text{out}}\times
H_{\text{out}}\times W_{\text{out}}\times C_{\text{in}}\times
H_{\text{kernel}}\times W_{\text{kernel}}\\\
\text{ACC}_{\text{Conv-l}}^{\text{FNN}}&=C_{\text{out}}\times
H_{\text{out}}\times W_{\text{out}}\\\
\text{MAC}_{\text{Conv-l}}^{\text{SNN}}&=T\times C_{\text{out}}\times
H_{\text{out}}\times W_{\text{out}}\\\
\text{ACC}_{\text{Conv-l}}^{\text{SNN}}&=\theta_{\text{l -
1}}\times(\lceil\frac{H_{\text{kernel}}}{S}\rceil)\times(\lceil\frac{W_{\text{kernel}}}{S}\rceil)\times
C_{\text{out}}\\\ &+T\times C_{\text{out}}\times H_{\text{out}}\times
W_{\text{out}}+\theta_{\text{l}}\\\ \end{split}$ (1)

In FNNs, the integration of dense input matrixes requires a MAC operation for
each element of the convolution kernels. That is described in the first row of
Eq. 1. Additionally, the integration of synaptic biases as ACC operations,
since it does not require multiplication with input activation. There is one
bias per output neuron as shown in the second row of Eq. 1.

On the other hand, for an SNN the input activations are sparse binary
matrices. The number of operations of the layer $l$, depends on the number of
input and output spikes of that layer, noted $\theta_{\text{l - 1}}$ and
$\theta_{\text{l}}$. Since spikes are binary and, they are integrated via ACC
operations in contrast with FNNs. Each input spike causes one ACC operation
per element of each filter, as shown in the first term of the fourth row
($\text{ACC}_{\text{Conv-l}}^{\text{SNN}}$) of Eq. 1. The second term accounts
for the bias added to each membrane potential at each timestep. The third term
accounts for the membrane potential reset whenever an output spike is
generated. Additionally, SNNs may involve a membrane potential leakage (i.e.
LIF neurons), which is modeled by an additional MAC operations for each output
neuron, which is repeated at each timestep. This is depicted in the last two
rows of Eq. 1.

####  III-A2 Fully-connected layers

The same reasoning is applied to FC layers. For a given layer $l$ the number
of input and output neurons is noted $N_{in}$ and $N_{out}$ respectively. The
equations for the number of MAC and ACC operations attributable to synaptic
operations in FC layers are summarized in Eq. 2.

$\begin{split}\text{MAC}_{\text{FC-l}}^{\text{FNN}}&=N_{\text{in}}\times
N_{\text{out}}\\\ \text{ACC}_{\text{FC-l}}^{\text{FNN}}&=N_{\text{out}}\\\
\text{MAC}_{\text{FC-l}}^{\text{SNN}}&=N_{\text{out}}\times T\\\
\text{ACC}_{\text{FC-l}}^{\text{SNN}}&=\theta_{\text{l - 1}}\times
N_{\text{in}}\times N_{\text{out}}+T\times N_{\text{out}}\\\ \end{split}$ (2)

###  III-B Memory cost

In order to provide an evaluation of the energy used by both the FNN and the
SNN, multiple assumptions have to be made. Without these assumptions, results
could vastly vary between different unconstrained hardware implementations.
Each layer of the FNN is assumed to have its own local (non-shared) memory. As
a result, activations need to be kept in memory (i.e. I/O buffers) for all
layers. The data flow between layers of the SNN is assumed to be sparse and
asynchronous. Therefore, messages of incoming spikes must be buffered in a
FIFO queue for each layer. Additionally, the SNN must keep the membrane
potentials for all layers between timesteps. In both cases, we assume that all
the memory is akin to local SRAM, including weights in order to support a
reconfigurable architecture. Additionally, there is no local caching in a
register bank. Only registers for the operands and a local accumulator are
present and are excluded from this evaluation. All data is assumed to be
represented with the same number of bits, including the messages describing a
spike.

####  III-B1 Memory accesses

Data flowing from and to the memory is an important sink of energy. We attempt
to describe each read or write operation of both the FNN and the SNN for each
layer in order to evaluate the possible energy savings from using an SNN,
which is mainly a result of its sparsity. Equations are provided for a single
input for the FNN and a single timestep for the SNN.

##### Read operations to inputs

For a formal Conv layer, each output position matches with read operations
from all input channels and all positions for which the kernel applies. For a
formal FC layer, the number of read operations for the input data is equal to
the number of inputs $N_{\text{in}}$.

$\displaystyle RdIn^{\text{FNN}}_{\text{Conv}}=$ $\displaystyle
C_{\text{in}}\times C_{\text{out}}\times H_{\text{out}}\times
W_{\text{out}}\times W_{\text{kernel}}\times H_{\text{kernel}}$ (3)
$\displaystyle RdIn^{\text{FNN}}_{\text{FC}}=$ $\displaystyle N_{\text{in}}$
(4)

For the SNN, the read operations in the queue directly depends on the number
of incoming spikes $\theta_{\text{l-1}}$ and must be measured during
inference:

$RdIn^{\text{SNN}}=\theta_{\text{l-1}}$ (5)

##### Read operations to parameters

In a formal Conv layer, each output position matches with read operations for
all the weights in all filters associated to all input channels. The biases
generate additional reads for all output positions and all filters. For an FC
layer, every weight corresponding to all output neurons $N_{\text{out}}$ and
all inputs $N_{\text{in}}$. The biases cause additional read for all output
neurons.

$\displaystyle\begin{split}RdParam^{\text{FNN}}_{\text{Conv}}=&(C_{\text{in}}\times
W_{\text{kernel}}\times H_{\text{kernel}}+1)\\\ \times&C_{\text{out}}\times
W_{\text{out}}\times H_{\text{out}}\\\
RdParam^{\text{FNN}}_{\text{FC}}=&(N_{\text{in}}+1)\times
N_{\text{out}}\end{split}$ (6)

In an spiking Conv layer, all received spikes $\theta_{\text{l-1}}$ will
trigger a read for all output filters and all associated output positions
(i.e. of the dimensions of the kernel). Biases for all output positions and
filters must still be read. For an FC layer, the number of read operations for
parameters is similar to an SNN except that weights are only read for all
input spikes $\theta_{\text{l-1}}$. Biases must still be read for all output
neurons $N_{\text{out}}$.

$\displaystyle RdParam^{\text{SNN}}_{\text{Conv}}=$
$\displaystyle\theta_{\text{l-1}}\times C_{\text{out}}\times
W_{\text{kernel}}\times H_{\text{kernel}}$ $\displaystyle+$ $\displaystyle
C_{\text{out}}\times W_{\text{out}}\times H_{\text{out}}$ (7) $\displaystyle
RdParam^{\text{SNN}}_{\text{FC}}=$ $\displaystyle\theta_{\text{l-1}}\times
N_{\text{out}}+N_{\text{out}}$ (8)

##### Read operations to potentials

There is no membrane potential to update in an FNN so there is no associated
read operation.

In a spiking Conv layer, the membrane potentials corresponding to all output
positions affected by each input (i.e. of the dimensions of the kernel) in all
filters must be read in order to update them. Biases need to be applied
separately and therefore generate an additional read operation at each
timestep for all output positions and all filters. For FC layers, the
potentials of all output neurons are read for each input. Biases are applied
separately and therefore generate an additional read operation at each
timestep for all output neurons.

$\displaystyle RdPot_{\text{Conv}}=$ $\displaystyle\theta_{\text{l-1}}\times
C_{\text{out}}\times W_{\text{kernel}}\times H_{\text{kernel}}$
$\displaystyle+C_{\text{out}}\times H_{\text{out}}\times W_{\text{out}}$ (9)
$\displaystyle RdPot_{\text{FC}}=$ $\displaystyle(\theta_{\text{l-1}}+1)\times
N_{\text{out}}$ (10)

##### Write operations to outputs

In an formal Conv layer, each output position in all filters require a write
operation. For an FC layer, each output neuron require a write operation. In
both cases, the output is assumed to be fully computed in the local
accumulator, including bias, before being written to RAM.

$\displaystyle WrOut^{\text{FNN}}_{\text{Conv}}=$ $\displaystyle
C_{\text{out}}\times H_{\text{out}}\times W_{\text{out}}$ (11) $\displaystyle
WrOut^{\text{FNN}}_{\text{FC}}=$ $\displaystyle N_{\text{out}}$ (12)

For the SNN, the write operations to the queue directly depends on the number
of generated spikes $N_{\text{output}}$ and must be measured during inference:

$WrOut=N_{\text{output}}$ (13)

##### Write operations to potentials

There is no membrane potential to update in an FNN so there is no associated
write operation.

In an spiking Conv layer, the membrane potentials corresponding to all output
positions affected by each input (i.e. of the dimensions of the kernel) in all
filters must be written to in order to update them. Additionally, the biases
must also be written separately to the potentials for all output positions and
all filters at each new timestep. For an FC layer, the potentials of all
output neurons must be written to for each input. Additionally, the biases
must also be written to the potentials for all output neurons separately at
each new timestep.

$\displaystyle WrPot_{\text{Conv}}=$ $\displaystyle\theta_{\text{l-1}}\times
C_{\text{out}}\times W_{\text{kernel}}\times H_{\text{kernel}}$
$\displaystyle+C_{\text{out}}\times H_{\text{out}}\times W_{\text{out}}$ (14)
$\displaystyle WrPot_{\text{FC}}=$ $\displaystyle\theta_{\text{l-1}}\times
N_{\text{out}}+N_{\text{out}}$ (15)

###  III-C Addressing in Sparse vs. Dense Convolutions

In this subsection, we evaluate the cost of addressing in FNNs and SNNs. The
first uses dense processing, in which all input synapses are stimulated at the
same time. On the other hand, the second uses sparse processing, in which
synapses are sparsely stimulated across time. Let us begin with convolution
layers. In order to simplify the following matter, we consider convolutions
with a “same” padding (input and output feature maps of same sizes) and a
stride of $S$. In FNNs, a kernel scans all its possible positions (depending
on padding, stride…) on the input sample and generates a dense output feature-
map. In such dense convolutions, computation is performed sequentially and
addresses can be computed by incrementing only an index (by 1 or $S$) assuming
the memory is contiguous and ordered the same way it is processed. Thus, one
index runs through the input, one index runs through the output and one index
runs through the weights. In SNNs, sparse convolutions are performed
asynchronously upon reception of input spikes, thus the kernel positions (i.e.
output neuron addresses) must be calculated each time a spike is received. In
a sparse representation, computation is performed non-sequentially with no
prior knowledge of which output position is affected by an incoming spike.
Computing the initial output position requires two multiplications.
Thereafter, the computation of the remaining positions are computed by
incrementing an index assuming the memory is contiguous and ordered as for
FNNs. There is only one index running through the kernel weights. The cost of
addressing in number ACC and MAC operations in spiking and formal convolution
layer are summarized in Equation 16.

$\begin{split}\text{ACC}_{\text{Addr-
Conv-l}}^{\text{FNN}}&=C_{\text{in}}\times H_{\text{in}}\times
W_{\text{in}}+C_{\text{out}}\times H_{\text{out}}\times
W_{\text{out}}+C_{\text{out}}\\\ &\times H_{\text{kernel}}\times
W_{\text{kernel}}\\\ \text{MAC}_{\text{Addr-
Conv-l}}^{\text{SNN}}&=\theta_{\text{l-1}}\times 2\\\ \text{ACC}_{\text{Addr-
Conv-l}}^{\text{SNN}}&=\theta_{\text{l-1}}\times C_{\text{out}}\times
H_{\text{kernel}}\times W_{\text{kernel}}\\\ \end{split}$ (16)

Where $C$ are the numbers of channels, $W$ the widths and $H$ the heights, of
respectively the inputs when the index is in, outputs when it is out and
kernels when it is kernel. $\theta_{\text{l-1}}$ is the number of input
spikes.

The same reasoning is applied to fully-connected layers. In FNNs, one index
runs through the input, and another index runs through both the output. In
SNNs, one single index runs through the output upon receiving an input spikes.
This yields Eq. 17.

$\begin{split}\text{ACC}_{\text{Addr-
FC-l}}^{\text{FNN}}&=N_{\text{in}}+N_{\text{out}}\\\ \text{ACC}_{\text{Addr-
FC-l}}^{\text{SNN}}&=\theta_{\text{l-1}}\times N_{\text{out}}\\\ \end{split}$
(17)

Where $N_{\text{in}}$ and $N_{\text{out}}$ are resepctively the number of
input and output neurons.

###  III-D Energy consumption metric

In this section, we combine the equations obtained for computation, memory
accesses and addressing in a global Energy evaluation metric. For this
purpose, we multiply the energy cost of each operation by its number of
occurrences, according to the metrics computed in subsections III-A, III-B and
III-C. Our model can be summarized as shown in Eq. 18:

$E=E_{\text{mem}}+E_{\text{ops+addr}}$ (18)

Where $E^{\text{mem}}$ is the energy consumption of memory accesses,
$E^{\text{ops}}$ that of synaptic operations, and $E^{\text{addr}}$ that of
addressing mechanisms.

Those elements are computed based on the metrics proposed in the above
subsections, as shown in Eq. 19 for memory accesses, and Eq. 20 for addressing
and synaptic operations. Equations are not repeated for those two last
elements since they are identical.

$\begin{split}E_{\text{mem}}^{\text{FNN}}&=(RdIn^{\text{FNN}}+RdParam^{\text{FNN}})\times
E_{\text{RdRAM}}\\\ &+WrOut^{\text{FNN}}\times E_{\text{WrRAM}}\\\
E_{\text{mem}}^{\text{SNN}}&=(RdIn^{\text{SNN}}+RdParam^{\text{SNN}}+RdPot)\times
E_{\text{RdRAM}}\\\ &+(WrOut^{\text{SNN}}+WrPot)\times E_{\text{WrRAM}}\\\
\end{split}$ (19)

With $E_{\text{RdRAM}}$ and $E_{\text{WrRAM}}$ the energy for a single read
and a single write operation in RAM, respectively. In our computation, we
assume that $E_{\text{RdRAM}}=E_{\text{WrRAM}}$ for simplicity purpose.

$\begin{split}E_{\text{ops+addr}}^{\text{FNN}}&=(E_{\text{ADD}}+E_{\text{MUL}})\times\text{MAC}_{\text{ops+addr}}^{\text{FNN}}\\\
&+E_{\text{ADD}}\times\text{ACC}_{\text{ops+addr}}^{\text{FNN}}\\\
E_{\text{ops+addr}}^{\text{SNN}}&=(E_{\text{ADD}}+E_{\text{MUL}})\times\text{MAC}_{\text{ops+addr}}^{\text{SNN}}\\\
&+E_{\text{ADD}}\times\text{ACC}_{\text{ops+addr}}^{\text{SNN}}\\\
\end{split}$ (20)

Where $E_{\text{ADD}}$ and $E_{\text{MUL}}$ are the energy cost of single
additions and multiplications respectively.

The energy consumption of single operations (addition, multiplication and
memory accesses) are drawn from the literature [15] for 45nm CMOS technology.
For addition and multiplication with 32-bit integers, we use respectively
$0.1pJ$ and $3.1pJ$. For SRAM memory accesses, we compute a linear
interpolation function based on 3 particular values : 8 kB ($10pJ$), 32 kB
($20pJ$) and 1 MB ($100pJ$). This function enables to compute the energy cost
of a memory access knowing the memory size (i.e. knowing the network hyper-
parameters).

##  IV Methods

###  IV-A Spike coding

The role of spike coding is to convert input pixels into spikes, which are in
turn transmitted to an SNN for information processing. In the _Direct
encoding_ scheme, a spiking neuron (e.g. LIF or IF) located in the first layer
(encoding) of the network is fed with a constant or dynamic input through $T$
timesteps. The first layer thus converts the analog input into spike trains.
In this paper, we propose three different uses of such encoding which are
detailed below.

####  IV-A1 Static Frame-based data encoding

The first method is Static Frame-based data encoding, in which the raw input
data is directly broadcast to the first layer. The network input size is thus
identical to that of the input sample, each of which is presented repeatedly
during $T$ timesteps. This method is adapted to every type of conventional
data. The major drawback of this method is that the whole processing must be
repeated over several timesteps (bias integration, membrane leakage…).

####  IV-A2 Dynamic Frame-based data encoding

The second method is Dynamic Frame-based data encoding, adapted to temporal
signals. The input data is split into several chunks along the temporal
dimension. The network input size is the same as the size of a chunk, which
are presented successively at the input, one per timestep. This approach has
two main benefits: reducing the input size and no longer considering timesteps
mechanisms as an additional cost.

####  IV-A3 Event-based data

Event-based data does not require a specific encoding since events can be
interpreted as spikes. However, in order to process event-based data in modern
deep learning models, we need to convert them into a dense representation. To
process events with FNNs, we simply sum all events occurring during the sample
duration $d$ to reconstruct a single frame, containing integer values. On the
other hand, when using SNNs, we accumulate events over $T$ time windows (i.e.
timesteps) lasting $\Delta t=\frac{d}{T}$ to reconstruct frames. This type of
representation is called a voxel grid [16], where each voxel represents a
pixel and a time interval. We added the constraint that the accumulation is a
simple OR operation: if at least one event is present during the time window
then its value will be 1 [17]. This way, all of our event frames stays binary
in order to leverage the efficiency of spiking neural networks running on
specialized hardware [18].

###  IV-B Organization of the output layer

Both our FNN and SNN for static frame-based data use a traditional final
fully-connected layer. On the other hand, our models for dynamic frame-based
and event-based data use a specific final classification layer as the feature
maps are not sufficiently reduced to be flattened before the final fully-
connected layer.

We followed the approach used in [17]. The output layer of our SNNs is simply
composed of a batch normalization layer, a $1\times 1$ convolution outputting
$num\\_classes$ channels and a final layer of LIF neurons. The final
predictions are then obtained by summing all output spikes first in the
spatial dimension and time dimension. We therefore obtain a tensor with a
spatial size of $1\times 1$ with $num\\_classes$ channels, which is equivalent
to the output of conventional fully-connected layers. The 1D convolution in
the final layer enables to avoid the use of e.g. average pooling to reduce the
spatial dimension as it would be incompatible with spikes computations. We
used the same approach for the equivalent FNNs but without summation along the
time axis since it does not exist.

##  V Experiments and Results

###  V-A Datasets and Models

####  V-A1 Static frame-based data

The CIFAR-10 dataset is made of 60000 32x32 RGB images representing 10
classes. For the SNN, CIFAR-10 samples are repeated as input over $T=4$
timesteps, following the static frame-based data encoding described in Section
IV-A. For this task, we use a VGG-16 architecture described in [19]. We
dropped the max pooling layers by using a stride of 2 in their preceding
convolution and we added batch normalization layers after each convolution.

####  V-A2 Dynamic frame-based data

The Google Speech Commands V2 dataset is a dataset of audio signals sampled at
16 kHz composed of 1-second recordings of 35 spoken keywords. We performed
data augmentation by randomly changing the speed of the raw audio signals. The
raw data are preprocessed to obtain images that can be fed to CNNs. We used 10
MFC Coefficients, FFT of size 1024, a window size of 640 with a hop of 320,
and a padding of 320 on both sides. This results in a 48x10 image interpreted
as 1D data truncated to 48 samples with 10 channels. For the SNN, we divided
this temporal data in $T=2$ timesteps, each of size 24. To tackle this
classification task, we designed a 4-layers CNN with the following topology:
48c3 - 48c3 - 96c3 - 35c1. Each convolutional layer has a stride of 1 and was
followed by a batch normalization layer.

####  V-A3 Event-based data

The Prophesee NCARS dataset [20] is a classification dataset composed of 24k
samples of length 100ms captured with a Prophesee GEN1 event camera mounted
behind the windshield of a moving car. The samples represent either a car or
background. We resized all the samples to a size of $64\times 64$ pixels using
nearest neighbor interpolation to keep our inputs binary. For SNNs, we divided
each sample in $T=5$ timesteps, while all the events were summed into a single
frame for the CNN. We proposed a variant of the classical Tiny VGG-11
architecture [19] that uses 4 times fewer channels in each convolution layers,
reducing the number of parameters and calculations. Once again, we dropped the
max pooling layers by using a stride of 2 in their preceding convolution and
we added batch normalization layers after each convolution. Finally, we
replaced the final 3 fully-connected layers by our output layer described in
Section IV-B.

###  V-B Results

We trained our FNNs using PyTorch, and our SNNs using SpikingJelly [21] with
surrogate gradient learning. The models were trained over 50 epochs for GSC
and NCARS, and 300 for CIFAR-10. All presented results represent the average
over 5 runs. The performance of our networks were measured by their
classification accuracy. We also measured the spike rate of our SNNs,
corresponding to the average number of spikes per synapse in the network.
Since computations are only performed when there is a spike, this has a direct
impact on the SNN energy consumption within our metric. These results are
summarized in Table I.

TABLE I: Accuracy and Activity comparisons between our proposed SNNs and CNNs on CIFAR-10, Google Speech Commands V2, and Prophesee NCARS. Dataset | Network | #Params | Activation | Acc. | Spike Rate
---|---|---|---|---|---
CIFAR-10 | VGG-16 | 15.2M | ReLU | 0.951 | –
IF | 0.884 | 0.10
GSC | 4-layers CNN | 29k | ReLU | 0.936 | –
LIF | 0.918 | 0.14
NCARS | Tiny VGG-11 | 356k | ReLU | 0.934 | –
LIF | 0.935 | 0.08

For dynamic and event data, SNNs are able to reach equivalent or close
accuracies to their FNN counterparts, a result never shown experimentally
before on these datasets. On the CIFAR-10 dataset, our SNN reaches lower
accuracy than the FNN. This is coherent with state of the art results, but
should be improved in further works. Still, all of our SNNs reach these
performance while having a very low spike rate, on average each neuron of a
model spikes less than 0.14 times per inference for the three datasets.

Using the metrics proposed in Section III-B, we were able to precisely
estimate the energy consumption of our models. In our model, I/O buffers
between SNN layers are FIFOs able to store 1000 32-bit elements. This
assumption has been validated through hardware simulation using SPLEAT
architecture [18]. On the other hand, the full feature maps are stored in
SRAMs between FNN layers. The results are illustrated in Fig. 1 and detailed
in Table II.

Figure 1:  Estimation of energy consumption for 45nm CMOS technology.  TABLE II: Energy consumption estimations of our SNNs and their equivalent FNNs on the 3 datasets for 45nm CMOS technology using the metrics proposed in Section III-B. |  | CIFAR10 | GSC | NCARS
---|---|---|---|---
SNN | FNN | SNN | FNN | SNN | FNN
Memory Accesses (nJ) | Potentials | 1.12E+6 | – | 3.50E+3 | – | 2.69E+5 | –
Weights | 4.60E+5 | 5.59E+6 | 1.73E+3 | 1.46E+4 | 8.10E+4 | 1.11E+6
Bias | 2.79E+2 | 6.97E+1 | 6.00E+0 | 3.00E+0 | 3.71E+1 | 7.43E+0
In/Out | 7.65E+2 | 6.09E+6 | 3.90E+1 | 1.50E+4 | 2.88E+2 | 1.54E+6
Total | 1.58E+6 | 1.17E+7 | 5.27E+3 | 2.96E+4 | 3.50E+5 | 2.64E+6
Synaptic Op. (nJ) | 2.41E+3 | 1.29E+06 | 4.08E+1 | 3.53E+3 | 6.15E+3 | 2.67E+5
Addressing (nJ) | 1.97E+2 | 4.94E+1 | 9.28E+0 | 1.93E+0 | 5.05E+1 | 7.65E+1
Total (nJ) | 1.58E+6 | 1.24E+7 | 5.32E+3 | 3.32E+4 | 3.57E+5 | 2.86E+6
$E^{\text{FNN}}/E^{\text{SNN}}$ | 8.19 | 6.22 | 8.17

The total energy consumption is dominated by the cost of memory accesses,
which is yet unduly neglected in most metrics of the literature. While SNNs
have additional memory accesses for updating the neuron potentials, it also
requires fewer memory accesses for the weights and the I/O. Moreover, the size
of I/O buffers are often much smaller in SNNs than in FNNs, since the first
only requires FIFOs of 1000 elements whereas the second requires storing the
full feature maps. The lower number of spikes combined with the absence of
multiplication result in an energy consumption of synaptic operations two
order of magnitude lower for SNNs than for FNNs. In the end, the total energy
consumption of our SNNs is between 6.25 and 8.02 times lower than their FNN
counterparts, a promising result for the implementation of SNNs on specialized
hardware.

##  VI Conclusion

Neuromorphic Engineering is based on the assumption that event-based
processing is the key to mimic the unparalleled energy efficiency of the
biological brain. However, this assumption remains to be proven. The goal of
this work was to settle this question through a generic and accurate energy
estimation metric, independent from low-level implementation choices and
hardware targets. Our proposed analytical model is based on three types of
operations occurring in hardware neural network implementations: synaptic
operations, memory accesses and addressing mechanisms. This metric was applied
to three datasets representative of three characteristic data types. In all
three cases, spiking implementation could bring major energy savings compared
to formal ones as our SNNs are respectively 8.19x, 6.22x and 8.17x more
efficient on CIFAR-10, GSC and NCARS, while producing near state-of-the-art
accuracy for the last two.

Future works will include a confrontation of those results with actual energy
consumption measurements, using our own SNN hardware architecture (SPLEAT
[18]) and other state-of-the-art deep learning accelerators. Moreover, further
work is required on our CIFAR-10 model, in order to reach state-of-the-art
accuracy and thus increase the fairness of the comparison. Additionally, we
will study the impact of various quantization schemes on energy consumption.

## References

  * [1] L. Khacef, N. Abderrahmane, and B. Miramond, “Confronting machine-learning with neuroscience for neuromorphic architectures design,” in Int. Joint Conf. on Neural Netw., 2018.
  * [2] B. Rueckauer et al., “Nxtf: An api and compiler for deep spiking neural networks on intel loihi,” 2021.
  * [3] E. Lemaire, Modélisation et exploration d’architectures neuromorphiques pour les systèmes embarqués haute-performance.  PhD thesis, Univ. Côte d’Azur, 2022.
  * [4] E. Lemaire, B. Miramond, S. Bilavarn, H. Saoud, and N. Abderrahmane, “Synaptic activity and hardware footprint of spiking neural networks in digital neuromorphic systems,” ACM Trans. on Embedded Computing Syst., 2022.
  * [5] S. R. Kheradpisheh and T. Masquelier, “Temporal backpropagation for spiking neural networks with one spike per neuron,” Int. J. of Neural Syst., vol. 30, no. 06, p. 2050027, 2020.
  * [6] R. Zimmer, T. Pellegrini, S. F. Singh, and T. Masquelier, “Technical report: supervised training of convolutional spiking neural networks with pytorch.” 2019\.
  * [7] T. Pellegrini, R. Zimmer, and T. Masquelier, “Low-activity supervised convolutional spiking neural networks applied to speech commands recognition,” in 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 97–103, IEEE, 2021.
  * [8] A. Amir et al., “A low power, fully event-based gesture recognition system,” in IEEE Conf. on Comput. Vision and Pattern Recognition, pp. 7243–7252, 2017.
  * [9] J. Ding, Z. Yu, Y. Tian, and T. Huang, “Optimal ANN-SNN conversion for fast and accurate inference in deep spiking neural networks,” in Int. Joint Conf. on Artif. Intell., pp. 2328–2336, 2021.
  * [10] E. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks,” IEEE Signal Process. Magazine, vol. 36, pp. 51–63, 2019\.
  * [11] S. Barchid, J. Mennesson, J. Eshraghian, C. Djéraba, and M. Bennamoun, “Spiking neural networks for frame-based and event-based single object localization,” 2022.
  * [12] S. Kundu, G. Datta, M. Pedram, and P. A. Beerel, “Spike-thrift: Towards energy-efficient deep spiking neural networks by limiting spiking activity via attention-guided compression,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3953–3962, 2021.
  * [13] L. Deng et al., “Rethinking the performance comparison between SNNs and ANNs,” Neural Netw., vol. 121, pp. 294–307, 2020.
  * [14] S. Davidson and S. B. Furber, “Comparison of artificial and spiking neural networks on digital hardware,” Frontiers in Neuroscience, vol. 15, p. 651141, 2021.
  * [15] N. P. Jouppi et al., “Ten lessons from three generations shaped google’s tpuv4i: Industrial product,” in ACM/IEEE Annu. Int. Symp. on Comput. Architecture, pp. 1–14, 2021.
  * [16] P. Bardow, A. J. Davison, and S. Leutenegger, “Simultaneous optical flow and intensity estimation from an event camera,” in IEEE Conf. on Comput. Vision and Pattern Recognition, pp. 884–892, 2016.
  * [17] L. Cordone, B. Miramond, and P. Thierion, “Object detection with spiking neural networks on automotive event data,” in Int. Joint Conf. on Neural Networks, 2022.
  * [18] N. Abderrahmane, B. Miramond, E. Kervennic, and A. Girard, “Spleat: Spiking low-power event-based architecture for in-orbit processing of satellite imagery,” in Int. Joint Conf. on Neural Networks, 2022.
  * [19] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Int. Conf. on Learning Representations, 2015.
  * [20] A. Sironi, M. Brambilla, N. Bourdis, X. Lagorce, and R. Benosman, “Hats: Histograms of averaged time surfaces for robust event-based object classification,” in IEEE Conf. on Comput. Vision and Pattern Recognition, June 2018.
  * [21] W. Fang, Y. Chen, J. Ding, D. Chen, Z. Yu, H. Zhou, Y. Tian, and other contributors, “Spikingjelly.” https://github.com/fangwei123456/spikingjelly, 2020.  Accessed: 2022-07-29.