# Fully Spiking Variational Autoencoder

Hiromichi Kamata 1, Yusuke Mukuta 1,2, Tatsuya Harada 1,2

###### Abstract

Spiking neural networks (SNNs) can be run on neuromorphic devices with ultra-
high speed and ultra-low energy consumption because of their binary and event-
driven nature. Therefore, SNNs are expected to have various applications,
including as generative models being running on edge devices to create high-
quality images. In this study, we build a variational autoencoder (VAE) with
SNN to enable image generation. VAE is known for its stability among
generative models; recently, its quality advanced. In vanilla VAE, the latent
space is represented as a normal distribution, and floating-point calculations
are required in sampling. However, this is not possible in SNNs because all
features must be binary time series data. Therefore, we constructed the latent
space with an autoregressive SNN model, and randomly selected samples from its
output to sample the latent variables. This allows the latent variables to
follow the Bernoulli process and allows variational learning. Thus, we build
the Fully Spiking Variational Autoencoder where all modules are constructed
with SNN. To the best of our knowledge, we are the first to build a VAE only
with SNN layers. We experimented with several datasets, and confirmed that it
can generate images with the same or better quality compared to conventional
ANNs. The code is available at https://github.com/kamata1729/FullySpikingVAE.

## Introduction

Recently, artificial neural networks (ANNs) have been evolving rapidly, and
have achieved considerable success in computer vision and NLP. However, ANNs
often require significant computational resources, which is a challenge in
situations where computational resources are limited, such as on edge devices.

Spiking neural networks (SNNs) are neural networks that more accurately mimic
the structure of a biological brain than ANNs; notably, SNNs are referred to
as the third generation of artificial intelligence (Maass 1997). In a SNN, all
information is represented as binary time series data, and is driven by event-
based processing. Therefore, SNNs can run with ultra-high speed and ultra-low
energy consumption on neuromorphic devices, such as Loihi (Davies et al.
2018), TrueNorth (Akopyan et al. 2015), and Neurogrid (Benjamin et al. 2014).
For example, on TrueNorth, the computational time is approximately 1/100 lower
and the energy consumption is approximately 1/100,000 times lower than on
conventional ANNs (Cassidy et al. 2014).

With the recent breakthroughs on ANNs, research on SNNs has been progressing
rapidly. Additionally, SNNs are outperforming ANNs in accuracy in MNIST,
CIFAR10, and ImageNet classification tasks (Zheng et al. 2021; Zhang and Li
2020). Moreover, SNNs are used for object detection (Kim et al. 2020), sound
classification (Wu et al. 2018), optical flow estimation (Lee et al. 2020);
however, their applications are still limited.

Figure 1: Illustration of our FSVAE. Entire model is constructed with SNNs.
All features are represented as spike trains, and the latent spike trains
follow Bernoulli processes.

In particular, image generation models based on SNNs have not been studied
sufficiently. Spiking GAN (Kotariya and Ganguly 2021) built a generator and
discriminator with shallow SNNs, and generated images of handwritten digits by
adversarial learning. However, its generation quality was low, and some
undesired images were generated that could not be interpreted as numbers. In
(Skatchkovsky, Simeone, and Jang 2021), SNN was used as the encoder and ANN as
the decoder to build a VAE (Kingma and Welling 2014); however, the main focus
of their research was efficient spike encoding, and not the image generation
task.

In ANNs, image generation models have been extensively studied and can
generate high-quality images (Razavi, van den Oord, and Vinyals 2019; Karras
et al. 2020). However, in general, image generation models are computationally
expensive, and some problems must be solved for edge devices, or for real-time
generation. If SNNs can generate images comparably to ANNs, their high speed
and low energy consumption can solve these problems.

Therefore, we propose Fully Spiking Variational Autoencoder (FSVAE), which can
generate images with the same or better quality than ANN. VAEs are known for
their stability among generative models and are related to the learning
mechanism of the biological brain (Han et al. 2018). Hence, it is compatible
to build VAE with SNN. In our FSVAE, we built the entire model in SNN, so that
it can be implemented in a neuromorphic device in the future. We conducted
experiments using MNIST (Deng 2012), FashionMNIST (Xiao, Rasul, and Vollgraf
2017), CIFAR10 (Krizhevsky and Hinton 2009), and CelebA (Liu et al. 2015), and
confirmed that FSVAE can generate images of equal or better quality than ANN
VAE of the same structure. FSVAE can be implemented on neuromorphic devices in
the future, and is expected to improve in terms of speed and energy
consumption.

The most difficult aspect of creating VAEs in SNNs is how to create the latent
space. In ANN VAEs, the latent space is often represented as a normal
distribution. However, within the framework of SNNs, sampling from a normal
distribution is not possible because all features must be binary time series
data. Therefore, we propose the autoregressive Bernoulli spike sampling.
First, we incorporated the idea of VRNN (Chung et al. 2015) into SNN, and
built prior and posterior models with autoregressive SNNs. The latent
variables are randomly selected from the output of the autoregressive SNNs,
which enables sampling from the Bernoulli processes. This can be realized on
neuromorphic devices because it does not require floating-point calculations
during sampling as in ANNs, and sampling using a random number generator is
possible on actual neuromorphic devices (Wen et al. 2016; Davies et al. 2018).
In addition, the latent variables can be sampled sequentially; thus, they can
be input to the decoder incrementally, which saves time.

The main contributions of this study are summarized as follows.

  * •

We propose the autoregressive Bernoulli spike sampling, which uses
autoregressive SNNs and constructs the latent space as Bernoulli processes.
This sampling method is feasible within the framework of SNN.

  * •

We propose Fully Spiking Variational Autoencoder (FSVAE), where all modules
are constructed in the SNN.

  * •

We experimented with multiple datasets; FSVAE could generate images of equal
or better quality than ANN VAE of the same architecture.

## Related Work

### Development of SNNs

SNNs are neural networks that accurately mimic the structure of the biological
brain. In the biological brain, information is transmitted as spike trains
(binary time series data with only on/off). This information is transmitted
between neurons via synapses, and subsequently, the neuron’s membrane
potential changes. When it exceeds a threshold, it fires and becomes a spike
train to the next neuron.

SNNs mimic these characteristics of the biological brain, modeling biological
neurons using differential equations and representing all features as spike
trains. This allows SNNs to run faster and asynchronously, because they
require fewer floating-point computations and only need computations when the
input spike arrives. SNNs can be considered as recurrent neural networks
(RNNs) with the membrane potential as its internal state.

Learning algorithms for SNNs have been studied extensively recently. (Diehl
and Cook 2015) used a two-layer SNN to recognize MNIST with STDP, an
unsupervised learning rule, and achieved 95% accuracy. Later, (Wu et al. 2019)
made it possible to train a deep SNN with backpropagation. Recently, (Zhang
and Li 2020) exceeded the ANN’s accuracy for MNIST and CIFAR10 (Krizhevsky and
Hinton 2009) in only 5 timesteps (length of spike trains). (Zheng et al. 2021)
has even higher accuracy in 2 timesteps for CIFAR10 and ImageNet (Deng et al.
2009).

### Spike Neuron Model

Figure 2: LIF neuron. When spike trains of the previous layer neurons enter,
the internal membrane potential $u_{t}$ changes by Eq. (2). If $u_{t}$ exceeds
$V_{\mathrm{th}}$, it fires a spike $o_{t}=1$; otherwise, $o_{t}=0$.

Although there are several learning algorithms for SNNs, in this study, we
follow (Zheng et al. 2021), which currently has the highest recognition
accuracy. First, as a neuron model, we use the iterative leaky integrate-and-
fire (LIF) model (Wu et al. 2019), which is a LIF model (Stein and Hodgkin
1967) solved using the Euler method.

$\displaystyle u_{t}=\tau_{\mathrm{decay}}u_{t-1}+x_{t}$ (1)

where $u_{t}$ is a membrane potential, $x_{t}$ is a presynaptic input, and
$\tau_{\mathrm{decay}}$ is a fixed decay factor.

When $u_{t}$ exceeds a certain threshold $V_{\mathrm{th}}$, the neuron fires
and outputs $o_{t}=1$. Then, $u_{t}$ is reset to $u_{rest}=0$. This can be
written as follows:

$\displaystyle u_{t,n}$
$\displaystyle=\tau_{\mathrm{decay}}u_{t-1,n}(1-o_{t-1,n})+x_{t,n-1}$ (2)
$\displaystyle o_{t,n}$ $\displaystyle=H(u_{t,n}-V_{\mathrm{th}})$ (3)

Here, $u_{t,n}$ is the membrane potential of the $n$th layer, and $o_{t,n}$ is
its binary output. $H$ is the heaviside step function. Input $x_{t,n}$ is
described as a weighted sum of spikes from neurons in the previous layer,
$x_{t,n-1}=\sum_{j}w^{j}o_{t,n-1}^{j}$. By changing the connection way of
$w^{j}$ , we can implement convolution layers, FC layers, etc.

The next step is to enable learning with backpropagation. As Eq. (3) is non-
differentiable, we approximate it as follows:

$\displaystyle\frac{\partial o_{t,n}}{\partial
u_{t,n}}=\frac{1}{a}\mathrm{sign}\left(|u_{t,n}-V_{\mathrm{th}}|<\frac{a}{2}\right)$
(4)

Figure 3: Overview of FSVAE. (a) During the training, the input image $x$ is
spike encoded to $\bm{x}_{1:T}$, which is passed through the SNN encoder to
obtain $\bm{x}_{1:T}^{E}$. In addition to $\bm{x}_{t}^{E}$, posterior takes
the previously generated latent variables $\bm{z}_{t-1}$ as input, and
sequentially outputs $\bm{z}_{t}$. The lower right figure shows this process
in detail. Here, $f_{q}$ is the SNN model described in Figure 4. In prior,
only $\bm{z}_{t-1}$ is used to generate $\bm{z}_{t}$. Next, $\bm{z}_{t}$ is
sequentially input to the SNN decoder, which outputs $\hat{\bm{x}}_{1:T}$ and
decodes it to obtain the reconstructed image $\hat{x}$. For the loss, we take
the reconstruction error of $x$ and $\hat{x}$ and the MMD of the posterior and
prior. (b) During the sampling, the image is generated from $\bm{z}_{1:T}$
sampled in prior.

### Variational Autoencoder

Variational Autoencoder (VAE) (Kingma and Welling 2014) is a generative model
that explicitly assumes a distribution of the latent variable $z$ over input
$x$. Typically, the distribution $p(x|z)$ is represented by deep neural
networks, so its inverse transformation is approximated by a simple
approximate posterior $q(z|x)$. This allows us to calculate the evidence lower
bound (ELBO) of the log likelihood.

$\displaystyle\log p(x)\geq\mathbb{E}_{q(z|x)}[\log
p(x|z)]-\mathrm{KL}[q(z|x)||p(z)]$ (5)

where $\mathrm{KL}[Q||P]$ is the Kullback–Leibler (KL) divergence for
distributions Q and P. In $q(z|x)$, reparameterization trick is used to sample
from $\mathcal{N}(\mu(x),\mathrm{diag}(\sigma(x)^{2}))$.

VAEs have stable learning among generative models, and can be applied to
various tasks, such as anomaly detection (An and Cho 2015). As VAE could
generate high-quality images (Razavi, van den Oord, and Vinyals 2019; Vahdat
and Kautz 2020), we aimed to build a VAE using SNN.

### Variational Reccurent Neural Network

Variational Reccurent Neural Network (VRNN) (Chung et al. 2015) is a VAE for
time series data. Its posterior and prior distributions are set as follows:

$\displaystyle q(\bm{z}_{1:T}|\bm{x}_{1:T})$
$\displaystyle=\prod_{t=1}^{T}q(\bm{z}_{t}|\bm{x}_{\leq t},\bm{z}_{<t})$ (6)
$\displaystyle p(\bm{z}_{1:T}|\bm{x}_{1:T-1})$
$\displaystyle=\prod_{t=1}^{T}p(\bm{z}_{t}|\bm{x}_{<t},\bm{z}_{<t})$ (7)

Here, $q(\bm{z}_{t}|\bm{x}_{\leq t},\bm{z}_{<t})$ and
$p(\bm{z}_{t}|\bm{x}_{<t}\bm{z}_{<t})$ are defined using LSTM (Hochreiter and
Schmidhuber 1997). When sampling, prior inputs $z_{t-1}$ to the decoder to
reconstruct $x_{t-1}$, which is used to generate $z_{t}$ repeatedly.

As SNN is a type of RNNs, we use VRNN to build FSVAE.

### Generative models in SNN

Spiking GAN (Kotariya and Ganguly 2021) uses two-layer SNNs to construct a
generator and discriminator to train a GAN; however, the quality of the
generated image is low. One reason for this is that the time-to-first spike
encoding cannot grasp the entire image in the middle of spike trains. In
addition, as the learning of SNN is unstable, it would be difficult to perform
adversarial learning without regularization.

### Applying VAE on SNN

Some studies partially used SNN to create VAEs. In (Stewart et al. 2021),
human gesture videos captured with a DVS camera were input to an SNN encoder,
and the latent variables were generated from the membrane potential of the
output neuron; the ANN decoder reconstructed the input from that. Their
research aimed to generate pseudo labels for new gestures, not generate
images. Moreover, as the decoder was built with ANN, the entire model could
not be implemented on neuromorphic devices. Similarly, (Skatchkovsky, Simeone,
and Jang 2021) had the same problem because their decoder is also built with
ANN.

In SVAE (Talafha et al. 2020), after training a VAE built on ANN, they
converted it to SNN to perform unsupervised learning using STDP, and then,
converted it again to ANN to improve the quality of the generated images.
Thus, image generation with SNN was not studied.

Some studies used probabilistic neurons to perform variational learning
(Rezende, Wierstra, and Gerstner 2011; Bagheri 2019). However, as
probabilistic neuron uses the sampling from a Poisson process, it requires
additional sampling modules. Therefore, we created an entire image generation
model in deterministic SNN, which is more commonly used.

### Autoregressive Model for Spike Train Modeling

When constructing VAEs with SNNs, prior and posterior distributions are
required to generate spike trains based on a stochastic process. In a related
study, MMD-GLM (Arribas, Zhao, and Park 2020) modeled a real biological spike
train as a Poisson process using an autoregressive ANN. The Mazimum Mean
Discrepancy (MMD) was used to measure the consistency with actual spike
trains. This is because using KL divergence may cause runaway self-excitation.

In this study, we propose a method for modeling spike trains using an
autoregressive SNN in prior and posterior distributions. The details are
described in the next section.

## Proposed Method

### Overview of FSVAE

The detailed network architecture can be found in Supplementary Material A.

Figure 3 shows the model overview. The input image $x$ is transformed into
spike trains $\bm{x}_{1:T}$ using direct input encoding (Rueckauer et al.
2017), and is subsequently input to the SNN encoder. From the output spike
trains of Encoder $\bm{x}^{E}_{1:T}$, the posterior outputs the latent spike
trains $\bm{z}_{1:T}$ incrementally. Then, the SNN decoder generates output
spike trains $\hat{\bm{x}}_{1:T}$, which is decoded to obtain the
reconstructed image $\hat{x}$. When sampling, prior generates $\bm{z}_{1:T}$
incrementally and inputs it into the SNN decoder to generate image $\hat{x}$.

### Autoregressive Bernoulli Spike Sampling

We define the posterior and prior probability distributions of posterior and
prior as follows:

$\displaystyle q(\bm{z}_{1:T}|\bm{x}_{1:T})$
$\displaystyle=\prod_{t=1}^{T}q(\bm{z}_{t}|\bm{x}_{\leq t},\bm{z}_{<t})$ (8)
$\displaystyle p(\bm{z}_{1:T})$
$\displaystyle=\prod_{t=1}^{T}p(\bm{z}_{t}|\bm{z}_{<t})$ (9)

We need to model $q(\bm{z}_{t}|\bm{x}_{\leq t},\bm{z}_{<t})$ and
$p(\bm{z}_{t}|\bm{z}_{<t})$ with SNNs. Notably, all SNN features must be
binary time series data. Therefore, we cannot use the reparameterization trick
to sample from the normal distribution as in the conventional VAE.

Consequently, we define $q(\bm{z}_{t}|\bm{x}_{\leq t},\bm{z}_{<t})$ and
$p(\bm{z}_{t}|\bm{z}_{<t})$ as Bernoulli distributions which take binary
values. First, we need to generate $\bm{z}_{t}$ sequentially from
$\bm{z}_{<t}$; thus, we use the autoregressive SNN model. By randomly
selecting one by one from its output, we can sample from a Bernoulli
distribution. The overview of this sampling method is shown in Figure 4.

Figure 4: Autoregressive Bernoulli spike sampling for prior and posterior
distributions. The input is $\bm{z}_{t-1}$ for prior, and
$(\bm{z}_{t-1},\bm{x}^{E}_{t})$ for posterior. We sample from the Bernoulli
distribution by randomly selecting $C$ channels from $kC$ channels of the
output.

When the autoregressive SNN model of $q(\bm{z}_{t}|\bm{x}_{\leq
t},\bm{z}_{<t})$ is $f_{q}$ and that of $p(\bm{z}_{t}|\bm{z}_{<t})$ is
$f_{p}$, their outputs are below.

$\displaystyle\bm{\zeta}_{q,t}\coloneqq$ $\displaystyle
f_{q}(\bm{z}_{q,t-1},\bm{x}_{t}^{E};\Theta_{q,t})\in\\{0,1\\}^{kC}$ (10)
$\displaystyle\bm{\zeta}_{p,t}\coloneqq$ $\displaystyle
f_{p}(\bm{z}_{p,t-1};\Theta_{p,t})\in\\{0,1\\}^{kC}$ (11)

where $C$ is the dimension of $\bm{z}_{t}$, $k$ is a natural number starting
from 2. $\bm{x}_{t}^{E}$ is the output of the encoder. $\Theta_{q,t}$ and
$\Theta_{p,t}$ are the sets of membrane potentials of the neurons in $f_{q}$
and $f_{p}$, which are updated by the input.

Sampling is performed by randomly selecting one for every $k$ value:

$\displaystyle\bm{z}_{q,t,c}$
$\displaystyle=\mathrm{random\\_select}(\bm{\zeta}_{q,t}[k(c-1):kc])$ (12)
$\displaystyle\bm{z}_{p,t,c}$
$\displaystyle=\mathrm{random\\_select}(\bm{\zeta}_{p,t}[k(c-1):kc])$ (13)

By doing this in $1\leq c\leq C$, we can sample
$\bm{z}_{q,t},\bm{z}_{p,t}\in\\{0,1\\}^{C}$.

In TrueNorth, one of the neuromorphic devices, there is a pseudo random number
generator mechanism that can replace the synaptic weight randomly (Wen et al.
2016), which makes this sampling method feasible.

This is equivalent to sampling from the following Bernoulli distribution.

$\displaystyle\bm{z}_{q,t}|\bm{x}_{t},\bm{z}_{q,t-1}\sim Ber(\bm{\pi}_{q,t})$
(14) $\displaystyle\bm{z}_{p,t}|\bm{z}_{p,t-1}\sim Ber(\bm{\pi}_{p,t})$ (15)
$\displaystyle where$
$\displaystyle\left\\{\begin{array}[]{ll}\bm{\pi}_{q,t,c}=\mathrm{mean}(\bm{\zeta}_{q,t}[k(c-1):kc])\\\
\bm{\pi}_{p,t,c}=\mathrm{mean}(\bm{\zeta}_{p,t}[k(c-1):kc])\end{array}\right.$
(18)

Therefore,

$\displaystyle q(\bm{z}_{t}|\bm{x}_{\leq t},\bm{z}_{<t})=Ber(\bm{\pi}_{q,t})$
(19) $\displaystyle p(\bm{z}_{t}|\bm{z}_{<t})=Ber(\bm{\pi}_{p,t})$ (20)

### Spike to Image Decoding using Membrane Potential

Sampled $\bm{z}_{t}$ is sequentially input to the SNN decoder, which outputs
the spike trains $\hat{\bm{x}}_{1:T}\in\\{0,1\\}^{C_{\mathrm{out}}\times
H\times W\times T}$. We need to decode this into the reconstructed image
$\hat{x}\in\mathbb{R}^{C_{\mathrm{out}}\times H\times W}$. To utilize the
framework of SNN, we use the membrane potential of the output neuron for
spike-to-image decoding. We use non-firing neurons in the output layer, and
convert $\hat{\bm{x}}_{1:T}$ into a real value by measuring its membrane
potential at the last time $T$. It is given by Eq. (2) as follows:

$\displaystyle u_{T}^{\mathrm{out}}$
$\displaystyle=\tau_{\mathrm{out}}u_{T-1}^{\mathrm{out}}+\hat{x}_{T}$
$\displaystyle=\tau_{\mathrm{out}}^{2}u_{T-2}^{\mathrm{out}}+\tau_{\mathrm{out}}\hat{x}_{T-1}+\hat{x}_{T}=\sum_{t=1}^{T}\tau_{\mathrm{out}}^{T-t}\hat{x}_{t}$
(21)

We set $\tau_{\mathrm{out}}=0.8$, and obtain the real-valued reconstructed
image as $\hat{x}=\tanh(u_{T}^{\mathrm{out}})$.

### Loss Function

The ELBO is as follows:

$\displaystyle ELBO=$
$\displaystyle\mathbb{E}_{q(\bm{z}_{1:T}|\bm{x}_{1:T})}[\log
p(\bm{x}_{1:T}|\bm{z}_{1:T})]$
$\displaystyle-\mathrm{KL}[q(\bm{z}_{1:T}|\bm{x}_{1:T})||p(\bm{z}_{1:T})]$
(22)

The first term is the reconstruction loss, which is $\mathrm{MSE}(x,\hat{x})$
as in the usual VAE. The second term, KL divergence, represents the closeness
of the prior and posterior probability distributions. Traditional VAEs use KL
divergence, but we use MMD, which has been shown to be a more suitable
distance metric for spike trains in MMD-GLM (Arribas, Zhao, and Park 2020).
MMD can be written using the kernel function $k$ as follows:

$\displaystyle\mathrm{MMD}^{2}[q(\bm{z}_{1:T}|\bm{x}_{1:T}),p(\bm{z}_{1:T})]$
$\displaystyle=$ $\displaystyle\underset{\bm{z},\bm{z}^{\prime}\sim
q}{\mathbb{E}}[k(\bm{z}_{1:T},\bm{z}^{\prime}_{1:T})]+\underset{\bm{z},\bm{z}^{\prime}\sim
p}{\mathbb{E}}[k(\bm{z}_{1:T},\bm{z}^{\prime}_{1:T})]$
$\displaystyle-2\underset{\bm{z}\sim q,\bm{z}^{\prime}\sim
p}{\mathbb{E}}[k(\bm{z}_{1:T},\bm{z}^{\prime}_{1:T})]$ (23)

We set
$k(\bm{z}_{1:T},\bm{z}^{\prime}_{1:T})=\sum_{t}\mathrm{PSP}(\bm{z}_{\leq
t})\mathrm{PSP}(\bm{z}^{\prime}_{\leq t})$, as the model based kernel in MMD-
GLM. The $\mathrm{PSP}$ stands for a postsynaptic potential function that can
capture the time-series nature of spike trains (Zenke and Ganguli 2018). We
use the first-order synaptic model (Zhang and Li 2020) as the $\mathrm{PSP}$.
The following update formula is used to calculate the
$\mathrm{PSP}(\bm{z}_{\leq t})$.

$\displaystyle\mathrm{PSP}(\bm{z}_{\leq
t})=\left(1-\frac{1}{\tau_{\mathrm{syn}}}\right)\mathrm{PSP}(\bm{z}_{\leq
t-1})+\frac{1}{\tau_{\mathrm{syn}}}\bm{z}_{t}$ (24)

where $\tau_{\mathrm{syn}}$ is the synaptic time constant. We set
$\mathrm{PSP}(\bm{z}_{\leq 0})=0$.

This gives us Eq. (23), as follows. The detailed derivation can be found in
Supplementary Material B.

$\displaystyle\mathrm{MMD}^{2}[q(\bm{z}_{1:T}|\bm{x}_{1:T}),p(\bm{z}_{1:T})]$
$\displaystyle=$
$\displaystyle\sum_{t=1}^{T}\|\mathrm{PSP}(\underset{\bm{z}\sim
q}{\mathbb{E}}[\bm{z}_{\leq t}])-\mathrm{PSP}(\underset{\bm{z}\sim
p}{\mathbb{E}}[\bm{z}_{\leq t}])\|^{2}$ (25) $\displaystyle=$
$\displaystyle\sum_{t=1}^{T}\|\mathrm{PSP}(\bm{\pi}_{q,\leq
t})-\mathrm{PSP}(\bm{\pi}_{p,\leq t})\|^{2}$ (26)

The loss function is calculated as follows:

$\displaystyle\mathcal{L}=\mathrm{MSE}(x,\hat{x})+\sum_{t=1}^{T}\|\mathrm{PSP}(\bm{\pi}_{q,\leq
t})-\mathrm{PSP}(\bm{\pi}_{p,\leq t})\|^{2}$ (27)

Dataset | Model |  Reconstruction Loss$\searrow$ |  Inception Score$\nearrow$ | Fréchet Distance$\searrow$
---|---|---|---|---
Inception (FID) | Autoencoder
MNIST | ANN | 0.048 | 5.947 | 112.5 | 17.09
FSVAE (Ours) | 0.031 | 6.209 | 97.06 | 35.54
Fashion MNIST | ANN | 0.050 | 4.252 | 123.7 | 18.08
FSVAE (Ours) | 0.031 | 4.551 | 90.12 | 15.75
CIFAR10 | ANN | 0.105 | 2.591 | 229.6 | 196.9
FSVAE (Ours) | 0.066 | 2.945 | 175.5 | 133.9
CelebA | ANN | 0.059 | 3.231 | 92.53 | 156.9
FSVAE (Ours) | 0.051 | 3.697 | 101.6 | 112.9
Table 1: Results for each dataset. In all datasets, our model outperforms ANN
in the inception score. Moreover, our model outperforms MNIST and Fashion
MNIST in FID, CIFAR10 in all metrics, and CelebA in Autoencoder’s Fréchet
distance. Reconstruction losses are better for our model in all datasets.

## Experiments

We implemented FSVAE in PyTorch (Paszke et al. 2019), and evaluated it using
MNIST, Fashion MNIST, CIFAR10, and CelebA. The results are summarized in Table
1.

### Datasets

For MNIST and Fashion MNIST, we used 60,000 images for training and 10,000
images for evaluation. The input images were resized to 32$\times$32\. For
CIFAR10, we used 50,000 images for training and 10,000 images for evaluation.
For CelebA, we used 162,770 images for training and 19,962 images for
evaluation. The input images were resized to 64$\times$64.

### Network Architecture

The SNN encoder comprises several convolutional layers, each with
kernel_size=3 and stride=2. The number of layers is 4 for MNIST, Fashion MNIST
and CIFAR10, and 5 for CelebA. After each layer, we set a tdBN (Zheng et al.
2021), and then, input the feature to the LIF neuron to obtain the output
spike trains. The encoder’s output is $\bm{x}_{t}^{E}\in\\{0,1\\}^{C}$ with
latent dimension $C=128$. We combine it with $\bm{z}_{t-1}\in\\{0,1\\}^{C}$,
and input it to the posterior model; thus, its input dimension is
$128+128=256$. Additionally, we set $\bm{z}_{0}=\bm{0}$. The posterior model
comprises three FC layers, and increases the number of channels by a factor of
$k=20$. $\bm{z}_{t}$ is sampled from it by the proposed autoregressive
Bernoulli spike sampling. We repeat this process $T=16$ times. The prior model
has the same model architecture; however, as the input is only $\bm{z}_{t-1}$,
its input dimension is 128.

Sampled $\bm{z}_{t}$ is input to the SNN decoder, which contains the same
number of deconvolution layers as the encoder. Decoder’s output is spike
trains of the same size as the input. Finally, we perform spike-to-image
decoding to obtain the reconstructed image, according to Eq. (21).

### Training Settings

We use AdamW optimizer (Loshchilov and Hutter 2019), which trains 150 epochs
with a learning rate of 0.001 and a weight decay of 0.001. The batch size is
250. In prior model, teacher forcing (Williams and Zipser 1989) is used to
stabilize training, so that the prior’s input is $\bm{z}_{q,t}$, which is
sampled from the posterior model. In addition, to prevent posterior collapse,
scheduled sampling (Bengio et al. 2015) is performed. With a certain
probability, we input $\bm{z}_{p,t}$ to the prior instead of $\bm{z}_{q,t}$.
This probability varies linearly from 0 to 0.3 during training.

Figure 5: Generated images of ANN VAE and our FSVAE (SNN).

### Evaluation Metrics

The quality of the sampled images is measured by the inception score (Salimans
et al. 2016) and FID (Heusel et al. 2017; Parmar, Zhang, and Zhu 2021).
However, as FID considers the output of the ImageNet pretrained inception
model, it may not work well on datasets such as MNIST, which have a different
domain from ImageNet. Consequently, we trained an autoencoder on each dataset
beforehand, and used it to measure the Fréchet distance of the autoencoder’s
latent variables between sampled and real images. We sampled 5,000 images to
measure the distance. As a comparison method, we prepared vanilla VAEs of the
same network architecture built with ANN, and trained on the same settings.

Table 1 shows that our FSVAE outperforms ANN in the inception score for all
datasets. For MNIST and Fashion MNIST, our model outperforms in terms of FID,
for CIFAR10 in all metrics, and for CelebA in the Fréchet distance of the
pretrained autoencoder. As SNNs can only use binary values, it is difficult to
perform complex tasks; in contrast, our FSVAE achieves equal or higher scores
than ANNs in image generation.

#### Computational Cost

We measured how many floating-point additions and multiplications were
required for inferring a single image. We summarized the results in Table 2.
The number of additions is 6.8 times less for ANN, but the number of
multiplications is 14.8 times less for SNN. In general, multiplication is more
expensive than addition, so fewer multiplications are preferable. Moreover, as
SNNs are expected to be about 100 times faster than ANNs when implemented in
neuromorphic device, FSVAE can significantly outperform ANN VAE in terms of
speed.

Model | Computational complexity
---|---
Addition | Multiplication
ANN | $\bm{7.4\times 10^{9}}$ | $7.4\times 10^{9}$
FSVAE (Ours) | $5.0\times 10^{10}$ | $\bm{5.6\times 10^{8}}$
Table 2: Comparison of the amount of computation required to infer a single
image in MNIST.

Figure 6 shows the change in FID depending on the timestep and $k$, the number
of output choices. The best FID was obtained when the timestep was 16. A small
number of timesteps does not have enough expressive power, whereas a large
number of timesteps makes the latent space too large; thus, the best timestep
was determined to be 16. Moreover, FID does not change so much with $k$, but
it is best when $k=20$.

Figure 6: Left: relationship between the timestep and FID score of generated
images in MNIST. Right: relationship between $k$ (multiplier for the channel)
and FID in MNIST.

#### Ablation Study

The results are summarized in Table 3. When calculating KLD, $\epsilon=0.01$
was added to $\bm{\pi}_{q,t}$ and $\bm{\pi}_{p,t}$ to avoid divergence. The
best FID score was obtained using MMD as the loss function and applying
$\mathrm{PSP}$ to its kernel.

KLD | MMD | apply $\mathrm{PSP}$ | FID$\searrow$
---|---|---|---
✓ |  |  | 114.3
| ✓ |  | 106.0
| ✓ | ✓ | 101.6
Table 3: Ablation study on CelebA. KLD and MMD indicate the distance metric
used in the loss function. We also measured whether to apply $\mathrm{PSP}$ to
the MMD kernel.

### Qualitative Evaluation

Figure 5 shows examples of the generated images. In CIFAR10 and Fashion MNIST,
the ANN VAE generated blurry and hazy images, whereas our FSVAE generated
clearer images. In CelebA, FSVAE generated images with more distinct
background areas than ANN. This is because the latent variables of FSVAE are
discrete spike trains, thus it can avoid posterior collapse, as VQ-VAE (van
den Oord, Vinyals, and Kavukcuoglu 2017).

Figure 7 shows the reconstructed images of Fashion MNIST. FSVAE reconstruct
images more clearly than ANN, especially the details. This is because that the
latent variable of FSVAE is discrete. Posterior collapse is caused by the
latent variables being ignored by the decoder. With discrete latent variables,
every latent variable becomes meaningful, which can prevent posterior
collapse.

Figure 7: Reconstructed images of Fashion MNIST.

## Conclusion

In this study, we proposed FSVAE, which allows image generation on SNNs with
equal or higher quality than ANNs. We proposed autoregressive Bernoulli spike
sampling, which is a spike sampling strategy that can be implemented on
neuromorphic devices. This sampling method is used in prior and posterior
distribution, and we model the latent spike trains as a Bernoulli process.
Experiments on multiple datasets show that FSVAE can generate images with
equal or higher quality than ANN VAE of the same architecture. As SNNs can be
significantly faster on neuromorphic devices, FSVAE can significantly
outperform ANN VAE in terms of speed. In the future, when we incorporate the
recent VAE researches, we will soon be able to achieve high-resolution image
generation with SNNs.

## Acknowledgments

This work was partially supported by JST AIP Acceleration Research JPMJCR20U3,
Moonshot R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR2015, and Basic
Research Grant (Super AI) of Institute for AI and Beyond of the University of
Tokyo. We would like to thank Yang Li and Muyuan Xu for some helpful advice of
SNNs. We also thank Tomoyuki Takahata, Yusuke Mori, and Atsuhiro Noguchi for
helpful discussions.

## References

  * Akopyan et al. (2015) Akopyan, F.; Sawada, J.; Cassidy, A.; Alvarez-Icaza, R.; Arthur, J. V.; Merolla, P.; Imam, N.; Nakamura, Y. Y.; Datta, P.; Nam, G.; Taba, B.; Beakes, M. P.; Brezzo, B.; Kuang, J. B.; Manohar, R.; Risk, W. P.; Jackson, B. L.; and Modha, D. S. 2015.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 34(10): 1537–1557.
  * An and Cho (2015) An, J.; and Cho, S. 2015.  Variational Autoencoder based Anomaly Detection using Reconstruction Probability.  _Special Lecture on IE_ , 2(1): 1–18.
  * Arribas, Zhao, and Park (2020) Arribas, D. M.; Zhao, Y.; and Park, I. M. 2020.  Rescuing neural spike train models from bad MLE.  In _NeurIPS_.
  * Bagheri (2019) Bagheri, A. 2019.  _Probabilistic Spiking Neural Networks : Supervised, Unsupervised and Adversarial Trainings_.  Ph.D. thesis, New Jersey Institute of Technology.
  * Bengio et al. (2015) Bengio, S.; Vinyals, O.; Jaitly, N.; and Shazeer, N. 2015.  Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.  In _NeurIPS_.
  * Benjamin et al. (2014) Benjamin, B. V.; Gao, P.; McQuinn, E.; Choudhary, S.; Chandrasekaran, A.; Bussat, J.; Alvarez-Icaza, R.; Arthur, J. V.; Merolla, P.; and Boahen, K. 2014\.  Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural Simulations.  _Proceedings of the IEEE_ , 102(5): 699–716.
  * Cassidy et al. (2014) Cassidy, A. S.; Alvarez-Icaza, R.; Akopyan, F.; Sawada, J.; Arthur, J. V.; Merolla, P.; Datta, P.; Tallada, M. G.; Taba, B.; Andreopoulos, A.; Amir, A.; Esser, S. K.; Kusnitz, J.; Appuswamy, R.; Haymes, C.; Brezzo, B.; Moussalli, R.; Bellofatto, R.; Baks, C. W.; Mastro, M.; Schleupen, K.; Cox, C. E.; Inoue, K.; Millman, S. E.; Imam, N.; McQuinn, E.; Nakamura, Y. Y.; Vo, I.; Guok, C.; Nguyen, D.; Lekuch, S.; Asaad, S. W.; Friedman, D. J.; Jackson, B. L.; Flickner, M.; Risk, W. P.; Manohar, R.; and Modha, D. S. 2014.  Real-Time Scalable Cortical Computing at 46 Giga-Synaptic OPS/Watt with ~100× Speedup in Time-to-Solution and ~100, 000× Reduction in Energy-to-Solution.  In _Supercomputing Conference_.
  * Chung et al. (2015) Chung, J.; Kastner, K.; Dinh, L.; Goel, K.; Courville, A. C.; and Bengio, Y. 2015\.  A Recurrent Latent Variable Model for Sequential Data.  In _NeurIPS_.
  * Davies et al. (2018) Davies, M.; Srinivasa, N.; Lin, T.; Chinya, G. N.; Cao, Y.; Choday, S. H.; Dimou, G. D.; Joshi, P.; Imam, N.; Jain, S.; Liao, Y.; Lin, C.; Lines, A.; Liu, R.; Mathaikutty, D.; McCoy, S.; Paul, A.; Tse, J.; Venkataramanan, G.; Weng, Y.; Wild, A.; Yang, Y.; and Wang, H. 2018.  Loihi: A Neuromorphic Manycore Processor with On-Chip Learning.  _IEEE Micro_ , 38(1): 82–99.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.; Li, K.; and Li, F. 2009.  ImageNet: A Large-Scale Hierarchical Image Database.  In _CVPR_.
  * Deng (2012) Deng, L. 2012.  The MNIST Database of Handwritten Digit Images for Machine Learning Research.  _IEEE Signal Processing Magazine_ , 29(6): 141–142.
  * Diehl and Cook (2015) Diehl, P. U.; and Cook, M. 2015.  Unsupervised Learning of Digit Recognition using Spike-Timing-Dependent Plasticity.  _Frontiers in Computational Neuroscience_ , 9: 99.
  * Han et al. (2018) Han, K.; Wen, H.; Shi, J.; Lu, K.-H.; Zhang, Y.; and Liu, Z. 2018.  Variational Autoencoder: An Unsupervised Model for Modeling and Decoding fMRI Activity in Visual Cortex.  _bioRxiv_.
  * Heusel et al. (2017) Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017\.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.  In _NeurIPS_.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long Short-Term Memory.  _Neural Computation_ , 9(8): 1735–1780.
  * Karras et al. (2020) Karras, T.; Laine, S.; Aittala, M.; Hellsten, J.; Lehtinen, J.; and Aila, T. 2020\.  Analyzing and Improving the Image Quality of StyleGAN.  In _CVPR_.
  * Kim et al. (2020) Kim, S. J.; Park, S.; Na, B.; and Yoon, S. 2020.  Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection.  In _AAAI_.
  * Kingma and Welling (2014) Kingma, D. P.; and Welling, M. 2014.  Auto-Encoding Variational Bayes.  In _ICLR_.
  * Kotariya and Ganguly (2021) Kotariya, V.; and Ganguly, U. 2021.  Spiking-GAN: A Spiking Generative Adversarial Network Using Time-To-First-Spike Coding.  arXiv:2106.15420.
  * Krizhevsky and Hinton (2009) Krizhevsky, A.; and Hinton, G. 2009.  Learning Multiple Layers of Features from Tiny Images.  Technical report, University of Toronto.
  * Lee et al. (2020) Lee, C.; Kosta, A.; Zhu, A. Z.; Chaney, K.; Daniilidis, K.; and Roy, K. 2020.  Spike-FlowNet: Event-Based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks.  In _ECCV_.
  * Liu et al. (2015) Liu, Z.; Luo, P.; Wang, X.; and Tang, X. 2015.  Deep Learning Face Attributes in the Wild.  In _ICCV_.
  * Loshchilov and Hutter (2019) Loshchilov, I.; and Hutter, F. 2019.  Decoupled Weight Decay Regularization.  In _ICLR_.
  * Maass (1997) Maass, W. 1997.  Networks of spiking neurons: The third generation of neural network models.  _Neural Networks_ , 10(9): 1659–1671.
  * Parmar, Zhang, and Zhu (2021) Parmar, G.; Zhang, R.; and Zhu, J. 2021.  On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation.  arXiv:2104.11222.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; Köpf, A.; Yang, E.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J.; and Chintala, S. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _NeurIPS_.
  * Razavi, van den Oord, and Vinyals (2019) Razavi, A.; van den Oord, A.; and Vinyals, O. 2019.  Generating Diverse High-Fidelity Images with VQ-VAE-2.  In _NeurIPS_.
  * Rezende, Wierstra, and Gerstner (2011) Rezende, D. J.; Wierstra, D.; and Gerstner, W. 2011.  Variational Learning for Recurrent Spiking Networks.  In _NeurIPS_.
  * Rueckauer et al. (2017) Rueckauer, B.; Lungu, I.-A.; Hu, Y.; Pfeiffer, M.; and Liu, S.-C. 2017.  Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification.  _Frontiers in Neuroscience_ , 11: 682.
  * Salimans et al. (2016) Salimans, T.; Goodfellow, I. J.; Zaremba, W.; Cheung, V.; Radford, A.; and Chen, X. 2016.  Improved Techniques for Training GANs.  In _NeurIPS_.
  * Skatchkovsky, Simeone, and Jang (2021) Skatchkovsky, N.; Simeone, O.; and Jang, H. 2021.  Learning to Time-Decode in Spiking Neural Networks Through the Information Bottleneck.  In _NeurIPS_.
  * Stein and Hodgkin (1967) Stein, R. B.; and Hodgkin, A. L. 1967.  The Frequency of Nerve Action Potentials Generated by Applied Currents.  _Proceedings of the Royal Society of London. Series B. Biological Sciences_ , 167(1006): 64–86.
  * Stewart et al. (2021) Stewart, K.; Danielescu, A.; Supic, L.; Shea, T. M.; and Neftci, E. 2021.  Gesture Similarity Analysis on Event Data Using a Hybrid Guided Variational Auto Encoder.  arXiv:2104.00165.
  * Talafha et al. (2020) Talafha, S.; Rekabdar, B.; Mousas, C.; and Ekenna, C. 2020.  Biologically Inspired Sleep Algorithm for Variational Auto-Encoders.  In _ISVC_.
  * Vahdat and Kautz (2020) Vahdat, A.; and Kautz, J. 2020.  NVAE: A Deep Hierarchical Variational Autoencoder.  In _NeurIPS_.
  * van den Oord, Vinyals, and Kavukcuoglu (2017) van den Oord, A.; Vinyals, O.; and Kavukcuoglu, K. 2017.  Neural Discrete Representation Learning.  In _NeurIPS_.
  * Wen et al. (2016) Wen, W.; Wu, C.; Wang, Y.; Nixon, K. W.; Wu, Q.; Barnell, M.; Li, H.; and Chen, Y. 2016.  A New Learning Method for Inference Accuracy, Core Accupation, and Performance Co-optimization on TrueNorth Chip.  In _DAC_.
  * Williams and Zipser (1989) Williams, R. J.; and Zipser, D. 1989.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks.  _Neural Computation_ , 1(2): 270–280.
  * Wu et al. (2018) Wu, J.; Chua, Y.; Zhang, M.; Li, H.; and Tan, K. C. 2018.  A Spiking Neural Network Framework for Robust Sound Classification.  _Frontiers in Neuroscience_ , 12: 836.
  * Wu et al. (2019) Wu, Y.; Deng, L.; Li, G.; Zhu, J.; Xie, Y.; and Shi, L. 2019.  Direct Training for Spiking Neural Networks: Faster, Larger, Better.  In _AAAI_.
  * Xiao, Rasul, and Vollgraf (2017) Xiao, H.; Rasul, K.; and Vollgraf, R. 2017.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms.  https://github.com/zalandoresearch/fashion-mnist.  Accessed: 2021-09-08, arXiv:1708.07747.
  * Zenke and Ganguli (2018) Zenke, F.; and Ganguli, S. 2018.  SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks.  _Neural Computation_ , 30(6).
  * Zhang and Li (2020) Zhang, W.; and Li, P. 2020.  Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networks.  In _NeurIPS_.
  * Zheng et al. (2021) Zheng, H.; Wu, Y.; Deng, L.; Hu, Y.; and Li, G. 2021.  Going Deeper With Directly-Trained Larger Spiking Neural Networks.  In _AAAI_.

Figure 8: Detailed architecture of FSVAE. After each layer a tdBN layer LIF
neuron is inserted. In our settings, timstep $T$ is set to 16, latent
dimension $C$ to 128, and $k$ to 20.

## Supplementary Material

## A Network Architecture

Figure 8 shows the detailed architecture of our FSVAE. For the CelebA dataset
we added an additional layer to both encoder and decoder. The architecture of
ANN VAE is the same as FSVAE, except for prior and posterior.

## B Derivation of MMD

In this section, we describe the derivation of Eq. (24).

We have $k(\bm{z}_{1:T},\bm{z}^{\prime}_{1:T})=\sum_{t}h(\bm{z}_{\leq
t})h(\bm{z}^{\prime}_{\leq t})$, so Eq. (21) is as follows:

$\displaystyle\mathrm{MMD}^{2}[q(\bm{z}_{1:T}|\bm{x}_{1:T}),p(\bm{z}_{1:T})]$
$\displaystyle=$
$\displaystyle\sum_{t}\left(\underset{\bm{z},\bm{z}^{\prime}\sim
q}{\mathbb{E}}[\mathrm{PSP}(\bm{z}_{\leq t})\mathrm{PSP}(\bm{z}^{\prime}_{\leq
t})]\right.$ $\displaystyle\qquad+\underset{\bm{z},\bm{z}^{\prime}\sim
p}{\mathbb{E}}[\mathrm{PSP}(\bm{z}_{\leq t})\mathrm{PSP}(\bm{z}^{\prime}_{\leq
t})]$ $\displaystyle\qquad\left.-2\underset{\bm{z}\sim q,\bm{z}^{\prime}\sim
p}{\mathbb{E}}[\mathrm{PSP}(\bm{z}_{\leq t})\mathrm{PSP}(\bm{z}^{\prime}_{\leq
t})]\right)$ $\displaystyle=$ $\displaystyle\sum_{t}\left(\underset{\bm{z}\sim
q}{\mathbb{E}}[\mathrm{PSP}(\bm{z}_{\leq t})]^{2}+\underset{\bm{z}\sim
p}{\mathbb{E}}[\mathrm{PSP}(\bm{z}_{\leq t})]^{2}\right.$
$\displaystyle\qquad\left.-2\underset{\bm{z}\sim
q}{\mathbb{E}}[\mathrm{PSP}(\bm{z}_{\leq t})]\underset{\bm{z}\sim
p}{\mathbb{E}}[\mathrm{PSP}(\bm{z}_{\leq t})]^{2}\right)$ $\displaystyle=$
$\displaystyle\sum_{t}\left\|\underset{\bm{z}\sim
q}{\mathbb{E}}[\mathrm{PSP}(\bm{z}_{\leq t})]-\underset{\bm{z}\sim
p}{\mathbb{E}}[\mathrm{PSP}(\bm{z}_{\leq t})]\right\|^{2}$

Here, according to Eq. (22), we can obtain $\mathrm{PSP}$ as

$\displaystyle\mathrm{PSP}(\bm{z}_{\leq
t})=\sum_{i=0}^{t}\frac{1}{\tau_{\mathrm{syn}}}\left(1-\frac{1}{\tau_{\mathrm{syn}}}\right)^{i}\bm{z}_{t-i}.$

Thus, we have $\mathbb{E}[\mathrm{PSP}(\bm{z}_{\leq
t})]=\mathrm{PSP}(\mathbb{E}[\bm{z}_{\leq t}])$. Therefore, MMD is as follows:

$\displaystyle\mathrm{MMD}^{2}[q(\bm{z}_{1:T}|\bm{x}_{1:T}),p(\bm{z}_{1:T})]$
$\displaystyle=$
$\displaystyle\sum_{t}\left\|\mathrm{PSP}(\underset{\bm{z}\sim
q}{\mathbb{E}}[\bm{z}_{\leq t}])-\mathrm{PSP}(\underset{\bm{z}\sim
p}{\mathbb{E}}[\bm{z}_{\leq t}])\right\|^{2}$ $\displaystyle=$
$\displaystyle\sum_{t}\left\|\mathrm{PSP}(\bm{\pi}_{q,\leq
t})-\mathrm{PSP}(\bm{\pi}_{p,\leq t})\right\|^{2}$

## C Derivation of KLD

In the ablation study, we used KLD as the distance metric between the prior
and posterior distribution. The KLD is calculated as follows:

$\displaystyle\mathrm{KL}[q(\bm{z}_{1:T})\|p(\bm{z}_{1:T})]$ $\displaystyle=$
$\displaystyle\sum_{t=1}^{T}\mathrm{KL}[q(\bm{z}_{t}|\bm{x}_{\leq
t},\bm{z}_{<t})\|p(\bm{z}_{t}|\bm{z}_{<t})]$ $\displaystyle=$
$\displaystyle\sum_{t=1}^{T}\mathrm{KL}[Ber(\bm{\pi}_{q,t}\|Ber(\bm{\pi}_{p,t})]$
$\displaystyle=$
$\displaystyle\sum_{t=1}^{T}\left(\pi_{q,t,c}\log\frac{\pi_{q,t,c}}{\pi_{p,t,c}}+(1-\pi_{q,t,c})\log\frac{1-\pi_{q,t,c}}{1-\pi_{p,t,c}}\right)$

To avoid divergence, we added $\epsilon=0.01$ to $\pi_{q,t,c}$ and
$\pi_{p,t,c}$.

## D Analysis of Spike Activities

Figure 9 shows the input images and their corresponding latent spike trains.
It can be seen that some spike train fire more periodically while others fire
more burst-like. This diversity allows the spike train to encode a variety of
different images.

Figure 9: Input images and their corresponding latent spike trains.

Figure 10 shows the average firing rates of each layer. Each firing rate is
stable, and contributes to stable learning.

Figure 10: Firing rates in each layer of FSVAE on CelebA

## E Codes for Algorithms

We show the detailed code for autoregressive Bernoulli spike sampling for
posterior in Algorithm 1, and prior in Algorithm 2. Additionally, Algorithm 3
shows the overall training and sampling processes.

Algorithm 1 Autoregressive Bernoulli Spike Sampling for Posterior

Input:
$\bm{x}_{1:T}^{E}\in\\{0,1\\}^{C\times T}$: outputs from SNN encoder,
Parameter:
$f_{q}$: autoregressive SNN for posterior,
$\Theta_{q}$: sets of membrane potentials of $f_{q}$.
Output:
$\bm{z}_{q,1:T}\in\\{0,1\\}^{C\times T}$: latent spike trains.
$\bm{\zeta}_{q,1:T}\in\\{0,1\\}^{kC\times T}$: outputs of $f_{q}$

1:function PosteriorSampling($\bm{x}_{1:T}^{E}$)

2: Let $\bm{z}_{q,0}=\bm{0}\in\\{0,1\\}^{C}$

3: for $t=1$ to $t=T$ do

4:
$\bm{\zeta}_{q,t}=f_{q}(\bm{z}_{q,t-1},\bm{x}_{t}^{E};\Theta_{q,t})\in\\{0,1\\}^{kC}$

5: for $c=1$ to $c=C$ do

6: $\bm{z}_{q,t,c}=\mathrm{random\\_select}(\bm{\zeta}_{q,t}[k(c-1):kc])$

7: end for

8: end for

9: return $\bm{z}_{q,1:T}$, $\bm{\zeta}_{q,1:T}$

10:end function

Algorithm 2 Autoregressive Bernoulli Spike Sampling for Prior

Input: None
Parameter:
$f_{p}$: autoregressive SNN for prior,
$\Theta_{p}$: sets of membrane potentials of $f_{p}$.
Output:
$\bm{z}_{p,1:T}\in\\{0,1\\}^{C\times T}$: latent spike trains.
$\bm{\zeta}_{p,1:T}\in\\{0,1\\}^{kC\times T}$: outputs of $f_{p}$

1:function PriorSampling()

2: Let $\bm{z}_{p,0}=\bm{0}\in\\{0,1\\}^{C}$

3: for $t=1$ to $t=T$ do

4: $\bm{\zeta}_{p,t}=f_{p}(\bm{z}_{p,t-1};\Theta_{p,t})\in\\{0,1\\}^{kC}$

5: for $c=1$ to $c=C$ do

6: $\bm{z}_{p,t,c}=\mathrm{random\\_select}(\bm{\zeta}_{p,t}[k(c-1):kc])$

7: end for

8: end for

9: return $\bm{z}_{p,1:T}$, $\bm{\zeta}_{p,1:T}$

10:end function

Algorithm 3 Overall Training and Sampling Algorithm

Input: $x$: input image
Output: $\hat{x}$: reconstructed image

1:function Training($x$)

2: $\bm{x}_{1:T}$ = DirectInputEncoding($x$)

3: $\bm{x}_{1:T}^{E}$ = SNNEncoder($\bm{x}_{1:T}$)

4: $\bm{z}_{q,1:T}$, $\bm{\zeta}_{q,1:T}$ =
PosteriorSampling($\bm{x}_{1:T}^{E}$)

5: $\bm{z}_{p,1:T}$, $\bm{\zeta}_{p,1:T}$ = PriorSampling()

6: $\hat{\bm{x}}_{1:T}$ = SNNDecoder($\bm{z}_{q,1:T}$)

7: $\hat{x}$ = SpikeToImageDecode($\hat{\bm{x}}_{1:T}$)

8: calculate $\mathcal{L}$ with ($x$, $\hat{x}$, $\bm{\zeta}_{q,1:T}$,
$\bm{\zeta}_{p,1:T}$) according to Eq. (25)

9: backward with $\mathcal{L}$ and update parameters

10: return $\hat{x}$

11:end function

Input: None
Output: $\hat{x}$: generated image

1:function Sampling()

2: $\bm{z}_{p,1:T}$, $\bm{\zeta}_{p,1:T}$ = PriorSampling()

3: $\hat{\bm{x}}_{1:T}$ = SNNDecoder($\bm{z}_{p,1:T}$)

4: $\hat{x}$ = SpikeToImageDecode($\hat{\bm{x}}_{1:T}$)

5: return $\hat{x}$

6:end function

## F Reconstructed Images

Figure 11: Reconstructed images of MNIST. Figure 12: Reconstructed images of
Fashion MNIST. Figure 13: Reconstructed images of CIFAR10. Figure 14:
Reconstructed images of CelebA.

## G Generated Images

Figure 15: Generated images of MNIST. Figure 16: Generated images of Fashion
MNIST. Figure 17: Generated images of CIFAR10. Figure 18: Generated images of
CelebA.


* [CDGP] P. Candelas, X. De La Ossa, P. Green, and L. Parkes A pair of Calabi-Yau manifolds as an exactly soluble superconformal field theory, Nuclear Physics B359 (1991), 21-74 and in Essays on Mirror Manifolds (S. T. Yau ed), International Press, Hong Kong, 1992, 31-95.
  * [CCF] C. Casagrande, G. Codogni and A. Fanelli The blow-up of $\mathbb{P}^{4}$ at $8$ points and its Fano model, via vector bundles on a del Pezzo surface Revista Matemática Complutense 32 (2019), 475-–529.
  * [CT] A.M. Castravet and J. Tevelev, Hilbert’s 14th problem and Cox rings, Compos. Math. 142 (2006), no. 6, 1479–1498.
  * [CK] D. Cox and S. Katz, Mirror Symmetry and Algebraic Geometry, American Mathematical Society, Mathematical Surveys and Monographs, 1999, Vol.68.
  * [C] H. Clemens, Double solids, Adv. Math. 47 (1983), 107 - 230.
  * [D83] I. V. Dolgachev, _Weyl Groups And Cremona Transformations_. Proceedings of Symposia in Pure Mathematics, Vol. 40 (1983), Part I, 283–294.
  * [D08] I. V. Dolgachev, _Reflection Groups In Algebraic Geometry_. Bulletin of the AMS, Vol. 45, Number 1, (January 2008), 1–60.
  * [DolgOrt88] I. V. Dolgachev and David Ortland, _Point Sets In Projective Spaces and Theta Functions_. Asterisque No. 165 (1988).
  * [DO] O. Dumitrescu and B. Osserman, An observation on (-1)-curves on rational surfaces, Proceedings of the American Mathematical Society 147 (4), 1391 - 1398, (2019).
  * [DM1] O. Dumitrescu and R. Miranda, Cremona orbits in $\mathbb{P}^{4}$ and Applications, https://arxiv.org/pdf/2103.08040.pdf.
  * [DM2] O. Dumitrescu and R. Miranda, Coxeter systems and Weyl group action, in preparation.
  * [DP1] O. Dumitrescu and E. Postinghel: _Vanishing Theorems for Linearly Obstructed Divisors_. Journal of Algebra, Vol. 477 (2017), pp. 312–359
  * [DP] O. Dumitrescu and N. Priddis, On $(-1)$ classes, https://arxiv.org/pdf/1905.00074.pdf.
  * [EH] D. Eisenbud and J. Harris, _3264 And All That: A Second Course In Algebraic Geometry_. (2016) Cambridge University Press.
  * [EV] D. Eisenbud and A. Van de Ven: _On the Normal Bundles of Smooth Rational Space Curves_. Math. Annalen 256, (1981), pp 453–463.
  * [F] R. Friedman, On threefolds with trivial canonical bundle, Complex Geometry and Lie Theory (Proc. Symp. Pure Math., vol. 53), American Mathematical Society, Providence, 1991, pp. 103 - 134
  * [Hu] J. E. Humphreys, Reflection Groups and Coxeter Groups, Cambridge Studies in Advanced Mathematics No. 29 (1990), Cambridge University Press.
  * [K1] M. Kontsevich, Mirror Symmetry in dimension 3,
Séminaire Bourbaki 1995, 47ème année 1994-95, no. 801.

  * [K2] M. Kontsevich, Ennumeration of rational curves via torus action,
The Moduli Space of Curves, by R.Dijkgraaf, C.Faber and G. Van der Geer.
Progress in Mathematics 129, 335-368 Birkhauser, Boston, MA, 1995.

  * [KPY] M. Kontsevich, V. Pestun and Y. Tschinkel Equivariant birational geometry and modular symbols, 56 pp., to appear in Journ. EMS, (2019)
  * [LU] A. Laface and L. Ugaglia, On a class of special linear systems on $\mathbb{P}^{3}$, Trans. Amer. Math. Soc. 358 (2006), no. 12, 5485-–5500 (electronic).
  * [LU2] A. Laface and L. Ugaglia, On multiples of divisors associated to Veronese embeddings with defective secant variety, Bull. Belg. Math. Soc. Simon Stevin 16 (2009), no. 5, Linear systems and subschemes, 933–-942.
  * [L] J. Lesieutre, The diminished base locus is not always closed Compositio Mathematica, Volume 150 , Issue 10 , October 2014 , pp. 1729 - 1741.
  * [Mu] S. Mukai, Finite generation of the Nagata invariant rings in A-D-E cases, RIMS Preprint n. 1502, Kyoto, 2005.
  * [N1] M. Nagata, On the fourteenth problem of Hilbert, Proc. Internat. Congress Math., Cambridge University Press, 1958, 459–-462.
  * [N2] M. Nagata, On the 14-th problem of Hilbert, Amer. J. Math., 81, 1959, 766-–772.
  * [N3] M. Nagata, On rational surfaces, II, Mem. Coll. Sci. Univ. Kyoto Ser. A Math., 1960, 33, 271–293.
  * [N4] M, Nagata, Lectures on The Fourteenth Problem of Hilbert, Tata Institute of Fundamental Research Lectures on Mathematics, 31, Bombay, 1965, Tata Institute of Fundamental Research.