Complete classification of torsion of elliptic curves over quadratic
cyclotomic fields
Filip Najman

_Department of Mathematics, University of Zagreb,
Bijenička cesta 30, 10000 Zagreb, Croatia _

###### Abstract

In a previous paper ([10]), the author examined the possible torsions of an
elliptic curve over the quadratic fields $\mathbb{Q}(i)$ and
$\mathbb{Q}(\sqrt{-3})$. Although all the possible torsions were found if the
elliptic curve has rational coefficients, we were unable to eliminate some
possibilities for the torsion if the elliptic curve has coefficients that are
not rational. In this note, by finding all the points of two hyperelliptic
curves over $\mathbb{Q}(i)$ and $\mathbb{Q}(\sqrt{-3})$, we solve this problem
completely and thus obtain a classification of all possible torsions of
elliptic curves over $\mathbb{Q}(i)$ and $\mathbb{Q}(\sqrt{-3})$.

00footnotetext: Mathematics subject classification (2000) 11G05, 11G30, 14H40

1 Introduction

For an elliptic curve $E$ over a number field $K$, it is well known, by the
Mordell-Weil theorem, that the set $E(K)$ of $K$-rational points on $E$ is a
finitely generated Abelian group. The group $E(K)$ is isomorphic to
$T\oplus\mathbb{Z}^{r}$, where $r$ is a non-negative integer and $T$ is the
torsion subgroup. When $K=\mathbb{Q}$, by Mazur’s Theorem, the torsion
subgroup is either cyclic of order $m$, where $1\leq m\leq 10$ or $m=12$, or
of the form $\mathbb{Z}_{2}\oplus\mathbb{Z}_{2m}$, where $1\leq m\leq 4$.
If $K$ is a quadratic field, then the following theorem classifies the
possible torsions.

######  Theorem(Kamienny, [3], Kenku and Momose, [4]).

Let $K$ be a quadratic field and $E$ an elliptic curve over $K$. Then the
torsion subgroup $E(K)_{tors}$ of $E(K)$ is isomorphic to one of the following
$26$ groups:

$\mathbb{Z}_{m},\text{ for }1\leq m\leq 18,\ m\neq 17,$
$\mathbb{Z}_{2}\oplus\mathbb{Z}_{2m},\text{ for }1\leq m\leq 6,$
$\mathbb{Z}_{3}\oplus\mathbb{Z}_{3m},\text{ for }m=1,2$
$\mathbb{Z}_{4}\oplus\mathbb{Z}_{4}.$

Moreover, the only quadratic field over which torsion
$\mathbb{Z}_{4}\oplus\mathbb{Z}_{4}$ appears is $\mathbb{Q}(i)$ and the only
quadratic field over which torsions $\mathbb{Z}_{3}\oplus\mathbb{Z}_{3}$ and
$\mathbb{Z}_{3}\oplus\mathbb{Z}_{6}$ appear is $\mathbb{Q}(\sqrt{-3})$.
In [2], Theorem 3.5, it is proved that if we let the quadratic fields vary,
then all of the 26 torsion subgroups appear infinitely often.
Unfortunately, if we fix a quadratic field, this theorem does not tell us
which of the 26 listed groups actually appear as torsion subgroups. In [10],
we took this approach, fixing the fields $\mathbb{Q}(i)$ and
$\mathbb{Q}(\sqrt{-3})$. These fields are somewhat special among quadratic
fields, as they are the only cyclotomic quadratic fields
($\mathbb{Q}(i)=\mathbb{Q}(\zeta_{4})$ and
$\mathbb{Q}(\sqrt{-3})=\mathbb{Q}(\zeta_{6})$, where $\zeta_{n}$ is a
primitive $n$-th root of unity). Also, as already mentioned, over each of
these fields, torsion subgroups appear that appear over no other fields. Note
that the rings of integers of both these fields are unique factorization
domains.

The main results of [10] are given in the following theorem.

######  Theorem 1.

  * (i)

Let $E$ be an elliptic curve with rational coefficients. Then
$E(\mathbb{Q}(i))_{tors}$ is either one of the groups from Mazur’s Theorem or
$\mathbb{Z}_{4}\oplus\mathbb{Z}_{4}$.

  * (ii)

Let $E$ be an elliptic curve defined over $\mathbb{Q}(i)$. Then
$E(\mathbb{Q}(i))_{tors}$ is either one of the groups from Mazur’s Theorem,
$\mathbb{Z}_{4}\oplus\mathbb{Z}_{4}$ or $\mathbb{Z}_{13}$.

  * (iii)

Let $E$ be an elliptic curve with rational coefficients. Then
$E(\mathbb{Q}(\sqrt{-3}))_{tors}$ is either one of the groups from Mazur’s
Theorem, $\mathbb{Z}_{3}\oplus\mathbb{Z}_{3}$ or
$\mathbb{Z}_{3}\oplus\mathbb{Z}_{6}$.

  * (iv)

Let $E$ be an elliptic curve defined over $\mathbb{Q}(\sqrt{-3})$. Then
$E(\mathbb{Q}(\sqrt{-3}))_{tors}$ is either one of the groups from Mazur’s
Theorem, $\mathbb{Z}_{3}\oplus\mathbb{Z}_{3}$,
$\mathbb{Z}_{3}\oplus\mathbb{Z}_{6}$, $\mathbb{Z}_{13}$ or $\mathbb{Z}_{18}$.

While it is not hard to show that (i) and (iii) are best possible, we
conjectured that the possible torsion $\mathbb{Z}_{13}$ could be removed from
(ii) and the possible torsions $\mathbb{Z}_{13}$ and $\mathbb{Z}_{18}$ could
be removed from (iv). In this note we prove this conjecture, thus proving the
following theorem.

######  Theorem 2.

  * (i)

Let $E$ be an elliptic curve defined over $\mathbb{Q}(i)$. Then
$E(\mathbb{Q}(i))_{tors}$ is either one of the groups from Mazur’s Theorem or
$\mathbb{Z}_{4}\oplus\mathbb{Z}_{4}$.

  * (ii)

Let $E$ be an elliptic curve defined over $\mathbb{Q}(\sqrt{-3})$. Then
$E(\mathbb{Q}(\sqrt{-3}))_{tors}$ is either one of the groups from Mazur’s
Theorem, $\mathbb{Z}_{3}\oplus\mathbb{Z}_{3}$ or
$\mathbb{Z}_{3}\oplus\mathbb{Z}_{6}$.

2 Torsion over $\mathbb{Q}(i)$ cannot be $\mathbb{Z}_{13}$

As seen from [12], case 2.5.2, page 30, elliptic curves with torsion
$\mathbb{Z}_{13}$ over $\mathbb{Q}(i)$ exist if and only if the curve $C_{1}$
defined by

$C_{1}:y^{2}=x^{6}-2x^{5}+x^{4}-2x^{3}+6x^{2}-4x+1$ (1)

has points over $\mathbb{Q}(i)$ satisfying

$x(x-1)(x^{3}-4x^{2}+x+1)\neq 0.$ (2)

As the points of a hyperelliptic curve have no structure, it is useful to
examine the Jacobian variety of a curve (see [1]). Let $J_{1}$ be the Jacobian
of $C_{1}$, $C_{1}^{(d)}$ the quadratic twist of $C_{1}$ by $d$ and
$J_{1}^{(d)}$ the Jacobian of $C_{1}^{(d)}$.

######  Lemma 3.

$J_{1}(\mathbb{Q}(i))\simeq\mathbb{Z}_{19}.$

_Proof:_
In MAGMA ([5]) there exists a implementation of $2$-descent on Jacobians (the
RankBounds function, see [13] for the algorithm), but unfortunately only over
$\mathbb{Q}$. However, we can compute $rank(J_{1}(\mathbb{Q}))=0$ and
$rank(J_{1}^{(-1)}(\mathbb{Q}))=0$, and thus

$rank(J_{1}(\mathbb{Q}(i)))=rank(J_{1}(\mathbb{Q}))+rank(J_{1}^{(-1)}(\mathbb{Q}))=0.$

Again for the torsion of the Jacobian, MAGMA has implemented functions only
over the rationals. For an algortihm for computing the torsion, see [11]. The
discriminant of the $C_{1}$ is $2^{20}\cdot 13^{2}$, so $2$ and $13$ are the
only rational primes with bad reduction.
We compute $J_{1}(\mathbb{Q})_{tors}\simeq\mathbb{Z}_{19}$. If $p$ is a
Gaussian prime of good reduction, then the prime-to-$p$ part of
$J_{1}(Q(i))_{tors}$ injects to $J_{1}(F_{p}(i))$. As $p\equiv 3\pmod{4}$
remains prime in $\mathbb{Z}[i]$ and $F_{p}(i)\simeq F_{p^{2}}$, by computing

$|J_{1}(F_{121})|=|J_{1}(F_{11}(i))|=7^{2}\cdot 19^{2},$
$|J_{1}(F_{529})|=|J_{1}(F_{23}(i))|=3\cdot 7\cdot 19\cdot 229,$
$|J_{1}(F_{961})|=|J_{1}(F_{31}(i))|=2^{8}\cdot 3^{2}\cdot 19^{2},$

we conclude that $|J_{1}(Q(i))_{tors}|=19$ and thus
$J_{1}(\mathbb{Q}(i))\simeq\mathbb{Z}_{19}$. ∎

######  Lemma 4.

$C_{1}(\mathbb{Q}(i))=\\{\infty_{+},\infty_{-},(1,\pm 1),(0,\pm 1)\\}.$

_Proof:_
The 18 non-zero elements of $J_{1}(\mathbb{Q})=J_{1}(\mathbb{Q}(i))$, in the
Mumford representation that MAGMA uses (see [8] for details, and [6] for the
particular implementation in MAGMA) are

$(x^{2},-2x+1,2),(x-1,-x^{3},2),(x,x^{3}-1,2),(x,x^{3}+1,2),(x^{2}-x,-1,2),$
$(x^{2}-x,-2x+1,2),(1,x^{3}-x^{2},2),(x^{2}-2x+1,x,2),(x-1,x^{3}-2,2),$
$(x-1,-x^{3}+2,2),(x^{2}-2x+1,-x,2),(1,-x^{3}+x^{2},2),(x^{2}-x,2x-1,2),$
$(x^{2}-x,1,2),(x,-x^{3}-1,2),(x,-x^{3}+1,2),(x-1,x^{3},2),(x^{2},2x-1,2),$

and we easily see that the points mentioned are the only ones on this curve. ∎

By proving Lemma 4, we have actually proven Theorem 2, (i).

3 Torsion over $\mathbb{Q}(\sqrt{-3})$ cannot be $\mathbb{Z}_{13}$ or
$\mathbb{Z}_{18}$

To prove that the torsion can not be $\mathbb{Z}_{13}$, we again have to prove
that there are no points on $C_{1}$ over $\mathbb{Q}(\sqrt{-3})$ satisfying
(2).

######  Lemma 5.

$J_{1}(\mathbb{Q}(\sqrt{-3}))\simeq\mathbb{Z}_{19}.$

_Proof:
_ We compute $rank(J_{1}^{(-3)}(\mathbb{Q}))=0$, and thus

$rank(J_{1}(\mathbb{Q}(\sqrt{-3})))=rank(J_{1}(\mathbb{Q}))+rank(J_{1}^{(-3)}(\mathbb{Q}))=0.$

As $5$ and $17$ are rational primes of good reduction that remain prime in
$\mathbb{Z}[\frac{1+\sqrt{-3}}{2}]$, and

$|J_{1}(F_{25})|=|J_{1}(F_{5}(\sqrt{-3}))|=19^{2},$
$|J_{1}(F_{289})|=|J_{1}(F_{17}(\sqrt{-3}))|=2^{6}\cdot 3^{2}\cdot 7\cdot 19,$

we conclude that $J_{1}(\mathbb{Q}(\sqrt{-3}))\simeq\mathbb{Z}_{19}.$ ∎

######  Lemma 6.

$C_{1}(\mathbb{Q}(\sqrt{-3}))=\\{\infty_{+},\infty_{-},(1,\pm 1),(0,\pm
1)\\}.$

_Proof:_
The proof is the same as the proof of Lemma 4. ∎

As seen from [12], case 2.5.6, pages 38–39, elliptic curves with torsion
$\mathbb{Z}_{18}$ over $\mathbb{Q}(\sqrt{-3})$ exist if and only if the curve
$C_{2}$ defined by

$y^{2}=x^{6}+2x^{5}+5x^{4}+10x^{3}+10x^{2}+4x+1$ (3)

has points over $\mathbb{Q}(\sqrt{-3})$ satisfying

$x(x+1)(x^{2}+x+1)(x^{3}-3x-1)\neq 0.$ (4)

Let $J_{2}$ be the Jacobian of $C_{2}$, $C_{2}^{(d)}$ the quadratic twist of
$C_{2}$ by $d$ and $J_{2}^{(d)}$ the Jacobian of $C_{2}^{(d)}$.

######  Lemma 7.

$J_{2}(\mathbb{Q}(\sqrt{-3}))\simeq\mathbb{Z}_{3}\oplus\mathbb{Z}_{21}.$

_Proof:
_ In MAGMA ([5]) we compute $rank(J_{2}(\mathbb{Q}))=0$ and
$rank(J_{2}^{(-3)}(\mathbb{Q}))=0$, and thus

$rank(J_{2}(\mathbb{Q}(\sqrt{-3})))=rank(J_{2}(\mathbb{Q}))+rank(J_{2}^{(-3)}(\mathbb{Q}))=0.$

The discriminant of $C_{2}$ is $2^{23}\cdot 3^{4}$, so $2$ and $3$ are the
only rational primes with bad reduction. We find the points in Mumford
representation $(x^{2}+x+1,-\sqrt{-3}x-\sqrt{-3},2)$ and $(1,-x^{3}-x^{2},2)$
on the Jacobian generate a group isomorphic to
$\mathbb{Z}_{3}\oplus\mathbb{Z}_{21}$.

As the rational primes $5$ and $11$ remain prime in
$\mathbb{Z}[\frac{1+\sqrt{-3}}{2}]$, we have

$|J_{2}(F_{25})|=|J_{2}(F_{5}(\sqrt{-3}))|=21^{2},$
$|J_{2}(F_{121})|=|J_{2}(F_{11}(\sqrt{-3}))|=2^{4}\cdot 3^{2}\cdot 7\cdot 13.$

We conclude that
$J_{2}(\mathbb{Q}(\sqrt{-3}))_{tors}\simeq\mathbb{Z}_{3}\oplus\mathbb{Z}_{21}$.
∎

######  Lemma 8.

$C_{2}(\mathbb{Q}(\sqrt{-3}))=\\{\infty_{+},\infty_{-},(-1,\pm 1),(0,\pm
1),(\frac{-1-\sqrt{-3}}{2},\pm\frac{-3-\sqrt{-3}}{2}),$
$(\frac{-1+\sqrt{-3}}{2},\pm\frac{-3+\sqrt{-3}}{2})\\}.$

_Proof:_
Searching throught the 62 non-zero elements of $J_{2}(\mathbb{Q}(\sqrt{-3}))$,
we see that the only points besides the obvious rational ones, satisfy
$x^{2}+x+1=0=y-x+1$, and hence we obtain our result. ∎

By Lemma 6 the torsion of an elliptic curve over $\mathbb{Q}(\sqrt{-3})$
cannot be $\mathbb{Z}_{13}$ and by Lemma 8, the torsion cannot be
$\mathbb{Z}_{18}$. Thus we have proved Theorem 2.

Remark The fact that $J_{1}(\mathbb{Q})_{tors}\simeq\mathbb{Z}_{19}$ and
$J_{2}(\mathbb{Q})_{tors}\simeq\mathbb{Z}_{21}$ was proven already 35 years
ago by Ogg in [9], and the existence of rational 19-torsion points on
$J_{1}(\mathbb{Q})$ was used by Mazur and Tate [7] to prove the non-existence
of rational points on elliptic curves of order 13 over $\mathbb{Q}$.

## References

  * [1] H. Cohen, G. Frey, _Handbook of Elliptic and Hyperelliptic Curve Cryptography_ , Discrete Math. Appl., Chapman & Hall/CRC, 2006\.
  * [2] D. Jeon, C. H. Kim and E. Park, _On the torsion of elliptic curves over quartic number fields_ , J. London Math. Soc. (2) 74 (2006), 1–12.
  * [3] S. Kamienny, _Torsion points on elliptic curves and $q$-coefficients of modular forms_, Invent. Math. 109 (1992), 221–229.
  * [4] M. A. Kenku and F. Momose, _Torsion points on elliptic curves defined over quadratic fields_ , Nagoya Math. J. 109 (1988), 125–149.
  * [5] The MAGMA computer algebra system is described in W. Bosma, J. Cannon, and Catherine Playoust, _The Magma algebra system. I. The user language_ , J. Symbolic Comput. 24 (1997), 235 -265.
  * [6] W. Bosma, J. Cannon, _Handbook of Magma Functions_ , http://www.msri.org/about/computing/docs/magma/Handbook.pdf
  * [7] B. Mazur and J. Tate, _Points of order 13 on elliptic curves_ , Invent. Math. 22 (1973/74), 41 -49.
  * [8] D. Mumford, _Tata lectures on theta II_ , volume 43 of _Progr. Math._ Birkhauser, 1984.
  * [9] A. Ogg, _Rational points on certain elliptic modular curves, Analytic number theory_ (Proc. Sympos. Pure Math., Vol XXIV, St. Louis Univ., St. Louis, Mo., 1972), pp. 221 -231, Amer. Math. Soc., Providence, R.I., 1973.
  * [10] F. Najman, _Torsion of elliptic curves over quadratic cyclotomic fields_ , Math J. Okayama U., to appear.
  * [11] B. Poonen, _Computing Torsion Points on Curves_ , Exp. Math. 10 (2001), 449–466.
  * [12] F. P. Rabarison, _Torsion et rang des courbes elliptiques definies sur les corps de nombres algebriques_ , Doctorat de Universite de Caen, 2008.
  * [13] M. Stoll, _Implementing 2-descent on Jacobians of hyperelliptic curves of genus two, II,_ Acta Arith. 98 (2001), 245–277.

FILIP NAJMAN
DEPARTMENT OF MATHEMTICS,
UNIVERSITY OF ZAGREB,
BIJENIČKA CESTA 30, 10000 ZAGREB,
CROATIA
_E-mail address:_<EMAIL_ADDRESS>


# Optimising Event-Driven Spiking Neural Network with
Regularisation and Cutoff

Dengyu Wu, Gaojie Jin, Han Yu, Xinping Yi and Xiaowei Huang
School of Electrical Engineering, Electronics and Computer Science
University of Liverpool
Liverpool, United Kingdom
{dengyu.wu<EMAIL_ADDRESS>
###### Abstract

Spiking neural networks (SNNs), next generation of artificial neural networks
(ANNs) that more closely mimic natural neural networks and are more energy
efficient, have achieved the accuracy close to its ANN counterparts, on
benchmark datasets such as CIFAR10/100 and ImageNet. However, comparing with
frame-based input (e.g., images), event-based inputs from e.g., Dynamic Vision
Sensor (DVS) can make a better use of SNNs thanks to the SNNs’ asynchronous
working mechanism. In this paper, we strengthen the marriage between SNNs and
event-based inputs with a proposal to consider anytime optimal inference SNNs,
or AOI-SNNs, which can terminate anytime during the inference to achieve
optimal inference result. Two novel optimisation techniques are presented to
achieve AOI-SNNs: a regularisation and a cutoff. The regularisation enables
the training and construction of SNNs with optimised performance, and the
cutoff technique optimises the inference of SNNs on event-driven inputs. We
conduct an extensive set of experiments on multiple benchmark event-based
datasets, including CIFAR10-DVS, N-Caltech101 and DVS128 Gesture. The
experimental results demonstrate that our techniques are superior to the
state-of-the-art with respect to the accuracy and latency.

##  1 Introduction

SNNs have recently attracted significant research and industrial interests
thanks to its energy efficiency and low latency [29], and there are
neuromorphic chips such as Loihi [7] and TrueNorth [1] on which SNNs can be
deployed. Mechanistically, SNNs mimic biological neurons, and the neurons
process and forward spikes independently. With such an asynchronous working
mechanism, only a (small) subset of neurons will be activated during
inference. That is, energy efficiency is inherent to SNNs.

The asynchronous mechanism also suggests that event-based input may make a
better use of SNNs. Actually, neuromorphic sensors such as Dynamic Vision
Sensor [25, 8, 16] and Dynamic Audio Sensor (DAS) [3] have been developed to
generate binary “events”, which are ideal inputs to SNN. For example, unlike
conventional frame-based cameras which measure the “absolute” brightness at a
constant rate, DVS cameras are bio-inspired sensors that _asynchronously_
measure per-pixel brightness changes (called “events”), and output a stream of
events that encode the time, location and sign of the brightness changes [16].
DVS reveals the sparsity and asychronicity in recognition systems for
computational efficiency [2, 27, 19]. To deal with event-based input, we
propose to consider anytime optimal inference SNNs, or AOI-SNNs, which allow
the termination at any time during the inference on a spike train (i.e., an
input) and return the best possible inference result. Such SNNs enable the
cutoff during the inference without (significantly) compromising the
performance, and thus can achieve the best in terms of accuracy and latency.

Figure 1: An illustrative diagram showing the regularisation for improving SNN
latency and the cutoff mechanism for reducing latency on Cifar10-DVS dataset.
Cutoff is triggered when $S_{gap}$ is greater than $\beta$, a value
dynamically determined by a confidence rate as introduced in Section 4.3.

Regarding the training of SNNs, a mainstream approach is through ANN-to-SNN
conversion, which adopts the mature training regime of ANNs to first train a
high-accuracy ANN, and then convert it into SNN. Such conversions via ANNs
have resulted in research to focus on achieving the near-zero conversion loss.
However, existing conversion methods [10, 6, 17] mostly conduct empirical
experiments on frame-based benchmark datasets such as ImageNet [9] and
CIFAR10/100 [20]. In this paper, we will focus on event-based input, and
therefore the AOI-SNNs, and explore effective training and inference methods
to improve accuracy and latency together.

When considering ANN-to-SNN conversions to deal with DVS inputs, there are two
possible ways. The first one aggregates the sparse events in the DVS stream
into a frame-based input, on which the SNN processes as a whole. This
resembles the ANN processing a static input (such as an image). As explained
in Section 3.1, the frame-based input will base on the average spike rate,
neglecting the spike timing information. The second is to directly work with
the event-based input, by considering e.g., AOI-SNNs. An obvious benefit is
that SNNs can exploit sparse events in the DVS input, enabling energy-
efficient operation and reduced latency. In addition, unlike frame-based
input, the event-based input does not need an encoder at or before the first
layer, which allows SNNs to operate asynchronously and achieve extra low-
latency (explained in Section 3.1).

This paper makes two key technical contributions. Firstly, we propose a
regularisation technique to influence the activation distribution during ANN
training, which results in an SNN that can classify with less input
information. As will be discussed in Related Work (Section 2), with our
proposed regulariser, we can train an ANN without clipping and do not need to
apply any quantisation-aware technique. Experiments in Section 5.2 show that
we can achieve better accuracy than the state-of-the-art methods on both
direct training and ANN-to-SNN conversion. Clipping (and quantisation-aware)
techniques have been the status quo in this area due to the recent progress
[23, 6, 10, 36] and our result suggests that there is an alternative, and
probably better, way to get an improved SNN. Instead of simulating non-
differentiable SNN activations during ANN training, our regulariser enables
the attainment of a better distribution of SNN current by actively
regularising the activations of the possible misclassifications. The
regulariser is based on a new theoretical result (Section 4.1) that a smaller
ratio of threshold voltages to average accumulated current can result in an
SNN that can achieve optimised performance at any time during the inference.

The second contribution is that, instead of setting the inference length to
always be $T_{total}$, we can explore an early cutoff mechanism that enables
the SNN model to automatically achieve optimal latency and energy efficiency.
As shown in Fig. 1, the SNN model will run a monitoring mechanism to determine
when it is sufficiently confident to make a decision. Once such a decision is
made at time $t<T_{total}$, a cutoff action is triggered so that the SNN will
not take future inputs until the time $T_{total}$. Therefore, it will lead to
lower latency because decision is made at time $t$ rather than $T_{total}$.

##  2 Related work

Table 1: Technical ingredients of different conversion methods. OE and QA denote Outlier Elimination and Quantisation-aware technique respectively. COE: Clipping for Outlier Elimination; ROE: Regularisation for Outlier Elimination. | Training | Inference
---|---|---
| OE through | Apply QA | Soft-reset | Additive Noise | Cutoff
[32] | - | - | $\surd$ | - | -
[10] | clipping (COE) | - | $\surd$ | - | -
[36] | clipping (COE) | - | $\surd$ | $\surd$ | -
[23] | clipping (COE) | $\surd$ | $\surd$ | - | -
[6] | clipping (COE) | $\surd$ | $\surd$ | $\surd$ | -
Ours | regularisation (ROE) | - | $\surd$ | $\surd$ | $\surd$
|  |  |  |  |

The application of SNNs to a data source can be separated into two phases:
training and inference. Broadly speaking, the training algorithms for SNNs can
be categorised into direct training (DT) and ANN-to-SNN conversion. Recently,
Spike-based Error Backpropagation [38, 39, 14, 11, 41] direct train a neural
network to process the temporal information of input spikes. However, either
direct training or conversion algorithm [21, 36] needs to collapse the input
spikes into frames for the training. More specifically, the first layer in
former SNN needs to wait for the full spike train within one frame to generate
one spike, while the latter can respond very fast as long as the SNN receives
spikes. Normally, the number of frames in direct training is kept small to
reduce training complexity and determines the latency of SNN in inference. In
contrast, ANN-to-SNN conversion can incorporate the maximum number of spikes
during training to consider the SNN with optimal latency.

For the ANN-to-SNN conversion, early studies [32, 12] use the maximum value of
activation to normalise the weights from ANN, and [34] proves that the
normalisation can also be achieved by greedily searching for the optimal
threshold using the input spike train. A unified conversion framework is
studied in [36]. Besides, there are hybrid methods [31, 30] that combine
conversion and direct training. Tandem Learning [37] leverages the gradient
from ANN to update SNN during training. The first two columns of Table 1
present the technical ingredients of different conversion methods for the
training phase. Recent work [10, 36] shows that, outlier elimination (OE) in
ANN activations can be implemented by applying _clipping_ operation after the
Rectified Linear Unit (ReLU). Based on this, [23, 6] further minimise the
quantisation error by Quantisation-aware (QA) training. Different from the
above methods, we develop a new regulariser to achieve the better performance
_without_ clipping, and moreover, noticeably, we are _free from_ applying QA
training.

For the inference phase, as indicated in the last two columns of Table 1, the
soft-reset mechanism [32] and the additive white noise to membrane potential
[10, 36, 6] can significantly increase the conversion efficiency. To the best
of our knowledge, there is no existing work on cutoff in the inference phase,
and our confidence-based method is the first of its kind.

##  3 Preliminary

In this section, we discuss the event-based input in spiking neuron and
introduce the ANN-to-SNN conversion. To facilitate the analysis, we use bold
symbol to represent vector, $l$ to denote the layer index, and $i$ to denote
the index of elements. For example, $\boldsymbol{a}^{l}$ is a vector and
$a_{i}^{l}$ is the $i$-th element in $\boldsymbol{a}^{l}$. Inference time $t$
represents the time length of input. $T_{total}$ denotes the maximum time
length of input and it can be various depending on dataset.
$\boldsymbol{W}^{l}$ is weight matrix at the $l$-th layer.

###  3.1 Integrate-and-fire model

Figure 2: Inference in integrate-and-fire (IF) neuron with _reset by
subtraction_ mechanism.

Conversion-based SNN uses integrate-and-fire (IF) neuron as the basic
computing unit to approximate ReLU in ANN [36]. Fig. 2 illustrates the
inference process in IF neurons. The input spike train $X_{i}(t)$ charges the
membrane potential $V_{i}(t)$ with weighted current. The weighted current and
bias current are translated from the weight $\boldsymbol{W}^{l}$ and bias
$\boldsymbol{b}^{l}$ in ANN. When $V_{i}(t)$ reaches the threshold $V_{thr}$,
the neuron will generate a spike and then reset the $V_{i}(t)$ by subtracting
$V_{thr}$. The _reset by subtraction_ mechanism was firstly suggested in [32]
to reduce information loss during inference. The dynamics of IF neuron can be
described as

$\boldsymbol{V}^{l}(t)=\left\\{\begin{array}[]{ll}\boldsymbol{V}^{l}(t-1)+\boldsymbol{Z}^{l}(t)-\boldsymbol{\theta}^{l}(t)V^{l}_{thr}&l>1\\\
\boldsymbol{Z}^{1}(t)&l=1\end{array}\right.$ (1)

where $\boldsymbol{\theta}^{l}(t)$ is a step function i.e.,
$\theta_{i}^{l}(t)=1$ if $V_{i}^{l}(t-1)+Z_{i}^{l}(t)\geq V^{l}_{thr}$ and
$\theta_{i}^{l}(t)=0$ otherwise. $\boldsymbol{Z^{l}}(t)$ is the input current
such that

$\displaystyle\boldsymbol{Z}^{l}(t)=\boldsymbol{W}^{l}\boldsymbol{\theta}^{l-1}(t)+\boldsymbol{b}^{l}\hskip
14.22636pt\text{ when }l>1.$ (2)

For the event-based inputs (e.g., from a DVS sensor) , $\boldsymbol{Z}^{l}(t)$
at the first layer, i.e., $\boldsymbol{Z}^{1}(t)$, can be initialised as

$\displaystyle\boldsymbol{Z}^{1}(t)=\boldsymbol{W}^{1}\boldsymbol{X}(t)+\boldsymbol{b}^{1}$
(3)

where $\boldsymbol{X}(t)$ is the time-dependent spike train, i.e., the input
may change the charging current with time during the inference. To consider
the temporal information, we split the spike train into $F$ frames and
duration of each frame is equal to $T=T_{total}/F$, where
$F\in\mathbb{Z^{+}}$. We write $\boldsymbol{\bar{X}_{f}}$ to represent the
average spiking rate of $f$-th frame, i.e.,
$\boldsymbol{\bar{X}}_{f}=1/N_{\max}\sum_{t=T\cdot(f-1)}^{T\cdot
f}\boldsymbol{X}(t)$ such that $N_{\max}$ is the maximum spikes of all
training frames in dataset $D$. The spiking resolution of $X(t)$ can be
roughly computed as $S_{r}=N_{\max}/T$. For event-based input, SNN can
manifest faster inference due to immediate response after receiving the first
spike, and it completes the inference whenever the spike train ends, i.e., at
$T_{total}$. The event-based benchmarks are further introduced in Section 5.1.
This characteristic makes it possible that the inference time is dynamic for
different inputs. In this paper, with the cutoff technique as in Section 4.3,
we will show that the average latency of the inference in SNN can be further
reduced (to some $t\leq T_{total}$).

###  3.2 Temporal training

Regarding the direct training of SNN in [14, 11], the resulted SNN can make
decision by averaging output spikes of consecutive frames. Assuming that the
inference of each frames is independent, such process can be approximated in
ANN training by letting the loss function be

$L_{TT}=\frac{1}{F}\sum_{f=1}^{F}L_{CE}(\boldsymbol{Y}_{f},\boldsymbol{\hat{Y}})$
(4)

where $\boldsymbol{Y}_{f}$ is output of $\boldsymbol{\bar{X}}_{f}$ after
softmax, $\boldsymbol{\hat{Y}}$ is the ground truth and $L_{CE}$ is cross-
entropy loss. Temporal training loss ($L_{TT}$) was suggested in [11] that
achieves better generalisation. To simplify the theoretical analysis, we let
$F=1$ in Section 3.3 & 4. The further explanation of temporal training is
given in supplementary, including the impact of $F$ on ANN training and
extension of theories to $F>1$. To ensure the independence between frames, the
membrane potential of hidden layer is reset after each frame, while that of
output layer is reset after the last frame, which is feasible in hardware
implementation [15, 18].

###  3.3 ANN-to-SNN conversion

The conversion method is mainly based on integrate-and-fire (IF) neuron, which
generates spikes depending on positive accumulated current, corresponding to
ReLU activation in ANN. An existing conversion method [36] uses current
normalisation methods by letting

$\displaystyle\frac{1}{T\cdot
S_{r}}\sum_{t=0}^{T}\boldsymbol{Z}^{1}(t)=\boldsymbol{a}^{1}$ (5)

where $\boldsymbol{a}^{1}$ is the output of ReLU activation at the first layer
of ANN. The spiking rate of each SNN neuron at layer $l$ is defined as
$\boldsymbol{r}^{l}(t)=\boldsymbol{N}^{l}(t)/t$, where $\boldsymbol{N}^{l}(t)$
is the number of spikes received up to time $t$ by neuron at layer $l$. The
relationship between spiking rate in SNN and activation in ANN has been
theoretically proved in [36], which gives

$\begin{array}[]{l}\boldsymbol{r}^{l}(t)=\frac{1}{V^{l}_{thr}}\Big{(}\boldsymbol{W}^{l}\boldsymbol{r}^{l-1}(t)+\boldsymbol{b}^{l}\Big{)}-\boldsymbol{\Delta}^{l}(t)\end{array}$
(6)

where
$\boldsymbol{\Delta}^{l}(t)\triangleq{\boldsymbol{V}^{l}(t)}/{(tS_{r}V^{l}_{thr})}$
represents the residual spiking rate. The spiking rate at the first layer can
be initialised as
$\boldsymbol{r}^{1}(t)=\boldsymbol{a}^{1}/V^{1}_{thr}-\boldsymbol{\Delta}^{1}(t)$.
Note that, we use $tS_{r}$ to represent the timestep in [36]. Then, the
current normalisation can be achieved by

$\tilde{\boldsymbol{W}^{l}}\leftarrow\boldsymbol{W}^{l},~{}\tilde{\boldsymbol{b}^{l}}\leftarrow\frac{1}{\lambda^{l-1}}\boldsymbol{b}^{l},~{}V^{l}_{thr}\leftarrow\frac{\lambda^{l}}{\lambda^{l-1}}$
(7)

where $\lambda^{l}$ be the maximum value of the activation at layer $l$. For
temporal training, the temporal input frames share the same $\lambda^{l}$.

##  4 Methods

We introduce two novel techniques: one is for the training and the other for
the inference. Section 4.1 presents the theoretical underpinning of the
regulariser, which in turn is introduced in Section 4.2. This is followed by
the introduction of cutoff mechanism in Section 4.3 for the inference.

###  4.1 Anytime optimal inference SNNs

Figure 3: Graphic illustration of the desired spiking rate
$\boldsymbol{r}_{d}^{l}$ and spiking rate $\boldsymbol{r}^{l}(t)$

The regulariser is based on an investigation into the design of AOI-SNNs. An
AOI-SNN is able to perform optimally under different settings on the inference
time for processing an input. When $t$ is large enough to make
$\boldsymbol{\Delta}^{l}(t)$ negligible, we define the desired spiking rate as
follows:

$\begin{array}[]{l}\boldsymbol{r}_{d}^{l}=\boldsymbol{a}^{l}/V^{1}_{thr}\end{array}$
(8)

We start from establishing a theoretical underpinning between spiking rate
$\boldsymbol{r}^{l}(t)$ and its desired value $\boldsymbol{r}_{d}^{l}$. Let
$\phi^{l}$ denote the angle between $\boldsymbol{r}_{d}^{l}$ and
$\boldsymbol{r}^{l}(t)$. Then, we follow [4] to use cosine similarity between
$\boldsymbol{r}_{d}^{l}$ and $\boldsymbol{r}^{l}(t)$, i.e., $\cos(\phi^{l})$,
for the measurement of the performance of SNN by $t$. Actually, [4] shows that
the cosine similarity between full precision and quantised neural network has
a high correlation with the final accuracy of the quantised neural network.
Similarly, we expect that higher cosine similarity between $r^{l}(t)$ and
$r^{l}_{d}$ can result in less accuracy drop by $t$.

The following theorem states that, for any $t$, the performance of the SNN is
of negative correlation with threshold $V_{thr}^{l}$, and positive correlation
with $L_{2}$ norm over $\boldsymbol{a}^{l}$, as stated in the below theorem.

###### Theorem 4.1

For any inference time, assuming that the residual spiking rate
$\boldsymbol{\Delta}^{l}(t)$ is independent from $\boldsymbol{r}_{d}^{l}$, the
cosine similarity between $\boldsymbol{r}_{d}^{l}$ and $\boldsymbol{r}^{l}(t)$
is inversely proportional to the ratio of threshold to average accumulated
current,

$\cos(\phi^{l})\propto\Big{(}\sqrt{n^{l}}\frac{V^{l}_{thr}}{\|\boldsymbol{a}^{l}\|_{2}}\Big{)}^{-1}$

where $n^{l}$ is the dimension of $\boldsymbol{a}^{l}$ and
$\|\boldsymbol{a}^{l}\|_{2}/\sqrt{n^{l}}$ denotes the average accumulated
current.

We give a proof sketch of the theorem. Because $\Delta^{l}(t)$ is independent
from $\boldsymbol{r}_{d}^{l}$, the angle between these two vectors tends to be
$\pi/2$ at high dimension. Then, by
$\boldsymbol{r}^{l}(t)=\boldsymbol{r}_{d}^{l}-\boldsymbol{\Delta}^{l}(t)$, we
get a right angle triangle with $\boldsymbol{r}_{d}^{l}$ and $\Delta^{l}(t)$
as the legs, and $\boldsymbol{r}^{l}(t)$ as the hypotenuse, as illustrated in
Fig. 3. Moreover, we have

$\cos(\phi^{l})=\frac{\|\boldsymbol{r}_{d}^{l}\|_{2}}{\|\boldsymbol{r}^{l}(t)\|_{2}}\geq\frac{\|\boldsymbol{r}_{d}^{l}\|_{2}}{\|\boldsymbol{r}_{d}^{l}\|_{2}+\|\boldsymbol{\Delta}^{l}(t)\|_{2}}$
(9)

We are interested in increasing the lower bound of Equation 9, so that we have
greater $\cos(\phi^{l})$ for different $t$. Combining with Equations (6) and
(8), we have

$\begin{split}\cos(\phi^{l})&\geq\frac{\|\boldsymbol{a}^{l}/V^{l}_{thr}\|_{2}}{\|\boldsymbol{a}^{l}/V^{l}_{thr}\|_{2}+\|\boldsymbol{V}^{l}(t)/{(tS_{r}V^{l}_{thr})\|_{2}}}\\\
&=\frac{\|\boldsymbol{a}^{l}\|_{2}}{\|\boldsymbol{a}^{l}\|_{2}+\|\boldsymbol{V}^{l}(t)/(tS_{r})\|_{2}}\end{split}$
(10)

Assuming that elements in $\boldsymbol{V}^{l}(t)$ satisfy uniform distribution
over the time $t$ and they are in $[0,V_{thr}]$, we can derive
$\mathbb{E}(\|V(t)/(tS_{r})\|_{2})\leq\sqrt{n^{l}}V_{thr}/(\sqrt{3}tS_{r})$
(proof in supplementary) Moreover, at high dimensions, the relative error made
as considering $\mathbb{E}(\|V(t)/(tS_{r})\|_{2})$ instead of the random
variable $\|V(t)/(tS_{r})\|_{2}$ becomes asymptotically negligible [5, 4].
Therefore, Equation 10 can be computed with the following lower bound

$\begin{split}\cos(\phi^{l})&\geq\frac{\|\boldsymbol{a}^{l}\|_{2}}{\|\boldsymbol{a}^{l}\|_{2}+\sqrt{n^{l}}V_{thr}^{l}/(\sqrt{3}tS_{r})}\\\
&=\frac{\sqrt{3}tS_{r}}{\sqrt{3}tS_{r}+\sqrt{n^{l}}V_{thr}^{l}/\|\boldsymbol{a}^{l}\|_{2}}\end{split}$
(11)

which explicitly explains that (1) the increase of $t$ to
$t\gg\sqrt{n^{l}}V_{thr}^{l}/\|\boldsymbol{a}^{l}\|_{2}$ can increase the
lower bound and (2) it is possible to minimise term
$\sqrt{n^{l}}V_{thr}^{l}/\|\boldsymbol{a}^{l}\|_{2}$ for developing an SNN
with optimised performance at any time during the inference. In other words,
an AOI-SNN expects a good (small) ratio of threshold voltage $V_{thr}^{l}$ to
average accumulated current, i.e., $\|\boldsymbol{a}^{l}\|_{2}/\sqrt{n^{l}}$,
while not degrading SNN classification performance. The point (2) corresponds
with the theorem.

###  4.2 Regulariser for outlier elimination (ROE)

This section shows how to design a regulariser based on Theorem 4.1. Recall
from Equation (7) that $V_{thr}^{l}$ is determined by $\lambda^{l}$ and
$\lambda^{l-1}$, where $\lambda^{l}$ is the maximum value of activation in the
$l$-th layer. To simplify the complexity of optimisation, the impact of
$1/\lambda^{l-1}$ is omitted and the ratio of threshold to expected current
approximately becomes proportional to
$\lambda^{l}/\|\boldsymbol{a}^{l}\|_{2}$. Therefore, we design a regulariser
to minimise term $\lambda^{l}/\|\boldsymbol{a}^{l}\|_{2}$ to develop an AOI-
SNN. Firstly, we use matrix $\boldsymbol{A}^{l}$ to represent a batch of
$\boldsymbol{a^{l}}$ during training. Secondly, we simply use maximum value in
$\boldsymbol{A}^{l}$ to approximate $\lambda^{l}$, i.e.,
$\lambda^{l}\approx\|\boldsymbol{A}^{l}\|_{\max}$. Then, we write
$\|\boldsymbol{A}^{l}\|_{2,q}=(\sum_{j}(\sum_{i}A_{ij}^{2})^{q/2})^{1/q}$ to
denotes the $L_{2,p}$ over $\boldsymbol{A}^{l}$, where $A_{ij}^{l}$ presents
$j$-th $\boldsymbol{a_{i}}^{l}$ in the batch and $q\in\mathbb{Z}$. Finally, we
can let the penalty term be the ratio between $\|\boldsymbol{A}^{l}\|_{\max}$
and $\|\boldsymbol{A}^{l}\|_{2,q}$ with scale constant $\sqrt{n^{l}}$, i.e.,

$R(\boldsymbol{A}^{l})=\sqrt{n^{l}}\frac{\|\boldsymbol{A}^{l}\|_{\max}}{\|(\boldsymbol{A}^{l})\|_{2,q}}$
(12)

We let $q$ be $-\infty$ so that the penalty term can focus on the inputs with
relatively small accumulated current in the batch. The final training
objective is

$L_{TT}+\alpha\sum_{l}\ln{(R(\boldsymbol{A}^{l})})$ (13)

where $\alpha$ is a hyper-parameter to balance two loss terms. Logarithm is
applied to reduce the impact from extremely large value. The regularisation-
based training is to train an ANN based on $\boldsymbol{\bar{X}}_{f}$
resulting in an SNN, then SNN operates with the event-based input (Equation
3). A small $R(\boldsymbol{A})^{l}$ implies that it is less possible for
$\lambda^{l}$ to be an outlier and $\|\boldsymbol{a^{l}}\|_{2}$ is generally
large.

###  4.3 Cutoff mechanism to reduce inference time

Thanks to the asynchronous working mechanism, event-driven SNNs can predict
when only part of the spike train is processed. Nevertheless, a naive cutoff
on the length of spike train (or the sampling time of event sensor) can easily
result in accuracy loss. In this section, we suggest a principled method to
determine the inference time. Technically, a new metric, called confidence
rate and denoted as $C(\hat{t},D\\{S_{gap}>\beta\\})$, is defined based on the
statistical characteristics of processing a set $D$ of inputs with respect to
the discrete inference time $\hat{t}$ and $S_{gap}$. $S_{gap}>\beta$ operates
as a condition to identify the samples in $D$ that are suitable for cutoff.
Actually, we are able to plot a curve of confidence rate
$C(\hat{t},D\\{S_{gap}>\beta\\})$ with respect to the time $\hat{t}$ and
$\beta$, respectively. During the processing of an individual input
$\boldsymbol{X}$, we will monitor another variable $S_{gap}$, and once it is
able to ensure the confidence rate can reach certain degree, an early cutoff
signal can be sent (see Fig. 1). The following provides the details.

We write $\boldsymbol{X}[\hat{t}]=\sum_{t=0}^{\hat{t}}\boldsymbol{X}(t)$ to
denote the accumulation of $\boldsymbol{X}(t)$ from 0 up to $\hat{t}$. Then,
we let $f(\boldsymbol{X}[\hat{t}])$ return the prediction of $f$ based on the
partial input $\boldsymbol{X}[\hat{t}]$. Based on this, we define a function

$g(\boldsymbol{X})=\arg\min_{\hat{t}}\\{\forall\hat{t}_{1}>\hat{t}:\mathbf{1}(f(\boldsymbol{X}[\hat{t}_{1}])=\boldsymbol{y})\\}$
(14)

to express the earliest time from which the model $f$ is able to confidently
and correctly classify according to the partial input. $\mathbf{1}(\cdot)$ is
the indicator function, i.e., $\mathbf{1}(x_{1}=x_{2})=1$ and
$\mathbf{1}(x_{1}\neq x_{2})=0$.
$\mathbf{1}(f(\boldsymbol{X}[\hat{t}_{1}])=\boldsymbol{y})$ suggests that
$f(\boldsymbol{X}[\hat{t}])$ is the same as the ground truth $\boldsymbol{y}$.
Then, recall that $\boldsymbol{N}^{L}(t)$ is the number of spikes received by
$t$ by the output layer $L$. We write $Top_{k}(\boldsymbol{N}^{L}(t))$ as the
top $k$ spikes that occur in some neuron of layer $L$. Then, we let

$S_{gap}=Top_{1}(\boldsymbol{N}^{L}(t))-Top_{2}(\boldsymbol{N}^{L}(t))$ (15)

be a variable denoting the gap of top-1 and top-2 number of spikes. A large
$S_{gap}$ implies little possibility of switching the prediction results
during inference. Then, we let $D\\{\cdot\\}$ denote the inputs in subset of
$D$ that satisfy a certain condition. Now, we can define the confidence rate
as follows:

$\begin{split}&\textit{Confidence rate: }C(\hat{t},D\\{S_{gap}>\beta\\})=\\\
&\frac{1}{|D\\{S_{gap}>\beta\\}|}\sum_{\boldsymbol{X}\in
D\\{S_{gap}>\beta\\}}(g(\boldsymbol{X})\leq\hat{t})\end{split}$ (16)

which intuitively computes the percentage of inputs in $D$ that can achieve
the prediction success on or before a prespecified time $\hat{t}$, i.e.,
$g(\boldsymbol{X})\leq\hat{t}$. $|D\\{S_{gap}>\beta\\}|$ denotes the number of
samples in $D$ satisfying the condition. It is not hard to see that, when
$\hat{t}=0$, $C(\hat{t},D\\{S_{gap}>\beta)$ is also $0$, and with the increase
of time $\hat{t}$, $C(\hat{t},D\\{S_{gap}>\beta)$ will also increase until
reaching $1$. Our algorithm searches for a minimum $\beta\in\mathbb{Z^{+}}$ at
a specific $\hat{t}$, as expressed in the following optimisation objective:

$\arg\min_{\beta}C(\hat{t},D\\{S_{gap}>\beta\\})\geq 1-\epsilon$ (17)

where $\epsilon$ is a pre-specified constant such that $1-\epsilon$ represents
an acceptable level of confidence for activating cut-off, and a set of $\beta$
is extracted under different $\hat{t}$ using training samples.

(a) Confidence w.r.t time ratio.

(b) Confidence w.r.t $\beta$ at 0.125 $T_{total}$, for SNN-ROE

Figure 4: Evaluation of confidence on Cifar10-DVS

Equation 16 is visualised in Fig. 4 that shows the impact of inference time
and $\beta$ on confidence. Time ratio denotes the normalised inference time.
We characterise the confidence metric with training samples and eventually use
testing samples for evaluation. Note that, on Cifar10-DVS, all model achieves
100% for training accuracy, however, they perform differently on confidence.
With regularisation, SNN-ROE can further improve the confidence than SNN-QA,
e.g., it is 0.01 higher at $0.125T_{total}$ and 0.07 higher at
$0.25T_{total}$. Therefore, SNN-ROE can have a better performance at any time
during the inference, as there are more inputs join the early cutoff. Fig.
4(b) presents that the input with large $S_{gap}$ has more consistent
prediction over time, which supports the use of $S_{gap}>\beta$ as the cutoff
condition.

##  5 Experiment

We implement the ROE and conduct an extensive set of experiments to validate
it. We consider its comparison with the state-of-the-art CNN-to-SNN conversion
methods. In this section, ‘SNN-QA’ denotes the method in [6], which includes
both COE and QA during training, and outperforms the other methods on image
input. In contrast, ‘SNN-COE’ denotes the SNN with only COE. Our proposed
method is denoted by ‘SNN-ROE’. To reduce the accuracy loss during inference,
we followed [36, 6] to add extra current $V^{l}_{thr}/2$ to each neuron.

Our method is validated against three event-based datasets, e.g., Cifar10-DVS
[22], N-Caltech101 [28] and DVS128 Gesture [2]. As [6] did not cover the
event-based input, we replicate their work as our baseline for comparison and
set the quantisation length of SNN-QA to 16 for all datasets, which yields
optimal performance. Note that, we use original input from DVS camera without
any pre-processing for the inference so that SNN can remain asynchronous to
the input events.

For SNN training, we inherit the conversion methods and most of the notation
from [36, 6]. Particularly, the relationship between ReLU and IF is from [36]
and the relationship between quantised ReLU and IF is from [6]. Moreover, ROE
minimisation is based on [36] and operates as a penalty term during ANN
training. Additionally, we conduct an experiment that cutoff can be extended
to other neuron model, such as Leaky integrate-and-fire (LIF) model. We train
such SNN directly using the surrogate gradient and loss function from [14,
13], denoted by ’SNN-LIF’. Normally, SNN-LIF has a pre-configured maximum
timestep during training. To implement cutoff in SNN-LIF, we can compute the
accumulated events with respect to timestep to characterise the confidence
metric (recall the Equations in Section 4.3). Further details of training
setup, e.g., network architecture, are described in supplementary.

###  5.1 Event-based datasets

The samples in the event-based datasets record the event addresses with on/off
events over a period of time. For Cifar10-DVS, it consists of 10,000 samples
extracted from Cifar10 [20]. Each sample has 128$\times$128 spatial
resolution. The length of each spike train is less equal to $1.3s$. For
N-Caltech101, it has 8709 samples categorised into 101 classes. The number of
samples in each class ranges from 31 to 800. The length of each spike train is
about $0.3s$. The width in x-direction does not exceed 240 pixels and in
y-direction does not exceed 180 pixels. For this two datasets, we use 90%
samples in each class for training and 10% for testing. DVS128 Gesture
consists of 1341 samples with 11 categories. Each sample is repetitive over
$6.0s$.

###  5.2 Experimental results

(a) Accuracy w.r.t time ratio

(b) Right predictions ratio w.r.t time ratio

Figure 5: Comparison of SNN on Cifar10-DVS. (a) Accuracy for full-length input
is shown in bracket and cutoff is based on
$\epsilon=\\{0.05,0.04.0.03,0.02,0.01,0.00\\}$. The results around red dash
line are summarised in Table 2. (b) The statistic data is extracted from
testing samples. The ratio of total right predictions is in bracket for
$\epsilon=0.00$.

This section presents a comparison between SNN-ROE, SNN-QA and SNN-COE on
accuracy w.r.t time ratio and performance improvement after cutoff. The
inference time $t=\text{Time Ratio}\times T_{total}$, where $T_{total}$ is
equal to 1.3$s$, 0.3$s$ and 1.2$s$ for Cifar10-DVS, N-Caltech101 and DVS128
Gesture respectively.

It is not hard to see, with cutoff, the performance of all models is improved
in Fig. 5(a) and Fig. 6. For example, the accuracy curve moves above its
original curve, which means that same accuracy can have less inference time
for same model. Fig. 5(b) indicates that the inference time of SNN becomes
adaptive after deploying cutoff across all tested models.

(a) Accuracy w.r.t time ratio, on N-Caltech101

(b) Accuracy w.r.t time ratio, on DVS128 Gesture

Figure 6: Comparison of SNN on different datasets. Accuracy for full-length
input is shown in bracket and cutoff is based on
$\epsilon=\\{0.05,0.04.0.03,0.02,0.01,0.00\\}$. The results around red dash
line are summarised in Table 2.

Thanks to the increase of confidence (recall the results in Fig. 4(a)), SNN-
ROE can have general higher accuracy before the time point (red dash line) and
it shows consistent results in different datasets. The confidence evaluation
for N-Caltech101 and DVS128 Gesture is provided in supplementary. It has been
argued in [41] that the temporal information in Cifar10-DVS is not the
dominant information, which is similar in N-Caltech101. Unlike N-Caltech101
and Cifar10-DVS, the correlation between the temporal events in DVS128 Gesture
is high. This phenomenon can also be observed from temporal training
experiment in supplementary. Therefore, we set $F=1$ on Cifar10-DVS and
N-Caltech101 for efficient training and $F=4$ on DVS128 Gesture to incorporate
temporal information. Similarly, we let maximum timestep in SNN-LIF be 8 for
Cifar10-DVS and N-Caltech101, and 24 for DVS128 Gesture. Moreover, the result
in Fig. 6(b) shows that cutoff can also improve SNN trained with $F>1$.

Table 2: Comparison with the exiting work on event-based datasets, using
direct training (DT), ANN-to-SNN conversion (CV) or graph neural network
(Graph). We present SNN-LIF, SNN-COE, SNN-QA and SNN-ROE with cutoff for
similar inference $t$.

Models | Methods | Acc. |  Ave. $t$
---|---|---|---
[26] | DT | 77.27% |  1.3$s$
[24] | Graph | 68.40% | -
[13] | DT | 74.40% |  1.3$s$
[14] | DT | 74.80% |  1.3$s$
[21] | CV | 66.40% | -
SNN-LIF (w/ cutoff) | DT | 74.00% |  0.5$s$
SNN-COE (w/ cutoff) | CV | 78.60% |  0.5$s$
SNN-QA (w/ cutoff) | CV | 79.90% |  0.5$s$
SNN-ROE (w/ cutoff) | CV | 81.20% | 0.5$s$
|  |  |
(a) Accuracy and latency on Cifar10-DVS.

Models | Methods | Acc. |  Ave. $t$
---|---|---|---
[33] | Graph | 66.80% | -
[24] | Graph | 76.10% | -
[35] | DT | 71.20% | -
[27] | DT | 74.50% |  0.3$s$
SNN-LIF (w/ cutoff) | DT | 77.54% |  0.1$s$
SNN-COE (w/ cutoff) | CV | 74.72% |  0.1$s$
SNN-QA (w/ cutoff) | CV | 78.00% |  0.1$s$
SNN-ROE (w/ cutoff) | CV | 78.00% | 0.1$s$
|  |  |
(b) Accuracy and latency on N-Caltech101.

Models | Methods | Acc. | Ave. $t$
---|---|---|---
[41] | DT | 97.57% | 1.2$s$
[40] | DT | 97.56% | 1.5$s$
[13] | DT | 97.92% | 6.0$s$
[14] | DT | 97.57% | 6.0$s$
[21] | CV | 95.56% | -
SNN-LIF (w/ cutoff) | DT | 96.69% | 0.8$s$
SNN-COE (w/ cutoff) | CV | 97.34% | 0.8$s$
SNN-QA | CV | 93.56% | 0.9$s$
SNN-ROE (w/ cutoff) | CV | 98.10% | 0.8$s$
|  |  |
(c) Accuracy and latency on DVS128 Gesture.

We collect the results around the time point in Table 2, which have similar
inference time, to show the performance of each model on cutoff. SNN-ROE
achieves the superior performance on accuracy and latency.

##  6 Conclusions

This paper promotes anytime optimal inference SNNs (AOI-SNNs), which maintain
the optimal performance throughout the inference stage, and therefore are
suitable for event-driven inputs such as those from dynamic vision sensor or
dynamic audio sensor. Two technical novelties are proposed to optimise the
attainment of AOI-SNNs, one for the training stage and the other for the
inference stage. Our experiments demonstrate the superior performance with
respect to the accuracy and latency, comparing to the state-of-the-art.
Moreover, the proposed cutoff is hardware-friendly and compatible with
different SNN neuron models (e.g. IF and LIF), as long as the prediction is
based on the number of output spikes. It provides a new dimension for
efficient computing in SNN and also highlights the advantage in SNN, which is
more adaptable in inference than traditional ANN.

## References

  * [1] F. Akopyan, J. Sawada, A. Cassidy, R. Alvarez-Icaza, J. Arthur, P. Merolla, N. Imam, Y. Nakamura, Nam Datta, P., G.J., and B. Taba.  Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip.  EEE transactions on computer-aided design of integrated circuits and systems, 34(10):1537–1557, 2015.
  * [2] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, et al.  A low power, fully event-based gesture recognition system.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7243–7252, 2017.
  * [3] Jithendar Anumula, Daniel Neil, Tobi Delbruck, and Shih-Chii Liu.  Feature representations for neuromorphic audio spike streams.  Frontiers in neuroscience, 12:23, 2018.
  * [4] Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry.  Scalable methods for 8-bit training of neural networks.  Advances in neural information processing systems, 31, 2018.
  * [5] Gérard Biau and David M Mason.  High-dimensional $p$ $p$-norms.  In Mathematical statistics and limit theorems, pages 21–40. Springer, 2015.
  * [6] Tong Bu, Wei Fang, Jianhao Ding, PENGLIN DAI, Zhaofei Yu, and Tiejun Huang.  Optimal ANN-SNN conversion for high-accuracy and ultra-low-latency spiking neural networks.  In International Conference on Learning Representations, 2022.
  * [7] M. Davies, N. Srinivasa, T.H. Lin, G. Chinya, Y. Cao, S.H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, and Y. Liao.  Loihi: A neuromorphic manycore processor with on-chip learning.  IEEE Micro, 38(1):82–99, 2018.
  * [8] Tobi Delbrück, Bernabe Linares-Barranco, Eugenio Culurciello, and Christoph Posch.  Activity-driven, event-based vision sensors.  In Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pages 2426–2429. IEEE, 2010.
  * [9] J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  2009 IEEE conference on computer vision and pattern recognition, 2(248-255):1430, 6 2009.
  * [10] Shikuang Deng and Shi Gu.  Optimal conversion of conventional artificial neural networks to spiking neural networks.  In International Conference on Learning Representations, 2021.
  * [11] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu.  Temporal efficient training of spiking neural network via gradient re-weighting.  In International Conference on Learning Representations, 2022.
  * [12] P.U. Diehl, D. Neil, J. Binas, M. Cook, S.C. Liu, and M. Pfeiffer.  Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing.  International Joint Conference on Neural Networks, pages 1–8, 7 2015.
  * [13] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian.  Deep residual learning in spiking neural networks.  Advances in Neural Information Processing Systems, 34:21056–21069, 2021.
  * [14] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian.  Incorporating learnable membrane time constant to enhance learning of spiking neural networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2661–2671, 2021.
  * [15] Charlotte Frenkel and Giacomo Indiveri.  Reckon: A 28nm sub-mm2 task-agnostic spiking recurrent neural network processor enabling on-chip learning over second-long timescales.  In 2022 IEEE International Solid-State Circuits Conference (ISSCC), volume 65, pages 1–3. IEEE, 2022.
  * [16] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al.  Event-based vision: A survey.  IEEE transactions on pattern analysis and machine intelligence, 44(1):154–180, 2020.
  * [17] B. Han, G. Srinivasan, and K. Roy.  Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13558–13567, 2020.
  * [18] Alireza Khodamoradi, Kristof Denolf, and Ryan Kastner.  S2n2: A fpga accelerator for streaming spiking neural networks.  In The 2021 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pages 194–205, 2021.
  * [19] Junho Kim, Jaehyeok Bae, Gangin Park, Dongsu Zhang, and Young Min Kim.  N-imagenet: Towards robust, fine-grained object recognition with event cameras.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2146–2156, 2021.
  * [20] A. Krizhevsky and G. Hinton.  Learning multiple layers of features from tiny images.  2009 IEEE conference on computer vision and pattern recognition, 2009.
  * [21] Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca.  Efficient processing of spatio-temporal data streams with spiking neural networks.  Frontiers in Neuroscience, 14:439, 2020.
  * [22] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi.  Cifar10-dvs: an event-stream dataset for object classification.  Frontiers in neuroscience, 11:309, 2017.
  * [23] Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu.  A free lunch from ann: Towards efficient, accurate spiking neural networks calibration.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6316–6325. PMLR, 18–24 Jul 2021.
  * [24] Yijin Li, Han Zhou, Bangbang Yang, Ye Zhang, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang.  Graph-based asynchronous event processing for rapid object recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 934–943, 2021.
  * [25] Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck.  A 128$\times$128 120 db 15$\mu$s latency asynchronous temporal contrast vision sensor.  IEEE journal of solid-state circuits, 43(2):566–576, 2008.
  * [26] Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo.  Training high-performance low-latency spiking neural networks by differentiation on spike representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12444–12453, 2022.
  * [27] Nico Messikommer, Daniel Gehrig, Antonio Loquercio, and Davide Scaramuzza.  Event-based asynchronous sparse convolutional networks.  In European Conference on Computer Vision, pages 415–431. Springer, 2020.
  * [28] Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor.  Converting static image datasets to spiking neuromorphic datasets using saccades.  Frontiers in neuroscience, 9:437, 2015.
  * [29] M. Pfeiffer and T. Pfeil.  Deep learning with spiking neurons: opportunities and challenges.  Frontiers in neuroscience, 12:774, 2018.
  * [30] Nitin Rathi and Kaushik Roy.  Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization.  IEEE Transactions on Neural Networks and Learning Systems, 2021\.
  * [31] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy.  Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation.  In International Conference on Learning Representations, 2020.
  * [32] B. Rueckauer, I.A. Lungu, M. Hu, Y.and Pfeiffer, and S.C. Liu.  Conversion of continuous-valued deep networks to efficient event-driven networks for image classification.  Frontiers in neuroscience, 11:682, 2017.
  * [33] Simon Schaefer, Daniel Gehrig, and Davide Scaramuzza.  Aegnn: Asynchronous event-based graph neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12371–12381, 2022.
  * [34] Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy.  Going deeper in spiking neural networks: Vgg and residual architectures.  Frontiers in neuroscience, 13:95, 2019.
  * [35] Xueyuan She, Saurabh Dash, and Saibal Mukhopadhyay.  Sequence approximation using feedforward spiking neural network for spatiotemporal learning: Theory and optimization methods.  In International Conference on Learning Representations, 2022.
  * [36] Dengyu Wu, Xinping Yi, and Xiaowei Huang.  A little energy goes a long way: Build an energy-efficient, accurate spiking neural network from convolutional neural network.  Frontiers in neuroscience, 16, 2022.
  * [37] Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang, Haizhou Li, and Kay Chen Tan.  Progressive tandem learning for pattern recognition with deep spiking neural networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [38] Y. Wu, L. Deng, G. Li, J. Zhu, and L. Shi.  Spatio-temporal backpropagation for training high-performance spiking neural networks.  Frontiers in neuroscience, 12(331), 5 2018.
  * [39] Y. Wu, L. Deng, G. Li, J. Zhu, Y. Xie, and L. Shi.  Direct training for spiking neural networks: Faster, larger, better.  In Proceedings of the AAAI Conference on Artificial Intelligence, 33:1311–1318, 7 2019.
  * [40] Zhenzhi Wu, Hehui Zhang, Yihan Lin, Guoqi Li, Meng Wang, and Ye Tang.  Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing.  IEEE Transactions on Neural Networks and Learning Systems, 2021\.
  * [41] Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, and Guoqi Li.  Temporal-wise attention spiking neural networks for event streams classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10221–10230, 2021.