U_{t}(x_{t})\rVert^{3})\lVert x_{t}-x^{*}\rVert$ (D.5)
$\displaystyle+4\gamma^{2}(\ell^{2}\lVert x_{t}-x^{*}\rVert^{4}+\lVert
U_{t}(x_{t})\rVert^{2}\lVert x_{t}-x^{*}\rVert^{2}),$ (D.6)

where we used in the second summand Eq. D.2 the Cauchy-Schwarz inequality,
Lipschitz continuity of the operator and the identinty $\lVert
x+y\rVert^{2}\leq 2\lVert x\rVert^{2}+2\lVert y\rVert^{2}$. For the third one
Eq. D.3 we used the identity $\lVert x+y\rVert^{4}\leq 8\lVert
x\rVert^{4}+8\lVert y\rVert^{4}$, Lipschitzness of the operator. For the
fourth one Eq. D.4 we used the quasi strong monotonicity of the operator. For
the firth one Eq. D.5 we used Cauchy-Schwarz inequality and the identity
$\lVert x+y\rVert^{2}\leq 2\lVert x\rVert^{2}+2\lVert y\rVert^{2}$ and
Lipschitzness of the operator. Thus in the right-hand side of the above
inequality we have constant terms, the $\lVert x_{t}-x^{*}\rVert^{4}$, $\lVert
x_{t}-x^{*}\rVert^{2}$ and $\lVert x_{t}-x^{*}\rVert$. Specifically, by
rearranging we get

$\displaystyle\lVert x_{t+1}-x^{*}\rVert^{4}\leq$ $\displaystyle\lVert
x_{t}-x^{*}\rVert^{4}(1+8\gamma^{2}\ell^{2}+8\gamma^{4}\ell^{4}-4\gamma\mu+16\gamma^{3}\ell^{3}+4\gamma^{2}\ell^{2})$
$\displaystyle+\lVert x_{t}-x^{*}\rVert^{2}(12\gamma^{2}\lVert
U_{t}(x_{t})\rVert^{2})$ $\displaystyle+\lVert
x_{t}-x^{*}\rVert(16\gamma^{3}\lVert U_{t}(x_{t})\rVert^{3}-4\lVert
x_{t}-x^{*}\rVert^{2}\langle U_{t}(x_{t}),x_{t}-x^{*}\rangle$
$\displaystyle+8\gamma^{4}\lVert U_{t}(x_{t})\rVert^{4}.$

Applying the expectation given the filtration $\mathcal{F}_{t}$ and setting
$\bar{\ell}=\max\\{\ell^{2},\ell^{3},\ell^{4}\\}$ we have

$\displaystyle\operatorname{\mathbb{E}}[\lVert
x_{t}-x^{*}\rVert^{4}\nonscript\,|\nonscript\,\mathopen{}\mathcal{F}_{t}]\leq$
$\displaystyle\operatorname{\mathbb{E}}[\lVert
x_{t+1}-x^{*}\rVert^{4}\nonscript\,|\nonscript\,\mathopen{}\mathcal{F}_{t}](1+16\bar{\ell}(\gamma^{2}+\gamma^{3}+\gamma^{4})-4\gamma\mu)$
$\displaystyle+\operatorname{\mathbb{E}}[\lVert
x_{t}-x^{*}\rVert^{2}\nonscript\,|\nonscript\,\mathopen{}\mathcal{F}_{t}](12\gamma^{2}\sigma^{2})$
$\displaystyle+\operatorname{\mathbb{E}}[\lVert
x_{t}-x^{*}\rVert\nonscript\,|\nonscript\,\mathopen{}\mathcal{F}_{t}](16\gamma^{3}{\delta_{\text{\tiny{
KYRT}}}}^{3})+8\gamma^{4}{\delta_{\text{\tiny{ KYRT}}}}^{4}.$

By choosing step-size such that

$\left\\{\begin{matrix}\gamma<1&\text{ for simplicity }\\\
16\bar{\ell}(\gamma^{2}+\gamma^{3}+\gamma^{4})-4\gamma\mu<-2\gamma\mu&\end{matrix}\right.$

we have that

$\displaystyle\operatorname{\mathbb{E}}[\lVert
x_{t+1}-x^{*}\rVert^{4}\nonscript\,|\nonscript\,\mathopen{}\mathcal{F}_{t}](2\gamma\mu)\leq$
$\displaystyle\operatorname{\mathbb{E}}[\lVert
x_{t}-x^{*}\rVert^{2}\nonscript\,|\nonscript\,\mathopen{}\mathcal{F}_{t}](12\gamma^{2}\sigma^{2})$
$\displaystyle+\operatorname{\mathbb{E}}[\lVert
x_{t}-x^{*}\rVert\nonscript\,|\nonscript\,\mathopen{}\mathcal{F}_{t}](16\gamma^{3}{\delta_{\text{\tiny{
KYRT}}}}^{3})+8\gamma^{4}{\delta_{\text{\tiny{ KYRT}}}}^{4}.$

Now consider $x\sim\pi_{\gamma}$ and let
$\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[\lVert
x-x^{*}\rVert^{k}]=\text{Mom}(k)$. Notice that the first moment is also
bounded by $\operatorname{\mathcal{O}}(\sqrt{\gamma/\mu})$ since from Eq. B.2
and Lipschitzness of the operator we have

$\lVert x_{t+1}-x^{*}\rVert^{2}\leq(1-2\mu\gamma+\gamma^{2}\ell^{2})\lVert
x_{t}-x^{*}\rVert^{2}+\lVert U_{t}(x_{t})\rVert^{2}$

Thus, combining all these we have

$\displaystyle\text{Mom}(4)2\mu\gamma\leq\text{Mom}(2)\operatorname{\mathcal{O}}(\gamma^{2})+\text{Mom}(1)\operatorname{\mathcal{O}}(\gamma^{3})+\operatorname{\mathcal{O}}(\gamma^{4}).$

equivalently

$\displaystyle\text{Mom}(4)\leq\text{Mom}(2)\operatorname{\mathcal{O}}(\gamma/\mu)+\text{Mom}(1)\operatorname{\mathcal{O}}(\gamma^{2}/\mu)+\operatorname{\mathcal{O}}(\gamma^{3}/\mu).$

But $\text{Mom}(2)\leq\operatorname{\mathcal{O}}(\gamma/\mu)$ and
$\text{Mom}(1)\leq\operatorname{\mathcal{O}}(\sqrt{\gamma/\mu})$, thus

$\text{Mom}(4)\leq\operatorname{\mathcal{O}}(\gamma^{2}/\mu^{2}),$

which implies that there exists $c\leq c_{0}\max\\{{\delta_{\text{\tiny{
KYRT}}}}^{3},{\delta_{\text{\tiny{ KYRT}}}}^{4},\sigma,\sigma^{2}\\}$ such
that

$\text{Mom}(4)\leq c\gamma^{2}/\mu^{2}.$

∎

######  Theorem D.2.

[Restated Theorem 2] Suppose Assumptions 1, 3, 2, 4, 5 and 7 hold. There
exists a threshold $\theta$ such that if $\gamma\in(0,\theta),$ (SGDA) admits
unique stationary distribution $\pi$, that depends on the choice of step-size,
and

$\operatorname{\mathbb{E}}_{x\sim\pi}[x]-x^{*}=\gamma\Delta(x^{*})+\operatorname{\mathcal{O}}(\gamma^{2}),$
(D.7)

where $\Delta(x^{*})$ is a vector independent of the choice of step-size
$\gamma$.

###### Proof.

Let
$\bar{x}=\int_{\mathbb{R}^{d}}x\pi_{\gamma}(x)\>dx=\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[x]$
and let
$\gamma<\min(\gamma_{\text{thresh}}^{\text{\ref{lem:moments}}},\gamma_{\text{thresh}}^{\text{\ref{thm:close-
app}}}):=\theta^{\prime}$ such that Lemma D.1 and Theorem C.1 hold. Assume
that we run (SGDA) $(x_{t})_{t\geq 0}$ and $x_{0}\sim\pi_{\gamma}$; since the
algorithm is initialized with the invariant distribution, then all the
iterations inevitably follow the invariant distribution. We start by applying
Taylor expansion, on the operator, of second and third order around the
solution $x^{*}$

$V(x)=\operatorname{\nabla}V(x^{*})\odot[x-x^{*}]+\frac{1}{2}\operatorname{\nabla}^{2}V(x^{*})\odot[x-x^{*}]^{2}+\text{Res}_{3}(x),$
(A) $V(x)=\operatorname{\nabla}V(x^{*})\odot[x-x^{*}]+\text{Res}_{2}(x),$ (B)

where $\text{Res}_{2}(x),\text{Res}_{3}(x)$ are the corresponding residuals of
the Taylor expansion for which it holds that
$\sup_{x\in\mathbb{R}^{d}}\\{\lVert\text{Res}_{3}(x)\rVert/\lVert
x-x^{*}\rVert^{3}\\}<\infty$ and
$\sup_{x\in\mathbb{R}^{d}}\\{\lVert\text{Res}_{2}(x)\rVert/\lVert
x-x^{*}\rVert^{2}\\}<\infty$. Notice also that

$\int_{x\in\mathbb{R}^{d}}\text{Res}_{3}(x)\pi_{\gamma}(x)\>dx<c_{3}\int_{x\in\mathbb{R}^{d}}\lVert
x-x^{*}\rVert^{3}\pi_{\gamma}(x)\>dx\leq
c_{3}\text{Mom}(3)\leq\operatorname{\mathcal{O}}(\gamma^{3/2}),$ (C)
$\int_{x\in\mathbb{R}^{d}}\text{Res}_{2}(x)\pi_{\gamma}(x)\>dx\leq
c_{2}\int_{x\in\mathbb{R}^{d}}\lVert x-x^{*}\rVert^{2}\pi_{\gamma}(x)\>dx\leq
c_{2}\text{Mom}(2)\leq\operatorname{\mathcal{O}}(\gamma).$ (D)

Additionally, by definition of (SGDA) we get that $x_{1}=x_{0}-\gamma
V(x_{0})-\gamma U_{0}(x_{0})$. Since $x_{0}\sim\pi_{\gamma}$ we have that
$x_{1}\sim\pi_{\gamma}$ and thus we have

$\operatorname{\mathbb{E}}_{x_{1}\sim\pi_{\gamma}}[x_{1}]=\operatorname{\mathbb{E}}_{x_{0}\sim\pi_{\gamma}}[x_{0}]-\gamma\operatorname{\mathbb{E}}_{x_{0}\sim\pi_{\gamma}}[V(x_{0})]-\gamma\operatorname{\mathbb{E}}_{x_{0}\sim\pi_{\gamma}}[U_{0}(x_{0})],$

which implies that

$\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[V(x)]=0.$ (E)

With these equations at hand, we proceed and take the expectation of (A) with
respect to the invariant distribution, combining also (C) and (E) and we get

$\operatorname{\nabla}V(x^{*})\odot[\bar{x}-x^{*}]+\dfrac{1}{2}\int_{x\in\mathbb{R}^{d}}\operatorname{\nabla}^{2}V(x^{*})\odot[x-x^{*}]^{2}\pi_{\gamma}(x)\>dx=\operatorname{\mathcal{O}}(\gamma^{3/2}).$
(D.8)

Again we focus on the first update of (SGDA) and we have

$\displaystyle x_{1}$ $\displaystyle=x_{0}-\gamma V(x_{0})-\gamma
U_{0}(x_{0})$ $\displaystyle x_{1}-x^{*}$
$\displaystyle=x_{0}-x^{*}-\gamma\left(\operatorname{\nabla}V(x^{*})\odot[x_{0}-x^{*}]+\text{Res}_{2}(x_{0})\right)-\gamma
U_{0}(x_{0})$ $\displaystyle x_{1}-x^{*}$
$\displaystyle=(I-\gamma(V(x^{*}))\odot[x_{0}-x^{*}]-\gamma\text{Res}_{2}(x_{0})-\gamma
U_{0}(x_{0}).$

We now compute $[x_{1}-x^{*}]^{2}=(x_{1}-x^{*})(x_{1}-x^{*})^{\top}$ and apply
the expectation with respect to the invariant distribution and the noise and
we have

$\displaystyle\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[[x-x^{*}]^{2}]=(I-\gamma\operatorname{\nabla}V(x^{*}))\odot\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[(x-x^{*})^{2}]\odot(I-\gamma\operatorname{\nabla}V(x^{*}))+\gamma^{2}\operatorname{\mathbb{E}}_{x_{0}\sim\pi_{\gamma}}[[U_{0}(x_{0})]^{2}]$
$\displaystyle+\operatorname{\mathcal{O}}\left(\underbrace{\gamma\int_{x\in\mathbb{R}^{d}}\text{Res}_{3}(x)\odot(I-\gamma(V(x^{*}))\odot[x_{0}-x^{*}]\pi_{\gamma}(x)\>dx+\gamma^{2}+\cdots}_{\gamma^{5/2}}\right).$

This leads to

$\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[[x-x^{*}]^{2}]=\gamma
Q(x^{*})\operatorname{\mathbb{E}}_{x_{0}\sim\pi_{\gamma}}[[U_{0}(x_{0})]^{2}]+\operatorname{\mathcal{O}}(\gamma^{3/2}),$

where $Q(x^{*}):=(\operatorname{\nabla}V(x^{*})\odot
I+I\odot\operatorname{\nabla}V(x^{*})-\gamma\operatorname{\nabla}V(x^{*})\odot\operatorname{\nabla}V(x^{*}))^{-1}$,
which is invertible since

$\operatorname{\nabla}V(x^{*})\odot
I+I\odot\operatorname{\nabla}V(x^{*})-\gamma\operatorname{\nabla}V(x^{*})\odot\operatorname{\nabla}V(x^{*})=\operatorname{\nabla}V(x^{*})\odot
M(x^{*})+M(x^{*})\odot\operatorname{\nabla}V(x^{*}),$

where $M(x^{*}):=I-\gamma/2\operatorname{\nabla}V(x^{*})$. By quasi-
monotonicity around $x^{*}$ and by choosing
$\gamma<\min(2L,\theta^{\prime}):=\theta$ we get that the tensor
$Q(\gamma^{*})$ is positive definite tensor.

By applying a second-order Taylor expansion about $x^{*}$ in
$Op(x):=[U_{t}(x)]^{2}$, and utilizing the same reasoning as above in
combination with the differentiability of the noise tensor (see Assumption 7),
we derive the following:

$\displaystyle\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[[U_{t}(x)]^{2}]$
$\displaystyle=[U_{t}(x^{*})]^{2}+\operatorname{\mathcal{O}}(\gamma)$ (D.9)
$\displaystyle\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[[U_{t}(x)]^{2}\odot[x-x^{*}]]$
$\displaystyle=[U_{t}(x^{*})]^{2}\odot[\operatorname{\mathbb{E}}_{x\sim\pi}[x]-x^{*}]+\operatorname{\mathcal{O}}(\gamma).$
(D.10)

Combining (D.8),(D.2),(D.9), we get that

$\bar{x}-x^{*}=-\dfrac{1}{2}[\operatorname{\nabla}V(x^{*})]^{-1}\odot\operatorname{\nabla}^{2}V(x^{*})\odot\big{(}\gamma
Q(x^{*})\operatorname{\mathbb{E}}_{x_{0}\sim\pi_{\gamma}}[[U_{0}(x_{0})]^{2}]+\operatorname{\mathcal{O}}(\gamma^{3/2})\big{)}+\operatorname{\mathcal{O}}(\gamma^{3/2}),$

which implies that

$\bar{x}-x^{*}=-\dfrac{1}{2}[\operatorname{\nabla}V(x^{*})]^{-1}\odot\operatorname{\nabla}^{2}V(x^{*})\odot\big{(}\gamma
Q(x^{*})\odot\\{[U_{t}(x^{*})]^{2}+\operatorname{\mathcal{O}}(\gamma)\\}+\operatorname{\mathcal{O}}(\gamma^{3/2})\big{)}+\operatorname{\mathcal{O}}(\gamma^{3/2}),$

or equivalently

$\bar{x}-x^{*}=\gamma\Delta(x^{*})+\operatorname{\mathcal{O}}(\gamma^{3/2}).$

The rest of the proof has the goal to improve the last term the order to
$\operatorname{\mathcal{O}}(\gamma^{2})$.

  1. 1.

We have seen that via (D.2),(D.9),:
$\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[[x-x^{*}]^{2}]=\gamma
Q(x^{*})\odot[U_{t}(x^{*})]+\gamma^{2}Q(x^{*})+o(\gamma^{2})$

  2. 2.

With similar calculations we can prove that:
$\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[[x-x^{*}]^{3}]=\gamma^{2}B(x^{*})+o(\gamma^{2})$

Using 4-th order taylor again we get the following equality

$\displaystyle x_{1}-x^{*}$ $\displaystyle=x_{0}-x^{*}$
$\displaystyle-\gamma\big{(}\operatorname{\nabla}V(x^{*})\odot[x-x^{*}]+\frac{1}{2!}\operatorname{\nabla}^{2}V(x^{*})\odot[x-x^{*}]^{2}$
$\displaystyle\quad\quad\quad+\frac{1}{3!}\operatorname{\nabla}^{3}V(x^{*})\odot[x-x^{*}]^{2}+\text{Res}_{4}(x)\big{)}$
$\displaystyle-\gamma U_{0}(x_{0})$

Applying expectation in the above equality and combining the bounds (1.) and
(2.), we have that

$\begin{Bmatrix}\operatorname{\nabla}V(x^{*})\odot[\bar{x}-x^{*}]+\dfrac{1}{2}\operatorname{\nabla}^{2}V(x^{*})\odot\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[[x-x^{*}]^{2}]\\\
\\\ +\\\ \\\
\frac{1}{3!}\operatorname{\nabla}^{3}V(x^{*})\odot\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[[x-x^{*}]^{3}]+\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[\text{Res}_{4}(x)]\end{Bmatrix}=0$
(D.11)

By applying the fourth-moment bound for
$\operatorname{\mathbb{E}}_{x\sim\pi_{\gamma}}[\text{Res}_{4}(x)]=\operatorname{\mathcal{O}}(\gamma^{2})$
we get the promised result. ∎


# RADIATE: A Radar Dataset for Automotive Perception
††thanks:

Marcel Sheeny1, Emanuele De Pellegrin1, Saptarshi Mukherjee1,
Alireza Ahrabian2, Sen Wang1, Andrew Wallace1  1Institute of Sensors, Signals
and Systems, Heriot-Watt University  2Hitachi Europe

###### Abstract

Datasets for autonomous cars are essential for the development and
benchmarking of perception systems. However, most existing datasets are
captured with camera and LiDAR sensors in good weather conditions. In this
paper, we present the RAdar Dataset In Adverse weaThEr (RADIATE), aiming to
facilitate research on object detection, tracking and scene understanding
using radar sensing for safe autonomous driving. RADIATE includes 3 hours of
annotated radar images with more than 200K labelled road actors in total, on
average about 4.6 instances per radar image. It covers 8 different categories
of actors in a variety of weather conditions (e.g., sun, night, rain, fog and
snow) and driving scenarios (e.g., parked, urban, motorway and suburban),
representing different levels of challenge. To the best of our knowledge, this
is the first public radar dataset which provides high-resolution radar images
on public roads with a large amount of road actors labelled. The data
collected in adverse weather, e.g., fog and snowfall, is unique. Some baseline
results of radar based object detection and recognition are given to show that
the use of radar data is promising for automotive applications in bad weather,
where vision and LiDAR can fail. RADIATE also has stereo images, 32-channel
LiDAR and GPS data, directed at other applications such as sensor fusion,
localisation and mapping. The public dataset can be accessed at
http://pro.hw.ac.uk/radiate/.

Figure 1: Examples from RADIATE. This dataset contains radar, stereo camera,
LiDAR and GPS data. It was collected in various weather conditions and driving
scenarios with 8 categories of annotated objects.

##  I Introduction

Autonomous driving research and development rely heavily on the use of public
datasets in the computer vision and robotics communities [1, 2, 3]. Camera and
LiDAR are the two primary perceptual sensors that are usually adopted.
However, since these are visible spectrum sensors, the data is affected
dramatically by bad weather, causing attenuation, multiple scattering and
turbulence [4, 5, 6, 7, 8]. On the other hand, a radar sensor is known to be
more robust in adverse conditions [9, 10, 11]. However, there are few public
radar datasets for automotive applications, especially in bad weather and with
object annotation.

In this paper, we present the RAdar Dataset In Adverse weaThEr (RADIATE) for
perceptual tasks in autonomous driving and driver assistance. It includes a
mixture of weather conditions and driving scenarios, representing different
levels of challenge. A high-resolution 79 GHz 360° radar is chosen as the main
sensor for object annotation. RADIATE includes 5 hours of radar images in
total, 3 hours of which are fully annotated. This gives RADIATE more than 200K
labelled object instances with 8 categories of road actors (i.e., car, van,
bus, truck, motorbike, bicycle, pedestrian and a group of pedestrians).
RADIATE also has stereo camera, LiDAR and GPS data collected simultaneously.
Some examples from RADIATE are shown in Figure 1.

Our work makes the following main contributions:

  * •

To the best of our knowledge, RADIATE is the first public radar dataset which
includes a large number of labelled road actors on public roads.

  * •

It includes multi-modal sensor data collected in challenging weather
conditions, such as dense fog and heavy snowfall. Camera, LiDAR and GPS data
are also provided for all sequences.

  * •

As an example of a use case, we demonstrate that RADIATE can be used for
robust vehicle detection in adverse weather, when optical sensors (camera and
LiDAR) fail.

##  II Related Work

There are many publicly available datasets for research into perception for
autonomous and assisted driving. The most popular is the KITTI dataset [1],
using cameras and a Velodyne HDL-64e LiDAR to provide data for several tasks,
such as object detection and tracking, odometry and semantic segmentation.
Although widely used as a benchmark, data was captured only in good weather
and appears now as rather small scale. Waymo [3] and Argo [12] are automotive
datasets which are larger than KITTI and provide more data variability. Some
data was collected in the rain and at night although adverse weather is not
their main research focus. Foggy Cityscapes [5] developed a synthetic foggy
dataset aimed at object recognition tasks but only images are provided. All
these datasets use only optical sensors.

TABLE I: Comparison of RADIATE with public automotive datasets that use radar
sensing.

Dataset | Size | Radar | Lidar | Camera | Night | Fog | Rain | Snow |  Object Detection |  Object Tracking | Odometry |  3D Annotations
---|---|---|---|---|---|---|---|---|---|---|---|---
nuScenes [2] | Large |  ✓ (Sparse Point Cloud) | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | ✓ | ✗ | ✓
Oxford Radar RobotCar [13] | Large |  ✓ (High-Res Radar Image) | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗
MulRan [14] | Large |  ✓ (High-Res Radar Image) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗
Astyx [15] | Small |  ✓ (Sparse Radar Point Cloud) | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓
RADIATE (Ours) | Large |  ✓ (High-Res Radar Image) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  ✗ (Pseudo-3D)

Radar, on the other hand, provides a sensing solution that is more resilient
to fog, rain and snow. It usually provides low-resolution images, which makes
it very challenging for object recognition or semantic segmentation. Current
automotive radar technology relies, usually on the Multiple Input Multiple
Output (MIMO) technique, which uses several transmitters and receivers to
measure the direction of arrival (DOA) [16]. Although this is inexpensive,
current configurations lack azimuth resolution, e.g. the cross-range image of
a commercial radar with 15° angular resolution is around 10 meters at 20-meter
distance. This means that a radar image provides insufficient detail for
object recognition or deatiled scene mapping. Scanning radar measures at each
azimuth using a moving antenna, providing better azimuth resolution. This type
of sensor has recently been developed to tackle radar based perception tasks
for automotive applications [9, 17, 18, 19].

For most datasets which provide radar data for automotive applications, the
radar is used only as a simple detector, e.g. NuScenes [2], to give sparse 2D
point clouds. Recently, the Oxford Robotcar [13] and MulRan datasets [14]
provided data collected from a scanning Navtech radar in various weather
conditions. However, they do not provide object annotations as the data was
designed primarily for Simultaneous Localisation and Mapping (SLAM) and place
recognition in long-term autonomy. The Astyx dataset [15] provides denser data
(compared to current MIMO technology) but with only about $500$ frames
annotated. It is also limited in terms of weather variability. Table I
compares existing public automotive radar datasets with RADIATE. To summarise,
although research into radar perception for autonomous driving has recently
been gaining popularity [20, 21, 22], there is no radar dataset with a large
set of actor annotations publicly available. We hope the introduction of
RADIATE can boost autonomous driving research in the community.

##  III The RADIATE Dataset

Figure 2: Sensor setup for data collection.

The RADIATE dataset was collected between February 2019 and February 2020. The
data collection system was created using the Robot Operating System (ROS)
[23]. From the ”rosbag” created by ROS, we extracted sensor information, each
with its respective timestamp. Figure 3 shows the folder structure used for
the dataset. To facilitate access, a RADIATE software development kit (SDK) is
released for data calibration, visualisation, and pre-processing.

Figure 3: Folder tree, annotation, metadata and timestamp structure of each
sequence

###  III-A Perception Sensors

RADIATE includes radar, LiDAR and stereo cameras. Figure 2 shows the sensor
setup and the extrinsic configuration on our vehicle. To support sensor
fusion, the sensors are calibrated (details in Section III-B).

#### Stereo Camera

An off-the-shelf ZED stereo camera is used. It is set at $672\times 376$ image
resolution at 15 frames per second for each camera. It is protected by a
waterproof housing for extreme weather. The images can be seriously blurred,
hazy or fully blocked due to rain drops, dense fog or heavy snow,
respectively. Some examples are shown in Figure 1.

#### LiDAR

A 32 channel, 10Hz, Velodyne HDL-32e LiDAR [24] is used to give 360° coverage.
Since the LiDAR signal can be severely attenuated and reflected by intervening
fog or snow [8], the data can be missing, noisy and incorrect.

#### Radar

RADIATE adopts the Navtech CTS350-X [25] radar. It is a scanning radar which
provides 360° high-resolution range-azimuth images. It has 100 meters maximum
operating range with 0.175m range resolution, 1.8° azimuth resolution and 1.8°
elevation resolution, Currently, it does not provide Doppler information.

###  III-B Sensor Calibration

Sensor calibration is required for multi-sensor fusion, feature and actor
correspondence. The intrinsic parameters and distortion coefficients of the
stereo camera are calibrated using the Matlab camera calibration toolbox [26].
Then, rectified images can be generated to calculate depths. In terms of
extrinsic calibration, the radar sensor is chosen as the origin of the local
coordinate frame as it is the main sensor. The extrinsic parameters for the
radar, camera and LiDAR are represented as 6 degree-of-freedom transformations
(translation and rotation). They are performed by first explicitly measuring
the distance between the sensors, and then fine-tuned by aligning measurements
between each pair of sensors. The sensor calibration parameters are provided
in a yaml file. The sensors operate at different frame rates and we simply
adopt the time of arrival of the data at each sensor as the timestamp.

###  III-C Data Collection Scenarios

We collected data in 7 different scenarios, i.e., sunny (parked),
sunny/overcast (urban), overcast (motorway), night (motorway), rain
(suburban), fog (suburban) and snow (suburban).

#### Sunny (Parked)

In this scenario the vehicle was parked at the side of the road, sensing the
surrounding actors passing by. This is intended as the easiest scenario for
object detection, target tracking and trajectory prediction. This was
collected in sunny weather.

#### Sunny/Overcast (Urban)

The urban scenario was collected in the city centre with busy traffic and
dense buildings. This is challenging since many road actors appear. The radar
collected in this scenario is also cluttered by numerous reflections from non-
road actors, such as trees, fences, bins and buildings, and multi-path
effects, increasing the challenge. Those sequences were collected in sunny and
overcast weather.

#### Overcast (Motorway)

The motorway scenario was captured on the Edinburgh city bypass. This can be
considered as relatively easy since most of the surrounding dynamic actors are
vehicles and the background is mostly very similar. This scenario was
collected in overcast weather.

#### Night (Motorway)

We collected data during night in a motorway scenario. Night is an adverse
scenario for passive optical sensors due to lack of illumination. Lidar and
radar were expected to behave well since they are active sensors and do not
depend on an external light source.

#### Rain (Suburban)

We collected 18 minutes of data in substantial rain. The collection took place
in a suburban scenario close to the university campus.

#### Fog (Suburban)

We found foggy data challenging to collect. Fog does not happen very often in
most places and it is very hard to predict when it will occur and how dense it
will be. In practice, we collected foggy data opportunistically when parked
and driving in suburban areas.

#### Snow (Suburban)

RADIATE includes 34 minutes of data in snow, in which 3 minutes are labelled.
Snowflakes are expected to interfere with LiDAR and camera data, and can also
affect radar images to a lesser extent. Moreover, heavy snowfall can block the
sensors within 3 minutes of data collection (see Figure 6). The data in snow
was collected in a suburban scenario.

Figure 4: Category distribution for each scenario.

Figure 5: Dataset length for driving scenarios and weather conditions (in
minutes).

Figure 6: Sensors covered in snow

###  III-D Labelling

Labelling radar images is challenging because it is not easy for a human to
recognise objects in radar data. This means most existing annotation tools
which are designed for a single sensor [27, 28, 29] are inadequate. Therefore,
a new annotation tool was developed to automatically correlate and visualise
multiple sensors through sensor calibration.

Eight different road actors are labelled in RADIATE: cars, vans, trucks,
buses, motorbikes, bicycle, pedestrian and group of pedestrians. 2D bounding
boxes were annotated on radar images after verifying the corresponding camera
images. Each bounding box is represented as (x,y,width,height,angle), where
(x,y) is the upper-left pixel locations of the bounding box, of given width
and height and counter-clockwise rotation angle. To achieve efficient
annotation, the CamShift tracker was used [30].

In total, RADIATE has more than $200$K bounding boxes over $44$K radar images,
an average 4.6 bounding boxes per image. The class distribution for each
driving scenario is illustrated in Figure 4. In Figure 6, the length and the
total number of objects in each scenario are given.

Figure 7: Data in various weather conditions. Top: Image with LiDAR points
projected. Middle: Radar with objects annotated. Bottom: LiDAR with objects
projected from radar annotation. Note both image and LiDAR images are degraded
in fog, rain and snow. The yellow circles encloses false LiDAR points caused
by snow flakes.

##  IV Example of Use: Radar based Vehicle Detection

One of the key perception capabilities of autonomous vehicles is to detect and
recognise road actors for safe navigation and decision making. To the best of
our knowledge, there is no existing work on radar based object detection for
autonomous driving in extreme weather. Therefore, we present as a use case the
first baseline results using RADIATE. Figure 7 shows some examples of our data
collected in adverse weather, in which trained networks fail to recognise
actors in the optical images. As shown in Figure 7. The radar images are less
affected so we would hypothesise that robust object detection is more likely
in all conditions.

Drawn from the data acquired in several weather conditions, RADIATE includes 3
exemplary data sets that can be used for evaluation for vehicle detection:

  * •

Training set in good weather: This only contains data in good weather, sunny
or overcast. It was created to validate whether sensors are capable to adapt
from good weather to bad weather conditions.

  * •

Training set in good and bad weather: This additional set includes data from
both good and bad weather conditions (night, rain, fog and snow). This was
used to develop algorithms for all weathers.

  * •

Test set: This test set includes data from both good and bad weather
conditions and is used for evaluation and benchmarking.

As a first baseline, we have performed evaluation of vehicle detection from
single images. We defined a vehicle as one of the following classes: car, van,
truck, bus, motorbike and bicycle. In Table II are shown the number of radar
images and vehicles used to train and test the network.

TABLE II: Number of images for each set defined. | #Radar Images | #Vehicles
---|---|---
Training Set in Good Weather | 23,091 | 106,931
Training Set in Good and Bad Weather | 9,760 | 39,647
Test Set | 11,289 | 42,707
Total | 44,140 | 147,005

###  IV-A Radar based Vehicle Detection in the Wild

We adopted the popular Faster R-CNN [31] architecture to demonstrate the use
of RADIATE for radar based object detection. Two modifications were made to
the original architecture to better suit radar detection:

  * •

Pre-defined sizes were used for anchor generation because vehicle volumes are
typically well-known and radar images provide metric scales, different from
camera images.

  * •

We modified the Region Proposal Network (RPN) from Faster R-CNN to output the
bounding box and a rotation angle which the bounding boxes are represented by
x, y, width, height, angle.

To investigate the impact of weather conditions, the models were trained with
the 2 different training datasets: data from only good and data from both good
and bad weather. ResNet-50 and ResNet-101 were chosen as backbone models [32].
The trained models were tested on a test set collected from all weather
conditions and driving scenarios. The metric used for evaluation was Average
Precision with Intersection over Union (IoU) equal to 0.5, which is the same
as the PASCAL VOC [33] and DOTA [34] evaluation metrics. Table III shows the
Average Precision (AP) results and Figure 8 shows the precision recall curve.
It can be seen that the AP difference between training with good weather and
good&bad weathers is marginal, which suggests that the weather conditions
cause no or only a subtle impact on radar based object detection. The heat
maps of the AP with respect to the radar image coordinates are also given in
Figure 9. Since the AP distribution of the model trained only on the good
weather data is very similar to the model trained with both good and bad
weather data, it further verifies that radar is promising for object detection
in all weathers. Regarding the results in each scenario, it is mainly biased
by the type of data, rather than the weather itself. The parked scenario is
shown to be the easiest, achieving close to 80% AP, potentially aided by the
consistency in the radar return from the environmental surround. Results in
snow and rain data performed more poorly. Examining Figure 7, the radar sensor
used was affected by rain changing the background pixel values. In fog, we
achieved considerably better results using radar. Since it is a challenging
scenario for optical sensors, radar is shown to be a good solution for dense
fog perception. In nighttime, motorway scenarios, the results were close to
the results in daytime. This was expected, since radar is an active sensor and
is not affected by the lack of illumination. Figure 10 illustrates some
qualitative results for radar based vehicle detection in various driving
scenarios and weather conditions, using the Faster R-CNN ResNet-101 trained in
good weather only.

TABLE III: Average Precision results on test set.

| Overall |  Sunny (Parked) |  Overcast (Motorway) |  Sunny/Overcast (Urban) |  Night (Motorway) |  Rain (Suburban) |  Fog (Suburban) |  Snow (Suburban)
---|---|---|---|---|---|---|---|---
ResNet-50 Trained on Good and Bad Weather | 45.77 | 78.99 | 42.06 | 36.12 | 54.71 | 33.53 | 48.24 | 12.81
ResNet-50 Trained on Good Weather | 45.31 | 78.15 | 47.06 | 37.04 | 51.80 | 26.45 | 47.25 | 5.47
ResNet-101 Trained on Good and Bad Weather | 46.55 | 79.72 | 44.23 | 35.45 | 64.29 | 31.96 | 51.22 | 8.14
ResNet-101 Trained on Good Weather | 45.84 | 78.88 | 41.91 | 30.36 | 40.49 | 29.18 | 48.30 | 11.16

Figure 8: Precision recall curves.

(a)

(b)

Figure 9: Heatmap of Average Precision. Figure 10: Qualitative results of
radar based vehicle detection.

##  V Conclusions

We have presented the new, labelled, large scale RADIATE dataset for research
into sensory data perception for autonomous and assisted driving. The sensor
setup, calibration and labelling processes are described, and we show examples
of data collected under several different weather and road scenarios. As an
example of use, we show how it can be used for radar based vehicle detection
in the wild. These results show that radar based object detection is less
affected by the weather, especially in foggy scenarios, where recognition from
LiDAR data fails at a very short range depending on the fog density. This
initial, baseline experiment demonstrated promising results, especially in
adverse conditions. In this preliminary study, we emphasise that motion is not
used for recognition, in part as actors may well be stationary, and in part
because no Doppler data is available, although frame to frame tracking is an
obvious avenue for further research provided a long term motion memory model
is employed. We hope RADIATE will facilitate research in the community,
particularly for robust perception in adverse weather.

## Acknowledgements

This work was supported by Jaguar Land Rover and the UK Engineering and
Physical Research Council, grant reference EP/N012402/1 (TASCC: Pervasive low-
TeraHz and Video Sensing for Car Autonomy and Driver Assistance (PATH CAD)).
We would like to thank Georgios Kalokyris for helping with the data
annotation. We also thank NVIDIA for the donation of the TITAN X GPU.

## References

  * [1] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _International Journal of Robotics Research (IJRR)_ , 2013\.
  * [2] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” _arXiv preprint arXiv:1903.11027_ , 2019.
  * [3] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine _et al._ , “Scalability in perception for autonomous driving: An open dataset benchmark,” _arXiv preprint arXiv:1912.04838_ , 2019.
  * [4] M. Pfennigbauer, C. Wolf, J. Weinkopf, and A. Ullrich, “Online waveform processing for demanding target situations,” in _Laser Radar Technology and Applications XIX; and Atmospheric Propagation XI_ , vol. 9080. International Society for Optics and Photonics, 2014, p. 90800J.
  * [5] C. Sakaridis, D. Dai, and L. Van Gool, “Semantic foggy scene understanding with synthetic data,” _International Journal of Computer Vision_ , vol. 126, no. 9, pp. 973–992, Sep 2018. [Online]. Available: https://doi.org/10.1007/s11263-018-1072-8
  * [6] M. Bijelic, T. Gruber, and W. Ritter, “A benchmark for lidar sensors in fog: Is detection breaking down?” in _2018 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2018, pp. 760–767.
  * [7] M. Kutila, P. Pyykönen, H. Holzhüter, M. Colomb, and P. Duthon, “Automotive lidar performance verification in fog and rain,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 1695–1701.
  * [8] A. M. Wallace, A. Halimi, and G. S. Buller, “Full waveform lidar for adverse weather conditions,” _IEEE Transactions on Vehicular Technology_ , vol. Early Access, 2020. [Online]. Available: https://ieeexplore.ieee.org/document/9076331
  * [9] L. Daniel, D. Phippen, E. Hoare, A. Stove, M. Cherniakov, and M. Gashinova, “Low-thz radar, lidar and optical imaging through artificially generated fog,” in _International Conference on Radar Systems (Radar 2017)_. IET, 2017, pp. 1–4.
  * [10] F. Norouzian, E. Marchetti, E. Hoare, M. Gashinova, C. Constantinou, P. Gardner, and M. Cherniakov, “Experimental study on low-thz automotive radar signal attenuation during snowfall,” _IET Radar, Sonar & Navigation_, vol. 13, no. 9, pp. 1421–1427, 2019.
  * [11] F. Norouzian, E. Marchetti, M. Gashinova, E. Hoare, C. Constantinou, P. Gardner, and M. Cherniakov, “Rain attenuation at millimeter wave and low-thz frequencies,” _IEEE Transactions on Antennas and Propagation_ , vol. 68, no. 1, pp. 421–431, 2019.
  * [12] M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, and J. Hays, “Argoverse: 3d tracking and forecasting with rich maps,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [13] D. Barnes, M. Gadd, P. Murcutt, P. Newman, and I. Posner, “The oxford radar robotcar dataset: A radar extension to the oxford robotcar dataset,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , Paris, 2020. [Online]. Available: https://arxiv.org/abs/1909.01300
  * [14] G. Kim, Y. S. Park, Y. Cho, J. Jeong, and A. Kim, “Mulran: Multimodal range dataset for urban place recognition,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , Paris, May 2020, accepted. To appear.
  * [15] M. Meyer and G. Kuschk, “Automotive radar dataset for deep learning based 3d object detection,” in _2019 16th European Radar Conference (EuRAD)_. IEEE, 2019, pp. 129–132.
  * [16] _Short Range Radar Reference Design Using AWR1642_ , April 2017. [Online]. Available: http://www.ti.com/lit/ug/tidud36b/tidud36b.pdf
  * [17] M. Sheeny, A. Wallace, and S. Wang, “300 ghz radar object recognition based on deep neural networks and transfer learning,” _IET Proceedings on Radar, Sonar and Navigation_ , 2020.
  * [18] ——, “Radio: Parameterized generative radar data augmentation for small datasets,” _Applied Sciences_ , vol. 10, no. 11, p. 3861, 2020.
  * [19] S. Mukherjee, S. Wang, and A. Wallace, “Interacting vehicle trajectory prediction with convolutional recurrent neural networks,” in _Proceedings of Internationa Conference on Robotics and Automation_ , 2020\.
  * [20] B. Major, D. Fontijne, A. Ansari, R. Teja Sukhavasi, R. Gowaikar, M. Hamilton, S. Lee, S. Grzechnik, and S. Subramanian, “Vehicle detection with automotive radar using deep learning on range-azimuth-doppler tensors,” in _The IEEE International Conference on Computer Vision (ICCV) Workshops_ , Oct 2019, pp. 0–0.
  * [21] L. Sless, B. El Shlomo, G. Cohen, and S. Oron, “Road scene understanding by occupancy grid learning from sparse radar clusters using semantic segmentation,” in _The IEEE International Conference on Computer Vision (ICCV) Workshops_ , Oct 2019.
  * [22] Y. Wang, Z. Jiang, X. Gao, J.-N. Hwang, G. Xing, and H. Liu, “Rodnet: Object detection under severe conditions using vision-radio cross-modal supervision,” _arXiv preprint arXiv:2003.01816_ , 2020.
  * [23] M. Quigley, K. Conley, B. P. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng, “Ros: an open-source robot operating system,” in _ICRA Workshop on Open Source Software_ , 2009.
  * [24] Velodyne, “Velodyne hdl-32e,” https://velodynelidar.com/products/hdl-32e/, 2020.
  * [25] Navtech, “Navtech radar technical specifications,” https://navtechradar.com/clearway-technical-specifications/, [Online; accessed April 03, 2020].
  * [26] “Matlab stereo camera calibration,” https://uk.mathworks.com/help/vision/ref/stereocameracalibrator-app.html, 2020, [Online; accessed April 03, 2020].
  * [27] “Cvat: Powerful and efficient computer vision annotation tool,” https://github.com/opencv/cvat.
  * [28] C. Vondrick, D. Patterson, and D. Ramanan, “Efficiently scaling up crowdsourced video annotation,” _International Journal of Computer Vision_ , pp. 1–21, 2013. [Online]. Available: http://dx.doi.org/10.1007/s11263-012-0564-1
  * [29] “Labelimg,” https://github.com/tzutalin/labelImg, [Online; accessed April 03, 2020].
  * [30] J. G. Allen, R. Y. Xu, J. S. Jin _et al._ , “Object tracking using camshift algorithm and multiple quantized feature spaces,” in _ACM International Conference Proceeding Series_ , vol. 100, 2004, pp. 3–7.
  * [31] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [32] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [33] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International journal of computer vision_ , vol. 88, no. 2, pp. 303–338, Jun 2010. [Online]. Available: https://doi.org/10.1007/s11263-009-0275-4
  * [34] G.-S. Xia, X. Bai, J. Ding, Z. Zhu, S. Belongie, J. Luo, M. Datcu, M. Pelillo, and L. Zhang, “Dota: A large-scale dataset for object detection in aerial images,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3974–3983.


# SPICEprop: Backpropagating Errors Through Memristive Spiking Neural Networks

Peng Zhou, Jason K. Eshraghian,
Dong-Uk Choi, and Sung-Mo Kang P. Zhou, D. Choi, and S. M. Kang are with the
Department of Electrical and Computer Engineering, UC, Santa Cruz, CA, USA.J.
K. Eshraghian is with the Dept. of Electrical Engineering and Computer
Science, University of Michigan, Ann Arbor, MI, USA.

###### Abstract

We present a fully memristive spiking neural network (MSNN) consisting of
novel memristive neurons trained using the backpropagation through time (BPTT)
learning rule. Gradient descent is applied directly to the memristive
integrate-and-fire (MIF) neuron designed using analog SPICE circuit models,
which generates distinct depolarization, hyperpolarization, and repolarization
voltage waveforms. Synaptic weights are trained by BPTT using the membrane
potential of the MIF neuron model and can be processed on memristive
crossbars. The natural spiking dynamics of the MIF neuron model are fully
differentiable, eliminating the need for gradient approximations that are
prevalent in the spiking neural network literature. Despite the added
complexity of training directly on SPICE circuit models, we achieve 97.58$\%$
accuracy on the MNIST testing dataset and 75.26$\%$ on the Fashion-MNIST
testing dataset, the highest accuracies among all fully MSNNs.

###### Index Terms:

Neuromorphic computing, memristor, spiking neural network, supervised
learning, backpropagation.

##  I Introduction

Spiking Neural Networks (SNNs), often referred to as the third generation of
neural networks, draw inspiration from biological neurons to enable heightened
efficiency over conventional neural networks. When processed on neuromorphic
hardware, SNNs have shown significant energy and latency savings due to their
activation sparsity, spike-based representations of data, and event-based data
processing [1, 2, 3, 4]. Recently, the remarkable success of backpropagation
in the training of Deep Neural Networks (DNNs) has broadened and inspired
research on SNNs. By unrolling the computational graph of spiking neuron
models, SNNs can take advantage of the backpropagation through time (BPTT)
algorithm [5] such that a global error is calculated in the final layer and
backpropagated across the full history of the SNN dynamics. By analogy, we
propose that circuit-level neuron models described by dynamical state
equations can also be trained in a similar manner. In doing so, the rich
dynamics of nonlinear circuit elements, such as memristors, can be fully
leveraged during the network optimization process.

The memristor, theoretically postulated by Chua in 1971 [6] and generalized by
Chua and Kang in 1976 [7], has become a commercially available technology that
can be integrated in the back-end-of-the-line (BEOL) of modern CMOS processes
[8, 9]. The non-volatile retention capacity of the memristor is often likened
to synaptic memory [10, 11, 12, 13], and its threshold-switching
characteristics have been exploited in spiking neuron models [14, 15, 16, 17].
However, much of the prior work on memristive spiking neural networks (MSNNs)
is constrained to using either memristive synapses or memristive neurons, but
not both. Memristive synaptic arrays are used only for non-volatile memory
retention, where networks are trained offline and thus, do not harness the
complex dynamics of memristors. Networks of memristive neurons (‘neuristors’),
on the other hand, are challenging to train and only used in small-scale
networks constrained to simple tasks, such as low-dimensional pattern
recognition solved via associative or local learning rules. The work in [18]
successfully integrates a fully memristive network, including memristive
synapses and neurons, to perform basic pattern recognition, but the memristors
used in this work are not accessible to the broader research community. The
work presented in [19] uses analog ReLU activations with memristive synapses
to achieve 93.63$\%$ testing accuracy on the MNIST dataset. As a non-spiking
network, analog signals are noise prone, whereas spikes are error tolerant
discrete events distributed in time.

In this paper, we scale up the complexity of learnable tasks in fully
memristive SNNs by directly applying gradient descent to the nonlinear state
evolution of memristive neurons and synapses. Both neurons and synapses are
modeled using memristors. The memristive integrate-and-fire (MIF) neuron model
is designed to achieve distinct depolarization, hyperpolarization, and
repolarization voltage phases with a minimal set of circuit elements.
Memristive synapses interconnect layers of neurons. To train the fully
memristive network, we propose SPICEprop, an application of error
backpropagation to large-scale networks derived from SPICE circuit models.
This enables dynamical, time-varying memristive neurons to achieve much higher
accuracy on data-driven tasks than has been previously reported with MSNNs. By
relying on the analog spiking characteristics that naturally occur in the MIF
neuron model, the non-differentiability of spike-based activations are
completely avoided. This means SPICEprop does not rely on surrogate gradient
techniques that are commonly used to train SNNs, which calculates biased
gradient estimators to circumvent the dead neuron problem [20]. To promote
broad accessibility of our methods, we use SPICE models of commercially
available, low-cost memristors to demonstrate the efficacy of SPICEprop in a
supervised deep learning framework.

##  II Methods

###  II-A MIF model

Our MSNN adopts the MIF neuron model shown in Fig. 1 [21]. The MIF neuron is
characterized by the differential equations below:

Figure 1: A memristive integrate-and-fire (MIF) neuron consists of two
memristors $M_{1}$ and $M_{2}$, connected to DC voltage sources $E_{\rm rest}$
and $E_{\rm reset}$, in parallel with a capacitor $C$. The MIF neuron is
provably minimal in generating a membrane potential traversing from a rest
voltage level to a threshold voltage level, then to a reset voltage level, and
then back to the rest potential when a current pulse is applied.

$\frac{{\mathrm{d}}v}{\mathrm{d}t}=\frac{I-G_{1}(v-E_{\rm
rest})-G_{2}(v-E_{\rm reset})}{C}$ (1a)
$\frac{{\mathrm{d}x_{1}}}{\mathrm{d}t}=\frac{1}{\tau_{1}}(\frac{1-x_{1}}{1+e^{\frac{v_{\rm
on1}-(v-E_{\rm rest})}{k_{th}}}}-\frac{x_{1}}{1+e^{\frac{(v-E_{\rm
rest})-v_{\rm off1}}{k_{th}}}})$ (1b)
$\frac{{\mathrm{d}x_{2}}}{\mathrm{d}t}=\frac{1}{\tau_{2}}(\frac{1-x_{2}}{1+e^{\frac{v_{\rm
on2}-(v-E_{\rm reset})}{k_{th}}}}-\frac{x_{2}}{1+e^{\frac{(v-E_{\rm
reset})-v_{\rm off2}}{k_{th}}}})$ (1c) $G_{1}=\frac{x_{1}}{R_{\rm
on1}}+\frac{1-x_{1}}{R_{\rm off1}}$ (1d) $G_{2}=\frac{x_{1}}{R_{\rm
on2}}+\frac{1-x_{2}}{R_{\rm off2}}$ (1e)

where $G_{1}$ and $G_{2}$ are the memductances, $x_{1}$ and $x_{2}$ are a pair
of internal states, $\tau_{1}$ and $\tau_{2}$ are time constants governing the
rate of change in internal states, and $k_{th}$ is the effective thermal
voltage. They are the characteristic variables of $M_{1}$ and $M_{2}$,
respectively. This system of equations mirrors several prominent SPICE
memristor models, and has been used to emulate the commercially available
KNOWMTM memristor.

For a physically plausible system, discrete spikes must be represented as
continuous current inputs. An input spike train of Dirac delta pulses
$\sum_{n}{\delta(t-t_{j}^{n})}$ is used to generate an alpha-shaped input
current $I$, modeled by Eq. (2):

$\displaystyle\tau_{syn}\frac{{\mathrm{d}I}}{\mathrm{d}t}$ $\displaystyle=a-I$
(2a) $\displaystyle\tau_{syn}\frac{{\mathrm{d}a}}{\mathrm{d}t}$
$\displaystyle=-a+W_{i}\cdot\sum_{n}{\delta(t-t_{i}^{n})}$ (2b)

where $a$ is an internal state variable, $\tau_{syn}$ is a time constant
governing the shape of the alpha current, $W_{i}$ is the synaptic weight
between presynaptic neuron $i$ and associated postsynaptic neuron.

The parameters used in this network are listed in Table I.

TABLE I: MIF circuit parameters Parameter | Value | Parameter | Value
---|---|---|---
$E_{\rm rest}$ | 0 mV | $E_{\rm reset}$ | 50 mV
$C$ | 100 pF | $k_{th}$ | 15 mV
$v_{\rm off_{1}}$, $v_{\rm off_{2}}$ | 5 mV |  $v_{\rm on_{1}}$, $v_{\rm on_{2}}$ | 110 mV
$R_{\rm off_{1}}$, $R_{\rm off_{2}}$ | 0.1 M$\Omega$ |  $R_{\rm on_{1}}$, $R_{\rm on_{2}}$ | 1 k$\Omega$
$\tau_{1}$, $\tau_{2}$ | 100 steps | $\tau_{syn}$ | 100 steps

###  II-B Numerical Integration of MIF Model

The SPICE model of the memristor used in the MIF neuron described in Eq. (1)
must be adapted to a recurrent form to be compatible in a deep learning
framework. SPICE simulators employ a variety of ordinary differential equation
(ODE) solvers, which, in the first order, are often equivalent to the backward
Euler method. For compatibility with BPTT, we adopt the forward Euler method
to find an approximate solution to the MIF network, as the forward Euler
method provides an explicit update equation. As such, we retain the temporal
dynamics of the MIF neuron, and also derive a discrete time representation of
the MIF neuron that can be unrolled across time steps for use with the BPTT
algorithm. The approximate solution of a MIF neuron model with alpha current
dynamics using the forward Euler method is provided in Eq. (3).

$a[t+1]=-\frac{a[t]}{\tau_{syn}}+a[t]+\sum_{j}{W_{j}\cdot S_{j}}$ (3a)
$I[t+1]=\frac{a[t]-I[t]}{\tau_{syn}}+I[t]$ (3b) $\displaystyle
x_{1}[t+1]=\frac{1}{\tau_{1}}\Big{(}\frac{1-x_{1}[t]}{1+e^{\frac{v_{\rm
on1}-(v[t]-E_{\rm rest})}{k_{th}}}}-$
$\displaystyle\frac{x_{1}[t]}{1+e^{\frac{(v[t]-E_{\rm rest})-v_{\rm
off1}}{k_{th}}}}\Big{)}+x_{1}[t]$ (3c) $\displaystyle
x_{2}[t+1]=\frac{1}{\tau_{2}}\Big{(}\frac{1-x_{2}[t]}{1+e^{\frac{v_{\rm
on2}-(v[t]-E_{\rm reset})}{k_{th}}}}-$
$\displaystyle\frac{x_{2}[t]}{1+e^{\frac{(v[t]-E_{\rm reset})-v_{\rm
off2}}{k_{th}}}}\Big{)}+x_{2}[t]$ (3d) $G_{1}[t+1]=\frac{x_{1}[t+1]}{R_{\rm
on1}}+\frac{1-x_{1}[t+1]}{R_{\rm off1}}$ (3e)
$G_{2}[t+1]=\frac{x_{2}[t+1]}{R_{\rm on2}}+\frac{1-x_{2}[t+1]}{R_{\rm off2}}$
(3f) $\displaystyle v[t+1]=\frac{I[t]-G_{1}[t](v[t]-E_{\rm
rest})-G_{2}[t](v[t]-E_{\rm reset})}{C}$ $\displaystyle+v[t]$ (3g)

Fig. 2 shows the simulation results across 1,000 time steps. Each time step is
of duration 0.01 ms and the total time simulated is 10 ms. The rest,
integrate, threshold, and reset performances are correctly simulated by a
SPICE simulator to match the dynamics of the alpha synapse current and the MIF
neuron modeled by the ODEs in Eq. (1) and (2).

Figure 2: Simulation results of MIF neuron solved using forward Euler
numerical integration. The results matches the quantitative dynamics of solved
by a SPICE simulator. Top: Alpha synaptic current dynamics. Middle: Internal
states $x_{1}$, $x_{2}$. Bottom: Voltage response $v$.

###  II-C Backpropagation Through Time in Memristive Spiking Neural Networks

The discrete-time solution in Eq. (3) is illustrated as an MSNN computational
graph in Fig. 3, resembling an unrolled recurrent neural network. The loss can
be calculated based on either spiking activity or the membrane potential. Two
observations support the use of the membrane potential. Firstly, an increase
in potential of the correct output neuron drives the neuron to spike, while
decreasing the potential of the incorrect neuron classes makes these neurons
silent. Applying the target to the membrane potential can serve as a proxy for
modulating the spiking behavior. Secondly, the MIF membrane potential is
autonomously reset after a spike is elicited without the need for extra
circuitry; i.e., the spike is part of the membrane potential waveform which is
weighted and propagated to downstream layers. Therefore, a fully analog MSNN
can be trained without reliance on additional surrogate gradient calculation
steps, as used in almost all multi-layer SNNs trained via error
backpropagation [20, 1].

Figure 3: Unrolled MSNN computational graph in discrete time.

###  II-D Network Architecture

To validate the use of MIF neurons in a deep learning framework, we used a
4-layer neural network shown in Fig. 4. In the input layer, there are
$28\times 28=784$ input units. The first hidden layer consists of 100 neurons
followed by a ReLU activation, and the second hidden layer contains 10 neurons
followed by a ReLU activation. Finally, the output layer consists of 10 MIF
neurons. Fully connected (all-to-all) weights are used between each layer.

Figure 4: The MSNN architecture.

##  III Results

The automatic differentiation engine from PyTorch v1.10.1 was used to perform
BPTT through the network in Python 3.8. For training and testing, we used the
MNIST dataset [22] which consists of 70,000 samples of handwritten digits.
During the training process, 1,000 time steps are simulated for each input
image sample. Over the 1,000 steps, the input pixel intensity is fed to the
network only at steps 0, 400, and 800 to promote sparse network activity. The
negative log-likelihood loss is applied to the membrane potential, i.e.,
$v(t)$ in the MIF circuit in Fig. 1, at each time step. In other words, the
goal of the network is to maximize the voltage across the MSNN circuit of the
correct class. The total loss is summed prior to backpropagating the error
through the SPICE memristor model. The Adam optimizer is utilized as it
performs well on both recurrent networks and stochastic problems [23]. For the
Adam optimizer, the learning rate is set to $10^{-4}$, the initial decay rates
for first and second order moments are set to $\beta_{1}=0.9$,
$\beta_{2}=0.999$. The learnable weights of dense layers are initialized by
sampling from a uniform distribution $U(-\sqrt{k},\sqrt{k})$, where
$k=\frac{1}{N_{i}}$ and $N_{i}$ is the number of input features to layer $i$.
The MSNN is trained for 40 epochs using a batch size of 200 samples. Accuracy
is evaluated at every 10 training iterations. For the MNIST dataset, the
training and testing accuracies across multiple epochs are shown in Fig. 5.
97.58$\%$ accuracy is achieved for the total MNIST test set. For a more
challenging task using real-world data, we also train our MSNN on the Fashion-
MNIST dataset, where we obtain 75.26$\%$ testing accuracy. This demonstrates
the first functional result on the Fashion-MNIST dataset using analog MSNNs,
and highlights the potential for MSNNs to be capable of moving beyond simple
MNIST classification.

(a)

(b)

Figure 5: Accuracy across epochs for training and testing processes for (a)
MNIST dataset (b) Fashion-MNIST dataset.

##  IV Discussion and Conclusion

We have confirmed the compatibility of circuit-level MSNNs with error-driven
supervised learning. Our primary results show that MIF neurons based on SPICE
models can be trained in a multi-layer neural network using BPTT, side-
stepping the need for surrogate gradient signals. To the best of our
knowledge, this is the first work to perform gradient descent directly on a
network of SPICE models of memristive neurons that emit analog spikes.

While MSNNs are extremely power efficient, they are also highly challenging to
train. We achieve state of the art accuracy on MNIST and Fashion-MNIST
datasets when compared to all other networks of spiking MSNNs. Prior attempts
on the MNIST dataset using memristive spiking neurons with error-driven
optimization [24] only achieved 83.24$\%$. The approach in [24] aims to fit
memristive neurons to LIF neuron models, whereas our approach directly relies
on the gradient of the naturally arising memristive dynamics during training,
in order to achieve the accuracy boost to 97.58$\%$.

Although SNNs add more latency for training due to the additional temporal
dimension, there are many reasons why this approach is valuable and holds
great potential:

  * •

SNNs can exploit sparse data connections and activations to reduce memory
access frequency, which we exploit by only presenting input data for 3 out of
the total 1,000 time steps of simulation. [25, 26, 17, 27].

  * •

A large-scale neural network based on the MIF neuron has a much higher density
compared with the conventional CMOS counterparts by taking advantage of
vertical BEOL CMOS integration.

  * •

Area and power can be saved as less peripheral circuits are needed by
eliminating the need for analog-to-digital converters (ADCs) and digital-to-
analog converters (DACs) in bit-line current summation approaches; i.e., all
spiking is a result of the natural evolution of the memristor’s state
dynamics.

  * •

The natural temporal dynamics of the MIF model allow for differentiable spike
propagation through a MSNN, eliminating the need for biased gradient
estimators such as surrogate gradients.

Future work will involve further increasing the complexity of the datasets,
accounting for the non-idealities present in memristive synapses, and
stochasticity that is not designed in the SPICE model yet.

## References

  * [1] J. K. Eshraghian, M. Ward, E. Neftci, X. Wang, G. Lenz, G. Dwivedi, M. Bennamoun, D. S. Jeong, and W. D. Lu, “Training spiking neural networks using lessons from deep learning,” _arXiv preprint arXiv:2109.12894_ , 2021\.
  * [2] M. R. Azghadi, C. Lammie, J. K. Eshraghian, M. Payvand, E. Donati, B. Linares-Barranco, and G. Indiveri, “Hardware implementation of deep network accelerators towards healthcare and biomedical applications,” _IEEE Transactions on Biomedical Circuits and Systems_ , vol. 14, no. 6, pp. 1138–1159, 2020.
  * [3] P. Zhou, D.-U. Choi, J. K. Eshraghian, and S.-M. Kang, “A fully memristive spiking neural network with unsupervised learning,” _arXiv preprint arXiv:2203.01416_ , 2022.
  * [4] P. Zhou and S. Abbaszadeh, “Towards real-time machine learning for anomaly detection,” in _2020 IEEE Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC)_. IEEE, pp. 1–3.
  * [5] P. J. Werbos, “Backpropagation through time: what it does and how to do it,” _Proceedings of the IEEE_ , vol. 78, no. 10, pp. 1550–1560, 1990.
  * [6] L. O. Chua, “Memristor-the missing circuit element,” _IEEE Transactions on Circuit Theory_ , vol. 18, no. 5, pp. 507–519, 1971.
  * [7] L. O. Chua and S. M. Kang, “Memristive devices and systems,” _Proceedings of the IEEE_ , vol. 64, no. 2, pp. 209–223, 1976.
  * [8] “eflash,” https://www.tsmc.com/english/dedicatedFoundry/technology/
specialty/eflash.

  * [9] “Mad200,” https://mycmp.fr/nvm-mad200/.
  * [10] S. H. Jo, T. Chang, I. Ebong, B. B. Bhadviya, P. Mazumder, and W. Lu, “Nanoscale memristor device as synapse in neuromorphic systems,” _Nano letters_ , vol. 10, no. 4, pp. 1297–1301, 2010.
  * [11] M. Rahimi Azghadi, Y.-C. Chen, J. K. Eshraghian, J. Chen, C.-Y. Lin, A. Amirsoleimani, A. Mehonic, A. J. Kenyon, B. Fowler, J. C. Lee _et al._ , “Complementary metal-oxide semiconductor and memristive hardware for neuromorphic computing,” _Advanced Intelligent Systems_ , vol. 2, no. 5, p. 1900189, 2020.
  * [12] F. Cai, J. M. Correll, S. H. Lee, Y. Lim, V. Bothra, Z. Zhang, M. P. Flynn, and W. D. Lu, “A fully integrated reprogrammable memristor–cmos system for efficient multiply–accumulate operations,” _Nature Electronics_ , vol. 2, no. 7, pp. 290–299, 2019.
  * [13] T. Serrano-Gotarredona, T. Masquelier, T. Prodromakis, G. Indiveri, and B. Linares-Barranco, “Stdp and stdp variations with memristors for spiking neuromorphic learning systems,” _Frontiers in neuroscience_ , vol. 7, p. 2, 2013.
  * [14] M. D. Pickett, G. Medeiros-Ribeiro, and R. S. Williams, “A scalable neuristor built with mott memristors,” _Nature materials_ , vol. 12, no. 2, pp. 114–117, 2013.
  * [15] J. K. Eshraghian, K. Cho, C. Zheng, M. Nam, H. H.-C. Iu, W. Lei, and K. Eshraghian, “Neuromorphic vision hybrid rram-cmos architecture,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , vol. 26, no. 12, pp. 2816–2829, 2018.
  * [16] X. Zhang, W. Wang, Q. Liu, X. Zhao, J. Wei, R. Cao, Z. Yao, X. Zhu, F. Zhang, H. Lv _et al._ , “An artificial neuron based on a threshold switching memristor,” _IEEE Electron Device Letters_ , vol. 39, no. 2, pp. 308–311, 2017.
  * [17] Y. Zhang, Z. Wang, J. Zhu, Y. Yang, M. Rao, W. Song, Y. Zhuo, X. Zhang, M. Cui, L. Shen _et al._ , “Brain-inspired computing with memristors: Challenges in devices, circuits, and systems,” _Applied Physics Reviews_ , vol. 7, no. 1, p. 011308, 2020.
  * [18] Z. Wang, S. Joshi, S. Savel’ev, W. Song, R. Midya, Y. Li, M. Rao, P. Yan, S. Asapu, Y. Zhuo _et al._ , “Fully memristive neural networks for pattern classification with unsupervised learning,” _Nature Electronics_ , vol. 1, no. 2, pp. 137–145, 2018.
  * [19] F. Kiani, J. Yin, Z. Wang, J. J. Yang, and Q. Xia, “A fully hardware-based memristive multilayer neural network,” _Science advances_ , vol. 7, no. 48, p. eabj4801, 2021.
  * [20] E. O. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks,” _IEEE Signal Processing Magazine_ , vol. 36, no. 6, pp. 51–63, 2019.
  * [21] S. M. Kang, D. Choi, J. K. Eshraghian, P. Zhou, J. Kim, B.-S. Kong, X. Zhu, A. S. Demirkol, A. Ascoli, R. Tetzlaff, and L. O. Chua, “How to build a memristive integrate-and-fire model for spiking neuronal signal generation,” _IEEE Transactions on Circuits and Systems I: Regular Papers_ , 2021.
  * [22] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [24] Q. Duan, Z. Jing, X. Zou, Y. Wang, K. Yang, T. Zhang, S. Wu, R. Huang, and Y. Yang, “Spiking neurons with spatiotemporal dynamics and gain modulation for monolithically integrated memristive neural networks,” _Nature communications_ , vol. 11, no. 1, pp. 1–13, 2020.
  * [25] C. D. Schuman, T. E. Potok, R. M. Patton, J. D. Birdwell, M. E. Dean, G. S. Rose, and J. S. Plank, “A survey of neuromorphic computing and neural networks in hardware,” _arXiv preprint arXiv:1705.06963_ , 2017.
  * [26] W. Wang, W. Song, P. Yao, Y. Li, J. Van Nostrand, Q. Qiu, D. Ielmini, and J. J. Yang, “Integration and co-design of memristive devices and algorithms for artificial intelligence,” _Iscience_ , p. 101809, 2020.
  * [27] L. A. Camuñas-Mesa, B. Linares-Barranco, and T. Serrano-Gotarredona, “Neuromorphic spiking neural networks and their memristor-cmos hardware implementations,” _Materials_ , vol. 12, no. 17, p. 2745, 2019.