# NU-AIR - A Neuromorphic Urban Aerial Dataset for Detection and Localization
of Pedestrians and Vehicles

Craig Iaboni, Thomas Kelly, Pramod Abichandani
New Jersey Institute of Technology
<EMAIL_ADDRESS>
###### Abstract

Annotated imagery capturing pedestrians and vehicles in an urban environment
can be used to train Neural Networks (NNs) for machine vision tasks. This
paper presents the first open-source aerial neuromorphic dataset that captures
pedestrians and vehicles moving in an urban environment. The dataset, titled
NU-AIR, features 70.75 minutes of event footage acquired with a 640 $\times$
480 resolution neuromorphic sensor mounted on a quadrotor operating in an
urban environment. Crowds of pedestrians, different types of vehicles, and
street scenes at a busy urban intersection are captured at different
elevations and illumination conditions. Manual bounding box annotations of
vehicles and pedestrians contained in the recordings are provided at a
frequency of 30 Hz, yielding 93,204 labels in total. Evaluation of the
dataset’s fidelity is performed by training three Spiking Neural Networks
(SNNs) and ten Deep Neural Networks (DNNs). The mean average precision (mAP)
accuracy results achieved for the testing set evaluations are on-par with
results reported for similar SNNs and DNNs on established neuromorphic
benchmark datasets. All data and Python code to voxelize the data and
subsequently train SNNs/DNNs has been open-sourced.

##  1 Introduction

Figure 1: (Top Left and Right) The DJI M-100 quadrotor was used to record
pedestrians and vehicles in urban environments. A safety rope was affixed to
the quadrotor during all flight operations. The recording quadrotor flew at
varying heights above a city intersection (Mid and Bottom Left), a walking
path (Mid Right), and a campus center (Bottom Right). Bounding boxes created
by manual annotators were drawn over frames.

Event cameras encode visual information as asynchronous events instead of
frames [22]. An event is a change in pixel illumination intensity and consists
of the pixel’s position $(x,y)$, the timestamp $t$ of the illumination change,
and the polarity $p$ indicating that the illumination intensity increased (p =
1) or decreased (p = 0). Event cameras have higher temporal resolution,
dynamic range and power efficiency compared to traditional frame-based
cameras, making them popular for applications in autonomous vehicles, and
robotics/IoT systems where low latency and robustness to lighting conditions
important.

Large-scale datasets are crucial for developing robust and effective computer
vision algorithms. Event cameras are relatively expensive compared to RGB
cameras, making it challenging for many researchers and practitioners to
access them. The availability of nueromorphic datasets is relatively low [47,
54, 39, 29, 28, 9, 5], and only a handful of them feature urban settings [41,
38, 53, 14, 46], with none of the existing neuromorphic urban datasets
providing aerial footage leaving a gap in the neuromorphic dataset landscape.
As depicted in Figure 1, this study adddresses this gap by introducing a
neuromorphic dataset consisting of event-recording segments captured from a
quadrotor-mounted event camera observing an urban environment during both
daytime and nighttime settings. The dataset is fully annotated, enabling the
development and evaluation of new event-based vision algorithms for urban
settings. The key contributions of this study are as follows:

  1. 1.

Open source Neuromorphic Urban Aerial (NU-Air) Dataset – The collection is
composed of 70.75 minutes of recording, which has been divided into 15 second
segments. A total of 283 event recording segments are provided. The dataset
was recorded from a quadrotor-mounted event camera observing three environment
variations ranging from campus, traffic intersection, and walking path scenes.
As such, the dataset provides a diverse set of visual stimuli for event-based
vision algorithm design work. Sequences are included for both daytime and
nighttime settings. Figure 1 depicts example stills from the dataset as well
the quadrotor platform in action. The collection is fully annotated for two
classes of detection: person and vehicle. Annotations are in the form of
upright rectangular bounding boxes and corresponding class labels. In total,
93204 annotations are provided.

  2. 2.

Evaluation of state-of-the-art neuromorphic vision algorithms: We evaluate 13
existing neuromorphic vision algorithms (3 SNNs and 10 DNNs) on our dataset
and report on their performance along with insights about the comparative
performance. In particular, Experimental evaluation of the dataset proposed in
this work has been performed by training Visual Geometry Group (VGG) [52],
DenseNet [27], and MobileNet [26] SNNs which have been previously benchmarked
in the literature on a large collection of neuromorphic data [12]. The results
of the evaluation are competitive to the benchmarking results for each trained
SNN.

The dataset as well as associated Python code to voxelize the dataset and
train 3 SNNs and 10 DNNs has been made publicly available through the
following repository link: https://bit.ly/nuair-data

##  2 Related Works

A number of datasets in the literature have reported on pedestrian and vehicle
detection in urban environments.

###  2.1  Frame-based Urban Datasets

Traditional frame-based cameras for object detection of real-world urban
scenes have been used to create several pedestrian detection datasets for
training and evaluation purposes, including ETH [19], INRIA [13], PRW [59],
TUD-Brussels [57], and Daimler [18]. Larger and more diverse collections such
as KITTI [23], Caltech-USA [16], UA-DETRAC [56], CityPersons [58], EuroCity
Persons [7], TRANCOS [24], and TJU-DHD [45] have allowed researchers and
practitioners to push the limits of algorithmic performance at scale. These
sets were recorded by driving or walking through urban areas and provide
annotated frames from video sequences.

Drone-mounted RGB cameras have been utilized to generate multiple real-world
aerial datasets. The Urban Drone Dataset (UDD) [10] features diverse urban
scenes from 10 video sequences captured in 4 cities in China. It includes
semantic annotations for 6 detection classes and comprises of 160 training and
45 validation images. The UAV123 [40] dataset includes 123 aerial video
sequences, with bounding box annotations provided for every sequence. The
VizDrone [60] dataset has 288 video clips and 10,209 static images captured
from drone-mounted cameras, manually annotated with over 2.6 million bounding
boxes. The OpenDD [8] dataset covers seven roundabouts in Germany, including
over 80,000 road bounding boxes in 62+ hours of data. The UAVDT [17] dataset
includes 100 video sequences with approximately 80,000 representative frames,
captured at various urban locations such as highways and crossings.

###  2.2  Event-based Camera Datasets

Early event-based datasets have been created by converting frame-based
datasets to event format for benchmarking tasks [44, 50, 21, 34]. The benefit
of these methods is that large datasets can be generated without manual
labeling. However, the low screen refresh rate of displays create unnatural
event sequences. As a result, recent efforts have centered on creating real-
world datasets for detection and recognition of persons, animals, objects,
robots, and quadrotors [5, 9, 54, 47, 39, 29, 28]. The majority of these
event-based datasets have been collected in controlled, indoor environments.

####  2.2.1 Real-World Urban Event Datasets

The following discussions highlight the fact that only a handful of studies
have provided open-source image/video datasets capturing urban settings [41,
38, 53, 14, 46].

  * •

The Scenes-DVS [41] set consists of recordings of long hikes in urban
environments, with sequences of indoor and outdoor environments exceeding 15
minutes long split into small 50ms sequences. Scene classification accuracy
was evaluated with a frame based single-layer and three multi-layer SNN
configurations (with varying hidden layer sizes) [42]. The highest accuracy
reported for the single-layer architecture and multi-layer architecture was
81.2% and 84.4%, respectively.

  * •

The PAFBenchmark pedestrian detection dataset [38] contains 4670 frame images
at 20ms time intervals with bounding box annotations of scenarios seen in
traffic surveillance tasks such as pedestrian overlapping, occlusion, and
collision.

  * •

N-Cars [53] is a large, real-world event-based dataset for car classification.
It is composed of 12,336 car samples and 11,693 non-car samples (background).
Each sample lasts 100ms. Classification accuracy for the two-class problem was
evaluated using a linear SVM classifier. 90% classification accuracy was
reported for this evaluation.

  * •

The Gen1 Automotive [14] contains 39 hours of open road and various driving
scenarios ranging from urban, highway, suburbs, and countryside scenes. Manual
bounding box annotations are provided for two classes of detection: pedestrain
and cars. The capture resolution of this set is 304 $\times$ 240\. The data
has not been evaluated for classification or detection accuracy in this study.

  * •

The 1 Megapixel Automotive [46] dataset includes 14 hours of annotated
vehicles and pedestrians captured within roadway scenes. This set is the
highest resolution neuromorphic object detection dataset from a vehicle-
mounted perspective to date, with a capture resolution of 1280 $\times$ 1024\.
A recurrent ConvLSTM architecture is used to evaluate the proposed dataset
[51]. A mAP of 0.43 was reported for automotive detection using the proposed
architecture.

The proposed dataset differs from previously reported datasets in that it
includes data collected in urban environments from a quadrotor-mounted
neuromorphic camera with accurate bounding box annotations for the multi-class
detection problem against complex and dynamic backgrounds.

The remainder of this paper is organized as follows. In Section 3, the
recording setup, annotation procedure, and dataset format are discussed. In
Section 4, the use of three SNNs and ten DNNs to benchmark the performance of
the proposed dataset is described. Section 5 concludes the paper with some
discussions and future directions.

Figure 2: The DJI Matrice M100 quadrotor with a forward-facing Prophesee
Gen3.1 VGA event camera was used for data collection.

##  3 Dataset

###  3.1 Recording Setup

This study used a Prophesee Gen3.1 VGA (640 $\times$ 480) resolution event
camera [4]. This camera operates on 1.8 V supplied via USB, with a 10 $mW$
power dissipation rating in low-power mode. The brightness intensity
measurements from the sensor were used to create gray-scale frames at a
frequency of 30 Hz, which were then manually annotated by human subjects to
generate ground-truth bounding boxes around areas of interest.

As shown in Figure 2, the camera was mounted on a DJI M100 quadrotor with a
$10^{\circ}$ downward mounting angle [1]. The camera was connected to an on-
board NVidia Jetson Xavier AGX computer for data recording via USB [3]. The
recording setup and quadrotor were powered by isolated batteries to ensure
reliable operations of the entire setup. The Jetson was powered by a 28800 mAh
capacity DC 12V power bank, while the quadrotor was equipped with a 5700 mAh
capacity TB48D battery [2]. A safety rope, as shown in Figure 1, was attached
to the quadrotor for all flight tests.

The data was collected from three different real-world scenes - a crowded
university campus center, a walking path with low foot traffic, and a city
intersection with high vehicle traffic in New Jersey, USA. The quadrotor was
translated (roll and pitch) and rotated (using yaw control) while collecting
the footage. The recordings ranged from one to five minutes in duration and
were captured over the course of one week in December 2022, at varying times
of the day to include different lighting and weather conditions. A total of
70.75 minutes, split among 14 recordings, were collected, yielding 238GB of
raw event data. In comparison, a gray-scale frame-based camera with the same
resolution and operating at a frequency of 120fps (i.e. 100 times lower
temporal resolution compared to the event camera) would have generated over
359GB of data 111Ignoring compression and assuming 1 byte per pixel.

Figure 3: The ratio of width to height, known as the aspect ratio, is
displayed for the manually labeled bounding boxes of pedestrians (top) and
vehicles (bottom) in the dataset. The histograms for training, validation, and
testing subsets of pedestrian and vehicle subjects are depicted.  Figure 4:
The diagonal distance is displayed for the manually labeled bounding boxes of
pedestrians (top) and vehicles (bottom) in the dataset. Objects smaller than
10 pixels have been ignored in the annotation process. The statistics for
training, validation, and testing subsets of pedestrian and vehicle subjects
are depicted.

###  3.2 Manual Annotation

By using the change detection events provided by the neuromorphic sensor, a
gray-scale image can be created at any desired time. Images created in this
way have the same temporal resolution as the events stream, and the time of
the last event used to generate them is associated to the image. The
annotations on the gray-scale images can be directly used as ground truth for
the events stream without the need for calibration or correction steps, as the
gray-scale images and events stream share the same pixel array [53].

High-frequency annotations are preferred to leverage the high temporal
resolution of the event camera. As such, gray-scale images in this set were
generated at 30Hz and given to human annotators to draw bounding boxes around
pedestrians and vehicles. Instructions were provided to the annotators to
minimize ambiguity and inconsistencies among annotations. Objects smaller than
10 pixels or partially occluded (less than 50% visible) were ignored. Cars,
buses, trucks, and other large vehicles were considered within the vehicle
category and were annotated as such. Persons observed within cars or buildings
were not annotated. A total of 66958 vehicle and 26246 pedestrians bounding
boxes have been annotated. The annotations produced were reviewed by two
experts to ensure location accuracy and subject correctness. Example gray-
scale images with manual annotations are shown in Figure 1.

###  3.3 Dataset Format

The dataset was captured across 14 continuous recording sessions. In order to
facilitate the training, the continuous recordings were divided into 15
seconds chunks, resulting in a total of 283 samples. The total samples were
divided into training, validation, and testing subsets with 197, 57, and 29
samples in each subset, respectively. To avoid overlap between subsets, events
from each continuous recording session were assigned to the same subset.

Each sample was provided in a binary .dat format, where events were encoded
using 4 bytes for the timestamps and 4 bytes for the position and the
polarity. Specifically, $14$ bits are used for the $x$ position, $14$ bits for
the $y$ position, and $1$ bit for the polarity. Python code to generate gray-
scale images from binary files are provided with the dataset.

Bounding box annotations follow a numpy format [25]. Each numpy array contains
the following fields:

  * •

$t_{s}$, timestamp of the box in microseconds

  * •

$x$, abscissa of the top left corner in pixels

  * •

$y$, ordinate of the top left corner in pixels

  * •

$w$, width of the box in pixels

  * •

$h$, height of the box in pixels

  * •

${class}_{{id}}$, class of the object: 0 for pedestrians and 1 for vehicles

Python code to load and view samples from the dataset with the corresponding
annotations visualized has been provided. Within the released code repository,
an example on how to apply the evaluation metrics on the dataset is provided.

##  4 Experimental Evaluation

The histogram of pedestrian and vehicle bounding box aspect ratio (box width
over box height), is displayed in Figure 3. The histograms have been computed
individually on the training, validation, and testing subsets. The histogram
data has been made available for readers to perform any desired statistical
analyses. It was observed that the training, validation, and testing subsets
showed similar statistics.

For all pedestrians bounding boxes, a mean aspect ratio of 0.46 was observed.
For all vehicle bounding boxes, a mean aspect ratio of 2.02 was observed.
These values are due to the consistent point of view of the quadrotor in which
pedestrians are captured upright from a rotated overhead perspective, and
vehicles are captured from the side or top.

Additionally, the histogram of pedestrian and vehicle bounding box diagonal
distance is shown in Figure 4. The diagonal distance of pedestrian and vehicle
bounding boxes follows a long tail distribution. The mean diagonal distance
for pedestrian bounding boxes was 47.75 pixels, while the mean diagonal for
vehicle bounding boxes was 90.17 pixels.

###  4.1 Experimental Evaluation

####  4.1.1 Voxel Cube Encoding

Voxel cubes were used to encode event data [12]. This approach offers
advantages over event encoding strategies such as event frames, two-channel
images, and time surfaces in that events are not collapsed to a 2D grid, thus
conserving both binarity and temporal information while allowing for batched
processing of events [22]. The voxel cube encoding method is inspired by voxel
grid encoding. Voxel grid encoding is an encoding method in which a series of
bins are used to convert the temporal dimension to a set of event volumes [6].
A voxel grid is represented by a 3D THW tensor of events, where T is the
number of timesteps, and H and W are the height and width of the data
respectively [6]. The accumulation of events within the time interval is
either a sum [20] or a binary accumulation [11].

In the case of binary accumulation, when events within the represented time
interval do not overlap on a particular voxel, the complete set of events can
be retrieved from the representation. When multiple events intersect on a
voxel, only the most recent event is represented in the voxel. Thus, temporal
information describing the frequency of the event occurrence at the given
voxel is lost, but the volume still reflects the distribution of events across
spatial and temporal dimensions.

By contrast, the voxel cube encoding technique subdivides each event volume
into smaller sub-volumes, referred to as voxel cubes. A voxel cube is
represented by a 4D CTHW tensor, where C is the number of channels, T is the
number of timesteps, and H and W are the height and width of the data
respectively [12]. The events within a sub-volume are assigned to a cell
depending on position (x,y) and to a sub-volume depending on timestamp (t).
Rather than counting event frequency like a histogram, we count each event
based on its temporal distance from the center of neighboring sub-volumes. The
value of each event is distributed over the two nearest sub-volumes, and
weighted by the temporal distance between the event and the centroids of
neighboring sub-volumes. In this way, temporal information is preserved as all
events that fall within a sub-volume give a varying contribution based on the
event timestamp.

####  4.1.2 Spiking Neural Networks

To assess the suitability of the dataset for object detection tasks, the
following three SNNs have been trained and their performance has been
evaluated in terms of accuracy.

  1. 1.

Spiking VGG [52]: VGG (Visual Geometry Group) is a convolutional neural
network that consists of up to 19 convolutional layers, followed by 3 fully-
connected layers. The spiking variant of VGG includes batch normalization
layers added before each spiking convolutional layer. The VGG SNN in this work
has 11 convolutional layers.

  2. 2.

Spiking DenseNet [27]: DenseNet is an architecture that promotes gradient
propagation by using channel-wise concatenations, which is an operation that
preserves spike representation. DenseNet configurations are defined by the
depth (number of layers within the network) and growth rate (how much new
information each layer contributes to the global state). For the spiking
variant, ReLU activations are replaced with Parametric Leaky Integrate-and-
Fire (PLIF). The PLIF neuron is a spiking neuron with learnable membrane
parameters, thus enhancing the expressiveness of the SNN [20]. The spiking
DenseNet architecture in this work used a depth of 120 convolution layers and
1 fully connected layer with a growth rate of 24.

  3. 3.

Spiking MobileNet [26]: MobileNet is a model intended for use in mobile
applications that use depth-wise separable convolutions which need fewer
parameters and computations compared to traditional convolutions. By varying
the number of input channels, MobileNet can be evaluated at various sizes. The
spiking version of MobileNet removes the activation function between the
depth-wise and point-wise convolutions, and positions all batch normalization
layers before the convolutional layers. The spiking MobileNet in this work
used 16 input channels.

SNN Methods | Params | ACC/ts | Sparsity | mAP (NU-AIR) | mAP [12]
---|---|---|---|---|---
VGG-11 + SSD | 12.6M | 11.1G | 23.1% | 0.17 | 0.17
DenseNet121-16 + SSD | 8.2M | 2.3G | 38.1% | 0.17 | 0.19
MobileNet-16 + SSD | 24.3M | 4.3G | 27.9% | 0.13 | 0.14
Table 1: Comparison of the COCO mean average precision (mAP) for three spiking neural networks (SNNs) trained on the presented dataset. The table displays the network size in millions of parameters, the number of accumulations per timestep, and the network’s sparsity. The results are compared to the performance of the Prophesee GEN1 dataset, as reported in [12]. DNN Methods | Params | mAP
---|---|---
YOLOv5n | 1.9M | 0.37
YOLOv5s | 7.2M | 0.39
YOLOv6n | 4.3M | 0.41
YOLOv6s | 17.2M | 0.39
YOLOv7t | 6.2M | 0.36
YOLOv7 | 36.9M | 0.53
YOLOv8n | 3.2M | 0.42
YOLOv8s | 11.2M | 0.44
Faster-RCNN | 19M | 0.42
RetinaNet | 36.4M | 0.37
Table 2: The COCO mean average precision (mAP) for 10 object detection models
trained on the presented dataset, including YOLOv5n, YOLOv5s, YOLOv6n,
YOLOv6s, YOLOv7t, YOLOv7, YOLOv8n, YOLOv8s, Faster-RCNN, and RetinaNet.

The SNNs are combined with Single Shot MultiBox Detector (SSD) bounding box
regression heads to perform object detection. The SSD object detection
framework consists of a spiking classifier backbone and multiple predictor
heads [37]. The heads take feature maps generated by the classifier at
different scales as input, and outputs bounding boxes with associated classes.
Readers are referred to [12] for further implementation details about the SNNs
used in this work.

For the task of object detection, the mean Average Precision (mAP) over 10 IoU
([0.5:0.05:0.95] – the COCO mAP) is reported [36]. However, assessing the
performance of SNNs is not limited to mAP, as multiple others features are
needed to take advantage of their benefits when embedded in specialized
hardware. For all results, the following metrics are reported:

  * •

Number of parameters: To meet the high memory constraints of embedded systems,
the network size is measured in terms of the number of parameters.

  * •

Number of accumulation operations (ACCs) : SNNs do not require multiplicative
operations, enabling substantial energy savings on specialized hardware. The
number of accumulation operations (ACCs) is reported to highlight potential
energy savings. All spiking convolutions operations amount to ACCs, and each
PLIF neuron only requires 1 ACC per timestep to update its potential. Batch
normalization layers are not included in the ACCs count.

  * •

Sparsity: The number of spikes emitted after each activation layer were
measured to represent the global sparsity of the network compared to an fully
dense equivalent Deep Neural Network (DNN). Processing events with SNNs
preserves the data sparsity [32]. On specialized hardware, computations are
only performed when there are spikes, therefore an highly sparse network would
consume less power than its dense counterpart. The sparsity is obtained by
averaging the number of spikes over the whole test set.

####  4.1.3 Frame-based DNNs

A total of 10 single-stage or two-stage frame-based object detection DNNs were
used in this study and their COCO mean average precision (mAP) metric has been
reported. The models include YOLOv5n [31], YOLOv5s [31], YOLOv6n [33], YOLOv6s
[33], YOLOv7t [55], YOLOv7 [55], YOLOv8n [30], YOLOv8s [30], Faster-RCNN [48],
and RetinaNet [35]. The evaluation involved selecting the two smallest
configurations of each variant of the YOLO object detector network, except for
YOLOv7, where the smallest two networks were named YOLOv7-tiny and YOLOv7. The
network size selection was made to evaluate the networks’ performances on edge
devices that compete with low-power neuromorphic hardware. For Faster-RCNN and
RetinaNet, only one variant each was evaluated. The number of parameters in
each model is reported in millions. The sparsity and ACC/ts metrics were not
reported as these networks do not use spiking principles or accumulation
operations.

###  4.2 Object Detection Results

####  4.2.1 SNN Results

For SNN object detection models, three configurations of classifier and SSD
network are used for evaluation. The COCO mAP accuracy results of the three
SNNs trained on this dataset are given in Table 1.

  * •

The VGG-11 + SSD network configuration achieved a COCO mAP of 0.17, which is
tied with DenseNet121-16 + SSD for the highest evaluation metric observed in
this work. The size of this network configuration was 12.6 million parameters,
with 11.1 billion accumulation operations per timestep, and a sparsity of
23.1%.

  * •

The DenseNet121-16 + SSD network configuration achieved a COCO mAP of 0.17.
The network contained 8.2 million parameters. The number of accumulation
operations per timestamp for this network was the least among the evaluated
SNNs at 2.3 billion. At 38.1% sparsity, this network was the most dense of the
evaluated SNNs.

  * •

The MobileNet-16 + SSD network configuration achieved a COCO mAP of 0.13. This
was the lowest reported mAP result among the three SNNs used for dataset
evaluation. This network configuration was the largest of the evaluation SNNs
at 24.3 million parameters. This network performed 4.3 billion accumulation
operations per timestep, with a sparsity of 27.9%.

####  4.2.2 DNN Results

Table 2 depicts the performance of various DNNs for the NU-AIR dataset. From
the table, it can be seen that YOLOv7 achieved the highest mAP of 0.53, and
RetinaNet achieved the lowest mAP of 0.37. An observation from the table is
that the model’s network size is related to its performance. In particular, we
can observe that the larger models, such as YOLOv7 and RetinaNet, achieved
higher mAP scores compared to smaller models such as YOLOv5n and YOLOv5s. This
observation is consistent with the general trend in deep learning, where
larger models tend to perform better (but end up requiring more computation
and memory resources) [49].

Additionally, it was observed that the YOLOv7-tiny variant achieved a lower
mAP score of 0.36 compared to its larger variant, YOLOv7, which achieved a
high mAP score of 0.53. This trend can also be seen for the results on YOLOv5
and YOLOv8 network variants. This observation suggests that, for similar
network architectures, reducing the network size results in a drop in
performance.

####  4.2.3 Discussion

For comparison, the SNN results achieved above are benchmarked against the
COCO mAP results achieved in [12]. The authors in [12] trained their model on
the Prophesee GEN1 dataset. It is observed that the SNNs trained on the NU-AIR
dataset achieve mAP close to the mAP of the comparison study [12]. The COCO
mAP of the VGG-11+SSD model reached 0.17, and is the highest accuracy reported
of the three trained models. However, [12] reported that DenseNet121-24+SSD
model achieved the highest accuracy in their work.

In terms of mAP, the VGG-11 + SSD and DenseNet121-16 configurations reported a
higher object detection performance than the MobileNet-16 + SSD configuration.
Note that the DenseNet121-16 configuration achieved the mAP score with a
smaller network and less accumulations than the VGG-11 architecture. This
trade-off in performance may be valuable on neuromorphic systems with
constrained amounts of neurons. However, VGG-11 uses the most sparse network
structure, which is best suited for use in power constrained embedded
hardware.

The lower performance of the MobileNet architecture was observed to be related
to the use of depth-wise separable convolutions. In [12], depth-wise separable
convolutions were shown to reduce performance in smaller networks (less than
32 input channels) when compared to normal convolutions. Although depth-wise
separable convolutions enable the use of a smaller number of network
parameters, thus making the training of the SNN with surrogate gradients a
simpler task, small networks were shown to benefit from the additional network
size for a 5% increase in accuracy. Thus, a larger variant of MobileNet
(larger than 16) can be used to yield an increase in network performance.

The use of convolution layers, which are particularly well-suited to
processing frame-based data, provides an performance advantage to the
evaluated ANNs which have used frame-based data as input and leverage GPU
computation during training [15].

####  4.2.4 Limitations

Some limitations of the presented work are discussed here. First, the SNNs
evaluated in this study were trained and evaluated on GPU hardware, which
differs from the low-power neuromorphic hardware that is the intended target
for this type of application. Therefore, the reported performance may not be
representative of how these models would perform in real-world edge devices.
The recently released Intel Loihi 2 could be used to further enhance the
dataset benchmarking experiments and results [43].

Second, the DNNs evaluated in this study were trained on frame-based grey-
scale images and did not leverage color information or the temporal
information of spikes. This limited approach may have negatively impacted the
object detection performance of these models.

Third, the data used to train and evaluate the models was collected from a
single city in New Jersey, which may limit the generalizability of the
reported results to other settings. Future work should evaluate the
performance of these models using data collected from multiple urban settings
to assess the generalizability of the models.

##  5 Conclusions

The first neuromorphic drone dataset for pedestrian/vehicle detection and
localization in urban environments was presented. The dataset was used to
train VGG, MobileNet, and DenseNet SNNs to evaluate object detection and
localization tasks. Ten frame-based single- and two-stage object detector DNNs
were used to evaluate the mAP of the dataset on object detection tasks. The
experimental accuracy results of SNNs were competitive with results observed
with matching networks trained on other large scale, contemporary datasets.
The dataset has been fully annotated and is open sourced along with all
accompanying Python code for the community. Future work will extend this study
to perform instance and semantic segmentation on a wider variety of urban
scenes.

## References

  * [1] Dji matrice m100: Quadcopter for developers.  https://www.dji.com/matrice100, Jan 2022.
  * [2] Matrice 100 tb48d battery.  https://store.dji.com/product/matrice-100-tb48d-battery, Jan 2022.
  * [3] Nvidia jetson agx xavier development kit.  https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-agx-xavier/, Jan 2022.
  * [4] Prophesee evk1 - gen3.1 vga.  https://docs.prophesee.ai/stable/hw/evk/gen3.html, Jan 2022.
  * [5] Arnon Amir, Brian Taba, David J. Berg, Timothy Melano, Jeffrey L. McKinstry, Carmelo di Nolfo, Tapan Kumar Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeffrey A. Kusnitz, Michael V. DeBole, Steven K. Esser, Tobi Delbrück, Myron Flickner, and Dharmendra S. Modha.  A low power, fully event-based gesture recognition system.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7388–7397, 2017.
  * [6] Patrick Bardow, Andrew J Davison, and Stefan Leutenegger.  Simultaneous optical flow and intensity estimation from an event camera.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 884–892, 2016.
  * [7] Markus Braun, Sebastian Krebs, Fabian Flohr, and Dariu M Gavrila.  Eurocity persons: A novel benchmark for person detection in traffic scenes.  IEEE transactions on pattern analysis and machine intelligence, 41(8):1844–1861, 2019.
  * [8] Antonia Breuer, Jan-Aike Termöhlen, Silviu Homoceanu, and Tim Fingscheidt.  opendd: A large-scale roundabout drone dataset.  In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1–6. IEEE, 2020.
  * [9] Shoushun Chen, Polina Akselrod, Bo Zhao, Jose Antonio Perez Carrasco, Bernabe Linares-Barranco, and Eugenio Culurciello.  Efficient feedforward categorization of objects and human postures with address-event image sensors.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(2):302–314, 2012.
  * [10] Yu Chen, Yao Wang, Peng Lu, Yisong Chen, and Guoping Wang.  Large-scale structure from motion with semantic constraints of aerial images.  In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 347–359. Springer, 2018.
  * [11] Loïc Cordone, Benoît Miramond, and Sonia Ferrante.  Learning from event cameras with sparse spiking convolutional neural networks.  In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2021.
  * [12] Loic Cordone, Benoît Miramond, and Phillipe Thierion.  Object detection with spiking neural networks on automotive event data.  In Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN), July 2022.
  * [13] Navneet Dalal and Bill Triggs.  Histograms of oriented gradients for human detection.  In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), volume 1, pages 886–893. Ieee, 2005.
  * [14] Pierre De Tournemire, Davide Nitti, Etienne Perot, Davide Migliore, and Amos Sironi.  A large scale event-based detection dataset for automotive.  arXiv preprint arXiv:2001.08499, 2020.
  * [15] Lei Deng, Yujie Wu, Xing Hu, Ling Liang, Yufei Ding, Guoqi Li, Guangshe Zhao, Peng Li, and Yuan Xie.  Rethinking the performance comparison between snns and anns.  Neural networks, 121:294–307, 2020.
  * [16] Piotr Dollár, Christian Wojek, Bernt Schiele, and Pietro Perona.  Pedestrian detection: A benchmark.  In 2009 IEEE conference on computer vision and pattern recognition, pages 304–311. IEEE, 2009.
  * [17] Dawei Du, Yuankai Qi, Hongyang Yu, Yifan Yang, Kaiwen Duan, Guorong Li, Weigang Zhang, Qingming Huang, and Qi Tian.  The unmanned aerial vehicle benchmark: Object detection and tracking.  In Proceedings of the European conference on computer vision (ECCV), pages 370–386, 2018.
  * [18] Markus Enzweiler and Dariu M Gavrila.  Monocular pedestrian detection: Survey and experiments.  IEEE transactions on pattern analysis and machine intelligence, 31(12):2179–2195, 2008.
  * [19] Andreas Ess, Bastian Leibe, Konrad Schindler, and Luc Van Gool.  A mobile vision system for robust multi-person tracking.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008.
  * [20] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian.  Incorporating learnable membrane time constant to enhance learning of spiking neural networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2661–2671, 2021.
  * [21] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  Computer Vision and Pattern Recognition Workshop, 2004.
  * [22] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al.  Event-based vision: A survey.  IEEE transactions on pattern analysis and machine intelligence, 44(1):154–180, 2020.
  * [23] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [24] Ricardo Guerrero-Gómez-Olmedo, Beatriz Torre-Jiménez, Roberto López-Sastre, Saturnino Maldonado-Bascón, and Daniel Onoro-Rubio.  Extremely overlapping vehicle counting.  In Pattern Recognition and Image Analysis: 7th Iberian Conference, IbPRIA 2015, Santiago de Compostela, Spain, June 17-19, 2015, Proceedings 7, pages 423–431. Springer, 2015.
  * [25] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant.  Array programming with NumPy.  Nature, 585(7825):357–362, Sept. 2020.
  * [26] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [27] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [28] Craig Iaboni, Deepan Lobo, Ji-Won Choi, and Pramod Abichandani.  Event-based motion capture system for online multi-quadrotor localization and tracking.  Sensors, 22(9):3240, 2022.
  * [29] Craig Iaboni, Himanshu Patel, Deepan Lobo, Ji-Won Choi, and Pramod Abichandani.  Event camera based real-time detection and tracking of indoor ground robots.  IEEE Access, 9:166588–166602, 2021.
  * [30] Glenn Jocher, Ayush Chaurasia, and Jing Qiu.  YOLO by Ultralytics, 1 2023.
  * [31] Glenn Jocher et al.  ultralytics/yolov5, Oct. 2020.
  * [32] Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca.  Efficient processing of spatio-temporal data streams with spiking neural networks.  Frontiers in Neuroscience, 14:439, 2020.
  * [33] Chuyi Li, Lulu Li, Hongliang Jiang, Kaiheng Weng, Yifei Geng, Liang Li, Zaidan Ke, Qingyuan Li, Meng Cheng, Weiqiang Nie, et al.  Yolov6: A single-stage object detection framework for industrial applications.  arXiv preprint arXiv:2209.02976, 2022.
  * [34] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi.  Cifar10-dvs: An event-stream dataset for object classification.  Frontiers in Neuroscience, 11, 2017.
  * [35] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [36] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [37] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 21–37. Springer, 2016.
  * [38] Shu Miao, Guang Chen, Xiangyu Ning, Yang Zi, Kejia Ren, Zhenshan Bing, and Alois C Knoll.  Neuromorphic benchmark datasets for pedestrian detection, action recognition, and fall detection.  Frontiers in neurorobotics, 13:38, 2019.
  * [39] Diederik Paul Moeys, Daniel Neil, Federico Corradi, Emmett Kerr, Philip Vance, Gautham Das, Sonya A. Coleman, Thomas M. McGinnity, Dermot Kerr, and Tobi Delbruck.  Pred18: Dataset and further experiments with davis event camera in predator-prey robot chasing.  2018\.
  * [40] Matthias Mueller, Neil Smith, and Bernard Ghanem.  A benchmark and simulator for uav tracking.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 445–461. Springer, 2016.
  * [41] Pablo Negri, Miguel Soto, Bernabé Linares-Barranco, and Teresa Serrano-Gotarredona.  Scene context classification with event-driven spiking deep neural networks.  In 2018 25th IEEE International Conference on Electronics, Circuits and Systems (ICECS), pages 569–572, 2018.
  * [42] Pablo Negri, Miguel Soto, Bernabé Linares-Barranco, and Teresa Serrano-Gotarredona.  Scene context classification with event-driven spiking deep neural networks.  In 2018 25th IEEE International Conference on Electronics, Circuits and Systems (ICECS), pages 569–572. IEEE, 2018.
  * [43] Garrick Orchard, E Paxon Frady, Daniel Ben Dayan Rubin, Sophia Sanborn, Sumit Bam Shrestha, Friedrich T Sommer, and Mike Davies.  Efficient neuromorphic signal processing with loihi 2.  In 2021 IEEE Workshop on Signal Processing Systems (SiPS), pages 254–259. IEEE, 2021.
  * [44] Garrick Orchard, Ajinkya Jayawant, Gregory K. Cohen, and Nitish Thakor.  Converting static image datasets to spiking neuromorphic datasets using saccades.  Frontiers in Neuroscience, 9, 2015.
  * [45] Yanwei Pang, Jiale Cao, Yazhao Li, Jin Xie, Hanqing Sun, and Jinfeng Gong.  Tju-dhd: A diverse high-resolution dataset for object detection.  IEEE Transactions on Image Processing, 30:207–219, 2020.
  * [46] Etienne Perot, Pierre De Tournemire, Davide Nitti, Jonathan Masci, and Amos Sironi.  Learning to detect objects with a 1 megapixel event camera.  Advances in Neural Information Processing Systems, 33:16639–16652, 2020.
  * [47] José Pérez-Carrasco, Bo Zhao, Carmen Serrano, Begoña Acha, Teresa Serrano-Gotarredona, Shoushun Chen, and Bernabé Linares-Barranco.  Mapping from frame-driven to frame-free event-driven vision systems by low-rate rate-coding and coincidence processing. application to feed forward convnets.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35:2706 – 2719, 11 2013.
  * [48] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [49] Jürgen Schmidhuber.  Deep learning in neural networks: An overview.  Neural networks, 61:85–117, 2015.
  * [50] Teresa Serrano-Gotarredona and Bernabé Linares-Barranco.  A 128x128 1.5% contrast sensitivity 0.9% fpn 3 µs latency 4 mw asynchronous frame-free dynamic vision sensor using transimpedance preamplifiers.  IEEE Journal of Solid-State Circuits, 48(3):827–838, 2013.
  * [51] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  Advances in neural information processing systems, 28, 2015.
  * [52] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [53] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman.  Hats: Histograms of averaged time surfaces for robust event-based object classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1731–1740, 2018.
  * [54] Ajay Vasudevan, Pablo Negri, Bernabé Linares-Barranco, and Teresa Serrano-Gotarredona.  Introduction and analysis of an event-based sign language dataset.  11 2020.
  * [55] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors.  arXiv preprint arXiv:2207.02696, 2022.
  * [56] Longyin Wen, Dawei Du, Zhaowei Cai, Zhen Lei, Ming-Ching Chang, Honggang Qi, Jongwoo Lim, Ming-Hsuan Yang, and Siwei Lyu.  Ua-detrac: A new benchmark and protocol for multi-object detection and tracking.  Computer Vision and Image Understanding, 193:102907, 2020.
  * [57] Christian Wojek, Stefan Walk, and Bernt Schiele.  Multi-cue onboard pedestrian detection.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 794–801. IEEE, 2009.
  * [58] Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele.  Citypersons: A diverse dataset for pedestrian detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3221, 2017.
  * [59] Liang Zheng, Hengheng Zhang, Shaoyan Sun, Manmohan Chandraker, Yi Yang, and Qi Tian.  Person re-identification in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1367–1376, 2017.
  * [60] Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling.  Detection and tracking meet drones challenge.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021.