reconstruction with target data. On the other hand, the authors of [32]
proposed the Domain Separation Network (DSN), comprising two encoders. One is
a shared-weight encoder aiming at learning to capture shared representations;
the second is a private encoder used for domain-specific components in each
domain. More, a shared decoder learns to reconstruct the input samples by
exploiting both the private and the shared representation.

###### Adversarial Reconstruction.

Adversarial reconstruction is inspired by dual learning [99], and it is
adopted in deep DA with the help of dual GANs. Zhu et al. [274] proposed Cycle
GAN, a technique for image-to-image translation without paired training
examples. This algorithm employs two generators, which learn a mapping $G:X\to
Y$ and the inverse mapping $F:Y\to X$. It exploits two discriminators $D_{X}$
and $D_{Y}$ and two losses; $D_{X}$ uses an adversarial loss to measure how
realistic the generated image is ($G(X)\thickapprox Y$ or $G(Y)\thickapprox
X$); $D_{Y}$ utilizes a cycle consistency loss (or reconstruction loss) to
measure how well the original input is reconstructed after a sequence of two
generations ($F(G(X))\thickapprox X$ or $G(F(Y))\thickapprox Y$). Similarly,
dual GAN [249] and disco GAN [126] have been proposed. The former one,
employed skip connections between mirrored downsampling and upsampling layers
[191, 116] for the generator, and the Markovian patch-GAN architecture [140]
for the discriminator. The second one exploited as the reconstruction loss
various distance functions such as Mean Squared Error (MSE), cosine distance
and hinge loss.

Deep DA strategies have been recently applied to other fields other than the
classification task. In particular, we discuss deep DA techniques suited for
face recognition, person re-identification, object detection, and semantic
segmentation.

###### Face recognition.

In face recognition, the domain shift may be represented by expressions,
poses, resolution, and others. For instance, since current benchmarks do not
perform well on young children due to a lack of training data, Xia et al.
[242] proposed to transfer the knowledge from adults to infants. More, the
authors of [106] exploited synthetic data having varying poses, increasing the
number of images in the source domain, and proposed an adversarial training
strategy to fill the gap between the synthetic data and the real-world ones.

###### Person re-identification.

Person re-identification (re-ID) aims at recognizing the identities of the
people between different cameras. Here, deep DA strategies can be helpful when
trained models on one dataset are directly employed in another dataset. Deng
et al. [67] proposed Similarity Preserving Generative Adversarial Network
(SPGAN). Inspired by Cycle GAN [274] and Siamese network, this solution
translates in an unsupervised fashion the labeled source image into the target
domain. Then it trains re-ID models with supervised learning that exploits the
translated images. On the other hand, Xiao et al. [243] introduced a dropout
algorithm able to discard useless neurons under a guide provided by the
characteristics of the domain.

###### Object detection.

Most deep DA techniques applied to the object detection task rely on two-stage
detectors (see also Section 2.2 for further details) because they produce, as
the first step, a set of regions that have therefore been classified in the
second stage. Considering this region selection mechanism being domain-
independent, deep DA strategies can be helpful in the second-stage
classification process to adapt to the target domain. In other words, deep DA
methods act in the second stage of the detection pipeline, where R-CNNs train
classifiers over the proposed regions. To this end, most works exploit the
limited images annotated with bounding boxes to learn to detect and, in
parallel, use weakly labeled data (such as image-level class labels) during
the second stage to adapt the detector to the target domain. For instance,
Hoffman et al. [104] proposed the Large-Scale Detection through Adaptation
(LSDA) network where, firstly, a classification layer is trained for the
target domain; then, a pre-trained source detection model is employed,
together with output layer adaptation strategies aiming at update the target
classification parameters. Instead, Chen et al. [45] considered the Faster
R-CNN detector incorporating an image-level and an instance-level adaptation
component in the detection pipeline; they minimized the domain discrepancy by
exploiting an adversarial strategy during the training phase.

###### Semantic segmentation.

Fully Convolutional Networks (FCNs) demonstrated to be effective for dense
per-pixel prediction; however, also their performance degraded because of
domain shift. Therefore, some works have been proposed to tackle it. Most of
them rely on deep unsupervised DA. Zhang et al. [267] exploited synthetic data
to enhance performance in real-world images using the global label
distribution loss of the images and local label distribution loss of the
landmark super-pixels in the target domain as a regularizer for the fine-
tuning of the network. On the other hand, Chen et al. [44] proposed a
framework for cross-city semantic segmentation. [107] employed a residual
network and adversarial training to make the source feature maps closer to the
target ones. The authors of [46] combine semantic segmentation and depth
estimation to boost the adaptation performance, providing to the discriminator
the segmentation and the depth prediction maps jointly.

### 2.4 Datasets and Annotation Types

The availability of training data for the supervised learning process is one
of the backbones for Computer Vision AI-based model development, and it is not
surprising that advances have been driven by the constant growth of available
well-labeled data and benchmarks. This section describes the datasets employed
directly or indirectly in the experiments presented in this work. Some of
these have been collected during this thesis and represent a concrete
scientific contribution to the Computer Vision community. Others were already
publicly available and have been exploited mainly for state-of-the-art
comparisons. We decide to group the datasets depending on their annotations.
In fact, it is worth noting that, in some cases, the annotations delineate the
task for which the datasets can be exploited and represent an indicator of the
human effort employed for their generation. Concerning the counting task, bare
minimum annotations correspond to provide the overall count of objects in each
training image. However, in this way, all spatial information is lost. This
lack of knowledge makes hard the training phase of the supervised models. The
next level of annotation is to specify the object position by putting a single
dot on each object instance in each image. Although the provided information
consists of weak localization of the objects, this knowledge can be profitably
placed in the training loop. More precise labels, like bounding boxes or per-
pixel annotations, lead to a considerable higher human effort for the
annotation procedure; on the other hand, these last labels carry more helpful
information, such as the dimension of the objects. Figure Figure 2.15 shows
some samples of annotations used for the counting task.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

?figurename? 2.15: Samples of annotations used for the counting task. Top row:
dot annotations, where dots indicate a coarse localization of the object
instances. Middle row: bounding box annotations, where rectangles localize the
objects also giving an estimate of their size. Bottom row: per-pixel
annotations, where per-pixel masks precisely identify the objects. More
informative labels lead to a higher human effort for the annotation procedure.

####  2.4.1 Dot-annotated Datasets

These datasets are characterized by labels consisting of dots placed on the
object centroids or over significant parts of the objects, for example, the
heads of the people. Dotting (e.g., pointing) is the natural way to count
objects for humans, at least when their number is high. Arguably, providing
dotted annotations for the training images is no harder for humans than giving
just the raw counts. Furthermore, dots provide additional information, e.g., a
spatial arrangement of the objects, that can be exploited in the supervised
training process. Overall, it should be noted that dotted annotation is less
labor-intensive than the bounding-box annotation, let alone pixel-accurate
annotation, commonly used by supervised techniques in many computer vision
tasks. On the other hand, these labels provide just a weak localization of the
objects but do not give additional crucial information, like their size.
Therefore, they represent a good trade-off between the information usable
during the training phase and the human effort in terms of labeling. In the
following, we briefly outline the dot-annotated datasets exploited in this
dissertation.

######  TRaffic ANd COngestionS (TRANCOS) [96].

The TRANCOS dataset is a public dataset containing 1244 dot-annotated images
of different congested traffic scenes captured by surveillance cameras, mainly
exploited for evaluating counting by density estimation approaches. The ground
truth density maps are generated by putting one Normal Gaussian kernel for
each dot present in the scene, having a value of $\sigma$ empirically decided
by the authors, who also provided the regions of interest (ROIs) for each
image.

###### VGG Cells [139].

This public dataset contains 200 RGB synthetic images simulating bacterial
cells from fluorescence-light microscopy at various focal distances. The size
of each image is fixed to 256×256×3 pixels, and the cells are designed to be
clustered and occluded with each other. The authors annotated the images by
putting dots in the cell centroids.

######  Modified Bone Marrow Cells (MBM Cells) [122].

This dataset contains RGB real microscopy images of human bone marrow with
various cell types stained blue, annotated with dots in their centroids. The
original image size of each image was 1200×1200×3 pixels, but the authors in
[57] split each of them into four images with the size of 600×600 pixels. The
images in this dataset have non-homogeneous tissue background and significant
cell shape variance.

######  GTA5 Crowd Counting (GCC) [233].

The GCC dataset is a large-scale and diverse synthetic crowd counting dataset
gathered from the video-game Grand Theft Auto V (GTA5) and automatically
annotated. It consists of 15,212 images, with a resolution of $1080\times
1920$, containing 7,625,843 automatically dot-annotated persons in 400
different scenarios with various locations, weather conditions, and crowd
densities. Compared with the existing datasets, GCC is a more large-scale
crowd counting dataset in both the number of images and persons.

###### ShanghaiTech [268].

The ShanghaiTech dataset is a large-scale crowd dataset of nearly 1,200
manually dot-annotated images with a total of 330,165 people with centers of
their heads. This dataset consists of two parts: part A, containing 482 images
crawled randomly from the Internet, and part B, composed of 716 images taken
from the busy streets of metropolitan areas in Shanghai. The crowd density
varies significantly between the two subsets, making this dataset more
challenging.

###### UCF-QNRF [114].

The UCF-QNRF dataset is a collection of images gathered from three sources:
Flickr, Web Search, and the Hajj footage. The authors performed the entire
annotation process in two stages, the first one for the labeling and the
second one for the verification, for a total of 2,000 human-hours spent
through to its completion. This dataset comprises 1,535 images with more than
1 million dot annotations on the centers of the heads of the pedestrian,
divided into training and test subsets.

###### NWPU-Crowd [234].

The NWPU dataset is a large-scale congested crowd counting and localization
dataset consisting of 5,109 images, in a total of 2,133,375 annotated heads
with points and boxes. Compared with other real-world datasets, it has the
most extensive density range. Another peculiarity of this dataset is that it
also comprises negative samples like high-density crowd images to assess the
robustness of models.

###### PNN.

This dataset consists of a collection of fluorescence microscopy images of
mice brain slices containing Perineuronal Nets (PNNs), extracellular matrix
aggregates surrounding the cell body of a large number of neurons throughout
the nervous system. It is worth noting that the non-trivial appearance of PNNs
causes difficulty in localizing them accurately, even for human experts. For
this reason, a small part of the dataset has been labeled by multiple raters;
nonetheless, the maximum agreement between raters is roughly 70%, highlighting
the need for an automated counting technique that accounts for uncertain
patterns. This dataset has been introduced on purpose in this dissertation,
specifically in Chapter 6, and it has been made available for the scientific
community at [52].

######  ADIpocyte Cells (ADI Cells) [57].

The ADI Cells dataset is a human subcutaneous adipose tissue dot-annotated
collection of microscopy images. It consists of 200 Regions Of Interest (ROI)
of 150×150×3 pixels in size sampled from high-resolution histology slides
representing adipocyte cells. The average cell count across all images is
$165\pm 44.2$, and the size of the biological structures can vary
dramatically, representing a challenging test case for automated cell counting
procedures.

######  Breast Cancer Dataset (BCData) [112].

This dataset is a recent collection of 1,338 images based on Ki-67 staining
with 181,074 dot-annotated cells divided into two classes (positive and
negative tumor cells, i.e., malignant and not malignant, respectively). Unlike
other datasets, BCData is not only large in scale concerning the labeled
objects but also considering the number of different unique patient cases
(that are 394). The size of each image is fixed to 640×640×3 pixels, and the
authors divided the dataset into training, validation, and testing split at a
ratio of approximately 6:1:3 (803, 133, and 402 images, respectively).

####  2.4.2 Bounding Box-annotated Datasets

These datasets are characterized by labels consisting of bounding boxes
localizing the objects. These annotations provide a more precise localization
than dots, although the considered regions often contain background portions
near the objects. On the other hand, the human effort for labeling is higher
than dot annotations. This category of datasets is commonly exploited for
detection and tracking tasks. In the following, we briefly describe the ones
used in this dissertation.

######  Webcam Traffic Video Dataset (WebCamT) [260].

The WebCamT dataset is a collection of traffic scenes recorded using city-
cameras. It is particularly challenging to analyze due to the low-resolution
$(352\times 240)$, high occlusion, and large perspective. It comprises about
40,000 images belonging to 10 different cameras and consequently having
different views.

######  Virtual Pedestrian Dataset (ViPeD) [14, 50].

ViPeD is a new synthetic dataset introduced in this thesis (specifically, in
Chapter 4), generated with the highly photo-realistic graphical engine of the
video game GTA V (Grand Theft Auto V) by Rockstar North and automatically
annotated with precise bounding boxes. It extends the Joint Track Auto (JTA)
dataset, presented in [72]. The dataset includes a total of about 500K images
extracted from 512 full-HD videos of different urban scenarios. These videos
are organized into a training set (256 videos) and a test set (the remaining
256 videos). The dataset is made freely available for the scientific community
at https://ciampluca.github.io/viped.

###### MOT17Det [164] and MOT20Det [65].

The MOT17Det and MOT20Det datasets are two collections of images (5,316 and
8,931, respectively), annotated with bounding boxes, taken from multiple
sequences describing crowded scenarios having different characteristics, like
viewpoints, weather conditions, and camera motions. They are suitable for
detection and tracking tasks. The authors provided training and test subsets,
but they released only the ground-truth labels for the former. The main
difference between MOT20Det compared to MOT17Det is that the first contains
more crowded scenarios.

###### CityPersons [264].

The CityPersons dataset consists of a set of stereo video sequences recorded
from a moving car in the streets of different cities in Germany and
neighboring countries. In particular, the authors provide 5,000 frames from 27
cities labeled with bounding boxes and split across train/validation/test
subsets.

###### CrowdHuman [199].

CrowdHuman is a benchmark dataset for pedestrian detection. It comprises
15,000, 4,370, and 5,000 images for training, validation, and testing,
respectively, describing diverse, crowded scenarios, with an average number of
persons in an image of 22.6. The authors annotated each human instance with a
head bounding box, a human visible-region bounding box, and a full-body
bounding box.

###### PRW [271].

The PRW dataset contains 11,816 frames where 932 different pedestrian
identities are annotated with their bounding boxes. The authors provide the
training and the test splits.

###### CUHK-SYSU [244].

The CUHK-SYSU is a large-scale benchmark dataset containing 18,184 images,
8,432 different identities, and 96,143 pedestrian bounding boxes. It is
divided into training and test subsets.

###### CrowdVisorPisa.

The CrowdVisorPisa dataset is a novel collection of images that we collected
and annotated on purpose for this dissertation. In particular, we stored 15
different sequences gathered from a webcam located in a public square of the
city of Pisa, Italy, each of which comprises ten images captured with a time
interval of 1 second. We manually labeled all frames, localizing the
pedestrian instances with bounding boxes. Furthermore, we also annotated each
sequence taking track of the different pedestrian entities entering or exiting
the scene. We divided the dataset into train and test splits, considering 10
and 5 different sequences. The former split is exploited to train the object
detector module, while the latter is to evaluate the performance of some
modules of our framework. It is worth noting that, due to camera positioning
not modifiable for local restrictions, this dataset represents a particularly
challenging scenario as people instances are small and sometimes difficult to
localize. A more detailed description is present in Chapter 7.

###### CrowdVisorPPE.

The CrowdVisorPPE is a new dataset introduced in this thesis comprising 54,017
images representing pedestrians with and without wearable Personal Protective
Equipment. Roughly half of the dataset are synthetic images procedurally
generated using the GTA V video game engine as in [20], whereas the other half
comprises real-world photographic images taken from the Web and manually
annotated. The PPE classes of interest, i.e., helmets, high-visibility vests
(HVVs), and face masks, are annotated with bounding boxes. A more detailed
description is present in Chapter 7.

###### PKLot [63].

The PKLot dataset contains 12,416 images of parking lots extracted from
surveillance camera frames. There are images of sunny, cloudy, and rainy days
and the parking spaces are labeled as occupied or empty. All images were
acquired at the parking lots of the Federal University of Parana (UFPR) and
the Pontical Catholic University of Parana (PUCPR), both located in Curitiba,
Brazil.

######  Car Parking Lot Dataset (CARPK) [109].

The CARPK dataset is the first large-scale aerial dataset for counting cars in
parking lots. It includes 989 and 459 training and test samples, respectively,
each of resolution $720\times 1,280$. The training images are taken from three
different parking lot scenes, while the test set is taken from a fourth scene.
The total number of car instances is 42,274 in the range [1,87] (i.e., from a
minimum of one car up to a maximum of 87 cars in the whole lot) and in the
test dataset is 47,500 in the range [2,188].

######  PUCPR+ [109].

The PUCPR+ dataset is published in the same paper as the CARPK dataset. It is
a subset of the PUCPR dataset [63], adapted by the authors for the counting
task. It contains images captured using a fixed camera from a height of the
10th floor of a building, which provides a slanted view of the parking lot.
This dataset has 100 and 25 training and test samples, respectively. Images
are taken under three different weather conditions (sunny, rainy, and cloudy),
resulting in different illuminations of the scene. The total number of car
instances in the training dataset is 1,299 in the range [0,331] and in the
test dataset is 3,920 in the range [1,328].

###### CNRPark+EXT [10, 11].

This dataset is a collection of images collected from the parking lots of the
National Research Council (CNR) campus in Pisa. It is comprised of roughly
150,000 images of parking spaces taken from nine smart cameras with different
viewpoints, on different days with different weather and light conditions, and
includes occlusion and shadow situations that make the occupancy detection
task challenging. It is employed for the lot occupancy detection task, and
each image represents a parking space that can be vacant or occupied.

####  2.4.3 Per-pixel Annotated Datasets

This class of datasets is characterized by labels involving the pixels of the
images. In particular, instance segmentation represents the more informative
annotation type, combining pixel-wise classification of the entire image and
the localization of the single detected object instances. However, this great
amount of information does not come for free. Indeed, instance segmentation is
the more costly and time-consuming labeling procedure in terms of human
effort. In the following, we briefly illustrate the datasets belonging to this
category used in this dissertation.

######  Common Objects in Context (COCO) [147].

The Microsoft Common Objects in Context (COCO) dataset is a collection of
images designed to represent a vast array of objects that we regularly
encounter in everyday life. It provides suitable data for several computer
vision tasks, like large-scale object detection, segmentation, and captioning,
comprising 883,331 labeled object instances belonging to 80 common object
categories in 121,408 images. In particular, each image is associated with an
instance-level pixel-wise segmentation of the object instances it depicts,
together with captions describing the whole scene. This dataset is considered
the gold standard benchmark for evaluating the performance of state-of-the-art
computer vision models. Still, it also provides a solid base dataset to train
models that can be then fine-tuned by exploiting custom datasets, to learn
other tasks.

######  Night and Day Instance Segmented Park (NDISPark).

This dataset was created on purpose in this thesis and made publicly available
at https://ciampluca.github.io/unsupervised
_counting. It is a small, manually annotated dataset for counting cars in
parking lots, consisting of about 250 images. This dataset is challenging and
describes most of the problematic situations we can find in a real scenario:
seven different cameras capture the images under various weather conditions
and viewing angles. Another challenging aspect is the presence of partial
occlusion patterns in many scenes, such as obstacles (trees, lampposts, other
cars) and shadowed cars. Furthermore, it is worth noting that images are taken
during the day and the night, showing utterly different lighting conditions
and that, unlike most counting datasets, the NDISPark dataset is precisely
annotated with instance segmentation labels, allowing us to generate accurate
ground truth density maps for the counting task since the size of the vehicles
is well-known. A more detailed description is provided in Chapter 5.

######  Grand Traffic Auto (GTA).

The GTA dataset was created on purpose in this dissertation and made publicly
available at https://ciampluca.github.io/unsupervised_counting. It is a vast
collection of about 15,000 synthetic images of urban traffic scenes collected
using the highly photo-realistic graphical engine of the GTA V - Grand Theft
Auto V video game. About half of them concern urban city areas, while the
remaining involve sub-urban areas and highways. To generate this dataset, we
designed a framework that automatically and precisely annotates the vehicles
present in the scene with per-pixel annotations. As in the NDISPark dataset,
the instance segmentation labels allow us to produce accurate ground truth
density maps for the counting task since the size of the vehicles is well-
known. A more detailed description is provided in Chapter 5.

##  ?chaptername? 3 Counting Vehicles in Embedded Vision Systems

Traffic-related issues are constantly increasing, and tomorrow’s cities cannot
be considered intelligent if they do not enable smart mobility. This concept
is becoming more critical since road congestion is growing, caused by the
increasing number of people traveling anywhere. Smart mobility applications,
such as smart parking and road traffic management, are widely employed
worldwide, making our cities more livable, benefiting our lives, reducing
costs, and improving energy usage. The ubiquity of video surveillance cameras
in modern cities and the significant development of Artificial Intelligence
provide new opportunities for the development of such helpful applications and
services for citizens.

Indeed, images are perhaps the best sensing modality to perceive and assess
the flow of vehicles in large areas. Like no other sensing mechanism, city
camera networks can monitor large areas while simultaneously providing visual
data to AI systems to extract relevant information from this deluge of data.
However, this application is often hampered by the limited computational
resources on disposable devices. Indeed, Deep Learning-based solutions,
including Convolutional Neural Network s, often require a considerable amount
of computational resources. To automatically learn a functional hierarchy of
features, these models are defined as deep, i.e., need to stack many
parametric transformations (called layers). Consequently, the model requires a
considerable computational and memory budget for the training and the
evaluation, often limiting the applications of this kind of solution in
restricted environments with limited power resources, such as IoT devices,
delegating complex data analysis to a centralized server.

In this chapter, we explore the adoption of Deep Learning (DL)-based solutions
for counting vehicles directly onboard embedded vision systems, i.e., devices
equipped with limited computational capabilities that can capture images,
process them extracting some knowledge, and eventually communicate with other
devices sending the elaborated information. In particular, we firstly propose
a DL solution to automatically detect and count vehicles in images taken from
a Unmanned Aerial Vehicle (UAV). We experimented over two real-world datasets
showing that our approach results in state-of-the-art performances, running at
a speed of 4 Frames Per Second (FPS) on an NVIDIA Jetson TX2 board. Then, in
another research activity, we introduce a novel vehicle counting solution
considering a different scenario, consisting of a parking area monitored by
multiple smart cameras. Indeed, in many real-world scenarios, one can benefit
from using multiple cameras to monitor the same parking lot from different
perspectives and viewpoints. Furthermore, multiple neighboring cameras can
also be helpful to cover a wider area. At the same time, such a setting
introduces issues related to merging the knowledge extracted from the single
cameras with partially overlapping Field Of View s. To this end, we propose a
multi-camera system that combines a CNN running onboard the smart cameras,
which can locate and count vehicles present in images belonging to the
individual FOV s, along with a decentralized approach, again running directly
on the disposable devices, that is responsible for analyzing and merging these
results, exploiting the geometry of the captured images to estimate the number
of vehicles present in the overlapped Field Of View s, and automatically
outputs the number of cars present in the entire parking area.

We organize the chapter into two distinct parts. In the first one,
corresponding to Section 3.1, we describe the vehicle counting solution that
analyzes images captured by a Unmanned Aerial Vehicle (UAV). In particular, we
introduce the topic, we review some works related to our, we describe the
proposed method, and we show the experiments and the obtained results. In the
second part of the chapter, corresponding to Section 3.2, we instead introduce
the decentralized multi-camera counting solution, we review some related
works, we explain our proposed method, and finally, we show the experimental
results.

The research presented in this chapter was published in [49, 12, 13, 53].

### 3.1 Counting Vehicles in Onboard UAV Imagery

####  3.1.1 Introduction

With the advent of Unmanned Aerial Vehicle s, new potential applications
emerge for unconstrained images and videos analysis for aerial view cameras,
ranging from agriculture to security. Visual tasks like object detection,
classification, or segmentation are the essential building blocks for many of
them. In this section, we address the counting problem for evaluating the
number of vehicles present in parking lots from UAV imagery, i.e., images of
parking areas taken from a drone view. In this scenario, the cars appear in
various orientations, often within the same scene, and can be partially
occluded by trees or bridges. Other challenges are represented by clutters
caused by motorbikes, buildings, or different light conditions. To this end,
we propose a real-time counting solution to be used directly onboard drones
equipped with low-power vision embedded systems. Specifically, we relied on an
existing CNN-based detection framework that we re-trained, specializing it for
this specific task and making it more suitable for working in scenarios
constrained by limited computational capabilities. We evaluated the
effectiveness and the reliability of our solution, testing it on two publicly
available car counting datasets: the CARPK dataset and the PUCPR+ dataset
[109]. In both cases, we achieved state-of-the-art counting errors. For our
tests, we used an NVIDIA Jetson TX2 Developer Kit, a power-efficient embedded
AI computing device running at 4 Frames Per Second (FPS). An overview of the
system architecture is shown in Figure 3.1

?figurename? 3.1: Overview of the system architecture. We localize and
counting vehicles from UAV imagery, directly onboard drones equipped with low-
power vision embedded systems.

####  3.1.2 Related Works

In this section, we briefly review some works related to our and with which we
have compared the obtained results. Prior methods for monitoring the parking
lot often assume that the locations of the monitored objects of a scene, e.g.,
the lots, are already known in advance, and the cameras are fixed. In this
setting, the counting task is cast as a classification problem in which the
network analyzes the single regions of images corresponding to the lots [4,
10, 11, 63]. However, these car counting methods are not directly applicable
in unconstrained drone videos. As stated in Section 2.1, most of the existing
counting approaches can be classified into two supervised learning-based
categories: i) counting by regression that tries to establish a direct mapping
(linear or not) from the image features to the number of objects present in
the scene or a corresponding density map (i.e., a continuous-valued function),
and ii) counting by detection where the goal is to localize instances of the
objects and then count them.

In [109] the authors proposed a detection-based solution, introducing a novel
Layout Proposal Network (LPN) that counts and localizes vehicles in drone
images, leveraging the spatial layout information (e.g., cars often park
regularly). Furthermore, they also presented the CARPK and the PUCPR+
datasets, two collections of images suitable for the vehicle counting task
that we exploited to evaluate the performance of our solution. In [168] the
authors presented instead ResCeption, a network that combines residual
learning with Inception-style layers [209] and is able to count vehicles in
one look. The authors of [158] proposed an approach that claims to be able to
count objects belonging to a generic class not decided a priori, i.e., a
class-agnostic counting network. To this end, the network must be adapted
using very few annotated data. They achieved this by recasting the counting
problem as an image matching problem, where counting instances is performed by
matching (self-similar patches) within the same image. This solution is tested
by counting bacterial cells, people and vehicles. Aich et al. in [7] proposed
a simple and effective way to improve one-look regression models for object
counting from images, enhancing them by regulating activation maps from the
final convolution layer of the network with coarse ground-truth activation
maps generated from simple dot annotations. They called this strategy Heatmap
Regulation (HR). These mentioned solutions are not designed to work on
embedded real-time devices due to time constraints and the limited available
computing resources. One prior method able to perform onboard computations is
described in [19], where the authors presented a detection-based solution
computationally inexpensive for vehicle detection in UAV imagery.

####  3.1.3 Method

We propose a vehicle counting solution from UAV imagery based on detections,
i.e., first, we localize the cars, and then we count the found instances. We
relied on YOLOv3 (You Only Look Once) [185], a popular deep Fully
Convolutional Network, employed in many detection framework, that we described
in Section 2.2.3. In a nutshell, unlike previous detectors based on a sliding-
window classifier or region proposals, YOLO addresses the detection problem by
exploiting the one-look paradigm, achieving higher inference speed, and making
it more suitable for working in limited computation capabilities scenarios.

YOLO v3, as well as its predecessors, divides the input images using a regular
$S\times S$ grid. For each cell of the grid, it predicts $B=3$ bounding boxes,
each of them encoded in a vector of
$(c,t_{x},t_{y},t_{w},t_{h})\times(p_{i},\dots,p_{num\\_classes})$ features,
where $c$ is the objectness of the associated bounding box, i.e., a sort of a
confidence measure of how likely a box is containing, or not, an object,
$t_{x},t_{y}$ are the features related to the offsets of the bounding box
center w.r.t. a corner of the grid cell, $t_{w},t_{h}$ are the features
related to the dimensions of the bounding box and
$(p_{i},\dots,p_{num\\_classes})$ are conditional probabilities,
$P(Class|Object)$, for each of the considered object classes. So, in the end,
the output of the network is a prediction consisting of a tensor of dimensions
$S\times S\times(B\times(5+num\\_classes))$. Moreover, YOLO v3, differently
from the previous versions, uses $3$ of these regular grids as outputs, making
detections at different scales and so predicting three of these tensors at
three different depths of the architecture. Figure 3.2 shows an example of
these three grids applied to a sample image. In particular, to perform these
computations, YOLO v3 uses a deep custom architecture that acts as the
backbone for the features extraction, called darknet-53, which is made up of
53 layers. The authors stacked onto it 53 additional layers, building an
architecture of 106 layers able to perform the detection task. The three
detections at the three different scales are made by the 82nd, the 94th and
the 106th layer, respectively, by applying $1\times 1$ detection kernels on
the feature maps.

In this work, we considered as a starting point a model of YOLO pre-trained on
the COCO dataset [147], a large collection of images describing complex
everyday scenes of common objects in their natural context, categorized in 80
different categories. Since this network is a generic objects detector, we
specialized it to localize and recognize object instances belonging to only a
specific category - i.e., the car category in our case. Specifically, we
extracted the weights of the first 81 layers of this pre-trained model since
these layers capture universal features (like curves and edges), which are
also relevant to our task. Then, we re-trained the detector by initialing the
first 81 layers with the previously extracted weights and the weights
associated with the remaining layers at random, exploiting annotated data
describing our specific scenario for the supervised learning. Thus, we got the
network to focus on learning dataset-specific features in the last layers.
Furthermore, in such a way, we reduced the dimension of the predicted output
tensors, from $S_{i}\times S_{i}\times(B\times(80+5))=S_{i}\times S_{i}\times
255$ to $S_{i}\times S_{i}\times(B\times(1+5))=S_{i}\times S_{i}\times 18)$,
where $S_{i}$ are the three different grids exploited by YOLO, saving memory
usage and making our solution more suitable for the limited computational
capabilities of the embedded vision system employed in our testing scenario.

?figurename? 3.2: Output vectors produced by the YOLO v3 network.YOLO v3
divides the input images in $S=3$ different regular grids, making detections
at different scales. For each cell of a grid it predicts $B=3$ bounding boxes
each of them encoded in a vector of
$(c,t_{x},t_{y},t_{w},t_{h})\times(p_{i},...,p_{n})$ features, where $c$ is
the objecteness of the associated bounding box, $t_{x},t_{y}$ are the features
related to the offsets of the bounding box center w.r.t. a corner of the grid
cell, $t_{w},t_{h}$ are the features related to the dimensions of the bounding
box and $(p_{i},...,p_{n})$ are conditional probabilities, $P(Class|Object)$,
for each of the considered $n$ object classes. At the end, YOLO outputs a
tensor of dimensions $S_{i}\times S_{i}\times(B\times(n+5))$, where $n$ is the
number of object classes and $S_{i}$ are the three different grids.

####  3.1.4 Experimental Evaluation

This section illustrates the employed experimental setup, the performed
experiments, and, finally, the obtained results. We start by describing the
exploited datasets and the testing scenario. Then we discuss the results
compared with the state-of-the-art ones. Following other counting benchmarks,
we used the Mean Absolute Error (MAE) and the Root Mean Squared Error (RMSE)
as the performance metrics (see also Section 2.1.2 for more details).
Furthermore, since we rely on a detection-based approach, we also exploit the
Precision and Recall metrics to take into account the quality of the
localization (see also Section 2.2.1).

##### Datasets

To train and evaluate our vehicle counting solution, we exploited two public
collections of images widely employed in the literature: the CARPK and the
PUCPR+ datasets, introduced in [109]. The CARPK dataset is the first large-
scale aerial dataset for counting cars in parking lots. It includes 989 and
459 training and test samples, respectively, each having resolution of
$720\times 1,280$. The training images are taken from three different parking
lot scenes, while the test set is taken from a fourth scene. The total number
of car instances is 42,274 in the range [1,87] (i.e., from a minimum of one
car up to a maximum of 87 vehicles in the whole lot) and in the test dataset
is 47,500 in the range [2,188]. The PUCPR+ dataset is published in the same
paper as the CARPK dataset. It is a subset of the PUCPR dataset introduced in
[63], adapted by the authors for the counting task. It contains images
captured using a fixed camera from a height of the 10th floor of a building,
which provides a slanted view of the parking lot. This dataset has 100 and 25
training and test samples, respectively. Images are taken under three
different weather conditions (sunny, rainy, and cloudy), resulting in
different scene illuminations. The total number of car instances in the
training dataset is 1,299 in the range [0,331] and in the test dataset is
3,920 in the range [1,328]. Some sample images from the dataset are shown in
Figure 3.3.

(a)

(b)

(c)

(d)

?figurename? 3.3: Examples from the CARPK (top row) and PUCPR+ (bottom row)
datasets. The CARPK dataset is an aerial dataset taken from a drone, while the
PUCPR+ dataset records a parking lot from a fixed camera located at the 10th
floor of a building. We exploit them to train and evaluate our vehicle
counting solution.

##### Experimental Scenario

We used the darknet implementation of YOLO 111https://pjreddie.com/darknet/.
We trained the network by exploiting the training subsets of the CARPK and
PUCPR+ datasets, obtaining two different models specific for the two
considered scenarios. Specifically, we picked up the best models in terms of
MAE over the validation splits. Concerning the CARPK dataset, we obtained the
validation subset by further dividing the training set, considering images
belonging to one of the three different scenes. On the other hand, we adopted
a 5-fold cross-validation for the PUCPR+ dataset since it is very small. For
the training phase, we exploited an NVIDIA GeForce RTX 2080 Ti GPU for 150
epochs, using Stochastic Gradient Descent (SGD) for the optimization with a
base learning rate of 0.001, momentum 0.9, and a weight decay of 0.0005. We
employed a data augmentation strategy over the training images to prevent
overfitting, randomly changing saturation, exposure (brightness), and hue
(color), applying a blurring filter, and changing the size and aspect ratio.

We evaluated our final solutions by testing them over the test subsets of the
CARPK and PUCPR+ datasets, also performing cross-dataset experiments, i.e.,
testing the model trained on CARPK over PUCPR+, and vice-versa. For this
phase, we exploited an NVIDIA Jetson TX2 Developer Kit as an embedded vision
device, simulating an autonomous system located onboard a UAV. This vision
board is equipped with two 64bit CPUs having two and four cores each, an
NVIDIA Pascal GPU with 256 CUDA cores, 4 GB of RAM shared between the system
and the graphics accelerator and a 32 GB solid-state storage volume, together
with a Camera Module able to capture images. The operating system is the
NVIDIA’s Linux4Tegra(L4T) distribution based on Ubuntu. The operational
autonomy can be guaranteed in a real scenario using, for example, a 3S 6000
mAh LiPo battery that provides a six-hour of autonomy, as stated in [27].

##### Results and Discussion

We report the obtained results in terms of MAE, RMSE, Precision, and Recall in
Table 3.1 and Table 3.2, concerning the CARPK and the PUCPR+ datasets,
respectively, comparing the gained performance against other state-of-the-art
approaches. On the other hand, Figure 3.4 shows some detection results.

?tablename? 3.1: State-of-the-art comparison on the CARPK test set (459 images
and 47,500 total vehicles)

Method | MAE | RMSE | Recall | Precision
---|---|---|---|---
One-Look Regression [168, 109] | 59.46 | 66.84 | - | -
Faster R-CNN [187, 109] | 47.45 | 57.55 | - | -
ShuffleDet [19] | 26.75 | 38.46 | - | -
Faster R-CNN (RPN-small) [187, 109] | 24.32 | 37.62 | - | -
Spatially Regularized RPN [109] | 23.80 | 36.79 | 57.5% | -
Our solution ∗ | 16.63 | 21.18 | - | -
GMN (3 images) [158] | 13.38 | 18.03 | 76.1% | 85.1%
VGG-GAP-HR [7] | 7.88 | 9.30 | - | -
GMN (full dataset) [158] | 7.48 | 9.90 | 88.4% | 91.8%
Our solution | 4.94 | 6.76 | 89.7% | 90.5%

* Trained using the PUCPR+ dataset

?tablename? 3.2: State-of-the-art comparison on the PUCPR+ test set (25 images
and 3,920 total vehicles)

Method | MAE | RMSE | Recall | Precision
---|---|---|---|---
Faster R-CNN [187, 109] | 156.76 | 200.59 | - | -
ShuffleDet [19] | 41.58 | 49.68 | - | -
Faster R-CNN (RPN-small) [187, 109] | 39.88 | 47.67 | - | -
Our solution ∗ | 28.16 | 41.20 | - | -
One-Look Regression [168, 109] | 21.88 | 36.73 | - | -
Spatially Regularized RPN [109] | 22.76 | 34.46 | 62.5% | -
VGG-GAP-HR [7] | 5.24 | 6.67 | - | -
Our solution | 2.80 | 4.53 | 95.3% | 95.2%

* Trained using the CARPK dataset

(a)

(b)

(c)

(d)

?figurename? 3.4: Some examples of the obtained results on the CARPK dataset
(top row) and the PUCPR+ dataset (bottom row). The detections are marked with
a bounding box and a label indicating the object class. In addition, we report
the total number of instances found together with the Ground Truth value.

We achieved state-of-the-art results in terms of counting errors in both
datasets, also ensuring high Precision and Recall values. The Generic Matching
Network (GMN) presented in [158] and tested on the CARPK dataset has the
advantage of being class agnostic. However, the error obtained using only
three training images is about three times larger than the one obtained with
our approach. The error using the entire training dataset is instead about two
times larger than the one we got, and in this case, the benefits of using a
class agnostic solution are less self-evident. The VGG-GAP-HR solution [7] has
instead the advantage of needing a dot-annotated training dataset, while our
solution needs bounding box labels. Dot-annotation is a less time-consuming
operation than the bounding box one, but the error of our solution is about
half for both datasets. Finally, comparing our approach with the one proposed
by [19] in terms of images processed per second, it is notable that the
algorithm in [19] is faster than our (14 FPS instead of 4 FPS). Still, the
error in [19] is pretty larger in both the exploited benchmarks compared to
ours. Finally, concerning the cross-dataset experiments, we experienced a
performance degradation due to the overfitting over a specific scenario;
however, counting errors remain comparable with the ones obtained using most
of the other approaches, thus showing capabilities by our solution to
generalize to new contexts.

####  3.1.5 Summary

In this first part of the chapter, we presented a useful tool based on a
state-of-the-art object detector for evaluating the number of vehicles present
in parking lots from UAV imagery. We tested our solution over two public
datasets, exploiting an NVIDIA Jetson TX2 as an embedded vision device having
limited computational capabilities that simulated an autonomous system located
onboard a drone. We compared the obtained results against other approaches
present in the literature, getting state-of-the-art performances.

### 3.2 Multi-Camera Vehicle Counting in Parking Lots

####  3.2.1 Introduction

In this second part of the chapter, we tackle the problem of estimating the
number of vehicles present in a parking lot using images captured by smart
cameras. Whereas classic solutions for car counting or parking lot occupancy
estimation are sensor-based (e.g., entrance-level photocells, per-space ground
sensors), camera-based solutions provide several advantages, such as a)
flexibility, as cameras can adapt to more challenging configurations of
parking spaces (e.g., undelimited parking lots with non-fixed spaces), b)
lower hardware and maintenance cost, as smart cameras can cost few tens of
euros while each monitoring multiple parking spaces, and c) multi-purpose, as
the same hardware can be used to perform additional tasks (e.g.,
surveillance). However, this vision-based counting task is challenging as the
process of understanding the captured images faces many problems, such as
shadows, light variation, weather conditions, and inter-object occlusions.

Although most of the existing solutions concerning visual vehicle counting
focus on the analysis of single images, in many real-world scenarios, one can
benefit from using multiple cameras to monitor the same parking lot from
different perspectives and viewpoints. Furthermore, multiple neighboring
cameras can also be helpful to cover a wider area. At the same time, such an
approach introduces issues related to merging the knowledge extracted from the
single cameras with partially overlapping Field Of View s (FOVs), as shown in
Figure 3.5. To this end, we propose a novel solution to improve car counting
when scaled up with multi-camera setups. We introduce a multi-camera system
that combines a state-of-the-art Convolutional Neural Network, which can
locate and count vehicles present in images belonging to individual cameras,
along with a decentralized geometry-based approach that is responsible for
aggregating the data gathered from all the devices and estimating the number
of cars present in the entire parking lot. A remarkable peculiarity of our
solution is that it performs the task directly on the disposable devices,
i.e., the smart cameras — vision systems with limited computational
capabilities able to capture images, extract information from them, make
decisions, and communicate with other devices.

We modeled our system as a graph where the nodes represent the smart cameras.
The total count is built exploiting the partial results computed in parallel
by the single cameras and propagated through messages. Hence, our system
scales better when the number of monitored parking spaces increases. Moreover,
our solution does not require any extra information about the monitored
parking area, such as the location of the parking spaces, nor any geometric
information about the camera positions in the parking lot. In short, it is a
flexible and ready-to-use solution that provides a simple “plug-and-play”
insertion of new cameras into the system.

?figurename? 3.5: An example of two cameras monitoring the same parking area
with partially overlapping fields of view. This redundancy provides robustness
and fault-tolerance, but it also raises the problem of aggregating knowledge
extracted from the individual cameras.

To validate our multi-camera solution, we employed and extended the CNRPark-
EXT dataset [11], a collection of images taken from the parking lot on the
campus of the National Research Council (CNR) in Pisa, Italy. Images have been
acquired by multiple cameras having partially overlapping fields of view and
describing challenging scenarios with different perspectives, illuminations,
weather conditions, and many occlusions. Since the annotations of this dataset
concerned single images, we extended it by re-labeling a part of it to be
consistent with our algorithm that instead considers the entire parking area.
We conducted extensive experiments testing the generalization capabilities of
the CNN responsible for detecting vehicles in single images and the
effectiveness of our multi-camera algorithm, demonstrating that our system is
robust and benefits from the redundant information deriving from the different
cameras improving the overall performance.

To summarize, the main contributions of this research are the following:

  * •

we introduce a novel multi-camera system able to automatically estimate the
number of cars present in the entire monitored parking area. It runs directly
on the disposable devices and combines a deep learning-based detector with a
decentralized technique that exploits the geometry of the captured images;

  * •

we extend the CNRPark-EXT dataset [11], a collection of images acquired by
multiple cameras having partially overlapping fields of view and describing
various parking lots, making it suitable with our scenario in which we
consider the whole parking area;

  * •

we show through experiments that our system is robust, flexible, and can
benefit from redundant information coming from different cameras while
improving overall performance.

####  3.2.2 Related Work

Parking lot monitor using visual data is not new, and other works have already
tackled it in the literature. In [10, 11], the same authors of the CNRPark-EXT
dataset presented a deep learning-based system for parking lot occupancy
detection that can run in a Raspberry Pi directly on board a smart camera. In
[171], the authors directly dealt with the issues deriving from the adoption
of a multi-camera system. In particular, they applied a homography to project
the detected vehicles from the plane of each camera to a common plane, where
they performed a perspective correction to correct matching between the
vehicle detections and the parking spots. Also, the authors in [228] proposed
a multi-camera system to classify parking spaces as vacant or occupied. In
this solution, the acquired images are processed onboard Raspberry Pi devices.
The extracted information about the status of parking spaces is then
transmitted to a central server, which evaluates the parking spaces in the
overlapping areas. Their algorithm is based on the histogram of oriented
gradients (HOG)[60] feature descriptor and Support Vector Machine (SVM)
classifiers. Since the HOG feature descriptor cannot adequately describe
rotated vehicles, the authors have provided a descriptor with additional
information about rotation to increase the system accuracy. However, these
solutions rely on prior knowledge of the monitored scene, like the position of
the parking spaces or some geometric information about the scene. In essence,
a preliminary annotation of the new areas and a new training phase of the
algorithm are often mandatory operations. As a consequence, these techniques
are not very flexible. On the other hand, we propose a simple yet effective
solution that does not need extra information about the monitored scene. The
smart cameras can automatically localize and count the vehicles present in
their field of view, propagating the single results to the other devices
through messages. A decentralized technique, again running directly onboard
the devices, is instead in charge of analyzing and merging these results,
exploiting the captured images geometry, and automatically outputs the number
of cars present in the entire parking area.

####  3.2.3 Method

##### Overview

We base our multi-camera counting algorithm on the parallel processing of each
of the smart cameras followed by the fusion of their results to estimate the
number of vehicles present in the entire parking area. Figure 3.6 shows an
example of our multi-camera counting system, together with its graphical
representation. We modeled our system as a graph $G$, comprised of $n$ nodes
$\nu_{i}$ and one Sink node $S$, $V=\\{\nu_{1},\nu_{2},\cdots,\nu_{n},S\\}$.
Each node $\nu_{i}$ represents an independent device, i.e., a smart camera in
our case. Two nodes $\nu_{i}$ and $\nu_{j}$ are considered neighbors if their
FOV s overlap, and in this case, a directed edge of the graph connects them.
Each device $\nu_{i}$ can capture images, localize and count the vehicles
present in its FOV exploiting a deep learning-based detector, and communicate
with its neighboring nodes through messages $m_{i}$ containing the cars
detections. Furthermore, each node $\nu_{i}$ can also run a local counting
algorithm in charge of computing partial counting results concerning the
estimation of the number of vehicles present in overlapped areas between its
FOV and the ones belonging to its neighbors. The fusion of the partial results
is performed by the Sink node $S$, which is also in charge of providing the
final result and synchronizing all the algorithm steps through synchronization
signals headed towards the other nodes $\nu_{i}$. On the other hand, the nodes
$\nu_{i}$ can also communicate through messages with the Sink node. Messages
can be of two types: i) messages $\eta_{i}$ containing the number of cars
captured by the node $\nu_{i}$ in its FOV, and ii) messages $\mu_{j,i}$
representing the partial counting estimation related to the overlapping area
between two neighboring nodes $\nu_{i}$ and $\nu_{j}$.

?figurename? 3.6: An example of our multi-camera counting system, with $n=5$
smart cameras. We model it as a graph $G$, comprised of $n$ nodes $\nu_{i}$
(one for each camera) and one Sink node $S$,
$V=\\{\nu_{1},\nu_{2},\cdots,\nu_{n},S\\}$. Each node $\nu_{i}$ can capture
images, localize and count the vehicles present in its FOV, and communicate
with its neighboring nodes through messages $m_{i}$ containing these
detections. Moreover, each node $\nu_{i}$ can run a local counting algorithm
in charge of computing partial counting results concerning the overlapped
areas between its FOV and the ones belonging to its neighbors, exploiting
images geometry. These partial results are sent through messages to the Sink
node $S$, which is responsible for their fusion and provides the final result.
Messages to $S$ can be of two types: i) $\eta_{i}$ containing the number of
cars captured by the node $\nu_{i}$ in its FOV, and ii) $\mu_{j,i}$
representing the partial counting estimation related to the overlapping area
between two neighboring nodes $\nu_{i}$ and $\nu_{j}$.

##### Initialization

This step is aimed at automatically initializing the system, estimating the
geometric relationship between each node (i.e., each scene monitored by a
smart camera) and its neighbors. The only hypotheses we impose are i) each
smart camera is aware of the IP addresses of its neighbors, i.e., the cameras
having the field of view overlapped with its own; ii) the Sink node $S$ is
aware of the IP addresses of all the smart cameras belonging to the system.

The Sink node $S$ starts the initialization phase, sending a synchronization
signal to the other nodes. Once received, each smart camera captures an image
of the scene it monitors and sends it to all its neighbors. Once a smart
camera $i$ receives an image from a neighboring camera $j$, it computes a
homographic transformation $H_{j,i}$ between the image $j$ and the image $i$
describing its monitored scene. This allows us to establish a correspondence
between the points belonging to the pair of images taken by the two cameras,
which will be used subsequently in the algorithm. We formalized the system
initialization for a generic node $\nu_{i}$ in the Algorithm 1.

However, finding this homography can be challenging because neighboring
cameras can have different angles of view, leading to a perspective distortion
between the images captured by them. Given a pair of neighboring nodes
$\nu_{i},\nu_{j}$, we employ a procedure that starts with finding the SIFT
[156] key-points and feature descriptors of the images $i,j$ captured by the
two nodes. Then, we match the two sets of feature descriptors performing the
David Lowe’s ratio test [156], and we further filter the matched feature
descriptors by keeping only the pairs whose euclidean distance is below a
given threshold. Finally, we obtain the homography transformation by applying
the RANdom SAmple Consensus (RANSAC) [78] algorithm to the filtered feature
descriptors. Figure 3.7 shows the concatenation of two neighboring images $i$
and $j$ in which we apply the found homographic matrix to the image $i$, to
have the same perspective as the image $j$.

Algorithm 1 : Initialization
At each Initialization Signal by $S$, each node $\nu_{i}$ performs the
following steps:

1:ReceiveInitSignal() $\triangleright$ waits the initialization signal from
$S$

2:image${}_{i}\leftarrow$ CameraCapture()

3:for each $j\in J$ do $\triangleright$ $J$ is the set of neighboring nodes of
node $\nu_{i}$

4: SendImage(imagei,$\nu_{j}$) $\triangleright$ sends imagei to node $\nu_{j}$

5: image${}_{j}\leftarrow$ ReceiveImage() $\triangleright$ receives imagej
from node $\nu_{j}$

6: $H_{j,i}=$ ComputeHomography(imagej, imagei)

?figurename? 3.7: Example of concatenation of two images using a homographic
transformation. It is also visible the overlapping area between them.

##### Local Counting Algorithm

The local counting algorithm runs directly and independently onboard the
disposable devices, i.e., the smart cameras. It combines a CNN in charge of
the localization and the estimation of the number of vehicles present in the
acquired single images, i.e., the contents of the messages $m_{i}$ and the
quantities $\eta_{i}$ shown in Figure 3.6, together with a geometric-based
approach responsible of estimating the number of vehicles present in the
overlapping areas between the nodes and their neighbors, i.e., the quantities
$\mu_{j,i}$ in Figure 3.6.

To count the vehicles, we exploited a detection-based approach since
regression-based techniques cannot precisely localize the objects present in
the scene and can eventually provide only a coarse position of the area in
which they are distributed (see also Section 2.1 for further details),
therefore resulting not suitable in our scenario where we need to know the
precise localization (with boundaries) of the detected vehicles. As in the
previous Section 3.1 where we proposed a solution able to estimate the number
of vehicles in UAV imagery, also in this research, we faced challenges due to
the imitated computational resources available in smart cameras. Here, we
relied on Mask R-CNN [101], a popular deep CNN for instance segmentation that
operates within the ‘recognition using regions paradigm [95], described in
Section 2.2.2. In particular, it extends the Faster R-CNN detector [187] by
adding a branch that outputs a binary mask indicating whether or not a given
pixel is part of an object. Briefly, in the first stage, a CNN acts as a
backbone, extracting the input image features. Starting from this feature
space, another CNN named Region Proposal Network (RPN) generates region
proposals that might contain objects. RPN slices pre-defined region boxes
(called anchors) over this space and ranks them, suggesting those most likely
containing objects. Once RPN produces the Regions of Interested (ROIs), they
might be of different sizes. Since it is hard to work on features having
different sizes, RPN reduces them into the same dimension using the Region of
Interest Pooling algorithm. Finally, these fixed-size proposals are processed
by two parallel CNN-based branches: one is responsible for classifying and
localizing the objects inside them with bounding boxes; the second produces a
binary mask that says whether or not a given pixel is part of an object. In
the end, given an input image, the network produces per-pixel masks localizing
the detected objects together with the associated labels classifying them. To
make our counting solution able to run efficiently directly on the devices, we
employed, as a backbone, the ResNet50 architecture, a lighter version of the
popular ResNet101 [100]. This simplification is also justified because the
more powerful version of Mask R-CNN based on the ResNet101 model was designed
for more complicated visual detection tasks than ours. Originally, Mask R-CNN
was trained on the COCO dataset [147] to detect and recognize 80 different
classes of everyday objects. In our case, we have to localize and identify
objects belonging to just one category (i.e., the vehicle category). To this
end, we further simplified the model by reducing the number of the final fully
convolutional layers responsible for the classification of the detected
objects, making the model lighter. Once we have localized the instances of the
objects, we count them estimating the number of vehicles present in the scene.

The local counting is triggered by the Sink node $S$, sending a
synchronization signal to all the smart cameras belonging to the system. Once
received the synchronization signal, each node $\nu_{i}$ captures an image
belonging to its underlying FOV, feeds it to the previously described CNN-
based counting technique, and obtains as output a set of masks masksi
localizing the vehicles present in the scene. The cardinality of this set of
masks corresponds to the number of cars present in the image, i.e., the
quantity $\eta_{i}$, that is sent through a message to the Sink node $S$.
Then, the node $\nu_{i}$ packs this set of masks masksi in a message $m_{i}$,
sends it to all its neighboring nodes $\nu_{j}$, and receives from them their
corresponding set of masks masksj packed in a message $m_{j}$. Once received a
message $m_{j}$, the node $\nu_{i}$ is responsible for analyzing the potential
vehicles present in the overlapped area between its FOV and the ones of the
nodes $\nu_{j}$. To this end, it employs the homographic transformation
$H_{j,i}$ computed during the system initialization, projecting the masks
belonging to the set masksj into its image plane, filtering them and
discarding the ones that overlap with the masks belonging to the set masksi
with a value of Intersection over Union (IoU) greater than a threshold that we
empirically found to be optimal at 0.2. These masks indeed localize vehicles
already detected, and that should not be considered a second time. On the
other hand, the cars left after this filtering are vehicles that were not
detected in the FOV underlying the node $\nu_{i}$, but instead found by the
node $\nu_{j}$, probably because of having a better view of this object.
Referring to our graph modeling the system and reported in Figure 3.6, the
number of the cars detected by $\nu_{j}$ and already localized by $\nu_{i}$
corresponds to the message $\mu_{j,i}$, that is sent to the Sink node $S$. We
detail all the described steps in the Algorithm 2 and in the Algorithm 3.

Algorithm 2 : Local Counting
At each Computational Signal by $S$, each node $\nu_{i}$ performs the
following steps:

1:ReceiveComputSignal() $\triangleright$ waits the computational signal from
$S$

2:image${}_{i}\leftarrow$ CameraCapture()

3:masks${}_{i}\leftarrow$ MaskRCNN(imagei)

4:$\eta_{i}\leftarrow\left|\text{masks}_{i}\right|$

5:SendMessage($\eta_{i},S$) $\triangleright$ sends $\eta_{i}$ to Sink node $S$

6:$m_{i}\leftarrow$ PackMessage(masksi) $\triangleright$ builds message
$m_{i}$ containing masksi

7:for each $j\in J$ do $\triangleright$ $J$ is the set of neighboring nodes of
node $\nu_{i}$

8: SendMessage($m_{i},\nu_{j}$) $\triangleright$ sends $m_{i}$ to node
$\nu_{j}$

9: $m_{j}\leftarrow$ ReceiveMessage() $\triangleright$ receives message
$m_{j}$ from node $\nu_{j}$

10: masks${}_{j}\leftarrow$ UnpackMessage($m_{j}$) $\triangleright$ unpacks
$m_{j}$ containing masksj

11: $\mu_{j,i}\leftarrow$ compute_$\mu$(masksi, masksj, $H_{j,i}$)

12: SendMessage($\mu_{j,i},S$) $\triangleright$ sends $\mu_{j,i}$ to Sink node
$S$

Algorithm 3 : Computation of $\mu$
$\mu$ represents the num of cars detected by $\nu_{j}$ and already detected by
$\nu_{i}$
Each node $\nu_{i}$ performs the following procedure:

1:procedure compute_$\mu$(masksi, masksj, $H_{j,i}$)

2: n_cars_already_detected $\leftarrow 0$

3: for each mask $\in$ masksj do

4: mask${}_{h}\leftarrow$ Project($H_{j,i}$, mask) $\triangleright$ projects
mask points on plane $i$

5: if maskh falls within imagei then

6:
mask${}_{\text{max}}\leftarrow\operatorname*{arg\,max}_{m\in\text{masks}_{i}}\text{IoU}(\text{mask}_{h},m)$

7: if IoU$(\text{mask}_{h},\text{mask}_{\text{max}})>\tau$ then

8: n_cars_already_detected ++

9: return n_cars_already_detected

##### Global Counting Algorithm

The global counting algorithm runs on the Sink node $S$, and it is responsible
for the fusion of the partial results coming from all the other nodes and for
finally outputting the number of cars present in the entire monitored parking
area.

This phase starts when $S$ receives all the $\eta_{i}$ and the $\mu_{j,i}$,
i.e., the number of vehicles estimated in the single FOV s and the estimation
of the number of cars already considered in the overlapping areas between
neighboring cameras, from all the nodes belonging to the system. In
particular, for each overlapped area shared between a pair of nodes
$\nu_{i},\nu_{j}$, the node $S$ receives two messages $\mu_{j,i}$ and
$\mu_{i,j}$, the contents of which are computed by the two nodes employing two
homographic transformations $H_{j,i}$ and $H_{i,j}$, respectively. These two
quantities can be potentially different. We choose the best value by
aggregating them, choosing between three different functions - max, min, and
mean, finding that the latter is the best one. Finally, the node $S$ builds
the final result, i.e., the estimation of the number of vehicles present in
the entire parking lot, by summing up all the $\eta_{i}$, and subtracting the
aggregated values. We detail all these steps in the Algorithm 4.

Algorithm 4 : Global Counting
The Sink node $S$ performs the following steps:

1:for each $(\mu_{i,j},\mu_{j,i})$ do

2: $\overline{\mu_{k}}\leftarrow$ Aggregate($\mu_{i,j},\mu_{j,i}$)

3:global_cars_count
$\leftarrow\sum_{n=1}^{N}\eta_{n}-\sum_{k=1}^{K}\overline{\mu_{k}}$
$\triangleright$ $N$ is the set of nodes, $K$ is the set of aggregations

####  3.2.4 Experimental Setup

This section describes the simulated scenario that we exploited for our
experiments. In particular, we extend the CNRPark-EXT dataset [11], adapting
it to be suitable for the counting task and so that it was usable for training
the vehicles counting CNN running on the smart cameras and applicable to
validate our multi-camera algorithm. Furthermore, we briefly describe the
PKLot dataset [63], a public dataset comprising parking lot scenes that we
exploit for further assessing the generalization capabilities of the local
vehicle counting network. Finally, we report some implementation details.

##### The CNRPark-EXT Dataset

We exploited the CNRPark-EXT public dataset introduced in [11], a collection
of annotated images of vacant and occupied parking spaces on the campus of the
National Research Council (CNR) in Pisa, Italy. This challenging dataset
represents most of the problematic situations that can be found in a real
scenario: nine different cameras capture the images under various weather
conditions, angles of view, light conditions, and many occlusions.
Furthermore, the cameras have their fields of view partially overlapped. Since
this dataset is specifically designed for parking lot occupancy detection, it
is not directly usable for the counting task. Indeed, each image, called
patch, contains one parking space labeled according to its occupancy status -
0 for vacant and 1 for occupied. Since this work aims at counting the cars
present in the parking area, we extended it by considering the full images and
adapting the ground truth to our purposes.

To train and evaluate the local vehicle counting based on Mask R-CNN, we
created a suitable label set. In this case, labels correspond to binary masks,
i.e., binary images identifying the polygons surrounding the vehicles we want
to detect. Since mask creation is a very time-consuming operation, differently
from [49], we considered the raw masks obtained directly from the bounding
boxes localizing the occupied parking spaces. The idea is that we do not need
precise polygons that identify the vehicles we want to detect. Still, we can
use the region within the delimiters that identify the occupied parking spaces
and the underlying part of the car.

On the other hand, to validate our multi-camera algorithm, we built a
simulated scenario considering some sequences of images belonging to different
cameras captured simultaneously. In other words, a sequence is defined as the
set of images captured by the different smart cameras that are monitoring the
parking area at the same moment. Hence, a sequence represents a snapshot of
the entire parking lot at a given timestamp, and it takes into account all the
spaces from the available different views. We manually annotated these
sequences to obtain the ground truth car counts. Specifically, we considered
the single images composing a sequence, counting the vehicles present in the
scenes, but taking care of accounting for them just once if they appear in
more than one view. We labeled six different sequences, two for each weather
condition, considering the images from camera2 to camera9. We did not consider
camera1 since it has small and particularly skewed field-of-view overlaps with
the others cameras, hindering the automatic homography estimation and the
subsequent projections.

##### The PKLot Dataset

To further validate the generalization capabilities of the CNN-based local
vehicles counting algorithm, we exploited an additional public dataset, named
PKLot [63]. In particular, this dataset is composed by three different
scenarios describing three different parking lot scenes - UFPR04, UFPR05 and
PUC. We considered only the first two subsets since the third one contains
images captured from a fixed camera located at the height of the 10th floor of
a building, which provides a slanted view of the parking lot and results in a
different setting without intra-vehicle occlusions. Like the CNRPark dataset,
also PKLot is specifically designed for the parking lot occupancy detection
task. Thus, we manually re-labeled the ground truth to our purposes, obtaining
a simulated scenario suitable for measuring the performance of our solution
concerning the counting task.

##### Implementation Details

We report in this section some implementation details concerning the Mask
R-CNN-based algorithm responsible for the prediction of the number of vehicles
in the single images. In particular, we trained the modified Mask R-CNN
initializing the weights of the ResNet50 backbone with the ones of a pre-
trained model on ImageNet [66], a popular dataset for classification tasks,
and the remaining ones at random. We freeze the backbone for the first ten
epochs, and then we trained the whole network for 20 additional epochs. To
prevent overfitting, we applied some standard augmentation techniques to the
training data: images are horizontally flipped with a 0.5 probability, then
their pixels are multiplied by a random value between 0.8 and 1.5, and
finally, they are blurred using a Gaussian kernel with a standard deviation of
a random value between 0 and 5. Then, to support training multiple images per
batch, we resized all pictures to the same size. If an image is not square, we
pad it with zeros to preserve the aspect ratio. In the end, we obtained images
of size $1024\times 1024$. At inference time, images are resized and padded
with zeros to get a square picture of size $1024\times 1024$, and no other
augmentations occur.

####  3.2.5 Experiments and Results

In this section, we report the experiments and the obtained results. First, we
evaluate the performance against other state-of-the-art solutions of the CNN-
based technique responsible for estimating the vehicle number in images
concerning the single-camera scenario, i.e., considering a single FOV. Then,
we validate the effectiveness of our multi-camera algorithm, testing it in the
simulated scenario previously described. Following other counting benchmarks,
for all the experiments we exploited the Mean Absolute Error (MAE), the Mean
Squared Error (MSE), and the Mean Absolute Relative Error (MARE) as
performance metrics, defined in Section 2.1.2.

##### Experiments on the Single-Camera Scenario

State-of-the-art comparison. We compared our solution with the results
obtained in [49], where a centralized detection-based counting approach has
been presented. It is based on the original version of Mask R-CNN having the
ResNet101 model as features extractor, which has been fine-tuned on a very
small manually annotated subset of the CNRPark-EXT dataset, starting from the
model pre-trained on the COCO dataset [147]. Although this solution is
computationally expensive and unsuitable for edge devices, it represents a
direct comparison in terms of counting on the same dataset. We also compared
our technique against the method proposed in [11], an approach for car parking
occupancy detection based on mAlexNet, a deep CNN designed explicitly for
smart cameras. This work represents an indirect method for counting cars in a
parking lot, as the counting problem is cast as a classification problem: if a
parking space is occupied, we increment the total number of cars. We
illustrate the results in Table 3.3, where we also report the performance
obtained using the Mask R-CNN network without a preliminary fine-tuning on the
CNRPark-EXT dataset. Our solution performed better than the other state-of-
the-art considered methods, considering all three counting metrics. In
particular, our approach outperformed the solution introduced in [49], despite
the latter employing a more deep and powerful CNN, and it is designed to be
used as a centralized-server solution. This is explained by the fact that in
[49], the authors fine-tuned the CNN using a tiny dataset. Consequently, the
algorithm overfits over the training data, and it cannot generalize over the
test subset. It is also worthy of notice that our CNN also outperformed the
mAlexNet network, even though the latter knows the exact location of the
parking spaces. Figure 3.8 shows some examples of images belonging to
different cameras and different weather conditions together with the masks
localizing the found vehicles.

?tablename? 3.3: Experiments on the single-camera scenario: comparison against SOTA. Left-side: results obtained using our counting solution compared with other state-of-the-art approaches; we get the best results on all the three considered counting metrics. Right-side: evaluation of the generalization capabilities on the PKLot dataset [63], using the model trained on the CNRPark-EXT dataset; we achieved an error that is approximately four times lower than the one obtained with the COCO pre-trained model. | CNRPark-EXT | PKLot
---|---|---
Method |  MAE |  MSE |  MARE |  MAE |  MSE |  MARE
[11] |  1.34 |  8.00 |  0.04 | -
[49] |  1.05 |  4.41 |  0.03 | -
ResNet50 Mask R-CNN |  11.20 |  247.40 |  0.30 |  16.90 |  522.40 |  0.48
Our solution |  0.49 |  1.04 |  0.01 |  4.56 |  33.88 |  0.13

(a) Image from Camera2

(b) Image from Camera8

?figurename? 3.8: Examples of vehicle localization and counting in the Single-
Camera Scenario. Images are taken from the CNRPark-EXT dataset. We report the
masks localizing detected cars and the estimate of their number.

Generalization capabilities. Errors in vehicle detection and counting are due
to many reasons, but critical points are different light conditions and
diverse perspectives. Weather conditions might produce significant
illumination changes since puddles and wet floors create a textural pattern
that may lead to an error, and sunbeams can create reflections on the car
windscreen, covering the majority of the images with saturated patterns. When
a CNN does not generalize well, it works well only in the conditions where it
was trained. To measure the robustness of our approach to these scenarios, we
performed two types of experiments exploiting the CNRPark-EXT dataset: i)
inter-weather and ii) inter-camera experiments. In the former, we trained our
CNN with images taken in one particular weather condition, and we computed the
performance metrics obtained on images having different weather conditions. In
particular, we performed three experiments, training respectively on the
Sunny, Overcast and Rainy subsets of the CNRPark-EXT dataset. In the latter,
we trained our algorithm employing images from one camera, and then we
computed the performance metrics on pictures captured by another camera. In
particular, we performed two experiments, training with images coming
respectively from camera1 and camera8. We chose these two cameras because they
are particularly representative since one has a side view of the parking lot
while the other has a pure front view. We report the results of the two
experiments in Table 3.4(a) and Table 3.4(b), respectively. We achieved a good
generalization in both the considered scenarios. We experienced a larger
amount of error when the CNN was trained and tested on two opposite weather
conditions, for instance, Sunny and Rainy, while the more accurate model was
the one trained on Overcast weather conditions. However, the performance
difference is quite small. On the other hand, in inter-camera experiments, the
model trained on camera8 is the best, and it has a slight drop in performance
only when tested on the camera1 subset. The model trained on the camera1
dataset performed in general worse. This is probably due to a bias in the
CNRPark-EXT dataset, where the majority of the images are captured from a
frontal viewpoint.

?tablename? 3.4: Experiments on the single-camera scenario: generalization
capabilities.

(a) Results of inter-weather experiments obtained when training on sunny, overcast, or rainy weather. | Sunny | Overcast | Rainy
---|---|---|---
Train Set |  MAE |  MSE |  MARE |  MAE |  MSE |  MARE |  MAE |  MSE |  MARE
Sunny |  - |  - |  - |  0.29 |  0.34 |  0.009 |  0.96 |  2.78 |  0.02
Overcast |  0.62 |  1.09 |  0.02 |  - |  - |  - |  0.56 |  1.26 |  0.01
Rainy |  0.84 |  1.65 |  0.02 |  0.49 |  0.65 |  0.01 |  - |  - |  -

(b) Results of inter-camera experiments obtained when training on camera 1 and camera 8. |  | Test Set
---|---|---
Metric | Train Set |  C1 |  C2 |  C3 |  C4 |  C5 |  C6 |  C7 |  C8 |  C9
MAE | C1 |  - |  0.77 |  1.21 |  2.53 |  3.26 |  2.57 |  2.88 |  2.88 |  1.54
C8 |  3.87 |  0.85 |  0.76 |  0.45 |  0.48 |  0.71 |  1.07 |  - |  0.41
MARE | C1 |  - |  0.08 |  0.05 |  0.06 |  0.07 |  0.05 |  0.06 |  0.05 |  0.05
C8 |  0.11 |  0.09 |  0.03 |  0.01 |  0.01 |  0.01 |  0.02 |  - |  0.01
MSE | C1 |  - |  1.48 |  2.91 |  10.61 |  20.24 |  13.50 |  19.82 |  17.30 |  7.19
C8 |  22.60 |  1.78 |  1.36 |  0.57 |  0.74 |  0.95 |  4.97 |  - |  2.13

Moreover, to further validate the generalization capabilities of our approach,
we considered our counting network trained on the entire training set of the
CNRPark-EXT dataset, and we tested it over a different dataset, the PKLot
dataset [63]. Results are shown in Table 3.3 where we also report the
performance obtained using the Mask R-CNN network without a preliminary fine-
tuning on the CNRPark-EXT dataset. As we can see, using our solution, we
achieved an error that is approximately four times lower than the one obtained
with the COCO pre-trained model.

##### Experiments on the Multi-Camera Scenario

To the best of our knowledge, there are no annotated datasets in the
literature suitable for evaluating counting algorithms operating on multiple
FOV-overlapping cameras. Hence, we performed our experiments on the extended
version of the CNRPark-EXT dataset, which we created on purpose in this
dissertation. To demonstrate that our algorithm can benefit from the redundant
information from the different cameras, we compared the obtained results
against a baseline and a simplified version of our algorithm.

In particular, we compared our solution against a system that is not aware of
the other overlapped areas, and so it just sums up all the vehicles detected
by all the cameras belonging to a sequence (Naïve Counting N). Then, we
considered a more conservative approach, where the nodes employed the
homographic transformations only with the purpose of black-masking the
overlapped areas (Overlap Masking M). This latter baseline then loses the
ability to take advantage of monitoring the same lots from different views.
However, it is still aware of the locations of the overlapping areas, and it
considers the vehicles inside them only once. Results are shown in Table 3.5.
Our solution obtained the best results compared to the considered baselines in
the three counting metrics and all the employed scenarios. We report the
errors concerning the considered six sequences of the CNRPark-EXT dataset,
together with the MAE, MSE, and MARE, which summarize the mean results
regarding all the scenarios. As an example, in Figure 3.9 we also report the
output of our multi-camera algorithm for a pair of images belonging to two
different cameras having a shared area in their field of view, where we
highlight in red and blue the masks projected from one camera to the other,
using the previously computed homographic transformations.

?tablename? 3.5: Experiments on the multi-camera scenario. We consider the
entire parking lot, comparing our solution against a baseline and a simplified
version of our algorithm. We report the errors obtained on the six considered
sequences (two for each weather condition) of the CNRPark-EXT dataset that we
extend on purpose.

| Error | Absolute Err. | Squared Err. | Relative Err. (%)
---|---|---|---|---
| N | M | O | N | M | O | N | M | O | N | M | O
Overcast-1 | 124 | -33 | 2 | 124 | 33 | 2 | 15,376 | 1,089 | 4 | 71.6 | 19.0 | 1.2
Overcast-2 | 131 | -26 | 1 | 131 | 26 | 1 | 17,161 | 676 | 1 | 76.1 | 15.1 | 0.6
Rainy-1 | 80 | -39 | -5 | 80 | 39 | 5 | 6,400 | 1,521 | 25 | 47.6 | 23.2 | 2.9
Rainy-2 | 105 | -44 | -5 | 105 | 44 | 5 | 11,025 | 1,936 | 25 | 54.4 | 22.8 | 2.6
Sunny-1 | 117 | -38 | 2 | 117 | 38 | 2 | 13,689 | 1,444 | 4 | 68.0 | 22.1 | 1.2
Sunny-2 | 113 | -37 | 2 | 113 | 38 | 2 | 12,769 | 1,444 | 4 | 66.1 | 22.2 | 1.2
Mean | 111.6 | -36.1 | -0.5 | 111.6 | 36.3 | 2.8 | 12,736.6 | 1,351.6 | 10.5 | 63.9 | 20.7 | 1.6

N: Naïve Counting; M: Overlap Masking; O: Ours (mean aggr., IoU Threshold
$\tau=0.2$)

(a) Image from Camera9

(b) Image from Camera8

?figurename? 3.9: Example of vehicle localization and counting in the Multi-
Camera Scenario. We consider a pair of images belonging to two different
cameras $i,j$ having a shared area in their FOV. We report in green the masks
localizing the vehicles detected by a camera in its own FOV, while in red and
blue, the masks projected from camera j to camera i and vice-versa, employing
the homographic transformations pre-computed during the system initialization.

####  3.2.6 Conclusion

In the second part of this chapter, we presented a distributed Artificial
Intelligence-based system that automatically counts the vehicles present in a
parking lot using images taken by multiple smart cameras. Unlike most of the
works in literature, we introduced a multi-camera approach that can estimate
the number of cars present in the entire parking area and not only in the
single captured images. The main peculiarities of this approach are that all
the computation is performed in a distributed manner at the edge of the
network and that there is no need for any extra information about the
monitored parking area, such as the location of the parking spaces, nor any
geometric information about the position of the cameras in the parking lot. We
modeled our system as a graph. The nodes, i.e., the smart cameras, are
responsible for estimating the number of cars present in their view and
merging data from nearby devices with an overlapping field of view. Our
solution is simple but effective, combining a Deep Learning technique with a
distributed geometry-based approach. We evaluated our algorithm on the
CNRPark-EXT dataset, which we specifically extended to show how we benefit
from redundant information from different cameras while improving overall
performance.

##  ?chaptername? 4 Virtual To Real Adaptation of Pedestrian Detectors

A key task in many intelligent video surveillance systems is pedestrian
detection, as it provides essential information for semantic understanding of
video. Accurate detection of individual instances of pedestrians in images
plays a vital role in a myriad of applications that can positively impact the
quality of human life. They range from video surveillance [26, 224], robotics,
automotive [83, 200] and assistive technologies to people with visual
disabilities [212], just to name a few. Moreover, pedestrian detection also
provides the main building block for people counting using detection-based
approaches. Once the pedestrian instances are localized, it is possible to
estimate the number of people present in the scene or a region of interest
inside it.

Convolutional Neural Network s-based methods [136] have recently demonstrated
their superiority compared to the approaches relying on hand-crafted features.
However, despite the recent advances, the pedestrian detection task remains a
challenging active research area in Computer Vision. One of the main issues
still affecting CNN-based approaches for pedestrian detection is related to
data scarcity. Indeed, the crux of CNN s is that to generalize well at
inference time, they require a massive amount of diverse well-labeled data
during the training phase, covering the widest number of different scenarios.
While there exist some large annotated generic datasets suitable for training
these supervised learning networks, such as ImageNet [66] and MS COCO [147],
in many real-world situations they are not enough. Hence, as a consequence, a
model that learned from these data usually experiences a drastic drop in
performance at inference time when applied to other scenarios never seen
during the training phase. This performance gap is due to the fact that the
network learns from data belonging to one domain (named the training or source
domain) and is then applied on another different domain (the test or target
domain), and is commonly referred as Domain Shift [215].

Since manually annotating new collections of images is expensive and requires
great human effort, it is unfeasible to label new target data every time to
supervise the model to acquire knowledge from the test domain. A recently
promising approach is to gather data from virtual world environments that
mimic as much as possible all the characteristics of the real-world scenarios
and where the annotations can be acquired with an automated process. To this
end, in this chapter, we introduce Virtual Pedestrian Dataset (ViPeD), a new
synthetic dataset generated with the highly photo-realistic graphical engine
of the video game GTA V (Grand Theft Auto V) by Rockstar North, that extends
the Joint Track Auto (JTA) dataset presented in [72]. ViPeD is the first
synthetic dataset suitable for the pedestrian detection task, annotated with
bounding boxes locating the instances of the people present in the scenes.

While synthetic data collections are very appealing, usually, when training
solely on a synthetic dataset, the model does not generalize well to real-
world data on which the pedestrian detector, in the end, shall be used. In
other words, the potentials inherent in synthetic data can only be partially
exploited. This is due to the Synthetic2Real Domain Shift. In this case, the
source and the target domains are the synthetic and the real-world,
respectively. The gap between the two data distributions is caused by the fact
that synthetic images’ appearance is still significantly different from that
observed in real-world images, even using current rendering techniques. To
mitigate this domain gap, in this chapter, we also propose two different
supervised Domain Adaptation (DA) methods, suitable for the pedestrian
detection task but possibly applicable to general object detection. The first
one consists of training the model by exploiting the synthetic data and then,
in a second step, fine-tuning it using real-world images. Instead, the second
one consists of an end-to-end training procedure in which we employed mixed
batches containing both synthetic and real-world data.

We conducted extensive experimentation to validate our approaches. First, we
tested the generalization capabilities of the detector over unseen scenarios.
We showed that we obtained better or comparable results when training with the
synthetic data compared to using the same model trained using only real-world
images, just taking advantage of the variety of ViPeD. Secondly, we
experimented with the two proposed supervised Domain Adaptation techniques to
boost the performance over specific real-world scenarios. We demonstrated that
we reduced the Synthetic2Real Domain Shift by bringing the two domains closer
together, thus achieving better results.

To summarize, the main contributions presented in this chapter are the
following:

  * •

we present Virtual Pedestrian Dataset (ViPeD), a new vast synthetic dataset
collected using the photo-realistic video game GTA V (Grand Theft Auto V) and
automatically annotated by the graphical engine. To the best of our knowledge,
it was the first synthetic dataset suitable for pedestrian detection;

  * •

we introduce two supervised Domain Adaptation techniques to mitigate the
Synthetic2Real Domain Shift existing between the synthetic and the real-world
images;

  * •

we demonstrate through extensive experimentation on various real-world
pedestrian detection datasets the generalization capabilities of the detector
trained using synthetic data, and that the two proposed DA solutions boost the
performance over specific real-world scenarios, bringing the synthetic and the
real-world domains closer.

The chapter is organized as follows. In Section 4.1, we review some works
related to the pedestrian detection task and the DA, focusing on the ones more
significant for our proposal. In Section 4.2, we describe ViPeD, our new
synthetic dataset. In Section 4.3, we present the DA strategies we employed to
contrast the Synthetic2Real Domain Shift. Section 4.4 shows the experimental
evaluation of our approaches.

The research presented in this chapter was published in [14, 50]. The code,
the models, and the dataset are made freely available at
https://ciampluca.github.io/viped.

### 4.1 Related Work

This section reviews some relevant works about the object and pedestrian
detection task. We also analyze some previous studies on DA, focusing on the
Synthetic2Real domain shift.

###### Pedestrian Detection

Pedestrian detection is highly related to object detection. It deals with
locating and recognizing instances of pedestrians’ specific class, usually in
images of urban environments, without considering group dynamics. We can
subdivide approaches for the pedestrian detection task into two main research
areas. The first class of detectors is based on handcrafted features, such as
ICF (Integral Channel Features) [22, 261, 262, 263, 169]. These methods can
usually rely on higher computational efficiency at the cost of lower accuracy.
On the other hand, more recently, Deep Neural Network approaches have been
explored. For example, [213, 248, 36] proposed some solutions based on the CNN
networks to detect pedestrians, even accounting for different scales.

###### Synthetic2Real Domain Adaptation

With the need for huge amounts of labeled data, synthetic datasets have
recently gained considerable interest. Some notable examples are GTA5 [188]
and SYNTHIA [192] for semantic segmentation. However, there is a non-
negligible domain gap between the synthetic and the real worlds, as mentioned
above. Many techniques try to fill this gap, using both supervised and
unsupervised approaches. A more exhaustive survey about deep learning DA
techniques is provided in Section 2.3 and in [232]. For example, authors in
[216] proposed an unsupervised domain adaptation solution for the segmentation
task. Authors in [72] created Joint Track Auto (JTA), a synthetic dataset
gathered from the highly photo-realistic video game GTA V. They demonstrated
that it is possible to reach excellent results on tasks such as people
tracking and pose estimation when validating on real data. In this work, we
extend this dataset, making it suitable for the pedestrian detection task.
Authors in [123, 161] have also focused on learning features from synthetic
data for the pedestrian detection task. Still, they did not consider deep
learning approaches, exploring only traditional detection techniques. In [28],
instead, the authors employed a synthetic dataset to train a CNN able to
detect objects belonging to different classes in a video. This CNN is
responsible only for the classification of the objects, while their detection
relies on a background subtraction algorithm based on Gaussian Mixture Models
(GMMs). The performance of this approach over real-world scenarios was
evaluated employing two pedestrian detection datasets, one of which, the 2D
MOT 2015 [135], is an older version of the dataset we used to carry out our
experiments. To the best of our knowledge, [121] and [28] are the closest
works to ours. In particular, they also used GTA V as the source for the
acquisition of the synthetic data, but they focused their efforts on the
vehicle detection task.

### 4.2 ViPeD

Virtual Pedestrian Dataset (ViPeD) is a new synthetic dataset generated with
the highly photo-realistic graphical engine of the video game GTA V (Grand
Theft Auto V) by Rockstar North. It extends Joint Track Auto (JTA) dataset,
presented in [72]. The dataset includes a total of about 500K images,
extracted from 512 full-HD videos of different urban scenarios. These videos
are organized into a training set (256 videos) and a test set (the remaining
256 videos).

While we could reuse the already existing JTA images, we needed to generate
suitable annotations for the pedestrian detection task. Indeed, the JTA
dataset provides only skeletal information, which is helpful in the pose
estimation and tracking tasks. Instead, in our scenario, we are required to
annotate each pedestrian with the four coordinates (x, y, w, h) delimiting its
minimum enclosing bounding box. Hence, we employed the already available JTA
images producing a new set of labels suitable for our task.

Estimating the precise bounding box surrounding each pedestrian instance can
be tricky, as we did not have access to the underlying GTA game engine. Other
works tried to overcome this problem by using some interesting work-around.
For example, [121] extracted the semantic masks around each object in the
scene and separated the instances by exploiting the depth information
available through the depth buffer.

Our solution relied instead on the skeletal information already provided by
the JTA annotations. Indeed, differently from [121], we dealt with multiple
instances of pedestrians in possibly highly crowded scenarios. In these cases,
the depth information may be insufficient for distinguishing two different
pedestrians, leading to possible severe bounding box estimation errors.

As a first approximation, we exploited the skeleton joints’ positions in
screen coordinates, directly available from the JTA annotations, for drawing
the minimum bounding box enclosing all the skeleton joints (green boxes in
LABEL:dataset_bbs). However, it can be noticed that the bounding boxes
produced using this simple procedure are undersized compared to the full-sized
pedestrian instance, as the skeleton always lays below the skin surface. We
solved this issue by constructing a more precise bounding box (blue boxes in
LABEL:dataset_bbs), obtained by estimating an amount of padding through a
simple heuristic.

In particular, we estimated the height of a pedestrian mesh, denoted as
$h_{m}$, from the height $h_{s}$ of its skeleton, through the formula:

$h_{m}=h_{s}+\frac{\alpha}{z},$ (4.1)

where $z$ is the distance of the pedestrian center of mass from the camera,
and $\alpha$ is a parameter that depends on the camera projection matrix.

(a)

(b)

?figurename? 4.1: (a) Pedestrians in the JTA (Joint Track Auto) dataset with
their skeletons. (b) Examples of annotations in the Virtual Pedestrian Dataset
(ViPeD) dataset; original bounding boxes are in green, while the sanitized
ones are in blue.

The $z$ value for each pedestrian was already part of the JTA annotations,
while $\alpha$ is unknown since we could not access the camera parameters.
Then, we evaluated $\alpha$ from Equation 4.1, estimating $h_{m}$ for a small
representative population of pedestrians. To this end, we isolated 50 random
pedestrians from different scenarios, and we manually annotated them with
their height in pixels units. At this point, it has been possible to recover
the value of $\alpha$ from Equation 4.1 performing a regression to find the
best fit.

The height padding depends basically only on the distance of the pedestrian
from the camera. Instead, the width is also linked to the specific pedestrian
pose. However, we found that we can ignore these pose-dependent effects while
still obtaining an excellent estimate by deriving the pedestrian width $w_{m}$
assuming no changes in the original bounding box aspect ratio. For this
reason, we derived $w_{m}$ from the computed $h_{m}$ as follows:

$w_{m}=h_{m}\frac{w_{s}}{h_{s}}=h_{m}r,$ (4.2)

where $r$ is the aspect ratio of the bounding box enclosing the skeleton. Some
examples of final estimated bounding boxes are shown in blue in
LABEL:dataset_bbs.

We then assessed the quality of the produced bounding boxes. In Figure 4.2, we
report a histogram depicting the distribution of the distances of the
pedestrians from the camera. We observed that human annotators tend not to
annotate pedestrians far more than a certain amount from the camera in real-
world datasets. We computed this distance limit by finding the minimum
bounding box height, in pixels, occurring in human annotations of the MOT17Det
[164] dataset, and seeing at what distance from the camera we reached this
bounding box limit size on the JTA annotations. We concluded that human
annotators did not include bounding boxes for pedestrians farther than 30–40
meters from the camera. Then, to be consistent with real-world datasets on
which we will validate our approach, we cleaned the produced bounding boxes by
pruning all the ones enclosing pedestrians farther than 40 meters.

?figurename? 4.2: Histogram of distances between pedestrians and cameras. We
show the distribution of the distances of the pedestrians from the camera
characterizing our Virtual Pedestrian Dataset (ViPeD).

In Figure 4.3, we report some examples of images of the ViPeD dataset together
with the sanitized bounding boxes.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

?figurename? 4.3: Samples of Virtual Pedestrian Dataset (ViPeD). We show some
images of our synthetic dataset together with the sanitized bounding boxes
localizing pedestrians..

### 4.3 Domain Adaptation for Synthetic2Real Pedestrian Detection

In this section, we describe the object detector and the domain adaptation
strategies we employed in this research. We exploited Faster R-CNN [187], a
widely used state-of-the-art object detector that we briefly review in Section
4.3.1 and that it is also described more in detail in Section 2.2.2. We
trained this CNN using ViPeD, our collection of synthetic images automatically
annotated, already outlined in the previous section. To mitigate the existing
domain shift between these data and the real-world ones, we proposed two
supervised Domain Adaptation techniques (for a better explanation of DA, see
also Section 2.3). The first one, described in Section 4.3.2, consists of
training the detector with the synthetic data and then fine-tuning it by
exploiting the real-world images. In the second approach, described in Section
4.3.3, we employed instead another supervised technique, called Balanced
Gradient Contribution (BGC) [192, 193], where we mixed the synthetic and the
real-world data during the training phase. Figure 4.4 and Figure 4.5 show an
overview of the two solutions.

####  4.3.1 Faster R-CNN Object Detector

We exploited Faster-RCNN [187] as object detector architecture. Our choice
fell on Faster R-CNN since it provides state-of-the-art performance.
Furthermore, we did not consider pedestrian detection-specific solutions since
the two proposed Domain Adaptation techniques can also be applied to other
tasks, accounting for another class of objects different from the pedestrian
one.

We described in detail Faster R-CNN in Section 2.2.2. It is a two-stage CNN-
based algorithm composed of different networks: the backbone, the Region
Proposal Network (RPN), and the Evaluation Network (EN). In the first stage, a
CNN acts as a backbone, extracting the input image features. Starting from
these features space, the RPN is in charge of generating region proposals that
might contain objects. Briefly, RPN slices pre-defined region boxes (called
anchors) over this space and ranks them, suggesting the ones most likely
containing objects. Once RPN produces the Regions Of Interests (ROIs), they
might be of different sizes. Since it is hard to work on features having
different sizes, RPN reduces them into the same dimension using the Region of
Interest Pooling algorithm. The EN finally processes these fixed-size
proposals, responsible for classifying and locating the objects inside them.
Then, given an input image, the EN network’s final outputs are class scores
and bounding box coordinates.

Faster R-CNN is then a versatile and modular network in which it is possible
to change the building blocks. Regarding the backbone, our choice fell on the
ResNet-50 network, a lighter version of the very popular ResNet-101 network
[100]. Indeed, Faster R-CNN with ResNet-50 can produce satisfactory detection
results compared to the low computational resources and the time required
during the training and test phases.

####  4.3.2 Domain Adaptation using Real-World Fine-Tuning

The first proposed DA solution relies on a Transfer Learning (TL) strategy. As
pointed out in [232] and in Section 2.3, DA is a particular Transfer Learning
(TL) case that employs labeled data in one or more relevant source domains to
execute the task in a target domain. In particular, the crucial point in this
methodology consists of fine-tuning a previously trained model with the
target-domain data.

We divided our fine-tuning methodology into two different steps. In the first
step, we considered as the baseline the Faster R-CNN detector described in the
above section, having a ResNet-50 backbone pre-trained on the COCO dataset
[147], a large collection of images depicting complex everyday scenes of
ordinary objects in their natural context, divided into 80 different
categories. Since this network is a generic object detector that can
distinguish between many different classes of objects, we modified the EN
building block to adapt the model to our purposes. In particular, we reduced
the last fully connected layers of the detector to recognize and locate object
instances belonging only to a specific category, i.e., the pedestrian
category. Then, we trained this modified Faster R-CNN-based network exploiting
our synthetic images of ViPeD, leaving all the model weights unfrozen during
this phase so that the back-propagation algorithm can tune them.

In the second step, we fine-tuned this pre-trained model using real-world
images as the target domain in the second step. So, in the end, the network
will have processed both source and target images, memorizing in its weights
information from both the domains. Figure 4.4 shows an overview of this
approach. This method, looking at real-world images in this last step, is
beneficial for boosting the performance of the detector in a specific real-
world target scenario.

?figurename? 4.4: Overview of the first domain adaptation technique. In a
first step, we train the detector using ViPeD, our synthetic collection of
images. Then, in a second step, we fine-tune the network using real-world
images.

####  4.3.3 Domain Adaptation using Balanced Gradient Contribution

The second DA approach is an end-to-end training, so it benefits from not
relying on a two-step process like the previous one.

As in the previous solution, we started with the modified Faster R-CNN
detector having the ResNet-50 backbone pre-trained on the COCO dataset. This
time, we trained the network using mixed batches, i.e., we employed batches
containing synthetic and real-world images simultaneously, given a fixed
mixing ratio. As explained in [193], the real-world data acts as a
regularization term over the synthetic data training loss. In particular, we
exploited batches composed of $2/3$ of synthetic images and $1/3$ of real-
world data. Thus, statistics from both domains are considered throughout the
entire procedure, creating a more accurate model for both. Again, during this
phase, we left all the network weights unfrozen so that the back-propagation
algorithm could modify the network parameters accordingly. Consequently, we
mitigated the Synthetic2Real Domain Shift straight in a single-step training.
Figure 4.5 shows an overview of this approach.

In the experiments, we employed this technique for boosting the performance of
the detector on a precise target scenario, using batches composed of the
synthetic data and the real-world images specific for the particular
considered scenes. Besides, we exploited this solution also for achieving
wider generalization capabilities, considering batches containing synthetic
images and generic real-world images containing pedestrians.

?figurename? 4.5: Overview of the second domain adaptation technique. We
mitigate the Synthetic2Real domain shift in a single-step training procedure,
employing mixed batches containing both synthetic and real images at the same
time.

### 4.4 Experimental Evaluation

In this section, we briefly report some details about the real-world datasets
exploited for the experiments. Then, we show and discuss the results
concerning the generalization capabilities of our detector trained using
ViPeD. Finally, we illustrate the performance of the two Domain Adaptation
techniques over specific real-world scenarios.

####  4.4.1 Real-World Datasets

MOT17Det [164] and MOT19Det [65] are datasets for pedestrian detection that
are particularly suitable for surveillance applications. They comprise a
collection of challenging images (5,316 and 8,931, respectively) taken from
multiple sequences with various crowded scenarios having different viewpoints,
weather conditions, and camera motions. The authors provided training and test
subsets, but they released only the ground-truth annotations belonging to the
former. The performance metrics concerning the test subsets are instead
available submitting results to their MOT Challenge website
(https://motchallenge.net). The main peculiarity of MOT19Det compared to
MOT17Det is the massive crowding of the collected scenarios.

CityPersons dataset [264] consists of a large and diverse set of stereo video
sequences recorded in streets of different cities in Germany and neighboring
countries. In particular, the authors provided 5,000 images from 27 cities
labeled with bounding boxes and divided across train/validation/test subsets.
This dataset is more focused on self-driving applications, and images are
collected from a moving car.

COCOPersons dataset is a split of the popular COCO dataset [147] comprising
images collected in general contexts belonging to 80 categories. We filtered
these images considering only the ones belonging to the persons category.
Hence, we obtained a new dataset of about 66,000 images containing at least
one pedestrian instance.

####  4.4.2 Experiments

We evaluated the detection performances using the standard mean Average
Precision (mAP) metric (see Section 2.2.1 for a more detailed explanation). In
particular, we considered the detection proposals having a score confidence
greater than 0.05. Then, we employed the COCO mAP [147] and the MOT AP metrics
[164], fixing the IoU threshold to 0.5 and varying only the detection
confidence threshold.

##### Testing Generalization Capabilities

To test the generalization capabilities, we trained the detector on a source
domain, and then we validated it on a different target domain. In particular,
we trained the model using a dataset, and then we tested it on another one. In
this way, we guaranteed that the two distributions were different and not
related.

In particular, we trained the modified Faster R-CNN-based detector described
in Section 4.3.1 using ViPeD. This procedure corresponds to the first step of
the previously described domain adaptation solution (see Section 4.3.2). We
evaluated this model using the real-world datasets MOT17Det, MOT19Det, and
CityPersons, defining three subsets containing images not present in the
training subset. To form a solid baseline for this experiment, we trained the
same detector using every one of the three real-world datasets, and then we
tested them over the remaining two. We also reported a further baseline
considering the detector trained only on the real-world general-purpose COCO
dataset, considering only the detections belonging to the person category.

We also experimented with the mixed-batch DA approach explained in Section
4.3.3, using the same evaluation protocol as before. We exploited batches
composed of $2/3$ of ViPeD and by the remaining $1/3$ of COCOPersons. We chose
the latter as the real-world dataset since it depicts humans in highly
heterogeneous scenarios, and it is not biased towards a specific application
(e.g., autonomous driving). Again, we evaluated this model testing on all the
three remaining real-world datasets.

We report the results in Table 4.1. Note that we omit results concerning a
specific dataset if employed during the training phase for a fair evaluation
of the overall generalization capabilities. In most cases, as we can see, our
network performed better than those trained using only the manually annotated
real-world datasets, taking advantage of the high variability and size of the
ViPeD dataset. In particular, concerning the MOT17Det dataset, all our
solutions trained with synthetic data outperformed those trained with real
ones. We obtained the best results using the mixed-batch approach. Considering
the MOT19Det dataset, we achieved the best result in training the detector
with ViPeD. CityPersons is the only dataset on which the algorithm maintained
higher performances when trained with real-world data. In particular, the
highest mAP on CityPersons is obtained when the detector is trained with the
MOT17Det dataset. However, in this case, the mixed-batch approach achieved
results comparable with the baselines.

?tablename? 4.1: Evaluation of the generalization capabilities. The first section of the table reports results obtained training the detector with real-world data, while the latter is related to the model trained over synthetic images. ViPeD \+ Real refer to the mixed batch experiments with $2/3$ ViPeD and $1/3$ of COCOPersons. Results are evaluated using the COCO mAP. We report in bold the best results. | Test Dataset
---|---
Training Dataset | MOT17Det | MOT19Det | CityPersons
COCO | 0.636 | 0.466 | 0.546
MOT17Det | - | 0.605 | 0.571
MOT19Det | 0.618 | - | 0.419
CityPersons | 0.710 | 0.488 | -
ViPeD | 0.721 | 0.629 | 0.516
ViPeD \+ Real | 0.733 | 0.582 | 0.546

##### Testing Domain Adaptation Techniques over Specific Real-world Scenarios

To test how the two proposed domain adaptation techniques behave when
considering specific target real-world scenarios, we considered the MOT17Det
and MOT19Det real-world datasets.

Regarding the fine-tuning DA approach, we considered as training sets those
proposed by the authors of [164, 65], and we obtained the evaluation of our
results over the test sets by submitting them to the Mot Challenge website.
For the mixed-batch DA solution, during the training phase, we injected in the
same batch $2/3$ of synthetic images from the ViPeD dataset and $1/3$ of real-
world images from the training subsets of the MOTDet17 or the MOT19Det
dataset. Again, we validated our results by submitting them to the MOT
Challenge website.

Table 4.2 and Table 4.3 report the results for the two considered scenarios.
We report our results together with the state-of-the-art approaches publicly
released in the MOT Challenges (at the time of writing). As we can see, the
two DA approaches can mitigate the Synthetic2Real Domain Shift. In both
datasets, we obtained an improvement in performance compared to the results in
Table 4.1. It is also worth noting that we achieved competitive results in
both scenarios compared to the state-of-the-art, reaching (at the time of
writing) the first and the second places in the leader boards of the MOT17Det
and MOT19Det challenges, respectively.

?tablename? 4.2: Evaluation of the two DA techniques on the MOT17Det dataset. FT-DA (Fine Tuning DA) is the first proposed solution, while MB-DA (Mixed Batch DA) is the second one. Results are evaluated using the MOT mean average precision (mAP). Method | MOT AP
---|---
YTLAB [36] | 0.89
KDNT [252] | 0.89
ViPeD FT-DA (our) | 0.89
ViPeD MB-DA (our) | 0.87
ZIZOM [145] | 0.81
SDP [248] | 0.81
FRCNN [187] | 0.72
?tablename? 4.3: Evaluation of the two DA techniques on the MOT19Det dataset. FT-DA (Fine Tuning DA) is the first proposed solution, while MB-DA (Mixed Batch DA) is the second one. Results are evaluated using the MOT mAP. Method | MOT AP
---|---
SRK_ODESA | 0.81
CVPR19_det | 0.80
Aaron | 0.79
PSdetect19 | 0.74
ViPeD FT-DA (our) | 0.80
ViPeD MB-DA (our) | 0.80

### 4.5 Summary

In this chapter, we addressed the pedestrian detection task by proposing a
CNN-based solution trained using synthetically generated data. The choice of
training a CNN using synthetic data is motivated by the fact that the network,
to generalize well, requires a considerable amount of manually annotated
images representing different scenarios. This procedure usually requires a
significant human effort, and it is error-prone. Consequently, often, training
of CNN s has to deal with data scarcity, resulting in performance degradation
when applied to new scenarios at inference time.

To this end, we introduced a synthetic dataset named Virtual Pedestrian
Dataset (ViPeD), containing a massive collection of images rendered from the
highly photo-realistic video game GTA V developed by Rockstar North and a full
set of precise bounding boxes annotations around all the visible pedestrians.
It is worth noting that, in this case, the labels have been automatically
collected without human intervention. To the best of our knowledge, it was the
first synthetic dataset suitable for training CNN-based pedestrian detectors.
However, the potential inherent in synthetic data can only be partially
exploited. Indeed, there is a domain gap between the synthetic and the real-
world data because synthetic images’ appearance is still significantly
different from that observed in real-world images, even using current
rendering techniques. To this end, we proposed two different supervised Domain
Adaptation techniques to mitigate this Synthetic2Real Domain Shift, which are
suitable for pedestrian detection and possibly applicable to more general
object detection tasks.

The experiments showed that, in most cases, the detector trained with the
synthetic data generalizes better on unseen scenarios than the same algorithm
trained using only the manually annotated real-world datasets. Moreover, the
two proposed DA approaches were able to mitigate the underlying differences
between the two worlds, obtaining a performance improvement in specific real-
world scenarios. In our opinion, the result of this research opens new
perspectives to address the scalability of pedestrian and object detection
methods for large physical systems with limited supervisory resources. Using
our freely available model trained using ViPeD, future researchers will have
at their disposal a detector able to localize instances of people over images
belonging to a multitude of different scenarios and, therefore, a system
robust to newly added sources of data. On the other hand, they will also have
the possibility of further specializing the detector to work over newly added
real-world scenarios using our two domain adaptation techniques, obtaining an
additional performance boost. However, the adoption of supervised domain
adaptation methods to mitigate the gap between the two data distributions has
a price since it still employs labeled data from the real-world domain. In the
next chapter, we will propose an unsupervised DA strategy, which overcomes
this problem, relying only on supervision in the synthetic domain and
inferring some knowledge from test data without using the labels.

##  ?chaptername? 5 Unsupervised Domain Adaptation for Traffic Density
Estimation and Counting

As seen in the previous chapters, with the advent of Convolutional Neural
Networks (CNNs), supervised learning has reached excellent results across many
Computer Vision application areas, such as object detection and counting.
However, as pointed out in Chapter 4, most CNN-based methods require a large
amount of well-labeled data for supervised learning and make a common
assumption: the training and testing data are drawn from the same
distribution. The direct transfer of the learned features between different
domains does not work very well because the distributions are different. Thus,
a model trained on one domain, named source, usually experiences a drastic
drop in performance when applied on another domain, named target. This problem
is commonly referred to as the Domain Shift. As we have seen, Domain
Adaptation (DA) is a common technique to address this problem. It adapts a
trained neural network by fine-tuning it with a new set of labeled data
belonging to the new distribution. However, in many real cases, gathering a
further collection of labeled data is expensive, especially for tasks that
imply per-pixel annotations, like semantic or instance segmentation. A
promising solution is to collect synthetic data where annotations are gathered
automatically by the graphical engine. Still, data coming from virtual worlds
cannot be fully exploited due to the Synthetic-to-Real Domain Shift, and
supervised adaptations using real-world data are often needed.

Unsupervised Domain Adaptation (UDA) addresses the domain shift problem
differently. It does not use labeled data from the target domain and relies
only on supervision in the source domain. Specifically, UDA takes a source
labeled dataset and a target unlabeled one. The challenge here is to
automatically infer some knowledge from the target data to reduce the gap
between the two domains.

This chapter proposes an end-to-end CNN-based UDA algorithm for traffic
density estimation and counting based on adversarial learning. Adversarial
learning is performed directly on the generated density maps, i.e., in the
output space, given that in this specific case, the output space contains
valuable information such as scene layout and context. We focused on vehicle
counting, but the approach is suitable for counting any other types of
objects. To the best of our knowledge, this was the first attempt to introduce
a UDA scheme for counting to reduce the gap between the source and the target
domain without using additional labels.

We conducted experiments considering different domain shifts and validating
our approach on various vehicle counting datasets. First, we employed two
existing datasets for traffic density estimation, WebCamT [260] and TRANCOS
[96]. We used images acquired by a specific subset of cameras as the source
domain. In contrast, to emphasize the domain shift problem, we represented the
target domain with images captured by a different subset of cameras, seeing
different perspectives and visual contexts. We called this type of domain
shift Camera2Camera domain shift. Comparisons with other techniques on these
datasets showed the superiority of our approach. To test our solution with
other types of domain shifts, we created and made publicly available two
additional datasets, described in the following. The first one was the Night
and Day Instance Segmented Park (NDISPark) dataset, consisting of images
captured by surveillance cameras in parking lots. Here, on the one hand,
source data include annotated images collected by various cameras during the
day. On the other hand, the unlabeled target domain contains images collected
during the night in the same scenarios. We called this domain shift Day2Night.
The second one was instead the Grand Traffic Auto (GTA) dataset, a vast
collection of synthetic images generated with the highly photo-realistic
graphical engine of the Grand Theft Auto V video game, developed by Rockstar
North. This dataset consists of urban traffic scenes, automatically and
precisely annotated with per-pixel annotations. To the best of our knowledge,
this was the first instance segmented synthetic dataset of traffic scenarios.
We used this dataset to train the counting algorithm. Then, we performed
Unsupervised Domain Adaptation to be able to count in real-world images. In
this case, the domain shift is represented by the Synthetic2Real difference.
Different from the previous chapter, in which we relied on a supervised
adaptation of the network using real-world data, here we dispensed the usage
of the annotations of the target data. Using the proposed UDA strategy, we
tackled the problem of data scarcity from two complementary sides: on the one
hand, we exploited the great variability of the synthetic data, while, on the
other hand, we mitigated the domain gap existing between the synthetic and the
real-world data in an unsupervised fashion. Figure 5.1 summarizes the
described domain shifts that we have addressed. In all the experiments, we
showed that our UDA technique always outperformed the non-domain adapted
models.

|  |  |
---|---|---|---
|  |  |
(a) | (b) | (c) | (d)
?figurename? 5.1: The Domain Shift scenarios that have been addressed in this
work. (a) Day2Night; (b) and (c) Camera2Camera; (d) Synthetic2Real. The first
row represents the labeled source domain, while the second represents the
unlabeled target one used for our unsupervised domain adaptation.

We summarize the main contributions presented in this chapter as follows:

  * •

we introduce a UDA algorithm for traffic density estimation and counting,
which can reduce the domain gap between a labeled source dataset and an
unlabeled target one. To the best of our knowledge, this was the first time
that UDA has been applied to counting;

  * •

we present two new datasets, both having instance segmentation annotations.
One was manually annotated, and it consists of images of parked cars collected
during the day and night. The second is a synthetic collection of images in
urban scenarios taken from a photo-realistic graphical engine, characterized
by per-pixel annotations gathered automatically;

  * •

we show through experimental evaluation over three different types of domain
shifts and various vehicle counting datasets that our UDA schema significantly
improved performances compared with the model without domain adaptation.

The chapter is organized as follows. In Section 5.1 we review some works
related to the UDA and the counting by density estimation task. Section 5.2
describes the employed datasets, emphasizing the two ones introduced in this
research. In Section 5.3 we introduce our UDA algorithm for traffic density
estimation and counting. In Section 5.4.1 we describe the experimental
settings, while in Section 5.4.2 we show the experiments and the obtained
results.

The research presented in this chapter was published in [51, 54]. All the
associated resources are freely available at
https://ciampluca.github.io/unsupervised_counting.

### 5.1 Related Works

This section reviews some previous work related to the Unsupervised Domain
Adaptation and the counting by density estimation task.

###### Unsupervised Domain Adaptation

Unsupervised Domain Adaptation (UDA) is a particular case of DA, in which we