rely only on supervision in the source domain without using the labels of the
target data. For a more comprehensive survey about DA see Section 2.3.
Traditional UDA approaches have been developed to address the problem of image
classification, and they try to align features across the two domains ([82],
[221]). However, as pointed out in [266], they do not perform well in other
tasks. More recent advances also involve the semantic segmentation task. In
this case, adversarial training for UDA is the most employed approach. It
includes two networks. The first predicts the segmentation maps for the input
source image. The second acts as a discriminator, taking the feature maps from
the segmentation network and trying to predict the input domain. The
adversarial loss, computed from the discriminator output, tries to make the
distributions of the two domains more similar. The first to apply such a
technique is [105]. More recently, the work proposed in [107] employs a
residual network and adversarial training to make the source feature maps
closer to the target ones. The authors of [46] combine semantic segmentation
and depth estimation to boost the adaptation performance, providing to the
discriminator the segmentation and the depth prediction maps jointly. Another
interesting work that inspired this paper is [216], where the authors applied
adversarial training to the output space taking advantage of the structural
consistency across domains.

###### Counting by Density Estimation

As already seen in Section 2.1, counting by density estimation is a supervised
learning approach that tries to establish a direct mapping from the image
features to a corresponding density map (i.e., a continuous-valued function),
skipping the challenging task of detecting instances of the objects. This
approach is particularly effective in very crowded scenarios where the
instances of the objects are sometimes not clearly visible due to occlusions.
Recently, Convolutional Neural Network s have been heavily exploited for
learning non-linear functions from crowd images to their corresponding density
maps, and a multitude of methods have been proposed in the literature. One of
the first works that employed a pure CNN to estimate the density and count
people in crowded contexts is presented by [30]. A more efficient structure is
proposed by [268] introducing a Multi-Column CNN-based architecture (MCNN) for
crowd counting. A similar idea is developed by [172] with a scale-aware,
multi-column counting model named Hydra-CNN able to estimate traffic densities
in congested scenes. More recently, the authors of [144] introduced CSRNet.
This CNN-based algorithm uses dilated kernels to deliver larger reception
fields and replace pooling operations. We employ this network as the baseline
in our work, and we briefly review its architecture in the next sections. The
main limitations of these approaches are due to the scarcity of data. As a
result, existing methods often suffer from overfitting, which leads to
performance degradation while transferring them to other scenes. Besides,
there is another inherent problem: the labels of these datasets are not very
accurate. Most of the existing datasets are dot-annotated. Consequently, the
ground truth density maps are just an approximation in which the sizes of the
objects are estimated using some heuristics. In this chapter, we address both
problems by proposing an UDA technique that exploits unlabeled data and by
introducing two new datasets with per-pixel annotations that allow the
creation of precise ground truth density maps.

### 5.2 Datasets

To prove the validity of our approach, we performed experiments on various
vehicle counting datasets, offering different domain shift characteristics.
Specifically, we exploited two existing datasets for traffic density
estimation: WebCamT [260] and TRANCOS [96]. Then, we used two additional
datasets that we created on purpose and made publicly available: the Night and
Day Instance Segmented Park (NDISPark) dataset and the Grand Traffic Auto
(GTA) dataset. Figure 5.2 shows some images belonging to these datasets,
together with the associated labels and the corresponding generated density
maps used for the counting task. In the following, we describe more in detail
each of them.

|  |
---|---|---
|  |
|  |
|  |
(a) | (b) | (c)
?figurename? 5.2: Some examples taken by the four datasets used in this work.
(a) Images; (b) Labels; (c) Density Maps generated from the labels. Each row
correspond to a specific dataset: from top to bottom, the NDISPark and the GTA
datasets introduced in this work, the WebCamT dataset [260] and the TRANCOS
dataset [96]. Note that the densities maps generated in our datasets are
accurate since we start from an instance segmentation annotations. Also notice
that, in the case of the GTA dataset, annotations are automatically generated
without human effort.

####  5.2.1 WebCamT Dataset

The WebCamT dataset is a collection of traffic scenes recorded using city-
cameras introduced by [260]. It is particularly challenging to analyze due to
the low-resolution $(352\times 240)$, high occlusion, and large perspective.
We considered a total of about 40,000 images belonging to 10 different cameras
and consequently having different views. We employed the existing bounding box
annotations of the dataset to generate ground truth density maps. In
particular, we considered one Gaussian Normal kernel for each vehicle present
in the scene, having a value of $\mu$ and $\sigma$ equal to the center and
proportional to the size of the bounding box surrounding the vehicle,
respectively. We used this dataset to test performance with the Camera2Camera
domain shift.

####  5.2.2 TRANCOS Dataset

The TRANCOS dataset is a public dataset containing 1244 dot-annotated images
of different congested traffic scenes captured by surveillance cameras,
introduced by [96]. The approximated ground truth density maps are generated
by putting one Normal Gaussian kernel for each dot present in the scene,
having a value of $\sigma$ empirically decided by the authors. They also
provided the regions of interest (ROIs) for each image. We used this dataset
to test performance with the Camera2Camera domain shift.

####  5.2.3 The Night and Day Instance Segmented Dataset

The Night and Day Instance Segmented Park (NDISPark) dataset was created by us
on purpose and made publicly available. It is a small, manually annotated
dataset for counting cars in parking lots, consisting of about 250 images.
This dataset is challenging and describes most of the problematic situations
we can find in a real scenario: seven different cameras capture the images
under various weather conditions and viewing angles. Another challenging
aspect is the presence of partial occlusion patterns in many scenes, such as
obstacles (trees, lampposts, other cars) and shadowed cars. Furthermore, it is
worth noting that images are taken during the day and the night, showing
utterly different lighting conditions, and that, unlike most counting
datasets, the NDISPark dataset is precisely annotated with instance
segmentation labels, which allowed us to generate accurate ground truth
density maps for the counting task since the sizes of the vehicles were well-
known. Figure 5.3 shows some samples of this dataset, together with the
labels.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

?figurename? 5.3: Samples of the Night and Day Instance Segmented Park
(NDISPark) dataset. We show images together with the associated per-pixel
annotations.

####  5.2.4 The Grand Traffic Auto Dataset

The Grand Traffic Auto (GTA) dataset was also created by us on purpose and
made publicly available. It is a vast collection of about 15,000 synthetic
images of urban traffic scenes collected using the highly photo-realistic
graphical engine of the GTA V - Grand Theft Auto V video game. About half of
them concern urban city areas, while the remaining involve sub-urban areas and
highways. To generate this dataset, we designed a framework that automatically
and precisely annotates the vehicles present in the scene with per-pixel
annotations. To the best of our knowledge, this was the first instance
segmentation synthetic dataset of city traffic scenarios. As in the NDISPark
dataset, the instance segmentation labels allowed us to produce accurate
ground truth density maps for the counting task since the sizes of the
vehicles were well-known. In Figure 5.4, we report some examples of images of
the GTA dataset together with the annotations.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

?figurename? 5.4: Samples of the Grand Traffic Auto (GTA) dataset. We show
images gathered from a video game together with the automatically-generated
per-pixel annotations.

### 5.3 UDA for Traffic Density Estimation

Our method relies on a CNN model trained end-to-end with adversarial learning
in the output space (i.e., the density maps), which contains rich information
such as scene layout and context. The peculiarity of our adversarial learning
scheme is that it forces the predicted density maps in the target domain to
have local similarities with the ones in the source domain.

Figure 5.5 depicts the proposed framework consisting of two modules: i) a CNN
that predicts traffic density maps, from which we estimate the number of
vehicles in the scene, and ii) a discriminator that identifies whether a
density map (received by the density map estimator) was generated from an
image of the source domain or the target domain. In the training phase, the
density map predictor learns to map images to densities based on annotated
data from the source domain. At the same time, it learns to predict realistic
density maps for the target domain by trying to fool the discriminator with an
adversarial loss. The output of the discriminator is a pixel-wise
classification of a low-resolution map, where each pixel corresponds to a
small region in the density map. Consequently, the output space is forced to
be locally similar for both the source and target domains. In the inference
phase, the discriminator is discarded, and only the density map predictor is
used for the target images. We describe each module and how it is trained in
the following subsections.

?figurename? 5.5: High-level architecture of our proposed algorithm. Given
$C\times H\times W$ images from source and target domains, we pass them
through the density map estimation network to obtain output predictions. A
density loss is computed for source predictions based on the ground truth. To
improve target predictions, a discriminator is used to locally classify
whether a density map belongs to the source or target domain. Then, an
adversarial loss is computed on the target prediction and is back-propagated
to the density map estimation and counting network.

####  5.3.1 Density Estimation Network

We formulate the counting task as a density map estimation problem [139]. The
density (intensity) of each pixel in the map depends on its proximity to a
vehicle centroid and the size of the vehicle in the image so that each vehicle
contributes with a total value of 1 to the map. Therefore, it provides
statistical information about the location of the vehicles and allows the
counting to be estimated by summing all density values (see also Section
2.1.1.)

This task is performed by a CNN-based model, whose goal is to automatically
determine the vehicle density map associated with a given input image.
Formally, the density map estimator, $\Psi:\mathbb{R}^{C\times H\times
W}\mapsto\mathbb{R}^{H\times W}$, transforms a $W\times H$ input image
$\mathcal{I}$ with $C$ channels, into a density map,
$D=\Psi(\mathcal{I})\in\mathbb{R}^{H\times W}$.

####  5.3.2 Discriminator Network

The discriminator network, denoted by $\Theta$, also consists of a CNN model.
It takes as input the density map, $D$, estimated by the network $\Psi$. Its
output is a lower resolution probability map where each pixel represents the
probability that the corresponding region (from the input density map) comes
either from the source or the target domain. The goal of the discriminator is
to learn to distinguish between density maps belonging to source or target
domains. Through an adversarial loss, this discriminator will, in turn, force
the density estimator to provide density maps with similar distributions in
both domains. In other words, the target domain density maps have to look
realistic, even though the network $\Psi$ was not trained with an annotated
training set from that domain.

####  5.3.3 Domain Adaptation Learning

The proposed framework is trained based on an alternate optimization of the
density estimation network, $\Psi$, and the discriminator network, $\Theta$.
Regarding the former, the training process relies on two components: i)
density estimation using pairs of images and ground truth density maps, which
we assume are only available in the source domain; and ii) adversarial
training, which aims to make the discriminator fail to distinguish between the
source and target domains. As for the latter, images from both domains are
used to train the discriminator on correctly classifying each pixel of the
probability map as either source or target.

To implement the above training procedure, we use two loss functions: one is
employed in the first step of the algorithm to train network $\Psi$, and the
other is used in the second step to train the discriminator $\Theta$. These
loss functions are detailed below.

###### Network $\Psi$ Training.

We formulate the loss function for $\Psi$ as the sum of two main components:

$\mathcal{L}(\mathcal{I^{\mathcal{S}}},\mathcal{I^{\mathcal{T}}})=\mathcal{L}_{density}(\mathcal{I^{\mathcal{S}}})+\lambda_{adv}\mathcal{L}_{adv}(\mathcal{I^{\mathcal{T}}}),$
(5.1)

where $\mathcal{L}_{density}$ is the loss computed using ground truth
annotations available in the source domain, while $\mathcal{L}_{adv}$ is the
adversarial loss that is responsible for making the distribution of the target
and the source domain closer to each other. In particular, we define the
density loss $\mathcal{L}_{density}$ as the mean square error between the
predicted and ground truth density maps, i.e.
$\mathcal{L}_{density}=MSE(D^{\mathcal{S}},D^{\mathcal{S\\_GT}})$.

To compute the adversarial loss $\mathcal{L}_{adv}$, we first forward the
images belonging to the target domain through network $\Psi$, to generate the
predicted density maps $D^{\mathcal{T}}$. Then, we forward $D^{\mathcal{T}}$
through network $\Theta$, to generate the probability map
$P=\Theta(\Psi(\mathcal{I^{\mathcal{T}}}))\in[0,1]^{H^{\prime}\times
W^{\prime}}$, where $H^{\prime}<H$ and $W^{\prime}<W$. The adversarial loss is
given by

$\mathcal{L}_{adv}(\mathcal{I^{\mathcal{T}}})=-\sum_{h,w}\log(P_{h,w}),$ (5.2)

where the subscript $h,w$ denotes a pixel in $P$. This loss makes the
distribution of $D^{\mathcal{T}}$ closer to $D^{\mathcal{S}}$ by forcing
$\Psi$ to fool the discriminator, through the maximization of the probability
of $D^{\mathcal{T}}$ being locally classified as belonging to the source
domain.

###### Network $\Theta$ Training.

Given an image $\mathcal{I}$ and the corresponding predicted density map $D$,
we feed $D$ as input to the fully-convolutional discriminator $\Theta$ to
obtain the probability map $P$. The discriminator is trained by comparing $P$
with the ground truth label map $Y\in\\{0,1\\}^{H^{\prime}\times W^{\prime}}$
using a pixel-wise binary cross-entropy loss

$\begin{split}\mathcal{L}_{disc}(\mathcal{I})=-\sum_{h,w}(1-Y_{h,w})\log(1-P_{h,w})+\\\
+Y_{h,w}log(P_{h,w}),\end{split}$ (5.3)

where $Y_{h,w}=0\;\;\forall\;h,w$ if $\mathcal{I}$ is taken from the target
domain and $Y_{h,w}=1$ otherwise.

### 5.4 Experimental Evaluation

####  5.4.1 Implementation Details

This section provides some implementation details concerning the two modules
making up our solution, i.e., the density map estimator and the discriminator.

###### Density Map Estimation and Counting Network.

We built our density map estimation network based on the Congested Scene
Recognition Network (CSRNet) [144]. Here we briefly review some of the
features characterizing this algorithm. For a more detailed description see
also Section 2.1.3. CSRNet provides a CNN-based method that can understand
highly congested scenes and perform accurate density estimation and counting.
It is composed of two major components. The authors use the well-known VGG-16
network [203] as the front-end for 2D feature extraction because of its strong
transfer learning ability. On the other hand, the back-end consists of dilated
kernels. The basic concept of dilated convolutions is to deliver larger
reception fields replacing the pooling operations. It is worth noting that the
max pool operation is responsible for losing quality in the density generation
procedure. Since the output size from VGG is reduced by a factor of 8 of the
original input size, we up-sampled the final output to compare it with the
ground truth density map.

###### Discriminator.

We used a Fully Convolutional Network similar to [182] and to [216], composed
of 5 convolution layers with kernel $4\times 4$ and stride of 2. The number of
channels are {64, 128, 256, 512, 1}, respectively. Each convolution layer is
followed by a leaky ReLU having a parameter equals to 0.2.

We implemented the whole system using the PyTorch framework on a single Nvidia
RTX 2080 GPU with 12 GB memory. To train the density estimator network and the
discriminator, we used Adam optimizer [127] with an initial learning rate set
to $10^{-5}$. During the training, it was crucial to balance the weight
between density and adversarial losses. A small value of $\lambda_{adv}$ may
not help the training process significantly. In contrast, a larger value may
propagate incorrect gradients to the density estimator. We empirically chose
the value of $\lambda_{adv}$ depending on the employed dataset.

####  5.4.2 Experiments and Results

We validated the proposed UDA method for traffic density estimation and
counting under different settings. First, we employed the NDISPark dataset,
and we tested the Day2Night domain shift; then, we utilized the WebCamT and
the TRANCOS datasets to take into account the Camera2Camera performance gap.
Finally, we used the GTA dataset to consider the Synthetic2Real domain
difference. For all the experiments, we based the evaluation of the models on
three metrics widely used for the counting task (see also Section 2.1.2 for
more details): (i) Mean Absolute Error (MAE) that measures the absolute count
error of each image; (ii) Mean Squared Error (MSE) that instead quantifies the
squared count error for each image; (iii) Mean Absolute Relative Error (MARE),
which measures the absolute count error divided by the true count. Results are
summarized in Table 5.1, while in the following, we discuss the results
obtained for every considered scenario. Finally, we show some examples of the
outputs obtained using our models, showing their visual quality. In
particular, Figure 5.6 illustrates the ground truth and the predicted density
maps for some random samples of the considered scenarios.

?tablename? 5.1: Experimental results obtained for the four considered domain shift. We employed three evaluation metrics: the Mean Absolute Error (MAE), the Mean Squared Error (MSE) and the Mean Absolute Relative Error (MARE). We achieved performance improvements for all the scenarios, considering all the three metrics.  |  MAE |  MSE |  MARE
---|---|---|---
Day2Night Domain Shift - NDISPark Dataset
Baseline - CSRNet [144] |  3.95 |  27.45 |  0.43
Our Approach |  3.49 |  20.90 |  0.39
Camera2Camera Domain Shift - WebCamT Dataset [260]
Baseline - CSRNet [144] |  3.24 |  16.83 |  0.21
Our Approach |  2.86 |  13.03 |  0.19
Camera2Camera Domain Shift - TRANCOS Dataset [96]
Hydra-CNN [172] |  10.99 |  68.70 |  0.71
FCN-MT [260] |  5.31 |  - |  0.85
LC-ResFCN [133] |  3.32 |  - |  -
Baseline - CSRNet [144] |  3.56 |  30.64 |  0.10
Our Approach |  3.30 |  23.60 |  0.08
Synthetic2Real Domain Shift - GTA Dataset
Baseline - CSRNet [144] |  4.10 |  25.83 |  0.28
Our Approach |  3.88 |  23.80 |  0.27
|  |  |
---|---|---|---
GT count: 56 | GT count: 13 | GT count: 35 | GT count: 12
|  |  |
Pred count: 53 | Pred count: 14 | Pred count: 38 | Pred count: 11
|  |  |
(a) | (b) | (c) | (d)
?figurename? 5.6: Examples of the predicted density maps in the considered
scenarios. (a) Day2Nigh Domain Shift using the NDISPark dataset; (b) and (c)
Camera2Camera Domain Shift employing the WebCamT and TRANCOS datasets,
respectively; (d) Synthetic2Real Domain Shift using the GTA dataset for the
training phase and the WebCamT dataset for testing on real images. First row:
input images; second row: ground truth; third row: predicted density maps.

###### Day2Night Domain Shift

In this scenario, we split the NDISPark dataset into train, validation, and
test subsets containing about 100, 50, and 100 images, respectively. The
former had only pictures taken during the day (source domain), while the
validation and the test subsets contained night images (target domain). To
fairly evaluate our method, we first considered the baseline model without the
domain adaptation module (i.e., putting the $\lambda_{adv}$ value to zero).
Then, we added the adversarial module to compare the results. In both cases,
we trained the network for 300 epochs, validating at each iteration. We chose
the best validation model in terms of MAE, and we tested it against the test
set. As shown in Table 5.1, using our solution, we obtained performance
improvements considering all three metrics.

###### Camera2Camera Domain Shift

In this case, we performed two sets of experiments to test the domain shift
that takes place when we consider a camera different from the ones used in the
training phase.

First, we considered the WebCamT dataset, and we split it into train,
validation, and test subsets. In the former, we accounted for about 25,000
images belonging to 7 cameras (source domain). In the last two, we considered
the remaining 15,000 pictures of 3 different cameras, having diverse contexts
and slightly different angles of view (target domain). We compared the
baseline and our solution when training for 20 epochs, validating it at each
iteration and choosing the best model in terms of MAE.

Second, we took into account the TRANCOS dataset. We split it into train,
validation, and test sets, following [96]. The train set represented the
source domain, while the other two belonged to the target domain and were
collected in different contexts. We trained our domain adaptation for 200
epochs, picking the best validation model in terms of MAE, and we evaluated it
against the test set. We compared the obtained results with the ones claimed
by [144] using only the state-of-the-art CSRNet algorithm (i.e., our baseline)
and with other state-of-the-art techniques present in the literature.

As shown in Table 5.1, we obtained performance improvements in both cases,
taking into account all three metrics. Considering the publicly available
TRANCOS dataset, we achieved superior results not only concerning the baseline
but also compared to the other considered approaches.

###### Synthetic2Real Domain Shift

In this scenario, we trained the algorithm using synthetic images. Then we
tested it on real-world data. In particular, we considered a subset of the GTA
dataset containing about 5,000 images of city traffic scenarios, and we used
it as the training set (source domain). On the other hand, we accounted for
the test and the validation subsets of the WebCamT dataset as the target
domain. We compared the results obtained using the baseline model and our
solution with the domain adaptation module. In both cases, we trained the
algorithm for 20 epochs, validating at each iteration. We chose the best model
in terms of MAE.

Again, as shown in Table 5.1, we achieved better results compared to the basic
model. We believe that this scenario is particularly interesting because we
obtained comparable results with the previous one, but this time without using
manual annotations neither in the source domain nor in the target one.

### 5.5 Summary

In this chapter, we tackled the problem of determining the density and the
number of objects present in large sets of images. The proposed methodology,
built on a CNN-based density estimator, was able to generalize to new data
sources for which there were no annotations available. We achieved this
generalization by exploiting an Unsupervised Domain Adaptation (UDA) strategy,
whereby a discriminator attached to the output forced similar density
distribution in the target and source domains. Experiments showed a
significant improvement relative to the performance of the model without
domain adaptation. To the best of our knowledge, we were the first to
introduce a UDA scheme for counting to reduce the gap between the source and
the target domain without using additional labels.

Another contribution is represented by the creation of two new per-pixel
annotated datasets made available to the scientific community. One of the two
novel datasets was a synthetic dataset created from a photo-realistic video
game. Here the labels were automatically assigned while interacting with the
API of the graphical engine. Using this synthetic dataset, we demonstrated
that it is possible to train a model with data gathered from virtual worlds
and to perform UDA toward a real-world scenario, obtaining outstanding
performance without using additional manual annotations. In other words, we
addressed the problem of data scarcity from two complementary sides: on the
one hand, we exploited the significant variability of the synthetic data,
while, on the other hand, we mitigated the domain gap existing between the
synthetic and the real-world images in an unsupervised fashion, thus without
using the labels of the test domain as in the previous chapter.

##  ?chaptername? 6 Counting Biological Structures with Raters’ Uncertainty

Detection and counting of biological structures are among the earliest fields
revolutionized by artificial neural networks now dominating state of the art.
Several vision models (mostly convolutional networks) have been successfully
adopted to localize, segment, and count cells or other structures from
microscopy images and even provide counting-density estimation particularly
effective in “crowded” scenarios. However, as already seen in the previous
chapters concerning other tasks and applications, the success of these methods
assumes the availability of a large-scale representative set of well-labeled
images. In this chapter, compared to the previous ones, we consider the
problem of data scarcity in a different setting. Whereas, in most cases,
objects to be counted can be unambiguously flagged by human raters, here we
investigate cell detection and counting under the assumption of weak multi-
rater labels, that is, in the presence of non-negligible disagreement between
multiple raters. This often occurs when trying to detect and count cells with
non-trivial patterns on a large scale, where several factors can produce weak
labels; raters can incur errors due to fatigue or inexperience (common when
hiring less-experienced raters to reduce labeling time) or have different
judgments that can span from conservative to liberal when assigning labels.

More reliable labels can be obtained by naively averaging the decisions taken
by several raters on the same data, i.e., multi-rating can be leveraged to
create stronger singular annotations. However, such data are expensive to
obtain and often available only in small quantities. On the other hand, given
the scale of training sets needed for deep learning methodologies and the
counting task, we consider here the case in which few expert raters, on a
limited labeling budget, tend to label new data rather than label the same
images more than once. This results in large, single-rater weakly labeled
datasets and only small multi-labeled subsets [37].

In this setting, we proposed a two-stage counting methodology for biological
structures in microscopy images, where each stage is devised to fully exploit
the annotations in each data subset. The first stage adopts existing solutions
on weakly-labeled data to detect and count cells. Specifically, we compared
three common CNN-based methodologies already present in the literature — a)
segment and count, b) detect and count, and c) count by density estimation.
The goal was to investigate their counting ability when trained with data
characterized by significant label noise from errors introduced by raters, and
to derive uncalibrated scores from the models’ output that have not been
designed to correlate with the quality of the predictions. In the second
stage, using a small set of multi-rater data, we defined a rescoring model
that refines predictions of the first stage, increasing the correlation
between the scores assigned by the model to the predictions and the raters’
agreement on the sample labels. We referred to scores produced in this stage
as calibrated scores, in contrast with the uncalibrated ones previously
assigned; these final scores can eventually be used to filter low-quality
predictions by practitioners. Advantages in operating in two-stage are
twofold: i) the localization of objects are decoupled from their scoring, thus
obtaining an overall improved counting model when the latter is fine-tuned
even on a few multiple raters’ judgments, and, ii) we can easily swap the
first stage with any state-of-the-art localization and counting method, making
the pipeline model-agnostic and “future proof” - any subsequent work can
simply plug-in the best detector and still use the proposed pipeline when
multi-rater data is available.

We evaluated the various stages of our pipeline on a novel weakly-labeled dot-
annotated dataset that we publicly released. It consists of a collection of
fluorescence microscopy images of mice brain slices containing Perineuronal
Nets (PNNs), extracellular matrix aggregates surrounding the cell body of a
large number of neurons throughout the nervous system. Multiple expert raters
have labeled a small part of the dataset; nonetheless, the maximum agreement
between raters is roughly 70%, highlighting the need for an automated counting
technique that accounts for uncertain patterns. We showed through experimental
evaluation that our proposed two-stage pipeline, independently from the
specific implementation of each stage, improve the performance of several
state-of-the-art counting methods on multiple ground-truth settings, from
liberal to conservative ones.

The main contributions presented in this chapter can be summarized as follows:

  * •

the proposal of a two-stage pipeline that improves biological structures
counting in weak-labels settings,

  * •

the introduction of a novel dot-annotated dataset for cell counting in
microscopy images (specifically, perineuronal nets) comprised of a large
weakly-labeled single-rater subset and a smaller multi-rater subset, and

  * •

the public release of the first pre-trained models for automatic perineuronal
nets counting in fluorescence images (at the best of our knowledge).

We organize the rest of the chapter as follows. We review related work to our
in Section 6.1. In Section 6.2, we describe the datasets used in our
experiments. Section 6.3 formalizes the proposed methodologies, while Section
6.4 outlines the performed experiments showing the obtained results.

The research presented in this chapter was published in [55, 52]. Code and
trained models are publicly available at
https://github.com/ciampluca/counting_perineuronal_nets.

### 6.1 Related Works

This section overviews some works related to ours concerning counting
approaches tailored to estimating the number of cells or structures in
microscopy images. We also review some works focusing on learning with multi-
rater data.

###### Microscope Cell Counting.

Counting biological structures like cells in microscopy images is a crucial
step to diagnose many diseases [226] and to understand cellular and molecular
mechanisms [208]. Several automatic cell counting methods have been proposed
over the years to facilitate this tedious and challenging task. Compared to a
typical counting task, microscopy images present different challenges, such as
low image contrast, significant cell shape and count variance, and
superposition of cells, leading to occlusions. As such, both detection-based
and regression-based methods have been proposed, each with the advantages and
the drawbacks already highlighted in Section 2.1. Basically, regression
techniques have shown superior performance compared to the detection-based
ones in crowded scenarios where the instances of the objects are sometimes not
clearly visible due to occlusions and clumps; however, they cannot precisely
localize the objects present in the scene, eventually providing only a coarse
position of the area in which they are distributed. A relevant example
belonging to the detection-based category is [16], where the authors
introduced a tree-structured discrete graphical model exploited to select and
label a set of non-overlapping regions in the image by a global optimization
of a classification score. More recently, the authors of [175] exploited the
popular Mask R-CNN [101] instance segmentation framework to detect overlapping
cells, whereas [69] used a CNN to segment biological structures from 3D
medical images. A comprehensive survey about deep learning algorithms used in
medical image analysis, including cell detection and segmentation in
microscopy images, is given by [149].

Recent efforts also focused on regression-based approaches that cope better
with overlapped objects and crowded scenarios. For example, [97] proposed SAU-
Net, an extension of the U-Net segmentation network [191] with a Self-
Attention module for counting by density regression. In [7], another
regression-based counting model is introduced, enhanced by regulating
activation maps from the final convolution layer of the network with coarse
ground-truth activation maps generated from simple dot annotations. More, in
[57], the authors proposed a novel deep neural network architecture adapted
from the Inception family [209] of networks called Count-ception. In [112],
the so-called CSRNet [144], a regression-based CNN suitable for counting
objects in several contexts (already employed in Chapter 5 for traffic density
estimation), is employed to estimate cell densities in immunohistochemically
stained sections of breast tissue. Jiang and Yu proposed two different
regression-based cell counting approaches [120, 118], again, based on the
estimation of density maps. Finally, authors in [102] presented another
regression model based on density estimation where auxiliary convolutional
neural networks are employed to assist in the training of intermediate layers.
Other regression-based strategies have also been devised to deal with densely
concentrated cells but still generating individual cell detections, such as
[74], [214], [129] and [246]. These approaches first generate intermediate
maps that indicate the likelihood of each pixel being the center of a cell in
the image, and then convert them into detections by applying some form of Non-
Maximum Suppression (NMS).

Concerning the automatic counting of PNNs, previous solutions are often based
on brittle hand-crafted computer vision pipelines, such as in [207]. To the
best of our knowledge, we are the first to use deep-learning solutions to
address the counting of perineuronal nets and its specific challenges, such as
the extreme inter-image variance of the number and the non-trivial appearance
of PNNs that cause difficulty to precisely count them, even for human experts.

###### Learning with multi-rater data

When dealing with multi-rater data, most existing methodologies apply simple
strategies like majority voting to obtain a unique set of ground-truth labels.
However, approaches exploiting multi-rater data more effectively exist and are
not new; in their seminal work, [62] proposed an Expectation-Maximization
algorithm to estimate raters’ error-rates in multinomial multi-rater data.
More recent works aim at modeling raters’ reliability for aggregating or
filtering labels, such as [189] and [258]. We refer the reader to [272] for a
review of approaches and challenges in inference with multi-rater data. The
recent trend is instead increasingly exploiting multi-rater data, when
possible, to increase data efficiency; in the biomedical context, Wei et al.
[238] proposed a curriculum learning approach on samples with increasing
raters’ agreement for histopathology image classification. In contrast,
Mirikharaji et al. [165] tackles skin-lesion segmentation by building multiple
models (one for each set of raters’ labels) and then aggregating models
predictions. To the best of our knowledge, the only proposed counting approach
dealing with multi-rater data is [17], where authors train a supervised
algorithm to count antarctic penguins in images dot-annotated by non-
professional volunteers; multi-rater labels are mainly exploited to estimate
the object scale, which varies wildly in their dataset (the diameter of a
penguin varies between 15 and 700 pixels) and is instead fixed in our
scenario. When dealing with constant scale objects, as in our microscopy
images scenario, their solution resembles [74], a segmentation-based approach
adopted and compared in this work. Moreover, instead of requiring large multi-
rater training sets, our approach is designed to train on a large single-rater
set plus a small multi-rater set, lowering the total labeling cost.

### 6.2 Datasets

In this section, we describe the employed datasets, summarized in Table 6.1.
We considered four publicly available single-rater datasets widely used in the
context of the microscope cell counting task that we exploit for comparing the
adopted counting architectures against the state of the art. Those have served
as baselines for our counting framework. Then, we illustrate our novel
collection of fluorescence microscopy images containing perineuronal nets
labeled by multiple professional raters, which we used for the experimental
evaluation of our two-stage counting pipeline.

?tablename? 6.1: Summary of datasets. We report some numerical characteristics on the top of the table. Below, we show a dataset image sample (for PNN, we show a 640x640 crop) and, in the last three rows, the associated targets exploited during training. Specifically, the targets are generated from dot annotations using different procedures: i) bounding boxes are produced by generating squares with side s, ii) density maps are built by superimposing Gaussian kernels $G_{\sigma}$, and iii) segmentation maps are generated drawing discs with radius r separated by background ridges. Bounding boxes, Gaussian kernels and discs are centered in the dot-annotated locations; the s, $\sigma$, and r parameters are fixed and dataset-specific, depending on the typical object size in the images. Targets in the multi-class BCData dataset are shown in false colors. |  |  |  |  | PNN
---|---|---|---|---|---
|  VGG [139] |  MBM [122] |  ADI [57] |  BCData [112] |  1 rater (PNN-SR) |  7 raters (PNN-MR)
subjects |  none (synthetic) |  8 |  N/A |  394 |  1 |  1
images |  200 |  44 |  200 |  1,338 |  25 |  12
size |  256$\times$256 |  600$\times$600 |  150$\times$150 |  640$\times$640 |  $\geq$8184$\times$6163 |  2000$\times$2000
objects |  35,192 |  5,553 |  29,684 |  181,074 |  34,620 |  2,351
obj./img. |  176$\pm$61 |  126$\pm$33 |  148$\pm$32 |  135$\pm$68 |  1,385$\pm$590 |  196$\pm$43
Sample |
Bounding Boxes |
Density Map |
Segmentation Map |

####  6.2.1 VGG Cells Dataset

This public dot-annotated dataset was introduced by [139]. It contains 200 RGB
synthetic images simulating bacterial cells from fluorescence-light microscopy
at various focal distances. Images have a fixed size of 256×256×3 pixels, and
the cells are designed to be clustered and occluded with each other.

####  6.2.2 MBM Cells Dataset

The Modified Bone Marrow (MBM) dataset contains 44 RGB dot-annotated
microscopy images of human bone marrow with various cell types stained blue.
The original dataset was collected by [122], acquiring 11 microscopy images
from the human bone marrow tissues of 8 different patients. The original
images are 1200×1200×3 pixels in size, but authors in [57] split each of them
into four images with the size of 600×600 pixels.

####  6.2.3 ADI Cells Dataset

The Adipocyte (ADI) dataset is a human subcutaneous adipose tissue dot-
annotated collection of microscopy images introduced by [57]. It consists of
200 Regions Of Interest (ROI) of 150×150×3 pixels in size sampled from high-
resolution histology slides representing adipocyte cells. The average cell
count across all images is $165\pm 44.2$, and the size of the biological
structures can vary dramatically, representing a challenging test case for
automated cell counting procedures.

####  6.2.4 BCData

The Breast tumor Cell Dataset (BCData) [112] is a recent collection of 1,338
images based on Ki-67 staining with 181,074 dot-annotated cells divided into
two classes (positive and negative tumor cells, i.e., malignant and not
malignant, respectively). Unlike other datasets, BCData is not only large in
scale concerning the labeled objects but also considering the number of
different unique patient cases (that are 394). The size of each image is fixed
to 640×640×3 pixels, and the authors divided the dataset into training,
validation, and testing split at a ratio of approximately 6:1:3 (803, 133, and
402 images, respectively).

####  6.2.5 PNN Dataset

Perineuronal Nets (PNNs) are extracellular matrix aggregates surrounding the
cell body of many neurons throughout the nervous system; their alterations are
associated with several physiological processes and pathological conditions,
e.g., psychiatric disorders such as schizophrenia [24, 163]. This contributed
to the increasing interest in PNNs research spanning various conditions and
animal models, including rodents [222, 29, 75], primates [167], and even human
brain samples [190]. We collected and publicly released [52] a novel dataset
of fluorescence microscopy images of mice brain slices containing annotations
for perineuronal nets 111The experiments were carried out in accordance with
the directives of the European Community Council (2011/63/EU) and approved by
the Italian Ministry of Health (Authorization # 621/2020-PR).. Specifically,
we obtained $50\mu m$ brain slices from C57BL6/J adult mice (transcardially
perfused with $4\%$ paraformaldehyde). PNNs were stained with a green
fluorescent marker by sequentially incubating them with biotinylated Wisteria
floribunda Lectin (WFA) and streptavidin Alexa Fluor™ 488 conjugate. We
acquired images with a fluorescence microscope (Zeiss Apotome.2). PNNs were
manually annotated by neuroscientists and biologists from the laboratory of
Prof. Pizzorusso, a leading expert in the field of the PNNs since 2002 [178].

The dataset is composed of two subsets — a large single-rater subset (PNN-SR)
and a smaller multi-rater subset (PNN-MR) — described below and depicted in
Figure 6.1.

1) PNN-SR: consists of 25 images having different sizes ranging from
$8184\times 6163$ to $15120\times 9477$ pixels. The extreme size of the images
makes their use impracticable by AI-based Computer Vision tools unless
dividing them into smaller regions. Among all the images, there are roughly
34k annotated PNNs, varying from a few dozens to some thousand per image,
depending on the considered portion of the brain. An expert manually created
annotations by putting a dot over the centroid of each identified PNN. Since
PNNs are often not easy to find and are subject to different judgments
depending on the rater, the training labels are sure to contain errors. Thus,
this subset can be considered weakly-annotated.

2) PNN-MS: comprises 12 microscopic images of $2000\times 2000$ pixels
representing different portions of a mouse brain, with a total of 2,532 dot-
annotated PNNs. The main peculiarity of this subset is that the annotation
procedure has been performed by seven different raters, showing a remarkable
discrepancy between the various judgments. As shown in Figure 6.1 (bottom-
right), more than $40\%$ of the PNN has not been annotated by the majority of
raters (3 or less of the 7 raters), expressing the difficulty of achieving
error-free assessments by a single rater.

?figurename? 6.1: PNN Dataset. Top Left. A sample from the single-rater subset
(PNN-SR) with dot annotations in red. Top Right. A sample from the multi-rater
subset (PNN-MR, labeled by 7 raters); the color of the circles encodes the
number of raters’ that identified that PNN following the legend on the below.
Bottom Left. Jaccard Index between the PNN sets found by each rater. Bottom
Right. Breakdown of the PNN-MR subset by raters’ agreement level. We show some
sample patches centered in the locations identifying PNNs, together with the
mean patches; percentages represent the fraction of the total number of PNNs
localized by $i$ raters, for $i=1,\dots,7$.

### 6.3 Methodology

Most counting approaches, both regression- or detection-based, can obtain good
detections of the objects and a good prediction of the total count when using
well-labeled training sets, as already demonstrated by cell counting
literature. However, under the presence of weak labels, these models tend to
detect also low-confidence or spurious patterns with high confidence for
multiple reasons; for example, regression-based models such as density-map
estimators do not model the confidence of a detected pattern, thus disabling
any filtering step, and detection-based approaches often assign overestimated
detection scores to maximize recall that does not correlate with the
“objectness” of the pattern. Although it is feasible to modify current models
to better express this correlation, training them would necessitate large
multi-rater datasets (where each pattern is labeled with a degree of
objectness or quality) that are expensive to obtain.

Here, we assume to have access to a large weakly-labeled single-rater dataset
and only a small multi-rater subset. With the former, we exploit the power of
existing counting solutions, and with the latter, we devise an additional
rescoring stage to cope with the problems discussed above. Specifically, we
model the counting task as a process (depicted in Figure 6.2) comprised of two
stages, each having its separate training phase. The first stage follows
standard approaches producing a set of coordinates localizing objects in the
input image. In the second stage, we consider the objects previously
localized, and we assign them an “objectness” score that correlates with the
raters’ agreement on their detection, i.e., a higher score indicates a higher
probability that most or all human raters detect that object. To do so, we
define a scorer module that inputs a small cropped patch containing the
previously localized objects and outputs a scalar score. We train it in a
supervised fashion with a small set of multi-rater data, where the agreement
between multiple raters reflects the pattern’s certainty. In practice, the
output of the scorer model provides a new “objectness” score that
practitioners can use to exclude or include samples from the total count.

We describe the two stages in more detail below.

?figurename? 6.2: Proposed counting pipeline. We model the task as a two-stage
process. In the first one, we detect the objects exploiting a localization
model $f_{\theta}$, previously trained on a large collection of dot-annotated
images that may have weak labels. In the second stage, we employ a scorer
model $g_{\phi}$ that assigns to the objects localized in the previous step an
“objectness” score, which we correlate with the pattern uncertainty quantified
by the agreement’s level.

####  6.3.1 Localization Stage

For this stage, we assume to have a collection of $N$ images with dot
annotations $\mathcal{X}=\\{(I_{1},\hat{L_{1}}),\dots,(I_{N},\hat{L_{N}})\\}$,
where $I_{i}$ is the $i$-th image and $\hat{L_{i}}$ is the set of coordinates
of the structures to be counted in image $I_{i}$ labeled by a human rater. We
assume $\mathcal{X}$ is large and may have weak labels, e.g., it may contain
spurious (false positives) and missing annotations (false negatives).

A localization model $f_{\theta}$ applied to the input image $I$ produces a
set of coordinates $L=\\{p_{1},\dots,p_{C}\ |\ p_{i}\in\mathbb{R}^{2}\\}$
localizing the objects to be counted. This model is trained using location
data $\mathcal{X}$ and can be implemented following several different
strategies; here, we test three successful approaches from the literature,
that are segmentation, detection, and density estimation, described below.

###### Localization by Segmentation

For this approach, we follow [74], i.e., we first produce a segmentation map
$S=f_{\theta}(I)\in[0,1]^{H\times W}$ for the input image $I$ having height
$H$ and width $W$. $S$ is then thresholded and further processed to extract
connected components. The centroids of those components form the output
localizations $L$. This solution can accommodate variable-shaped objects, but
segmentation annotations are usually very expensive to produce. Here, we
generate the target segmentation maps $\hat{S}\in[0,1]^{H\times W}$ by
imposing a disc centered in the dot-annotated position. The radius of the disc
is fixed and depends on the typical object size in the dataset. A narrow ridge
separates overlapping discs. In case of multiple object classes, the network
outputs one segmentation map per class, and target generation is performed
independently for each class. An example of a target segmentation map is
reported in Table 6.1. The model is trained to minimize the weighted binary
cross-entropy between pixels of the output and target maps; more weight is
assigned to more important pixels of the map, such as background ones near
foreground objects. More details of the generation procedure of segmentation
targets are available in [74]. As in [74], we implement $f_{\theta}$ as a
standard U-Net [191]. In the following, we will refer to this method as
S-UNet.

###### Localization by Detection

For this approach, we employ the Faster-RCNN model for visual object detection
[187] (see Section 2.2.2 for further details); $f_{\theta}$ produces a list of
bounding boxes following the standard two-stage detection paradigm. In the
first step, a Region Proposal Network (RPN) generates the region proposals
that might contain objects, slicing pre-defined region boxes (called anchors);
in the second step, these priors are refined, performing a regression to the
coordinates of bounding boxes precisely localizing the objects inside these
Regions of Interest (RoIs). The centers of the boxes comprise the final
localization of the entities. Targets are produced by generating squared
bounding boxes centered in the dot-annotated data with a fixed side, again,
depending on the typical object size in the dataset. We implement $f_{\theta}$
as a Faster-RCNN network with a Feature Pyramid Network module and a ResNet-50
backbone. From now on, we will refer to this method as FRCNN.

###### Localization by Density Estimation

We also tested density-estimation approaches known for delivering excellent
counting performances, especially in “crowded” scenarios. Using this approach,
we learn a regression model producing a density map
$D=f_{\theta}(I)\in\mathbb{R}^{H\times W}$ from an input image of height $H$
and width $W$. Each pixel of $D$ corresponds to the quantity of the objects
present at that precise point. Thus, the notion of density map loosely
corresponds to the physical/mathematical notion of density; the number of
objects $n$ in an image sub-region $P\subseteq I$ is estimated by integrating
$D$ over $P$, i.e., summing up pixel values in the considered region,
$n=\sum_{p\in P}D_{p}$. We refer the reader to Section 2.1.1 for further
details. Although these approaches are not intended to localize objects, a
coarse localization can be obtained by analyzing the estimated density map, in
particular by finding the top-$n$ maximum local peaks of it, as already done
in [245]. During training, the target density maps are produced by
superimposing Gaussian kernels $G_{\sigma}$ centered in the dot-annotated
locations; the spread parameter $\sigma$ is fixed and depends on the typical
object size in the dataset. In case of multiple object classes, the network
outputs one density map per class, and target generation is performed
independently for each class. An example of a target density map is reported
in Table 6.1. The model is trained by minimizing the mean squared error loss
between target and output density maps. We implement $f_{\theta}$ exploiting
the Congested Scene Recognition Network (CSRNet) [144], a CNN for density
estimation comprised of a modified VGG-16 network [203] for feature extraction
and a series of dilated convolutional layers [253] to extract deeper
information of saliency and, at the same time, maintaining the output
resolution (see Section 2.1.3 for further details). We will refer to this
method as D-CSRNet.

Once objects are localized, the “objectness” of each prediction needs to be
quantified to permit filtering of false positives or negatives that inevitably
leak from labeling errors. We cope with this task in the subsequent separate
stage, but for comparison, we also consider deriving objectness scores from
the three accounted localization models alone as a baseline. Among the
considered models, FRCNN is the only one that natively outputs a score
$\in[0,1]$ stating the probability of containing an object inside the bounding
box that we can use as objectness score. On the other hand, S-UNet and
D-CSRNet do not provide directly a score that can be used for filtering
predictions. For S-UNet, we derive a score $\in[0,1]$ associated with each
localized object by considering the maximum values of the connected components
found in the predicted segmentation map. Regarding the D-CSRNet model, we
instead infer a score by taking the local maximum peak values of the predicted
density map. However, none of these scores are defined to correlate with the
pattern uncertainty that instead needs to be explicitly modeled. We do that in
the second stage of our pipeline.

####  6.3.2 Scoring Stage

The goal of this stage is to define a model that scores the certainty of a
pattern; higher scores should represent objects localized by most human
raters, while lower scores should indicate dubious patterns.

Given the coordinates $p$ of an object in image $I$ localized with one of the
approaches in the previous stage, we define a scorer model $g_{\phi}$ that
assigns to the object a scalar objectness score $s=g_{\phi}(o)$, with $o$ the
squared sub-patch of the image $I$ centered in $p$ containing the object. To
train $g_{\phi}$, we assume to have a small set of images where objects have
been labeled by $K$ different raters; this produces a training set
$\mathcal{X}^{\prime}=\\{(o_{1},a_{1}),\dots,(o_{M},a_{M})\\}$, where
$o_{i}\in\mathbb{R}^{l\times l}$ is the image sub-region containing the $i$-th
localized object, and $a_{i}\in\\{0,\dots,K\\}$ is the raters’ agreement,
i.e., the number of raters who localized that object. Regions containing no
localized objects ($a=0$) are used as negative samples during training.

Although this rescoring stage is novel in counting pipelines, we can formulate
it as well-known problems and implement it following existing solutions. Below
we propose several methodologies that can be adopted for training the
$g_{\phi}$ model. It is worth noting that the $s$ score takes on different
values depending on the adopted method.

###### Agreement Regression (AR)

A simple baseline is directly regressing scores from the input patches. In
this formulation, $g_{\phi}$ produces a scalar output and is trained to
directly regress the normalized raters’ agreement $\nicefrac{{a}}{{K}}$ from
the object patch. Specifically, we minimize

$\mathcal{L}(\mathcal{X}^{\prime};\phi)=\frac{1}{2}\sum_{(o,a)\in\mathcal{X}^{\prime}}\left(\frac{a}{K}-g_{\phi}(o)\right)^{2}\,,$
(6.1)

where $o$ is a squared image patch containing a localized object and
$\nicefrac{{a}}{{K}}$ is the fraction of raters localizing that object.

###### Agreement Classification (AC)

Another simple baseline comprises classifying the input patches in agreement
levels. In this formulation, we consider the $K+1$ agreement values
$a\in\\{0,\dots,K\\}$ (including the $0$ value as background samples) as
separate classes into which objects can be classified. The model $g_{\phi}$
produces a $(K+1)$-way softmax output that is trained with standard cross-
entropy loss

$\mathcal{L}(\mathcal{X}^{\prime};\phi)=-\sum_{(o,a)\in\mathcal{X}^{\prime}}\log(g^{a}_{\phi}(o))\,,$
(6.2)

where $g^{i}_{\phi}(o)$ indicates the $i$-th output of the model. The final
scalar score $s$ is obtained as the (normalized) expected value of the class
over the output categorical distribution

$s(o)=\frac{1}{K}\sum_{i=0}^{K}i\cdot g^{i}_{\phi}(o)\,.$ (6.3)

###### Agreement Ordinal Regression (OR)

We formulate the scoring problem as an ordinal regression problem with $K+1$
ordered categories from the lowest to the highest agreement. Similarly to
agreement regression, $g_{\phi}$ produces a scalar output but is trained
following [176]. Along with model parameters, a set of $K$ ordered thresholds
$\Theta=\\{\theta_{i}\\}_{\scriptscriptstyle i=0}^{\scriptscriptstyle
K-1},\theta_{0}<\theta_{1}<\dots<\theta_{K-1}$ are defined as learnable
parameters. Given the model scalar output $s=g_{\phi}(o)$, we model

$P(a\leq k|o)=\sigma(\theta_{k}-s)\quad k=0,\dots,K-1\,,$ (6.4)

where $\sigma$ is the sigmoid function, and thus

$\displaystyle y_{k}(o)$ $\displaystyle=P(a=k|o)$ (6.5) $\displaystyle=P(a\leq
k|o)-P(a\leq k-1|o)$ $\displaystyle=\begin{cases}\sigma(\theta_{0}-s)&\text{if
}k=0\,,\\\ \sigma(\theta_{k}-s)-\sigma(\theta_{k-1}-s)&\text{if }k=1\dots
K-1\,\text{, and}\\\ 1-\sigma(\theta_{K-1}-s)&\text{if }k=K\,.\end{cases}$

The models parameters $\phi$ and thresholds $\Theta$ are optimized by
minimizing the negative log likelihood of observed samples

$\mathcal{L}(\mathcal{X}^{\prime};\phi,\Theta)=-\sum_{(o,a)\in\mathcal{X}^{\prime}}\log(y_{a}(o))\,.$
(6.6)

The values of $\theta_{i}$ are optionally clipped after each update to kept
them ordered. Once trained, we discard $\Theta$ and adopt only $g_{\phi}$ to
output the score $s$ for an object.

###### Agreement Rank Learning (RL)

Here, we model agreement by learning to rank a tuple of samples with
increasing agreement values. Our formulation instantiates a standard pairwise
learning to rank approach [35] with a custom sample loss definition.
Specifically, we still define a model with a scalar output $s=g_{\phi}(o)$,
but we employ a different training scheme; given a $(K+1)$-tuple of ordered
samples $O=(o_{0},\dots,o_{K})$ containing one sample per agreement class
(i.e., $o_{i}$ has an agreement value $a_{i}=i$), we ask our model to produce
scores $s_{i}=g_{\phi}(o_{i})$ that are sorted $s_{0}<s_{1}<\dots<s_{K}$.
Translating this constraint in a loss function for the single tuple, we obtain
a class-balanced pairwise margin loss

$\mathcal{L}(o_{0},\dots,o_{K};\phi)=\frac{1}{K}\sum_{i=1}^{K}\max(m-g_{\phi}(o_{i})+g_{\phi}(o_{i-1}),0)\,,$
(6.7)

where $m$ is a margin hyper-parameter empirically set to $0.1$. A dataset of
tuples is obtained by repeatedly drawing $K+1$ random samples, one for each
agreement class, from the training set $\mathcal{X}^{\prime}$. This has the
advantage to produce large training datasets even when dealing with a small
initial multi-rater dataset. The batch loss is obtained as the mean loss over
a batch of tuples.

### 6.4 Experiments and Results

In this section, we describe the experiments performed to validate our
approach and discuss the obtained results. We divided them into three parts.
First, we evaluated the considered counting architectures, i.e., the
segmentation-based S-UNet, the detection-based FRCNN, and the density-based
D-CSRNet approaches, against standard single-rater cell counting benchmarks.
The aim was to demonstrate that they work plausibly fine, i.e., they produced
comparable results against the state-of-the-art, warding off that results
provided by our counting pipeline are not due to a weak poorly-trained
baseline. Then, we evaluated the first stage of our pipeline, i.e., the
localization stage. Specifically, we performed experiments on our novel PNN
dataset, training the three adopted counting architectures with single-rater
data having significant label noise from errors introduced by raters. The goal
was to detect and count perineuronal nets under this weakly labeled setting,
deriving uncalibrated scores from the models’ output that have not been
designed to correlate with the quality of the predictions. Finally, we
performed experiments with our multi-rater PNN-MR subset to validate our
proposed second stage, i.e., the score calibration stage. Here, we refined
predictions of the previous stage, producing calibrated scores that increase
the correlation with the raters’ agreement. We compared it to several
baselines, showing that it improves counting performances when dealing with
uncertain patterns.

####  6.4.1 Evaluation of the adopted counting architectures

We evaluated the three adopted counting approaches against the state of the
art using VGG Cells, MBM Cells, ADI Cells, and BCData counting benchmarks
described in Section 6.2. All these collections of images are single-rater,
i.e., the final available labels belong to a single rater. Even when multiple
raters have been employed during the annotation procedure, the final
annotations are squashed into a single label per object. In other words,
multi-rater annotations are not leveraged if not for the creation of stronger
annotations at the expense of the dataset scale.

We followed the evaluation protocol introduced by [139] and adopted by most
subsequent works; we considered a testing subset fixed for all the experiments
(100 images for VGG Cells and ADI Cells, and 10 images for MBM Cells) and
training and validation subsets of varying size ($N$ images for each subset)
to simulate lower or higher numbers of labeled examples. Following previous
work, we set $N$ to 16, 32, and 50 for VGG Cells, to 10, 25, and 50 for ADI
Cells, and to 5, 10, 15 for MBM Cells. Concerning BCData, we instead used the
training, validation and testing splits provided by [112]. As performance
metrics, we adopted the widely used Mean Absolute Error (MAE) already defined
in Section 2.1.2. For VGG Cells, MBM Cells, and ADI Cells, we repeated the
experiment 10 times, randomly sampling different splits for each
configuration, and we reported the mean and standard deviation of the
evaluation metric. To check the consistency of the results on these random
splits, we also re-implemented the original FCRN-A method presented in [245],
thus performing an exact head-to-head comparison with the same samples being
used for training and testing. Concerning BCData, we reported the mean and the
standard deviation of the MAE calculated between 10 runs over the 402 images
comprising the test split, changing the random initialization seed each time.

Table 6.2 reports results on the four datasets. The density-based solution
D-CSRNet performed best among tested solutions on the VGG dataset and
comparably to state of the art. The other two adopted methods, i.e., the
segmentation-based S-UNet and the detection-based FRCNN, exhibited slightly
larger errors, consistently with their inherent limitations when applied to
“crowded” scenarios with occluded objects like VGG Cells. On the same grounds,
D-CSRNet achieved best performances on the BCData dataset, and the detection-
based FRCNN approach is the one that faced more difficulties. On the other
hand, in the MBM and the ADI datasets, where the challenges are more related
to the object shape variations, all the approaches showed competitive results,
outperforming state-of-the-art solutions in some cases (e.g., S-UNet in MBM
and FRCNN in ADI). Overall, the tested approaches performed in line with
state-of-the-art, and thus we proceeded to adopt them in the first
localization stage of our pipeline.

?tablename? 6.2: Comparison of the adopted architectures on standard single-
rater counting benchmarks. For VGG, MBM and ADI we varied the training and
validation subsets ($N$ images for each subset), repeating the experiments 10
times. For BCData we used the splits provided by [112], performing 10 runs
changing the seed each time. Mean$\pm$st.dev. of MAE is reported.

(a) VGG Cells (200 images in total - 100 test images)

Method |  N = 16 |  N = 32 |  N = 50
---|---|---|---
Arteta et al. [16] |  N/A |  5.06 $\pm$ 0.2 |  N/A
GMN [158] |  N/A |  3.6 $\pm$ 0.3 |  N/A
Lempitsky and Zisserman [139] |  3.8 $\pm$ 0.2 |  3.5 $\pm$ 0.2 |  N/A
VGG-GAP-HR [7] ∗ |  N/A |  2.95∗∗ |  2.67
SAU-Net [97] † |  N/A |  N/A |  2.6 $\pm$ 0.4
FCRN-A [245] |  3.4 $\pm$ 0.2 |  2.9 $\pm$ 0.2 |  2.9 $\pm$ 0.2‡
FCRN-A [245] § |  4.7 $\pm$ 0.7 |  3.3 $\pm$ 0.2 |  3.1 $\pm$ 0.1
Count-Ception [57] |  2.9 $\pm$ 0.5 |  2.4 $\pm$ 0.4 |  2.3 $\pm$ 0.4
CCF [119] |  2.8 $\pm$ 0.1 |  2.6 $\pm$ 0.1 |  2.6 $\pm$ 0.1
C-FCRN+Aux [102] $ |  2.3 $\pm$ 2.2
S-UNet [74] (our) |  7.7 $\pm$ 2.0 |  6.8 $\pm$ 1.0 |  4.5 $\pm$ 1.0
D-CSRNet [144] (our) |  3.7 $\pm$ 0.3 |  2.9 $\pm$ 0.3 |  2.5 $\pm$ 0.1
FRCNN [187] (our) |  9.3 $\pm$ 0.7 |  7.5 $\pm$ 0.6 |  7.0 $\pm$ 0.4

* They did not report standard deviation. ** They used a validation subet of $100-N$ images. $\dagger$ They did not use a test subset, but only a $100-N$ images validation subset. $\ddagger$ Reported in their work as N = 64. $\$$ They used a 5-fold cross validation-based evaluation protocol considering the whole dataset. $\mathsection$ Re-implemented in this work

(b) MBM Cells. (44 images in total - 10 test images)

Method |  N = 5 |  N = 10 |  N = 15
---|---|---|---
Xie et al. [246] ‡ |  36.3 $\pm$ 19.4
FCRN-A [245] $ |  28.9 $\pm$ 22.6 |  22.2 $\pm$ 11.6 |  21.3 $\pm$ 9.4
FCRN-A [245] § |  15.6 $\pm$ 4.3 |  12.4 $\pm$ 4.0 |  12.2 $\pm$ 2.9
Marsden et al. [162] ∗ |  23.6 $\pm$ 4.6 |  21.5 $\pm$ 4.2 |  20.5 $\pm$ 3.5
Count-Ception [57] |  12.6 $\pm$ 3.0 |  10.7 $\pm$ 2.5 |  8.8 $\pm$ 2.3
CCF [119] ∗ |  9.3 $\pm$ 1.4 |  8.9 $\pm$ 0.9 |  8.6 $\pm$ 0.3
C-FCRN+Aux [102] ∗∗ |  6.5 $\pm$ 5.2
SAU-Net [97] † |  N/A |  N/A |  5.7 $\pm$ 1.2
Jiang and Yu [120] |  8.2 $\pm$ 1.1 |  6.9 $\pm$ 0.9 |  6.0 $\pm$ 0.6
Jiang and Yu [118] |  - |  - |  6.0 $\pm$ 0.2
S-UNet [74] (our) |  5.5 $\pm$ 1.9 |  5.9 $\pm$ 4.2 |  5.7 $\pm$ 0.9
D-CSRNet [144] (our) |  9.4 $\pm$ 3.4 |  7.2 $\pm$ 2.0 |  6.4 $\pm$ 1.4
FRCNN [187] (our) |  9.3 $\pm$ 1.4 |  9.0 $\pm$ 1.4 |  8.6 $\pm$ 0.8

* They used 14 test images. ** They used a 5-fold cross validation-based evaluation protocol considering the whole dataset. $\dagger$ They did not use a test subset, but only a $44-N$ images validation subset. $\ddagger$ They used a train/test split of 8/3 using full-size images. $ Implemented by [57]. $\mathsection$ Re-implemented in this work.

(a) ADI Cells. (200 images in total - 100 test images)

Method |  N = 10 |  N = 25 |  N = 50
---|---|---|---
FCRN-A [245] § |  21.1 $\pm$ 4.7 |  13.1 $\pm$ 0.7 |  11.3 $\pm$ 1.1
Count-Ception [57] |  25.1 $\pm$ 2.9 |  21.9 $\pm$ 2.8 |  19.4 $\pm$ 2.2
CCF [119] |  16.9 $\pm$ 1.9 |  14.5 $\pm$ 0.4 |  14.5 $\pm$ 0.4
SAU-Net [97] † |  N/A |  N/A |  14.2 $\pm$ 1.6
Jiang and Yu [120] |  13.8 $\pm$ 0.7 |  11.6 $\pm$ 0.4 |  10.6 $\pm$ 0.3
Jiang and Yu [118] |  - |  - |  10.1 $\pm$ 0.1
S-UNet [74] (our) |  16.6 $\pm$ 5.5 |  13.6 $\pm$ 1.8 |  13.7 $\pm$ 4.9
D-CSRNet [144] (our) |  12.6 $\pm$ 1.3 |  10.8 $\pm$ 1.5 |  8.8 $\pm$ 1.0
FRCNN [187] (our) |  10.0 $\pm$ 0.9 |  9.1 $\pm$ 0.7 |  8.7 $\pm$ 0.8

$\dagger$ They did not use a test subset, but only a $200-N$ images validation
subset. $\mathsection$ Re-implemented in this work.

(b) BCData (1,338 images in total - 803 train, 133 val, 402 test); positive
and negative cells are malignant and not malignant tumor cells, respectively.

Method |  Positive |  Negative |  All
---|---|---|---
Sirinukunwattana et al. [206] ∗ † |  9.1 |  20.6 |  14.8
CSRNet [144] (integration) ∗ $ |  9.2 |  24.8 |  14.8
U-CSRNet [112] (integration) ∗ |  10.0 |  18.0 |  14.0
CSRNet [144] (detection) ∗ $ |  7.7 |  14.1 |  10.9
U-CSRNet [112] (detection) ∗ |  6.8 |  14.1 |  10.5
S-UNet [74] (our) |  8.3 $\pm$ 0.5 |  19.7 $\pm$ 0.9 |  14.4 $\pm$ 0.7
D-CSRNet [144] (our) |  8.3 $\pm$ 0.9 |  16.6 $\pm$ 1.4 |  12.5 $\pm$ 0.9
FRCNN [187] (our) |  10.3 $\pm$ 0.4 |  30.9 $\pm$ 2.3 |  20.6 $\pm$ 1.3

* They did not report standard deviation. $ Implemented by [112], they used ResNet-50 [100] instead of VGG-16 [203] for feature extraction. $\dagger$ Implemented by [112].

####  6.4.2 Localization Stage Evaluation

For these experiments, we applied the three previously evaluated solutions to
our novel PNN dataset, and we investigated their counting ability in the
presence of weakly-labeled data, i.e., under significant label noise
introduced by errors of raters.

We considered the large single-rater subset PNN-SR to train the models,
whereas we evaluated them on the multi-rater subset PNN-MR. Since the training
set contains weak labels, for each solution, we derived a scalar score $s$
from the models’ output that can be used to filter low-quality predictions. We
refer to the scores obtained in this stage as uncalibrated scores, since they
have not been designed to correlate with the quality of predictions (we did
this in the subsequent scoring stage). For S-UNet we set $s$ as the maximum
value of the connected component found in the segmentation map. For FRCNN, we
set $s$ as the classification score that the network already outputs together
with the regressed bounding box coordinates localizing the object. Finally,
for D-CSRNet, we considered as $s$ the value of the higher local peak in the
density maps localizing the object.

During the training phase, we split the data into training and validation
parts. We did not adopt the common per-image split strategy, as the number of
PNNs vastly varies depending on the particular considered brain slice (i.e.,
image), and thus this strategy would have produced unbalanced splits. Instead,
we split each image vertically in half, including one half in the training set
and the other in the validation set in an alternate fashion.

Due to the extreme size of the images, we processed them in patches. During
the training phase, we cropped squared randomly localized patches from
training images. We experimented with different patch sizes of 256, 320, 480,
640, and 800 pixels. At validation time, we divided and processed the image in
regularly-spaced overlapped patches of the same size used during training (see
also Figure 6.2), we reconstructed the global output by combining patch
predictions, and we computed metrics at the entire image level. For
segmentation-based and density-based solutions, image-level maps were obtained
by stitching back together the patch-level maps and taking the mean pixel
values in the overlap areas. For the detection-based solution, we performed
non-maximum suppression of all the bounding boxes predicted in the overlap
areas.

In Figure 6.3, we show the results obtained by the three solutions (one per
column) on the whole multi-rater PNN-MR subset when varying the patch size and
the threshold on the scalar score $s$. To compare the counting errors we used,
in addition to the MAE, the Mean Absolute Relative Error (MARE) that provides
a percentage relating the absolute counting error to the number of objects to
be counted, and the Grid Average Mean absolute Error (GAME) [96] that accounts
for localization errors, as it is computed by sub-dividing the image in
$4^{L}$ non-overlapping regions and computing the MAE in each of these sub-
regions (see also Section 2.1.2 for further details). As depicted in the first
three rows of Figure 6.3, patch size did not significantly influence the
performance of FRCNN and D-CSRNet. Thus, for these models, we suggest opting
for bigger patch sizes that reduce processing overhead. On the other hand, the
S-UNet solution was more sensitive to this aspect; due to artifacts in the
overlap regions of the segmentation map, different patch sizes induced
different score distributions that responded differently to score
thresholding. For S-UNet, the best performance is obtained with smaller patch
sizes together with more conservative threshold values. Note that the density-
based solution D-CSRNet was the most strained by weakly-labeled training data,
achieving the worst overall performance primarily due to a low recall.
Moreover, as expected, score thresholding was not effective since the density
peak value employed as the score was not expected to locate the PNN precisely
and correlate with its “visual quality”. Even when counting was performed via
density map integration, instead of peak localization and counting, the best
counting performance that D-CSRNet achieves is an MAE of $90.99$. Finally, in
the last row of Figure 6.3, we show Precision-Recall (PR) curves, with the
goal of better highlighting the influence of the considered threshold.
Precision was computed as the percentage of model predictions that
corresponded to a ground-truth object, whereas Recall was the percentage of
ground-truth objects identified by the model. Predictions and ground-truth
objects were matched using the Hungarian algorithm based on the 2D Euclidean
distance between pixel coordinates; we assigned an infinite cost to pairs
separated by a distance greater than 1.25 times the typical radius of objects.
As depicted in the figure, the PR curves concerning the FRCNN and the D-CSRNet
are monotonic, while the S-UNet does not have this property, again due to
artifacts in the segmentation maps and on merging/separating components.

So far, we have experimented on the entire PNN-MR subset, thus including every
PNN found by at least one rater in the ground-truth set. Next, we illustrate
how the trained models behave when asked to localize only PNNs on which at
least $a$ raters agree on their presence. Specifically, we defined four sets
of ground truth labels for PNN-MR comprising PNNs labeled by at least 1, 4, 5,
and 7 out of 7 raters, respectively, simulating different counting policies,
from more liberal to more conservative ones; the choice of 4 and 5 raters
reflects the rater’s agreement above $50\%$ and $70\%$, respectively, which
are two thresholds widely adopted to legitimize labels. In Table 6.4, we
report the results in terms of MAE, MARE, and GAME(3), obtained on these four
sets by the three tested models in their most effective combination of patch
size and threshold values. We observe that models tended to correctly identify
and count the PNNs found by more raters, as these are also the clearer and
easier-to-spot samples. Although all the models delivered a similar
performance at the higher agreement levels, at the lowest agreement level
($a\geq 1$), the D-CSRNet tended to have a higher error due to its inability
to achieve a high recall on low-agreement samples. We observe that FRCNN
tended to achieve lower errors when increasing the rater’s agreement $a$ with
respect to the other tested models. We also note that all models showed high
variability in MAE values computed on different test set images. We deem this
was due to particular brain regions where PNNs appeared dimmer and thus more
difficult for both models and human raters to cope with (see Figure 6.4).

?figurename? 6.3: PNN-MR: Impact of patch size and threshold to counting and localization performance on standard segmentation-based (S-UNet), detection-based (FRCNN), and density-based (D-CSRNet) approaches (without score calibration). ?tablename? 6.4: PNN-MR: Performance on different agreement levels. We report the mean $\pm$ st.dev. of MAE, MARE, and GAME(3) considering four sets of ground-truth labels for the whole PNN-MR dataset composed by objects labeled by any number of raters (Any), at least 50%, at least 70%, or all raters, respectively, to simulate different counting policies, from liberal to conservative ones. Models are trained once on the weakly-labeled PNN-SR dataset.  |  | Raters’ Agreement
---|---|---
|  |  Any $(a\geq 1)$ |  $\geq 50\%$ $(a\geq 4)$ |  $\geq 70\%$ $(a\geq 5)$ |  $100\%$ $(a=7)$
|  |  2,351 obj. |  1,384 obj. |  1,234 obj. |  880 obj.
MAE | S-UNet | $27.6\,\pm\,24.8$ | $15.1\,\pm\,14.1$ | $15.8\,\pm\,11.6$ | $13.8\,\pm\,12.8$
FRCNN | $27.8\,\pm\,21.6$ | $15.5\,\pm\,13.4$ | $13.3\,\pm\,12.6$ | $7.8\,\pm\,9.9$
D-CSRNet | $91.5\,\pm\,43.6$ | $21.1\,\pm\,23.0$ | $15.3\,\pm\,20.1$ | $10.9\,\pm\,8.9$
MARE (%) | S-UNet | $14.2\,\pm\,11.3$ | $12.6\,\pm\,12.9$ | $14.5\,\pm\,9.6$ | $19.0\,\pm\,13.3$
FRCNN | $14.9\,\pm\,11.0$ | $13.6\,\pm\,10.9$ | $12.7\,\pm\,12.5$ | $11.8\,\pm\,13.6$
D-CSRNet | $47.8\,\pm\,20.3$ | $18.9\,\pm\,18.0$ | $14.3\,\pm\,14.8$ | $15.7\,\pm\,11.7$
GAME(3) | S-UNet | $61.8\,\pm\,18.7$ | $36.8\,\pm\,15.2$ | $34.9\,\pm\,15.4$ | $29.0\,\pm\,13.3$
FRCNN | $60.2\,\pm\,12.9$ | $31.7\,\pm\,13.5$ | $28.5\,\pm\,12.6$ | $23.2\,\pm\,7.8$
D-CSRNet | $99.3\,\pm\,38.5$ | $41.2\,\pm\,18.4$ | $35.0\,\pm\,14.8$ | $28.1\,\pm\,11.8$
| Easy Sample | Hard Sample
---|---|---
|  $a\geq 1$ |  $a=7$ |  $a\geq 1$ |  $a=7$
Groundtruth |  |  |  |
S-UNet |  |  |  |
FRCNN |  |  |  |
D-CSRNet |  |  |  |
?figurename? 6.4: PNN-MR: Examples of localization predictions of tested
models. We show portions of two samples; an easy one (first two columns) and a
hard one (last two columns), with different ground truths defined by including
only PNNs with a minimum raters’ agreement $a$. In the first row, we highlight
the ground truth in yellow squares. In the rest of the rows, we indicate false
positives in purple, false negatives in cyan, and true positives in green,
with the corresponding ground-truth position drawn in red and connected via a
thin yellow line. Best viewed in electronic format.

####  6.4.3 Scoring Stage Evaluation

Here, we describe the experiments to evaluate the proposed additional scoring
stage. The goal was to produce new “objectness” scores that correlate with the
raters’ agreement; we refer to scores produced in this stage as calibrated
scores, in contrast with the uncalibrated ones derived in the previous stage.
This stage required a small multi-rater dataset to be used as a training set,
and thus we adopted the PNN-MR dataset for both the training and testing
phases. To this end, we randomly split the images comprising PNN-MR into
train, validation, and test sets following the widely employed 70/15/15
proportion. Table 6.5 reports the average number of objects per agreement
level in the PNN-MR train/validation/test splits.

?tablename? 6.5: PNN-MR Splits Statistics. Mean and standard deviation of number of objects in the five PNN-MR 70/15/15 random splits. | Raters’ Agreement
---|---
|  Any |  $\geq 50\%$ |  $\geq 70\%$ |  $100\%$
|  $(a\geq 1)$ |  $(a\geq 4)$ |  $(a\geq 5)$ |  $(a=7)$
Total | $2351$ | $1384$ | $1234$ | $880$
Train | $1167\,\pm\,70$ | $678\,\pm\,56$ | $606\,\pm\,50$ | $428\,\pm\,33$
Validation | $569\,\pm\,60$ | $351\,\pm\,38$ | $314\,\pm\,33$ | $232\,\pm\,28$
Test | $615\,\pm\,58$ | $356\,\pm\,27$ | $315\,\pm\,24$ | $220\,\pm\,25$

First, we assessed this scoring stage in a stand-alone way, considering it
independently from our overall counting pipeline. We implemented the scorer
model $g_{\phi}$ as a small convolutional network with 8 Conv-GroupNorm-ReLU
blocks followed by average pooling and a linear projection producing the
desired number of outputs (8 for Agreement Classification, 1 for the rest). We
trained each calibration methodology presented in Section 6.3.2 by providing,
as inputs, small patches around PNNs centered in the locations provided by the
ground-truth labels. Due to the limited size of the dataset, we performed five
runs with randomly generated splits. In Figure 6.5, we report the distribution
of (z-normalized) scores obtained by the tested methods on the PNN patches of
the test splits. In addition, we also report the distributions of the
uncalibrated scores obtained from the localization models as described in
Section 6.4.2. We notice that scores obtained by Agreement Ordinal Regression
(OR) and Agreement Rank Learning (RL) strategies behave best in terms of
correlation with the raters’ agreement $a$ achieving the highest Pearson’s
correlation index $r$ of 75%. Those are also the most data-efficient methods,
as they operate on pairs or tuples of samples, while Agreement Regression (AR)
and Classification (AC) seem to suffer from the limited number of samples.
Thus, opting for OR or RL is suggested, as multi-rater data is often limited.
Moreover, the mean scores per agreement level of OR and RL tend to follow a
steep regression line. In contrast, in other methods like D-CSRNet and AR, the
score distributions of nearby agreement levels are more overlapped and often
with non-monotonic means.

?figurename? 6.5: PNN-MR: Correlation between scores and raters’ agreement. We
show the distribution of (z-normalized) scores per agreement level, the
regression lines, and the Pearson correlation coefficient $r$ between scores
and raters’ agreement for each tested methods.

Next, we evaluated the effect of the scoring stage on the whole counting
pipeline. We first localized PNNs in test images with the three localization
models trained in Section 6.4.2; for this operation, we set a high-recall
threshold for each model, such that filtering can be postponed after rescoring
by $g_{\phi}$. We then scored the locations found using the trained scoring
models and evaluate counting performance for each combination of localization
and scorer models. Figure 6.6 and Table 6.6 compare the achieved performance
in terms of MAE when choosing the best threshold value for the rescored
predictions. As baseline, we report also the counting performance of using
uncalibrated scores, i.e., without using the proposed rescoring stage. Again,
we report results for different ground-truth settings defined by the minimum
desired raters’ agreement. S-UNet combined with Agreement Classification (AC)
achieves the best counting performance among most ground-truth configurations,
whereas Agreement Ordinal Regression (OR) is the second-best rescoring
solution. Despite providing significant boosts compared to other tested
rescoring approaches, AC can suffer when multi-rater samples are unbalanced
(or missing) among agreement levels, which is fairly common in this
application. In those cases, rank-based methods (OR or RL) are known to behave
better under these scenarios. However, we leave to future work the evaluation
of sample efficiency and of robustness to class unbalance. Note that for
S-UNet and FRCNN, score calibration generally improves the counting
performance, specifically for the former where we achieve MAE reductions up to
11.07. On the other hand, when adopting the D-CSRNet localization method, we
achieve an improvement only on the highest agreement test set; this is mainly
due to the limited recall of D-CSRNet on the PNN dataset.

?figurename? 6.6:  PNN-MR (Test Subset): Impact of the rescoring stage
$g_{\phi}$ on counting performance in terms of MAE. We show mean values $\pm$
standard deviation over five runs with randomized train/val/test splits of the
PNN-MR subset. Numerical values in tabular format can be found in Table 6.6.
?tablename? 6.6: PNN-MR (Test Subset): Impact of the rescoring stage
$g_{\phi}$ on counting performance in terms of MAE. We report mean and
standard deviation over five runs with randomized train/val/test splits of the
PNN-MR subset. ‘w/o’ indicates results without the rescoring stage, i.e.,
filtering is performed on method-specific scores extracted in the first
localization stage. Results are also showed using plots in Figure 6.6.

|  | Raters’ Agreement
---|---|---
|  |  Any $(a\geq 1)$ |  $\geq 50\%$ $(a\geq 4)$ |  $\geq 70\%$ $(a\geq 5)$ |  $100\%$ $(a=7)$
| $g_{\phi}$ |  |  |  |
S-UNet | w/o |  19.13 $\pm$ 11.63 |  14.67 $\pm$ 6.70 |  15.80 $\pm$ 4.98 |  13.87 $\pm$ 2.81
AR |  19.73 $\pm$ 14.44 |  11.20 $\pm$ 3.83 |  9.53 $\pm$ 4.80 |  7.27 $\pm$ 3.41
AC |  17.53 $\pm$ 6.93 |  6.00 $\pm$ 3.46 |  4.73 $\pm$ 3.75 |  3.73 $\pm$ 3.07
OR |  14.87 $\pm$ 10.98 |  7.00 $\pm$ 5.07 |  5.80 $\pm$ 4.96 |  5.13 $\pm$ 2.69
RL |  16.13 $\pm$ 10.48 |  6.93 $\pm$ 4.10 |  7.67 $\pm$ 3.39 |  6.07 $\pm$ 3.51
FRCNN | w/o |  10.07 $\pm$ 10.38 |  8.67 $\pm$ 5.33 |  8.33 $\pm$ 4.61 |  6.13 $\pm$ 2.51
AR |  13.73 $\pm$ 7.27 |  10.53 $\pm$ 2.73 |  9.40 $\pm$ 3.21 |  8.73 $\pm$ 3.47
AC |  11.27 $\pm$ 7.77 |  7.00 $\pm$ 5.55 |  6.33 $\pm$ 4.31 |  5.27$\pm$ 2.56
OR |  10.13 $\pm$ 5.73 |  6.40 $\pm$ 3.46 |  5.93 $\pm$ 3.47 |  4.13 $\pm$ 3.06
RL |  9.67$\pm$ 7.08 |  6.60 $\pm$ 3.43 |  8.00 $\pm$ 2.13 |  6.00 $\pm$ 2.66
D-CSRNet | w/o |  89.53 $\pm$ 27.92 |  15.67 $\pm$ 5.85 |  9.00 $\pm$ 3.85 |  8.33 $\pm$ 4.96
AR |  89.53 $\pm$ 27.92 |  18.33 $\pm$ 5.80 |  12.73 $\pm$ 5.28 |  7.40 $\pm$ 4.80
AC |  89.53 $\pm$ 27.92 |  16.47 $\pm$ 5.24 |  10.13 $\pm$ 3.55 |  5.53 $\pm$ 1.99
OR |  89.53 $\pm$ 27.92 |  15.67 $\pm$ 5.62 |  9.53 $\pm$ 2.90 |  5.53 $\pm$ 1.99
RL |  89.53 $\pm$ 27.92 |  15.73 $\pm$ 6.12 |  10.33 $\pm$ 3.55 |  5.93 $\pm$ 2.20

AR = Agreement Regression. AC = Agreement Classification. OR = Agreement
Ordinal Regression. RL = Agreement Rank Learning.

### 6.5 Summary

In this chapter, we tackled the task of counting biological structures from
microscopy images under the assumption that training datasets were
characterized by weak multi-rater labels, i.e., in the presence of non-
negligible disagreement between multiple raters. This is a hypothesis that
often occurs in medical images where intrinsically non-trivial patterns can
produce weak annotations due to raters’ judgment differences, even among
experts. More robust annotations can be obtained by aggregating and averaging
the decisions provided by multiple raters regarding the same data. However,
the scale of the counting task and the limited resources dedicated to the
labeling process put a damper on this solution since raters prefer to label
new data rather than annotate the same data more times. Consequently, the
challenge here has been to deal with weakly-labeled large sets of data, i.e.,
characterized by errors due to differences in raters’ judgment, and very
small, more reliable multi-rater data, from which it becomes crucial to make
the best fully exploiting the redundancy of the information represented by the
different raters’ opinions. Thus, as in the previous chapters, also in this
case, we faced the counting task in the presence of data scarcity, but in a
different setting that characterized medical images and, in general, all the
contexts where non-trivial patterns prevent the possibility of easily
identifying the objects we want to count.

In this setting, we proposed a two-stage counting strategy, where each stage
is devised to make the best of the annotations available in each data subset.
The first stage exploited large single-rater data to bootstrap state-of-the-
art counting methodologies; we evaluated three CNN-based methods i.e.,
segmentation-based S-UNet, detection-based FRCNN and density-based approaches
D-CSRNet. We showed that this step alone leads to sub-optimal results due to
the underlying noisy nature of the employed single-rater data. Thus, we
introduced a second rescoring stage that harnesses a small multi-rater subset
and refines the previously computed predictions.

We performed an extensive experimental evaluation of our pipeline on a novel
weakly-labeled dot-annotated dataset introduced on purpose, consisting of a
collection of fluorescence microscopy images of mice brains containing
biological structures. Results showed that rescoring strategies improved the
correlation between the scores and the raters’ agreement. Using the proposed
pipeline, we enhanced counting performance, in some cases significantly
reducing the MAE. Whereas even simple rescoring methods such as Agreement
Classification is beneficial, we deem the rank-based ones, like Agreement
Ordinal Regression and Agreement Rank Learning, to be also data-efficient and
robust to data unbalance, operating on pairs or tuples of samples. However, we
leave a rigorous evaluation of those aspects to future work.

##  ?chaptername? 7 An Embedded Toolset for Human Activity Monitoring in
Critical Environments

As occurs during a severe health emergency event, there exist scenarios in
which ensuring compliance to a set of guidelines becomes crucial to secure a
safe living environment in which human activities can be conducted. In fact,
as evidenced during the recent COVID-19 pandemic, wearing medical masks,
avoiding the creation of large gatherings in confined places, and keeping a
certain physical distance among people were the most common rules every
government applied in their jurisdiction territories. However, human
supervision could not always guarantee this task, especially in crowded scenes
where checking personal protection equipment or enforcing strict social
behaviors must be continuously assessed to preserve global health.

As we have already seen in the previous chapters, Computer Vision applications
have shown outstanding results in several daily life tasks. Automatic image
analysis aimed at classifying, locating, and counting objects, as well as
estimating the distance between different instances of objects, are typical
examples of applications of Computer Vision technology, which can be a
valuable tool to automatically monitor human activities in critical
environments through images captured by networked cameras.

In this chapter, we present an embedded modular Computer Vision-based and AI-
assisted system that puts into practice some of the techniques described in
the previous chapters, carrying out several tasks to help monitor individual
and collective human safety rules. We strive for a real-time but low-cost
system, thus complying with the compute- and storage-limited resources
availability typical of off-the-shelves embedded devices, where images are
captured and processed directly onboard. Our solution consists of multiple
modules, each responsible for a specific functionality that the user can
easily enable and configure. One of them aims at estimating the number of
people present in a region of interest, a piece of crucial information to
monitor the occupancy area. By measuring, and eventually limiting, the number
of people who can be in a specific site at any one time, it is possible to
reduce the likelihood of setting up gatherings. Other helpful capabilities
that our system makes available are the skill to estimate the so-called social
distance (i.e., the physical distance among pedestrians), as well as the
possibility to localize and classify Personal Protective Equipment (PPE) worn
by people (such as helmets, high-visibility clothing, and face masks) that the
World Health Organizations has recommended as one of the primary tools able to
curb the spread of the disease, like, for example, the recent COVID-19
pandemic.

To validate our solution, we test all the functionalities that our framework,
deployed on an embedded device, makes available, exploiting two novel datasets
that we collected and annotated on purpose and that represent another
scientific contribution. Specifically, we gathered a first dataset comprising
images captured by a smart camera located in a public square in the city of
Pisa, Italy, that represents a typical scenario for which it is crucial to
check compliance with the safety rules. Moreover, we collected and annotated a
second dataset comprising synthetic and real-world images containing
pedestrians with and without PPE, such as helmets, high-visibility vests, and
face masks. Experiments show that our system can effectively carry out all the
functionalities that the user can set up, providing to be a valuable asset to
automatically monitor compliance with safety rules.

To summarize, the main contributions introduced in this chapter are the
following:

  * •

We introduce an expandable and flexible Computer Vision-based and AI-assisted
embedded system, deployed in a real use-case scenario, capable of
automatically monitoring human activities in critical environments, where
individual and collective safety must be constantly checked. We base our
solution on modules responsible for specific tasks that the user can easily
configure and add to the whole system, making available many functionalities
such as estimating the number of pedestrians present in the monitored scene.

  * •

We collect and annotate two novel datasets that we exploit to validate our
framework. One, named CrowdVisorPisa, is gathered from a camera in a public
square of the city of Pisa, Italy, and represents a typical scenario for which
it can be essential to monitor compliance with the safety rules, such as the
respect of a ruled maximum number of people allowed to stay in the site.

  * •

We conduct experiments evaluating all the modules and the functionalities,
which our framework makes available _in an embedded and deployed off-the-shelf
device_ , showing that our solution may be a valid aid to monitor and handle
critical environments drastically reducing human supervision.

We organize the rest of the chapter as follows. We review similar works in
Section 7.1, and we introduce our modular framework and its plug-ins in
Section 7.2. In Section 7.3, we describe the exploited datasets along with the
adopted training procedures. In Section 7.4, we show our experiments, also
discussing and analyzing the obtained results.

The research presented in this chapter was published in [21].

### 7.1 Related Work

Due to the COVID-19 pandemic, many Computer Vision-based works have been
recently published to help monitor human activities analyzing images,
especially on the specific task of evaluating the social distance between
people. For example, the Inter-Homines system, presented in [73], evaluates in
real-time the contagion risk in a monitored area by analyzing video streams.
The system includes occlusion correction, homography transformation, and
people anonymization. People are located in the space exploiting the CenterNet
[273] object detector, and interpersonal distances are then calculated.
Results are evaluated on the JTA dataset [72] (i.e., in a virtual world). In
[198], the YOLO9000 detector [184] has been exploited to detect people, and
centroids of the found bounding boxes are computed to evaluate the distance
between them. Similarly, in [3], a platform for social distance tracking in
top perspective video frames based on YOLOv3 [185] was presented. Here too,
centroids of the bounding boxes are used to estimate distances. A subset of
the authors of [3] also presented in [2] a social distance framework based on
the Faster-RCNN detector [187]. Instead, YOLOv3 [185] to detect humans and
Deepsort [239] to track people are exploited in [180]. Experiments are
conducted on the Oxford town center surveillance footages [23]. Monitoring of
workers to detect social distancing violation that uses Mobilenet-V2 [197] to
detect people is introduced in [125].

Another task recently tackled in literature, again related to the COVID-19
pandemic, is face mask detection. For example, authors [128] present an edge
computing-based mask identification framework (ECMask). It consists of three
main stages: video restoration, face detection (inspired by FaceBoxes [265]),
and mask identification (based on Mobilenet-V2 [197]). Deep Learning models
were trained and evaluated on the Bus Drive Monitoring Dataset, which
unfortunately is not publicly available. Authors in [71] developed a deep
learning-based computer vision system able to perform face mask detection but
also face-hand interaction detection. A more comprehensive literature review
of applications of Artificial Intelligence in battling against COVID-19 is
given in [211] including social distancing and face mask detection.

Differently from most other works, in this chapter, we present a modular and
expandable Computer Vision-based embedded system that can fulfill multiple
tasks to help monitor compliance of individual and collective human safety
rules in critical scenarios, like the one caused by the COVID-19 pandemic. The
main peculiarities are that it runs directly on a low-cost computing device
and that the user can easily enable and configure the available
functionalities, ranging from estimating the number of people present in the
scene, computing social distances, or detecting PPE s, combining more modules
and building more complex tasks.

### 7.2 Modular Framework

The general purpose of our monitoring system is to be embeddable on low-cost
devices and, above all, to be expandable to different features in demanding
situations. To this end, we designed a framework able to orchestrate a set of
internal and user-defined _plugins_ , each dedicated to a single task.
Specifying inputs and outputs makes it possible to create a dependency graph.
Each sub-module represents a node, and each pair of matching input-output
represents an edge. In this way, given the desired output, a _topological
sort_ is executed to minimize and linearize the sequential execution of
computations. Although an easier solution may exist in the context of the
plugins, our methodology is relatively simple to implement and allows low-cost
systems to execute any complex compute graphs in a sequential and
semantically-correct way.

An overview of our modular framework is depicted in Figure Figure 7.1. Video
frames are taken at regular intervals from one or more cameras and processed
locally. Multiple video streams can be multiplexed and handled by a single
system instance. Current modules include a) Pedestrian Detector, b) Density-
based Pedestrian Counter, c) Instance-based Pedestrian Counter, d) Pedestrian
Tracker, e) PPE Detector, and f) Interpersonal Distance Measurer; Figure
Figure 7.2 exemplifies the results of the analyses performed by each module,
whereas their detailed description is reported in the following sections. All
the modules are toggleable; the Instance-based Pedestrian Counter, Pedestrian
Tracker, Interpersonal Distance Measurer, and PPE Detector modules depend on
the output of the Pedestrian Detector module and require it to be active.
Results of the active modules are combined and provided in JSON format to be
consumed by downstream services. Note that video frames are analyzed onboard
and never stored; this enables privacy-aware solutions where captured images
never leave the edge devices.

?figurename? 7.1: Overview of our modular framework. Multiple video streams
can be multiplexed and handled by a single instance. Results are generated in
JSON format and can be routed to downstream services. Our system is flexible
and expandable, as modules can be activated or deactivated depending on the
user’s needs, and novel functionalities can be introduced with additional
custom modules. ?figurename? 7.2: Visualization of output examples of the
modules currently available in our system. The outputs of each module are the
following. Pedestrian Detection & Instance-based Counting: list of pedestrian
bounding boxes and respective count. Pedestrian Tracking: numeric ID assigned
to detected pedestrians persisting through frames. Density-Based Pedestrian
Counting: estimated number of pedestrians (and, optionally, the density map).
Distance Measurement: IDs of groups of pedestrians violating a predefined
distance. PPE Detection: list of PPE bounding boxes detected per pedestrian.

###### Pedestrian Detector

The pedestrian detector is the system’s main component on which almost all
other plug-ins rely. Its primary purpose is to localize and classify
pedestrian instances from input images. These detections constitute the main
data that will be exploited, in different ways, by the other nodes of the
system.

We base our pedestrian detector on Faster R-CNN [187], a popular state-of-the-
art CNN-based object detection system belonging to the two-stage paradigm,
already described in Section 2.2.2, and that we also exploited in Chapter 4
and Chapter 6. We preferred a two-stage detector to single-shot detectors, as
we could easily extract features from the region proposal stage and make them
directly available for subsequent processing. We used the extracted features
when tracking pedestrians, and they could be used in other future modules,
such as cross-camera pedestrian re-identification. Moreover, Faster R-CNN is
widely adopted and usually guarantees a state-of-the-art detection
performance. However, the modularity of our system does not limit us to Faster
R-CNN, and another object detector can be easily adopted in the future, just
replacing the related module.

We specialize this detection system to localize pedestrian instances through a
supervised domain adaption technique that exploits several pedestrian
datasets, in a similar way to what we have done in Chapter 4.

###### Pedestrian Tracker

Object tracking can be an essential tool to increase the robustness to
spurious detections and achieve temporal consistency in video analysis. To
this end, we implement and apply an object tracker over pedestrian detection
to reidentify people among consecutive video frames. This step is beneficial
for assessing temporal rules, such as raising alarms after the same pedestrian
occupies a forbidden area for more than a predefined amount of time.

The implementation of the tracker follows the formulation of DeepSort [239];
it is based on SORT [25], a simple causal tracking algorithm for 2D objects in
which targets are represented by the position and area of the bounding box and
their speed of variation. The state of each target (also known as tracklet) is
updated with available detections using a Kalman filter framework. DeepSort
builds upon SORT by adding a matching scheme between predicted and actual
targets based on feature vectors that describe the appearance of tracked
objects; tracklets can be confirmed if the cosine score between feature
vectors of the predicted and actual target is above a programmable threshold.
Feature vectors in DeepSort must be provided by extracting representations
from detected regions with an additional pretrained network. In our
implementation, we avoid this step by reusing the feature vectors of detected
regions that the object detection network has already extracted; in
particular, we perform a Region of Interest (RoI) average pooling of the
features extracted by the CNN backbone using only the regions provided by the
pedestrian detection module.

###### Instance-based and Density-based Pedestrian Counter

In some scenarios where individual and collective safety has to be constantly
monitored, like people aggregations during the recent COVID-19 pandemic,
estimating the number of people present in a region of interest is crucial to
monitor the occupancy area. By measuring, and limiting, the number of people
who can visit a location at any one time, it is possible to reduce the
likelihood of setting up people gatherings and, consequently, minimize human
virus transmission. Our solution relies on a dedicated plug-in that can work
in two different modalities that the user can conveniently pick out, depending
on the considered scenario. The first one, named Instance-based Pedestrian
Counter, is better suited for not particularly crowded environments and relies
on the pedestrian detector component previously described. The second, named
Density-based Pedestrian Counter, is instead a more holistic approach more
appropriate for highly crowded scenarios; it aims to compute a mapping between
the features of the captured image and its pedestrian density maps, skipping
the detection of the single instances. The estimated number of people present
in the controlled area can then be obtained by integrating this density map.
More information about this technique are provided in Section Section 2.1.1.
We build our density map estimator upon the Congested Scene Recognition
Network (CSRNet) [144], a CNN-based algorithm that can understand highly
congested scenes and perform accurate density estimation, already described in
Section 2.1.3 and that we exploited also in Chapter 5 and Chapter 6, for
estimating urban traffic density and counting cells in microscopy images,
respectively.

###### Interpersonal Distance Measurer

A critical condition that must be kept under control in dynamic environments
where an infection is ongoing is represented by the physical distance among
individuals. In the case of air-borne diseases, it is thus very common to
issue rules to avoid people gatherings in confined places and keep a specific
reciprocal separation to contrast the spread of pathogen agents. Although
crowd counting is effective in monitoring aggregations, measuring distances
among people becomes critical during pandemic events. Assuming that
individuals mostly hang out on the same planar floor, we decided to measure
their actual distance by applying a simple pre-calibrating step to the fixed
monitoring camera, using a proper geometrical transformation that places
detected items on a common system of reference. Our solution is to pre-compute
a mapping between real points in the scene whose relative position is known
and their projection on the acquired frame image. This process is well known
in Computer Vision and consists of finding a _homography_ , i.e., a
perspective transformation that projects on two different points of view a set
of 3D points lying on the same plane.

###### Personal Protective Equipment Detector

A simple intervention for protecting health and well-being is wearing Personal
Protective Equipment (PPE); this is particularly true in dangerous working
environments, such as wearing harnesses and helmets on construction sites.
Still, it also became evident in light of the recent COVID-19 pandemic, where
wearing face masks can prevent infections. Therefore, we implement a module
dedicated to detecting worn PPE, essential for ensuring compliance with
regulations that imply personal protection.

Our solution for PPE detection follows the same methodology already adopted
for pedestrian detection: specifically, we assume the same detector
architecture based on Faster-RCNN. Differently from the pedestrian detector,
the PPE detector network takes as input a rather small image depicting a
pedestrian, and it is trained to distinguish and detect several classes of
worn PPE, i.e., surgical/face masks, helmets, and high-visibility vests. The
module’s input is provided by the pedestrian detection module: once detected,
the patch of the video frame depicting a pedestrian is given as input to the
PPE detector that provides bounding boxes of PPE if the pedestrian wears them.

Conceptually, the detection of PPE could be tackled by the object detector
module by adding the PPE classes to the base detection model. However, we
empirically noticed that merging the PPE detection with the pedestrian
detection module leads to performance degradation in both tasks. Using
separate modules provides a more flexible solution in which the input image
resolution of both detectors can be adjusted separately and better adapted to
the monitored scenario. For example, when wide areas are monitored, PPE
detection can be performed several times at each processed video frame
depending on how many pedestrians have been detected, while the PPE detection
network can be configured to operate on smaller input sizes, maintaining an
affordable computational cost.

### 7.3 Datasets and Architecture Adaptions

A key point in producing a verification system that can generalize on a broad
spectrum of working conditions is to generate a training set based on an
adequately large amount of environmental conditions. In our case, this means
being able to access a massive amount of images involving people under
different environmental scenarios. As we have already seen in the previous
chapters, this point represents one of the main challenges related to the CNN
s, and in general to all supervised learning approaches, since manually
annotating new collections of images is expensive and requires a notable human
effort. Thus, to train the modules of our framework, we build vast training
datasets, covering a multitude of different scenarios and contexts. We
consider images from public datasets when available, and we collect others
when needed. As we have already seen in Chapter 4 and Chapter 5, a promising
approach is to gather data from virtual world environments that mimics as much
as possible all the characteristics of the real-world scenarios, and where the
annotations can be acquired with an automated process. In this way, it is
possible to collect vast quantities of annotated data covering a massive
number of different scenarios. Thus, we also consider synthetic datasets like
ViPeD introduced in Chapter 4, providing a high variability of the training
data. Hereafter, we briefly describe the adopted datasets, dividing them
according to the module for which they are employed. Furthermore, we describe
the exploited training procedures, highlighting the changes we made to the
architectures to adapt them to our specific scenarios.

###### Pedestrian Detection

To train the pedestrian detector module, we use many popular publicly
available pedestrian detection datasets: MOT17Det [164], MOT20Det [65],
CityPersons [264], CrowdHuman [199], PRW [271] and CUHK-SYSU [244]. These
datasets are described in Section 2.4. We exploit also ViPeD, a synthetic
dataset introduced in Chapter 4. Furthermore, we introduce a novel dataset
that we collected and annotated on purpose, named CrowdVisorPisa, also
employed for evaluating our solution. In particular, it comprises 15 different
sequences gathered from a webcam located in a public square of the city of
Pisa, Italy, each of which consists of ten images captured with a time
interval of 1 second. We manually labeled all frames, localizing the
pedestrian instances with bounding boxes. More, we also annotated each
sequence taking track of the different pedestrian entities entering or exiting
the scene. We divided the dataset into train and test splits, considering 10
and 5 different sequences, respectively. The former split is exploited to
train the object detector module, while the latter is used to evaluate the
performance of some modules of our framework. It is worth noting that, due to
camera positioning not modifiable for local restrictions, this dataset
represents a particularly challenging scenario as people instances are small
and sometimes difficult to localize. A sample of the dataset is shown in
Figure 7.3.

?figurename? 7.3: A sample from our novel CrowdVisorPisa dataset. We show an
image together with bounding box annotations localizing pedestrians.

To make the pedestrian detector able to run efficiently directly on
computational- and resource-limited devices, we adopt a strategy similar to
the one described in Chapter 3, employing, as the backbone of Faster R-CNN,
the ResNet50 architecture, a lighter version of the popular ResNet101 [100],
and reducing the final fully-connected layers responsible for classifying the
detected objects since, in our case, we have to localize and identify objects
belonging to just one class (i.e., pedestrian). We call Light this modified
version of the pedestrian detector module to distinguish it from the Full
original one, having instead the ResNet101 backbone and a larger number of
fully connected layers. To train this module, we adopt the supervised _domain
adaptation_ strategy described in Chapter 4, where, during the training phase,
we mix the synthetic data taken from ViPeD, and the real-world images gathered
from the remaining datasets. In this way, we take advantage of the great
variability and size of ViPeD, and at the same time, we mitigate the existing
domain shift between these synthetic data and the real-world ones. In
particular, during the training phase, we exploit batches composed of $2/3$ of
synthetic images and $1/3$ of real-world data, thus considering statistics
from both domains throughout the entire procedure and where the real-world
data acts as a regularization term over the synthetic data training loss.

###### PPE Detection

To train and evaluate our PPE detection module, we collect and annotate a
novel dataset (named CrowdVisorPPE). It comprises 54,017 images representing
pedestrians with and without wearable PPE. Roughly half of the dataset
comprises synthetic images procedurally generated using the GTA V video game
engine as in [20], whereas the other half comprises photographic real-world
images taken from the Web and manually annotated. The PPE classes of interest,
i.e., helmets, high-visibility vests (HVVs), and face masks, are annotated
with bounding boxes. The real-world subset is the only source of face mask
instances since they are not available for rendering in GTA V. We hold out a
subset of real images as the test split, whereas synthetic images and the
remaining real ones form the training split.

As in the pedestrian detector, we adopt the Faster R-CNN model with the
ResNet50 backbone as the PPE detector architecture, and we use a similar
domain adaptation strategy exploiting synthetic and real data. The only
difference concerning the pedestrian detector is that we perform PPE detection
only on pre-segmented patches containing a single pedestrian instead of
searching for PPE in the full-frame. This simplifies the task for the model
and enables us to save computational budget by processing smaller images.

###### Crowd Counting by Density Estimation

To train the module responsible for crowd counting by density estimation, we
exploit many popular publicly available real-world datasets: ShanghaiTech
[268], UCF-QNRF [114] and NWPU-Crowd [234]. Furthermore, also in this case, we
exploit also a synthetic dataset, named GTA5 Crowd Counting (GCC) [233]. These
datasets are described in details in Section 2.4. To train this module, we
adopt the supervised domain adaptation strategy illustrated in Chapter 4,
consisting of training the network with the synthetic data and then fine-
tuning it exploiting the real-world images. In particular, we set the initial
weights of the network layers with values coming from a Gaussian distribution
with 0.01 standard deviation. Then, we train the network exploiting the GCC
dataset, and, finally, we fine-tune it using the real-world data.

### 7.4 Experiments and Results

We evaluate all the modules making up our framework, considering different
scenarios and exploiting appropriate metrics depending on the considered task.
For all the experiments, we consider the Light version of our object detector
module since it has shown similar performance compared with the Full version,
and it is more appropriate to be used in combination with low-cost and
computational-limited hardware.

Being our target a deployable monitoring system, we selected the NVIDIA Jetson
TX2 embedded device as the hardware host. It is composed of two 64 bit CPUs
with two and four cores each, an NVIDIA Pascal GPU with 256 CUDA cores, 8 GB
of RAM shared between the system and the graphics accelerator, and a 32 GB
solid-state storage volume. The operating system is the NVIDIA’s Linux4Tegra
(L4T) distribution based on Ubuntu. We installed Python 3.8 with OpenCV 4.5
and the deep learning framework PyTorch 1.8. As detailed in Table 7.1, memory
usage is kept within 5 GB of both system and GPU RAM. An external USB camera
completes the whole installation.

?tablename? 7.1: System and GPU Memory Usage in GB. PD = pedestrian detector model type; DC = whether the density-based counter module is active; PPE = whether the PPE detector module is active. The modular framework is assumed to always use the object detector in its _Light_ or _Full_ models, along with the enabled distance measure plug-in that consumes a fixed and negligible (less than 1 MB) amount of memory. Video stream size is $1173\times 880$ RGB pixels. System memory is calculated with /usr/bin/time -f "%M", GPU memory with torch.cuda.max_memory_allocated(). PD | DC | PPE |  SysRAM |  GpuRAM
---|---|---|---|---
Light | ✗ | ✗ |  2.36 |  0.55
✓ | ✗ |  2.44 |  0.86
✗ | ✓ |  2.35 |  2.10
✓ | ✓ |  2.44 |  2.20
Full | ✗ | ✗ |  2.51 |  0.62
✓ | ✗ |  2.52 |  0.94
✗ | ✓ |  2.51 |  2.20
✓ | ✓ |  2.51 |  2.30

####  7.4.1 Counting by Instances

In this setting, we test and evaluate the counting by instance functionality.
We consider our CrowdVisorPisa dataset and, in particular, the five sequences
belonging to the test subset, performing two different sets of experiments
over it; the first one involves only the pedestrian detector module, and the
second instead takes also into account the tracker module. More in detail, in
the first case, we evaluate the effectiveness of our framework to estimate the
number of people present in the single frames. On the other hand, in the
second scenario, we also consider the temporal relation existing between
consecutive images, tracking the found pedestrian instances over time.

We report in Figure Figure 7.4 the obtained results concerning the first
scenario. Each row of the figure represents a different sequence. The first
column shows the number of people that our detector module can localize for
each frame comprising a sequence, varying the detection threshold. On the
other hand, in the second column, we illustrate the errors in terms of
counting. We also report, for each sequence, the best Mean Absolute Error
(MAE), i.e., the mean of the sum of the absolute errors, obtained with a
specific threshold. As can be seen, we get an MAE close to 1 or 2, depending
on the considered scenario, demonstrating that the module provides a reliable
estimation of the number of pedestrians present in the monitored scene. The
optimal threshold may vary depending on the scenario and the desired behavior,
e.g., the user may prefer under- or over-estimation in case of errors. Due to
the empirical nature of its choice, in our system, we provide the dynamical
configuration of several parameters, including detection thresholds.

Number of detected people

Counting Errors

?figurename? 7.4: Evaluation of _counting by instances_ functionality of our
framework, considering the single still frames of the five test sequences of
our CrowdVisorPisa dataset. In the first column, we report the number of
people located by our detector, varying the detection thresholds. The black
line (GT) indicates the actual number of pedestrians in the frame. The second
column shows the counting errors and the best MAE obtained with a specific
detection threshold.

On the other hand, in Figure Figure 7.5, we show the results concerning the
second scenario. Each row of the figure corresponds to a different sequence.
We report the results about the single frames making up a sequence for three
different detection thresholds, one for each column. In particular, we
indicate the pedestrians that enter and exit from the scene at each frame,
exploiting the tracklets provided by the tracker module that represents the
recognized identities of the people instances over time. Solid lines represent
the actual number of people present in a frame over time, and green/red
candles represent the number of people entering/exiting the scene. Similarly,
dashed lines represent the number of people predicted by our system, and
yellow/blue candles represent the estimated number of people entering/exiting
the scene as predicted by the tracker module. We notice that with a low (resp.
high) threshold value, our system tends to overestimate (resp. underestimate)
the total number of people present in a sequence, thus finding its optimal
threshold values in the 0.5 - 0.6 range. We also note that false-positive
detections tend to create spurious peaks in the people count. However, they
often recovered in the immediately following frame.

?figurename? 7.5: Evaluation of the counting by instances functionality of our
framework, taking into account also the tracker module. We considered the five
test sequences of our CrowdVisorPisa dataset reported one for each row.
Columns represent the results obtained for three different detection
thresholds. For each plot, we show the pedestrians that enter and exit from
the scene, relying on the tracklets describing the recognized identities of
the people instances over time.

####  7.4.2 Counting by Density Estimation

We validate the counting by density estimation, performed by the Density
Counter module, by exploiting several test subsets belonging to publicly
available datasets (described in Section 7.3). This choice has been driven by
the fact that our CrowdVisorPisa dataset does not provide the needed features,
i.e., the scenarios are not highly crowded. Given that the annotation
procedure for labeling datasets having these characteristics is highly costly
in terms of manual human effort, we exploited some already existing
collections of annotated images available in the literature.

We report in Table 7.2 the obtained results in terms of MAE and Root Mean
Squared Error (RMSE). It is worth noting that, as a result of the squaring
operation, RMSE effectively penalizes large errors more heavily than small
ones, thus more suitable when outliers are particularly undesirable.
Furthermore, we also compute the Structural Similarity Index Measure (SSIM)
[237] to measure the density map quality, which measures images’ similarities
under three aspects: brightness, contrast, and structure. The value of SSIM is
in the $[0,1]$ range: the larger it is, the less distortion of the image is
measured (see also Section 2.1.2 for further details). Finally, in Figure 7.6
we show some examples of the considered images, together with the ground truth
and the predicted density maps.

?tablename? 7.2: Evaluation of the Pedestrian Detection (PD), Density-based Counter (DC), and PPE Detector modules, measured on the corresponding test sets. The mean Average Precision (mAP) measures the average precision of the detection when varying the score threshold in detection-based modules (PD, PPE). For DC, MAE and RMSE measure the counting error, while SSIM measures the quality of the predicted density map. PD | PPE | DC
---|---|---
mAP $\uparrow$ | mAP $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$ | SSIM $\uparrow$
0.836 | 0.606 | 92.28 | 365.4 | 0.79

Image

Groundtruth Density Map

Predicted Density Map

?figurename? 7.6: Some samples of the test subsets exploited for the
evaluation of our _counting by density_ estimation functionality. We show some
images together with the ground truth and the predicted density maps.
Integrating these density maps, i.e., summing up the pixel values, we can
obtain an estimation of the people present in the image.

####  7.4.3 Detecting Pedestrians and Personal Protection Equipment

Here, we validate the detection of pedestrian and worn PPE, performed
respectively by the Pedestrian Detector and the Personal Protection Equipment
Detector modules. For the former, we focus on the five test sequences of our
CrowdVisorPisa dataset, whereas for the latter, we consider the CrowdVisorPPE
test subset.

We evaluate the two modules using the mean Average Precision (mAP), a popular
metric in measuring the accuracy of object detectors that computes the average
precision value for recall values spanning 0 to 1 (see also Section 2.2.1 for
further details). We prefer the mAP over other threshold-dependent metrics,
such as True Positive Rate, False Positive Rate, or F1-score. Indeed,
threshold-dependent metrics are scenario- and application-specific; end-users
may accept different levels of false positive or negative rates and may decide
to tune thresholds differently depending on their needs. On the other hand,
mAP provides a unique metric summarizing the performance at multiple
operational points. We report the obtained results in Table 7.2, showing that
our modules can reach a mAP of 0.836 and 0.606 for the pedestrian detection
and the PPE detection tasks, respectively. Figure 7.7 shows some predictions
of PPE detections on the CrowdVisorPPE test set.

####  7.4.4 Measuring Interpersonal Distances

To establish a correspondence between the acquired image and a planar _metric_
surface onto which objects (i.e., pedestrians) positions can be evaluated, we
used a known-sized manhole in the monitored scene to calculate the homography.
The computed matrix is exploited to unwarp pixel position into real-world
relative locations. The homography reprojection is a closed-form mathematical
process, so no previous training is needed.

Our measuring results can be seen in Figure Figure 7.8: in the examples shown,
the precision of measurements is relative to both the initial calibration
(i.e., manhole real size mapped to its projection on the input image) and the
accuracy of the pedestrian bounding boxes (i.e., rectangles) predicted by the
object detector. We measured the manhole with an upper bound precision of 1 cm
and a pixel area error of about 3 cm, thus confining the overall measurements
below the 10 cm error. For pedestrian positions, we used the midpoint of the
lower edge of its predicted bounding box. As can be seen from the gridded
unprojection of the examples, results are consistent within the above error
gap.

?figurename? 7.7: Examples of predictions of the PPE Detection module on our
CrowdVisorPPE test set. PPE classes are color coded (helmet, high-visibility
vest, face mask), and the detection score is reported in parenthesis.

?figurename? 7.8: Examples of detections and distance warnings under different
lighting conditions. Each of the eight images represents, on its left side,
pedestrians detected and tracked in the example scenario, while showing on its
right side their 2D projection on a virtual planar surface obtained through
homography, with a reference 1-meter-spacing overlay grid. The green color
means a safe placement, and the red indicates violations of the 1-meter
physical distance rule. Some failure examples are outlined in white and
zoomed.

### 7.5 Summary

In this chapter, we presented, from a more practical perspective, a modular
framework based on Computer Vision and AI technologies, deployed in a real
use-case scenario on a low-cost off-the-shelf embedded platform and aimed at
monitoring human activities in critical conditions. Our goal was to provide a
system having the peculiarity to be expandable in the future, simply adding
new modules in charge of performing new functionalities that the user can
easily enable (or disable) according to their needs. In this way, our
framework turns out to be flexible since the tasks to be accomplished can
change in time, and it is always possible to insert a new module responsible
for performing it. As an effective setup, we implemented a set of visual-based
modules for pedestrian detection, tracking, aggregation counting (based on