# Pedestrian Detection: Domain Generalization, CNNs, Transformers and Beyond

Irtiza Hasan, Shengcai Liao, Jinpeng Li, Saad Ullah Akram, and Ling Shao,
Irtiza Hasan is with Group 42, UAE. Part of this work was done while he was a
researcher at Inception Institute of Artificial Intelligence.
<EMAIL_ADDRESS>Shengcai Liao is with Inception Institute of
artificial Intelligence, UAE. He is also the corresponding author.
<EMAIL_ADDRESS>Jinpeng Li is with Department of Computer
Science and Engineering, The Chinese University of Hong Kong. Saad Ullah Akram
is with Aalto Univeristy, Finland. National Center for Artificial
Intelligence, Riyadh, Saudi Arabia.

###### Abstract

Pedestrian detection is the cornerstone of many vision based applications,
starting from object tracking to video surveillance and more recently,
autonomous driving. With the rapid development of deep learning in object
detection, pedestrian detection has achieved very good performance in
traditional single-dataset training and evaluation setting. However, in this
study on generalizable pedestrian detectors, we show that, current pedestrian
detectors poorly handle even small domain shifts in cross-dataset evaluation.
We attribute the limited generalization to two main factors, the method and
the current sources of data. Regarding the method, we illustrate that biasness
present in the design choices (e.g anchor settings) of current pedestrian
detectors are the main contributing factor to the limited generalization. Most
modern pedestrian detectors are tailored towards target dataset, where they do
achieve high performance in traditional single training and testing pipeline,
but suffer a degrade in performance when evaluated through cross-dataset
evaluation. Consequently, a general object detector performs better in cross-
dataset evaluation compared with state of the art pedestrian detectors, due to
its generic design. As for the data, we show that the autonomous driving
benchmarks are monotonous in nature, that is, they are not diverse in
scenarios and dense in pedestrians. Therefore, benchmarks curated by crawling
the web (which contain diverse and dense scenarios), are an efficient source
of pre-training for providing a more robust representation. Accordingly, we
propose a progressive fine-tuning strategy which improves generalization.
Additionally, this work also investigate the recent Transformer Networks as
backbones to test generalization. We demonstrate that as of now, CNNs
outperform transformer networks in terms of generalization and absorbing large
scale datasets for learning robust representation. In conclusion, this paper
suggests a paradigm shift towards cross-dataset evaluation, for the next
generation of pedestrian detectors. Code and models can be accessed at
https://github.com/hasanirtiza/Pedestron.

###### Index Terms:

Pedestrian detection, Object detection, Generilizable pedestrian detection,
Autonomous driving, Surveillance

##  1 Introduction

Figure 1: Left: Pedestrian detection performance over the years for Caltech,
CityPersons and EuroCityPersons on the reasonable subset. EuroCityPersons was
released in 2018 but we include results of few older models on it as well.
Dotted line marks the human performance on Caltech. Right: We show comparison
between traditional single-dataset train and test evaluation on Caltech [13]
_vs._ cross-dataset evaluation for three pedestrian detectors and one general
object detector (Cascade R-CNN). Methods enclosed with bounding boxes are
trained on CityPersons [52] and evaluated on Caltech [13], while others are
trained on Caltech.

Pedestrian detection is a critical component of many real-world applications,
such as, autonomous driving [9, 18], robotic navigation, and video
surveillance [17]. It is one of the most heavily researched computer vision
problems and has seen huge improvement over the years. In last decade, like
most other computer vision problems, deep learning based techniques have
enabled significant progress for the pedestrian detection. Currently, on
certain benchmarks, the automated methods are approaching the human
performance as shown in Fig. 1 left. However, current methods are not very
robust and their performance varies significantly across the datasets as can
be seen in Table VII. Some widely-used pedestrian detection methods suffer
from over-fitting to source datasets, especially in the case of autonomous
driving, as shown in Fig. 1 right. The performance of these detectors on a
target dataset is considerably worse, even when trained on a relatively large
source dataset, without any obvious domain shift. This lack of robustness
limits the full potential of these methods as most applications utilizing
pedestrian detectors require very low failure rate.

So far, robustness and generalizability, despite being vital for most
applications, have not been the main focus in the pedestrian detection
research. Performance characteristics of pedestrian detectors for the same
source and target dataset have previously been investigated, but poor cross-
dataset performance has not been thoroughly investigated or discussed. In this
paper, we hypothesize that the poor cross-dataset performance is primarily due
to over-fitting caused by the fact that the current state-of-the-art
pedestrian detectors are heavily tailored for target datasets, which biases
their overall design towards the target datasets.

In addition, we hypothesize that the training datasets are generally not dense
in pedestrians, lack diversity of environments and scenerios, and have
relatively small number of pedestrians, which also limits the performance of
current pedestrian detectors. Almost all state-of-the-art pedestrian detection
methods are based on deep learning and their performance depend heavily on the
quantity and quality of data, and there is evidence that the performance on
some other computer vision tasks (e.g. image classification, object detection,
image segmentation) keeps improving at least up-to billions of samples [33].
All current pedestrian detection datasets for autonomous driving have at least
three main limitations. Firstly, they contain limited number of unique
pedestrians. Secondly, they have low pedestrian density, i.e. relatively small
proportion of pedestrians have challenging occlusions. The performance of
current method is significantly worse for these challenging occluded
pedestrians as can be seen in Table IV. Thirdly, these datasets have limited
diversity as they are captured during small number of sessions by a small
team, primarily for dataset creation. Thesedays, dashcam videos are widely
available online, e.g. youtube, facebook, etc, enabling the potential curation
of much more diverse and realistic datasets.

In recent years, few large and diverse person detection datasets, e.g.
CrowdHuman [39], WiderPerson [54] and Wider Pedestrian [31], have been created
using images and videos available online and through surveillance cameras.
These datasets advance the general person detection research significantly but
are not the most suitable datasets for pedestrian detection, as they contain
people in a lot more diverse scenarios, than are relevant for autonomous
driving. Nevertheless, they are still beneficial for learning a more general
and robust model of pedestrians as they contain more people per image, and
they are likely to contain more human poses, appearances and occlusion
scenarios, which is beneficial for autonomous driving scenarios, provided
current pedestrian detectors have the capacity to digest these large-scale
data.

In this paper, we investigate the performance characteristics of current
pedestrian detection methods in cross-dataset setting. We show that 1) the
existing methods fare poorly compared to general object detectors, without any
adaptations, when provided with larger and more diverse datasets, and 2) when
carefully trained, the state-of-the-art general object detectors, without any
pedestrian-specific adaptation on the target data, can significantly out-
perform pedestrian-specific detection methods on pedestrian detection task
(see Fig. 1 right). In addition, we propose a progressive training pipeline
for better utilization of general person datasets for improving the pedestrian
detection performance in autonomous driving scenarios. We show that by
progressively fine-tuning the models, from the dataset furthest from the
target domain to the dataset closest to the target domain, large gains in
performance can be achieved in terms of $MR^{-2}$ on reasonable subset of
Caltech (3.7$\%$) and CityPerson (1.5$\%$) without fine-tuning on the target
domain. These improvement hold true for models from all pedestrian detection
families that we tested such as Cascade R-CNN [8], Faster RCNN [38] and
embedded vision based backbones such as MobileNet [21] Finally, we also
compare the generaliztaion ability of CNNs against the recent transformer
network (Swin-Transformer) [30]. We illustrate that, despite superior
performance of Swin Transformer [30], it struggles when the domain shift is
large, in comparison with CNNs. To the best of our knowledge, this is the
first study to objectively illustrate this.

The paper is organized in the following way. Section 2 reviews the relevant
literature. We introduce datasets and evaluation protocol in Sec. 3. We
benchmark our baseline in Sec. 4. We test the generalization capabilities of
the pedestrian specific and general object detectors in Sec. 5, along with
qualitative results. Subsequently, we compare CNNs with Transformer networks
in Sec. 6. We also discuss effect of fine-tuning on the target set in Sec. 7.
Finally, conclude the paper in Section 8.

##  2 Related Work

Pedestrian detection. Prior to CNNs, the pioneering work of Viola and Jones
[43] which slid windows over all scales and locations motivated many
pedestrian detection methods. To better describe the features of pedestrians,
Histogram of Oriented Gradients (HOG) was presented in the work of Dalal and
Triggs [11]. The aggregate channel feature (ACF) leveraged features in
extended channels to improve the speed of pedestrian detection [12]. In
similar ways, pedestrian detectors in [52, 36] employed spatial pooling with
low-level features and filtered channel features, respectively. Nonetheless,
their performance and generalization ability were still limited by the hand-
crafted features.

As the great progress of Convolutional Neural Networks (CNNS), they dominated
the research field of generic object detection [38, 19, 42, 26] and
considerably improved the accuracy. The pedestrian detectors [1, 20, 7] also
benefit from this powerful paradigm. R-CNN detector [16] was utilized in some
of the pioneering efforts for pedestrian detection using CNNs [20, 51] and is
still widely employed in this research field. RPN+BF [50] combined Region
Proposal Network and boosted forest to enhance the performance of pedestrian
detection, which overcame the problems of poor resolution and imbalanced
classes in Faster RCNN [38]. Although the performance of RPN+BF was
outstanding, its learning ability was limited by the non-end-to-end trainable
architecture. Since the strong performance and high expandability of Faster
RCNN [38] , it inspired a broad spectrum of pedestrian detectors [55, 52, 6,
5, 34].

Some pedestrian detection methods designed more sophisticated architectures
and leveraged extra information to further boost the detection performance.
ALF [28] employed several progressive detection heads on Single Shot MultiBox
Detector (SSD) [26] to gradually refine the initial anchors, which inherited
the merit of high efficiency in single stage detectors and further improved
the detection accuracy. Inspired by the blob detection, CSP [29] reformulated
the pedestrian detection task into an anchor-free manner which only needed to
locate the center points and regress the scales of pedestrians without relying
on the complicated settings of anchor boxes. To improve the detection
performance on occluded pedestrians, extra information of the visible-area
bounding-box was utilized as the guidance of attention mask in MGAN [37].

Pedestrian detection benchmarks. Due to the large practical value of
pedestrian detection, a lot of works were devoted to create benchmarks to
promote the development of pedestrian detection, such as Daimler-DB [35],
TownCenter [2], USC [47], INRIA [11], ETH [14],and TUDBrussels [46], which
were all from surveillance scenarios and not suitable for applications in
autonomous driving. Recently, the great progress of pedestrian detection also
attracted the attention of autopilot systems, and several datasets were
created for this context, such as Caltech [13], KITTI [15], CityPersons [52]
and ECP [4]. The cameras in these datasets were typically installed on the
front windshield of cars to collect images from the similar field of views as
human drivers. Caltech [13] and CityPersons [52] are the most popular
benchmarks for recent learning-based pedestrian detectors, while their small
data sizes and monotonous scenarios limit their capabilities in training more
robust methods. To solve these limitations, ECP [4] dataset collected images
from diverse scenarios including various cities, all seasons, and day and
night times, which contains almost ten times more images and eight times more
persons than CityPersons [52]. Although ECP [4] has a much larger scale, it
still suffers from the low density of persons and high similarity of
background scenes, which could be the focus of future datasets. Thus, in this
work, we argue that the low density and diversity of these datasets constrains
the generalization ability of pedestrian detectors, while the web crawled
datasets, such as CrowdHuman [39], WiderPerson [54] and Wider Pedestrian [31],
including much diverse scenes and denser persons may increase the upper bound
of pedestrian detectors’ generalization ability.

Cross-dataset evaluation. Some existing works [52, 4, 39] explored the
relations between the performance of pedestrian detectors and training
datasets, whose purpose was to show how much performance advantage could be
obtained on target datasets by pre-training on more diverse and dense scenes
datasets. But, in this work, we aim to thoroughly evaluate the generalization
abilities of some popular pedestrian detection methods by using the cross-
dataset evaluation.

##  3 Experiments

###  3.1 Experimental Settings

TABLE I: Experimental settings. Setting | Height | Visibility
---|---|---
Reasonable | [50, inf] | [0.65, inf]
Small | [50, 75] | [0.65, inf]
Heavy | [50, inf] | [0.2, 0.65]
Heavy* | [50, inf] | [0.0, 0.65]
All | [20, inf] | [0.2, inf]
TABLE II: Datasets statistics. $\ddagger$ Fixed aspect-ratio for bounding boxes. | Caltech $\ddagger$ | CityPersons $\ddagger$ | ECP | CrowdHuman | Wider Pedestrian
---|---|---|---|---|---
images | 42,782 | 2,975 | 21,795 | 15,000 | 90,000
persons | 13,674 | 19,238 | 201,323 | 339,565 | 287,131
persons/image | 0.32 | 6.47 | 9.2 | 22.64 | 3.2
unique persons | 1,273 | 19,238 | 201,323 | 339,565 | 287,131
TABLE III: Evaluating generalization abilities of different backbones using our baseline detector. Backbone | Training | Testing | Reasonable
---|---|---|---
HRNet | WiderPedestrian + CrowdHuman | CityPersons | 12.8
ResNeXt | WiderPedestrian + CrowdHuman | CityPersons | 12.9
Resnet-101 | WiderPedestrian + CrowdHuman | CityPersons | 15.8
ResNet-50 | WiderPedestrian + CrowdHuman | CityPersons | 16.0

Datasets. We conduct extensive experiments on three public pedestrian
detection datasets which collectd from the scenario of autonomous driving to
evaluate and compare with the state-of-the-art pedestrian detection
algorithms. These three benchmarks include Caltech [13], CityPersons [52] and
EuroCity Persons [4], and are categorized into the _autonomous driving_
datasets in this work. Caltech [13] is one of the most popular dataset in the
research field of pedestrian detection. It recorded 10 hours of video in Los
Angeles, USA by a front-view camera of vehicle, and contained roughly 43K
images and 13K persons extracted from the video. We perform the evaluations on
the refined Caltech annotations from [51]. Compared to Caltech, CityPersons
[52] is built upon the dataset of Cityscapes, and contains more diverse
scenarios. It was recorded from the street scenes of different cities in and
close to Germany. CityPersons contains 2,975, 500, 1,575 images in the
training, validation and testing sets, and provides the full bounding-boxes
and visible bounding-boxes for 31k pedestrians. EuroCity Persons (ECP) [4] is
a recently released larege-scale dataset recoded in 31 different European
cities, which contains more diverse scenarios and is more challenging for
pedestrian detectors compared to the datasets of Caltech and CityPersons. It
contains two subsets of ECP day-time and ECP night-time based on the recording
time. ECP dataset has roughly 200K bounding-boxes. Similar to the evaluation
procedure in ECP [4], we conduct the experiments on the subset of ECP day-time
for the fair comparisons with existing literature. Unless other statement, all
experimental results are from the validation sets due to the frequent
submissions to online testing server is not allowed. Except to the _autonomous
driving_ datasets of Caltech, CityPersons and ECP, we further conduct the
experiments on two _web-crawled_ datasets of CrowdHuman and Wider
Pedestrian111Wider Pedestrian contains images from the scenarios of autonomous
driving and surveillance. The data provided in 2019 challenge was used in our
experiments. Data can be downloaded from :
https://competitions.codalab.org/competitions/20132 [31]. We provide more
details of above datasets in Table II.

Evaluation protocol. We evaluate the performance of pedestrian detectors by
the widely used metric of log average miss rate over False Positive Per Image
(FPPI) over range [$10^{-2}$, $10^{0}$] ($MR^{-2}$) on Caltech [13],
CityPersons [52] and ECP [4]. The experimental results on different occlusion
levels including Reasonable, Small, Heavy, Heavy*222In the case of
CityPersons, for the fair comparison with some previous methods under the same
setting, we also report the numbers under the visibility level between
[0.0,0.65] which is denoted as Heavy* occlusion. and All are reported unless
stated otherwise. Table I provides the specific settings of each set.

Cross-dataset evaluation. To evaluate the generalization ability of pedestrian
detectors, we perform cross-dataset evaluation by only using the training set
of dataset _A_ to train models and directly testing then on the
validation/testing set of dataset _B_. This training and testing procedure is
consistent for all experiments, and denoted as A$\rightarrow$B.

Baseline. Because most of the high performance pedestrian detectors on the
datasets of Caltech, CityPersons and ECP are built upon the two-stage
detectors of Faster/Mask R-CNN [38, 19], the more powerful multi-stage
detector of Cascade R-CNN [8] which also belongs to the R-CNN family is chosen
as the baselines. In this work, the terms of baseline and Cascade RCNN are
interchangeably used, which both means the same method of Cascade R-CNN [8].
Multiple detection heads are used step-by-step in the Cascade R-CNN to
gradually filter out the false positive anchors and generate more and more
high-quality proposals. We equipped different backbones on our baseline method
to evaluate its robustness, and the details of these experiments are shown in
Table III. Among them, ResNeXt [48] and HRNet [44] show the top ranked
performance. Unless other statements, HRNet [44] is used as the default
backbone network of our baseline. HRNet simultaneously processes multiple
levels feature maps in a parallel way retaining both low-level details and
high-level semantic information, which may greatly benefit the pedestrian
detection under the large scale variations.

##  4 Benchmarking

The benchmarking results of our baseline, Cascade R-CNN [8], on three
autonomous driving datasets including Caltech [13] dataset, CityPersons [52]
and on ECP [4] are presented in Table IV. Without any ”bells and whistles”,
our baseline achieved the performance comparable to the specially customized
pedestrian detectors on the datasets of Caltech and CityPersons.
Interestingly, the performance gap between our baseline and the state-of-the-
art algorithms changes as the sizes of datasets increase. The relative
performance of our baseline is lowest on the smallest dataset of Caltech and
significantly improves on the largest dataset of ECP.

TABLE IV: Benchmarking on autonomous driving datasets.

Method | Testing | Reasonable | Small | Heavy
---|---|---|---|---
ALFNet [28] | Caltech | 6.1 | 7.9 | 51.0
Rep Loss [45] | Caltech | 5.0 | 5.2 | 47.9
CSP [29] | Caltech | 5.0 | 6.8 | 46.6
Cascade R-CNN [8] | Caltech | 6.2 | 7.4 | 55.3
RepLoss [45] | CityPersons | 13.2 | - | -
ALFNet [28] | CityPersons | 12.0 | 19.0 | 48.1
CSP [29] | CityPersons | 11.0 | 16.0 | 39.4
Cascade R-CNN [8] | CityPersons | 11.2 | 14.0 | 37.1
Faster R-CNN [4] | ECP | 7.3 | 16.6 | 52.0
YOLOv3 [4] | ECP | 8.5 | 17.8 | 37.0
SSD [4] | ECP | 10.5 | 20.5 | 42.0
Cascade R-CNN [8] | ECP | 6.6 | 13.6 | 33.3

##  5 Generalization Capabilities

As previously mentioned, existing works evaluate the pedestrian detectors with
a traditional manner where training and evaluation data are from the same
domain dataset, i.e., within-dataset evaluation. However, we argue that this
algorithm development pipeline ignores the generalization capability of
pedestrian detectors and is easy to over-fit on a specific dataset. Thus, in
this work, we emphasize the importance of cross-dataset evaluation in the
design of pedestrian detectors. We can clearly show how well pedestrian
detectors perform on unseen domain by cross-dataset evaluation. Thus,
extensive cross-dataset experiments are conducted in this section to evaluate
the robustness of pedestrian detectors.

###  5.1 Dataset Illustrations

We showcase some examples of datasets related to pedestrian detection in
Figure 2. Top row depicts different scenarios in diverse and dense datasets
collected by crawling on web. Bottom row illustrates images from traditional
autonomous driving datasets. It can be observed that web-crawled datasets
provide more enriched representation of pedestrians, since they cover several
scenarios, such as different poses, illumination and different types of
occlusion. Whereas, autonomous driving benchmarks are monotonous in nature,
i.e. same background, view-point etc. Interestingly, ECP [4] and CityPersons
[52], illustrate striking resemblance (where the camera is mounted, image
resolution, geographical location etc.), this further stresses the point that
even when the target domains are not drastically different, current pedestrian
detectors do not generalize well (cf. Tables 5,6 and 7 in the paper).

Figure 2: Illustration of benchmarks. Top row shows images from diverse and
dense datasets, such as CrowdHuman [39] and Wider Pedestrian [31]. Bottom row
presents images of autonomous driving benchmarks, ECP [4], CityPersons [52]
Caltech [13] TABLE V: Cross dataset evaluation on Caltech and CityPersons.
A$\rightarrow$B refers to training on _A_ and testing on _B_.

Method | Bakcbone | CityPersons$\rightarrow$CityPersons | CityPersons$\rightarrow$Caltech | Caltech$\rightarrow$Caltech | Caltech$\rightarrow$CityPersons
---|---|---|---|---|---
FRCNN [52] | VGG-16 | 15.4 | 21.1 | 8.7 | 46.9
Vanilla FRCNN [52] | VGG-16 | 24.1 | 17.6 | 12.2 | 52.4
ALFNET [28] | ResNet-50 | 12.0 | 17.8 | 6.1 | 47.3
CSP [29] | ResNet-50 | 11.0 | 12.1 | 5.0 | 43.7
PRNet [41] | ResNet-50 | 10.8 | 10.7 | - | -
BGCNet [23] | HRNet | 8.8 | 10.2 | 4.1 | 41.4
Faster R-CNN [38] | ResNext-101 | 16.4 | 11.8 | 9.7 | 40.8
Cascade R-CNN [8] | HRNet | 11.2 | 8.8 | 6.2 | 36.5
Cascade R-CNN [8] | Swin Transformer | 9.2 | 9.1 | 8.0 | 41.9

TABLE VI: Cross dataset evaluation of (Casc. R-CNN and CSP) on Autonomous driving benchmarks. Both detectors are trained with HRNet as a backbone. Method | Training | Testing | Reasonable | Small | Heavy
---|---|---|---|---|---
Casc. RCNN | CityPersons | CityPersons | 11.2 | 14.0 | 37.0
CSP | CityPersons | CityPersons | 9.4 | 11.4 | 36.7
Casc. RCNN | ECP | CityPersons | 10.9 | 11.4 | 40.9
CSP | ECP | CityPersons | 11.5 | 16.6 | 38.2
Casc. RCNN | ECP | ECP | 6.9 | 12.6 | 33.1
CSP | ECP | ECP | 19.4 | 50.4 | 57.3
Casc. RCNN | CityPersons | ECP | 17.4 | 40.5 | 49.3
CSP | CityPersons | ECP | 19.6 | 51.0 | 56.4
Casc. RCNN | CityPersons | Caltech | 8.8 | 9.8 | 28.8
CSP | CityPersons | Caltech | 10.1 | 13.3 | 34.4
Casc. RCNN | ECP | Caltech | 8.1 | 9.6 | 29.9
CSP | ECP | Caltech | 10.4 | 13.7 | 31.3

###  5.2 Cross Dataset Evaluation of Existing State-of-the-Art

In this section we demonstrate that existing state-of-the art pedestrian
detectors generalize worse than general object detector. We show that this is
mainly due to the biases in the design of methods for the target set, even
when other factors, such as backbone, are kept consistent.

To see how well state-of-the-art pedestrian detectors generalize to different
datasets, we performed cross dataset evaluation of five state-of-the-art
pedestrian detectors and our baseline (Cascade RCNN) on CityPersons [52] and
Caltech [13] datasets. We evaluated recently proposed BGCNet [23], CSP [29],
PRNet [41], ALFNet [28] and FRCNN [52](tailored for pedestrian detection).
Furthermore, we added along with baseline, _Faster R-CNN[38]_, without bells
and whistles, but with a more recent backbone ResNext-101 [48] with FPN [24].
Moreover, we implemented a vanilla _FRCNN[52]_ with VGG-16 [40] as a backbone
and with no pedestrian specific adaptations proposed in [52] (namely quantized
anchors, input scaling, finer feature stride, adam solver, ignore region
handling, etc).

We present results for Caltech and CityPersons in Table V, respectively. We
also report results when training is done on target dataset for readability
purpose. For our results presented in Table V (Fourth column,
CityPersons$\rightarrow$Caltech), we trained each detector on CityPersons and
tested on Caltech. Similarly, in the last column of the Table V, all detectors
were trained on the Caltech and evaluated on CityPersons benchmark. As
expected, all methods suffer a performance drop when trained on CityPersons
and tested on Caltech. Particularly, BCGNet [23], CSP [29], ALFNet [28] and
FRCNN [52] degraded by more than 100 $\%$ (in comparison with fifth column,
Caltech$\rightarrow$Caltech). Whereas in the case of Cascade R-CNN [8],
performance remained comparable to the model trained and tested on target set.
Since, CityPersons is a relatively diverse and dense dataset in comparison
with Caltech, this performance deterioration cannot be linked to dataset scale
and crowd density. This illustrates better generalization ability of general
object detectors over state-of-the-art pedestrian detectors. Moreover, it is
noteworthy that BGCNet [23] like the Cascade R-CNN [8], also uses HRNet [44]
as a backbone, making it directly comparably to the Cascade R-CNN [8].

Importantly, pedestrian specific FRCNN [52] performs worse in cross dataset
(fourth column only), compared with its direct variant vanilla FRCNN. The only
difference between between the two being pedestrian specific adaptations for
the target set, highlighting the bias in the design of tailored pedestrian
detectors.

Similarly, standard Faster R-CNN [38], though performs worse than FRCNN [52]
when trained and tested on the target dataset, it performs better than FRCNN
[52] when it is evaluated on Caltech without any training on Caltech.

It is noteworthy that Faster R-CNN [38] outperforms state-of-the-art
pedestrian detectors (except for BGCNet [23]) as well in cross dataset
evaluation, presented in Table V. We again attribute this to the _bias_
present in the design of current state-of-the-art pedestrian detectors, which
are tailored for specific datasets and therefore limit their generalization
ability. Moreover, a significant performance drop for all methods (though
ranking is preserved except for vanilla FRCNN), including Cascade R-CNN [8],
can be seen in Table V, last column. However, this performance drop is
attributed to lack of diversity and density of the Caltech dataset. Caltech
dataset has less annotations than CityPersons and number of people per frame
is less than 1 as reported in Table II. However, still it is important to
highlight, even when trained on a limited dataset, usually general object
detectors are better at generalization than state-of-the-art pedestrian
detectors. Interestingly, Faster R-CNN’s [38] error is nearly twice as high as
that of BGCNet [23] in within-dataset evaluation, whereas it outperforms in
BGCNet [23] in cross-dataset evaluation.

As discussed previously, most pedestrian detection methods are extensions of
general object detectors (FR-CNN, SSD, etc.). However, they adapt to the task
of pedestrian detection. These adaptations are often too specific to the
dataset or detector/backbones (e.g. anchor settings [52, 28], finer stride
[52], additional annotations [55, 37], constraining aspect-ratios and fixed
body-line annotation [29, 23] etc.). These adaptations usually limit the
generalization as shown in Table V, also discussed, task specific
configurations of anchors limits generalization as discussed in [27].

###  5.3 Autonomous Driving Datasets for Generalization

We show that the general object detectors outperforms existing pedestrian
detection methods (such as CSP [29]) as learning a generic feature
representation for pedestrians, even when they are trained on the large
dataset (such as ECP) and tested on the small dataset (such as Caltech).
Furthermore, detectors achieve higher generalization from larger and denser
autonomous driving datasets.

As shown in the last section, cross dataset evaluation can shed light on the
generalization capabilities of pedestrian detectors. Moreover, the
characteristic of dataset is also an important determinant for the model
generalization. The intrinsic nature of the real world can be more effectively
draw by diverse datasets [4]. Consequently, such datasets potentially provide
the chances for pedestrian detectors to learn more generic feature
representation to robustly tackle the domain shifts. Instead of exploring the
impact of dataset in generalization as existing studies [4, 39, 54], we aim at
presenting a detailed comparison of a general object detector and state-of-
the-art pedestrian detection methods when the training and testing datasets
are varied. For fair comparsions, the backbone of CSP [29] is replaced from
ResNet-50 to HRNet [44]. As shown in the second row of Table VI, this change
improves the performance of CSP from 11.0$\%$$MR^{-2}$ to 9.4$\%$$MR^{-2}$

First, we train Cascade RCNN and CSP on the pedestrian detection dataset of
ECP which contains more countries and cities and is the largest benchmark with
regard to pedestrian density and diversity from the context of autonomous
driving. CityPersons [52] is chosen as the evaluation benchmark, and results
are shown in the third and fourth row of Table VI, respectively. It is clear
that in the case of providing the same backbone, Cascade RCNN generalizes
better on CityPersons than CSP in the reasonable setting. Considering CSP
significantly outperforms Cascade RCNN by nearly 2$\%$ $MR^{-2}$ points when
they are evaluated in the within-dataset setting, it is surprising to see that
the results are turn over.

Secondly, we train CSP and Cascade RCNN on CityPersons and evaluate them on
ECP [4] to further study the generalization abilities of them under different
diverse degrees of training datasets. Similarly, when training dataset is with
low diversity, Cascade RCNN still outperforms than CSP. The performance
difference is 10.5 $\%$ $MR^{-2}$, 7.1 $\%$ $MR^{-2}$ and 2.2 $\%$ $MR^{-2}$
in the small, heavy and reasonable settings, respectively.

Finally, we combine CityPersons and ECP as the training data and perform the
evaluate on the Caltech which is the smallest data source. The results of
Cascade RCNN and CSP in all settings are shown in the last four rows of Table
VI. We conclude that when we use a diverse and dense training dataset, ECP
Cascade RCNN has more robust performance than CSP on all evaluating subsets.

TABLE VII: Benchmarking with CrowdHuman and Wider Pedestrian dataset. Method | Training | Testing | Reasonable | Small | Heavy
---|---|---|---|---|---
Casc. RCNN | CrowdHuman | Caltech | 3.4 | 11.2 | 32.3
CSP | CrowdHuman | Caltech | 4.8 | 5.7 | 31.9
Casc. RCNN | CrowdHuman | CityPersons | 15.1 | 21.4 | 49.8
CSP | CrowdHuman | CityPersons | 11.8 | 18.3 | 44.8
Casc. RCNN | CrowdHuman | ECP | 17.9 | 36.5 | 56.9
CSP | CrowdHuman | ECP | 19.8 | 48.9 | 60.1
Casc. RCNN | Wider Pedestrian | Caltech | 3.2 | 10.8 | 31.7
CSP | Wider Pedestrian | Caltech | 3.4 | 3.0 | 29.5
Casc. RCNN | Wider Pedestrian | CityPersons | 16.0 | 21.6 | 57.4
CSP | Wider Pedestrian | CityPersons | 17.0 | 22.4 | 58.2
Casc. RCNN | Wider Pedestrian | ECP | 16.1 | 32.8 | 58.0
CSP | Wider Pedestrian | ECP | 24.1 | 62.6 | 76.7
TABLE VIII: Investigating the effect on performance when CrowdHuman, Wider Pedestrian and ECP are merged and Cascade R-CNN [8] is trained only on the merged dataset. Method | Training | Testing | Reasonable | Small | Heavy
---|---|---|---|---|---
Casc. RCNN | CrowdHuman $\rightarrow$ ECP | CP | 10.3 | 12.6 | 40.7
CSP | CrowdHuman $\rightarrow$ ECP | CP | 10.4 | 10.0 | 36.2
Casc. RCNN | Wider Pedestrian $\rightarrow$ ECP | CP | 9.7 | 11.8 | 37.7
CSP | Wider Pedestrian $\rightarrow$ ECP | CP | 9.8 | 14.6 | 35.4
Casc. RCNN | CrowdHuman $\rightarrow$ ECP | Caltech | 2.9 | 11.4 | 30.8
CSP | CrowdHuman $\rightarrow$ ECP | Caltech | 11.0 | 14.7 | 32.2
Casc. RCNN | Wider Pedestrian $\rightarrow$ ECP | Caltech | 2.5 | 9.9 | 31.0
CSP | Wider Pedestrian $\rightarrow$ ECP | Caltech | 8.6 | 12.0 | 30.3

###  5.4 Diverse General Person Detection Datasets for Generalization

We study how much performance improvement dense and diverse datasets can
bring. When the testing source is small datasets from the context of
autonomous driving, such as Caltech [13], diverse and dense datasets are still
beneficial for generalization even under large domain gaps between training
and evaluation datasets. Moreover, diverse and dense datasets can bring more
benefits to the general object detection methods, such as Cascade RCNN than
the specially tailored pedestrian detectors, such as CSP.

CrowdHuman [39] and Wider Pedestrian [31] datasets are two diverse and dense
pedestrian detection datasets collected from web-crawling and surveillance
cameras. Unlike the autonomous driving datasets, the crowd density and
scenario diversities are large in CrowdHuman [39] and Wider Pedestrian [31]
since they include images in diverse sources, such as street views and
surveillance, which increases the data diversity from a different form. Thus,
they are idea sources to pre-train models. We pre-train Cascade R-CNN [8] and
CSP [29] on CrowdHuman [39] and Wider Pedestrian [31] datasets, and shows
corresponding results in Table VII. It can be seen that pre-training
significantly boost the performance of pedestrian detectors. When tested on
Caltech dataset, Cascade R-CNN outperforms all previous methods which are only
trained on Caltech, and the test error is reduced nearly by half. The trend of
performance improvement is observed on the results of CSP [29], although its
improvement is less than Cascaded R-CNN. The performance of either Cascade
RCNN or CSP are not improved on CityPersons [52], when they are trained on
CrowdHuman [39]. This is reasonable due to CityPersons [52] is more difficult
than Caltech [13] with regard to density and diversity. Similar trends are
observed in Table VI when detectors are tested on ECP [4]. Training on
CityPersons [52] brings better performance than training on CrowdHuman [39].
From the bottom half of Table VII, we can see that general object detector
benefits more from training on Wider Pedestrian [31]. We hypothesis this is
because that Wider Pedestrian [31] is with larger scale and more similar to
target domain than CrowdHuman [39]. The domain difference is reflected in the
scenarios of images where CrowdHuman [39] includes web-crawled persons with
diverse poses while Wider Pedestrian [31] are mainly from street views and
surveillance cameras.

###  5.5 Progressive Training Pipeline

We propose a progressive training pipeline to take full charge of multi-source
datasets and thus further improve the pedestrian detection performance. This
pipeline first trains detectors on a dataset that is farther from target
domain but general diverse enough and then fine-tune them on a dataset which
is similar to the target domain.

Extensive experiments are conducted to demonstrate the value of progressive
training. To be in line with the study described in the previous section,
target domain dataset is not touched, and only the training subset of each
corresponding dataset is used in our pipeline. We use the symbol of _A_
$\rightarrow$ _B_ to denote pre-training model on _A_ dataset and fine-tuning
it on _B_ dataset. Besides, two datasets of _A_ and _B_ could be directly
merged together to train the model, which is denoted as _A_ \+ _B_. In this
section, Caltech [13] and CityPersons [52] datasets are respectively used as
the evaluation benchmark, and corresponding results are shown in Table VIII.
The upper part of Table VIII clearly shows that the performance of Cascade
RCNN can be significantly improved by the progressive training pipeline.
Noticeably, without training on CityPersons [52] dataset, Cascade R-CNN
achieves comparable results as the state-of-the-art detectors through the
progressive training pipeline of Wider Pedestrian [31] $\rightarrow$ ECP [4].
Besides, progressive training also helps Cascade R-CNN achieve new state-of-
the-art results on the Caltech [13] dataset. It is worth noting that our
performance on Caltech [13] is very close to the human baseline (0.88).

Finally, we show the experimental results of directly merging all datasets in
the third and fourth rows of Table VIII. This training strategy can also
improve the performance but it still cannot reach the performance of
progressive training pipeline, which demonstrates the value of pre-training on
general dataset and then fine-tuning on the autonomous driving dataset.
Without touching the data in target domain, our progressive training helps to
effectively improve the pedestrian detection performance of state-of-the-art
detectors. These experiments demonstrate that our training pipeline explores a
way to significantly improve the generalization capability of Cascade R-CNN
and makes it on a level with state-of-the-art detectors on CityPersons [52]
and achieve best performance on Caltech [13].

###  5.6 Application Oriented Models

In this section, we conducted experiments to show that pre-training on dense
and diverse datasets can help a light-weight neural network architecture,
MobileNet [21], to achieve competitive results as state-of-the-art detectors,
such as CSP, on CityPersons [52] dataset.

The computational cost and model size of pedestrian detectors are important
factors in many real-world applications, such as drones and autonomous driving
cars, which require real-time detection and are usually run on limited
hardware. To study whether progressive training pipeline is still effective in
improving the performance of light-weight backbone, we conduct experiments
with a widely used light-weight neural network backbone, MobileNet [21] v2,
proposed for embedded and mobile computer vision tasks.

We replace the backbone of Cascade R-CNN [8] as a MobileNet [21], and present
the results on CityPersons [52] in Table IX. To establish reference
performance, we train and test MobileNet [21] on CityPersons [52], and show
its results in the first row of Table IX. Intuitively, the performance of
MobileNet [21] is lower than HRNet [44]. But, the results demonstrate that
progressive training by pre-training MobileNet [21] on CrowdHuman [39] and
then fine-tuning on ECP [4] still effectively improves the detection
performance on CityPersons [52]. Moreover, further performance improvement can
be achieved by replacing the pre-training dataset of CrowdHuman [39] as Wider
Pedestrian [31]. As shown in the first and forth rows, our progressive
pipeline of Wider Pedestrian [31] $\rightarrow$ ECP [4] improves the
performance of 0.6$\%$ $MR^{-2}$n the reasonable subset of CityPersons [52].
The results in Table IX and VII both demonstrate that pre-training on Wider
Pedestrian [31] dataset can bring larger performance improvement than
CrowdHuman [39] when the evaluation benchmark is CityPersons [52]. This is
because Wider Pedestrian [31] includes scenario of autonomous driving and
shares more common characteristics with target domain. It is worth noting that
our progressive training pipeline makes Cascade R-CNN with a light-weight
backbone of MobileNet [21] approach the performance of state-of-the-art
method, CSP [29], which is equipped with a much larger backbone of ResNet-50.

TABLE IX: Investigating the performance of embedded vision model, when pre-
trained on diverse and dense datasets.

Training | Testing | Reasonable | Small | Heavy
---|---|---|---|---
CP | CP | 12.0 | 15.3 | 47.8
ECP | CP | 19.1 | 19.3 | 51.3
CrowdHuman$\rightarrow$ECP | CP | 11.9 | 15.7 | 48.9
Wider Pedestrian$\rightarrow$ECP | CP | 11.4 | 14.6 | 43.4

###  5.7 Qualitative Results

Figure 3: Qualitative results of Cascade R-CNN on different benchmarks. Top
row includes example from Caltech, bottom left CityPersons and ECP on bottom
right. $\dagger$ Figure 4: Cascade R-CNN $\dagger$ across different scenarios,
such as summer, winter, rain and low-illumination, illustrating the robustness
of the general object detector.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 5: Generalization comparison between CSP [29] with HRNet and Cascade
RCNN[8]. Both methods are trained on CityPersons and evaluated on ECP. Images
with Yellow bounding box are CSP’s detections where as magenta bounding box
are Cascade RCNN’s detections. Dotted green line illustrates instances where
CSP fails to detect.

We show detection quality of Cascade R-CNN$\dagger$ on different datasets in
Figure 3. Top row contain results from Caltech and bottom row show images from
CityPersons and ECP. One could observe that Cascade R-CNN$\dagger$ is robust
to crowd density, as presented images show several instances of occlusion,
people walking in close vicinity, etc. Furthermore, we extracted images from
different regions across the globe under varying conditions, shown in Figure,
4. Four different conditions are showcased, for example, Netherlands shows
rainy conditions, where people are wearing jackets with hood, rain coats and
holding umbrellas. In Italy, summer season is illustrated and people are
walking in city-center in close vicinity, often occluding each other. Winter
season can be seen in Germany, as snow is visible. Finally, France illustrates
a low-illumination scenario, where car headlights can be seen illuminating the
scene (also bringing reflections and shadows into play). Aim of this figure is
to demonstrate the robustness of Cascade R-CNN$\dagger$, since the pre-
training is done on web-crawled datasets and thanks to the several diverse and
dense scenes, Cascade R-CNN$\dagger$ has learnt a representation capable of
handling several real-world scenarios efficiently.

Finally, we show qualitatively in Figure 5, how pedestrian detectors such as
CSP lacks in generalization compared with a general object detector such as
Cascade RCNN. In the figure 5, we trained both detectors with HRNet on
CityPersons and evaluate on ECP. We picked cases from different cities, under
varying lighting conditions (afternoon, evening etc.) and under different
weathers such as sunny, raining or snowing. Common failure cases for CSP
includes low-illumination/blurry pedestrians Fig.5 (a) and (d), small-scale
pedestrians Fig.5 (g) to (l), and occlusion Fig.5 (b, c, e, f, k). On the
contrary, Cascade RCNN seems to be robust to such domain shifts and handles
the above mentioned challenging scenarios better than CSP. We argue that
probable reasons behind CSP’s poor generalizations stems from the fact that it
is a single stage detector without feature alignment compared to two-stage
detector like Cascade RCNN. Feature alignment improves generalization [10].
Moreover, two-stage detectors have specific module for feature-alignment (ROI-
Align), therefore leading them to more aligned features on unseen domains.
This explicit feature alignment is absent in CSP and on unseen domains leads
to failure cases, such as occlusion and small-scale pedestrians, where feature
alignment becomes pivotal for precise localization.

##  6 How Transformers Fare With Respect To CNN

In this section, we discuss the results when CNNs are pitched against the
recent Transformer Networks. Intuitively, transformer networks do outperform
CNN based backbones in direct and cross dataset evaluation. However, when the
domain gap is large and we increase the size of the training dataset along
with more sources of pre-training, we quantitatively illustrate that CNNs are
more robust to domain shifts and have better ability to digest data, compared
with transformer networks. To the best of our knowledge, this is one of the
first studies that objectively illustrates this.

In order to make comparison fair, we used the same detector, Cascade R-CNN, in
both backbone networks HRNet and Swin Transformer. We included Swin
Transformer for evaluation, since, it has achieved state of the art
performance on general object detection benchmarks. Therefore, we start by
benchmarking in Table X. It can be observed that Swin Transformer outperforms
HRNet on direct and cross-dataset evaluation. Which is intuitive as, Swin
Transformer also outperforms HRNet on general object detection as well, thanks
to the shited window based self attention which captures a more powerful
representation compared with a CNN based backbone such as the HRNet. However,
when the domain shift is large, sources of pre-training are fixed to
CrowdHuman and Wider Pedestrian and tested on autonomous driving oriented
datasets, we notice the Swin Transformer gets outperformed by HRNet
significantly, across all settings as shown in TableXI. This finding
illustrate that CNNs are more tolerant to domain shifts, especially, if the
shifts are not subtle. Inline with the studies in sections above, we test Swin
Transformer, through our progressive training pipeline. In table XII, we pre-
train on diverse general person detection dataset and finetune on ECP which is
closer to the target domain. Except for the third row, HRNet outperformed Swin
Tranformers on all datasets. This trend persists even when we add the target
set to the progressive training pipeline as shown in Table XIII. We attribute
this to the fact that potentially one of the main reasons are that the hyper-
parameters of transformer networks are not as optmized as they for CNNs.
Moreover, transformer networks are also more prone to overfitting compared
with the CNNs. Nonetheless, it is still relatively early for transformer
networks compared with CNNs, which have been around and used extensively for
nearly a decade and their hyperparameters are optimized quite thoroughly.

TABLE X: Cross dataset evaluation of HRNet and Swin-Trans. on Autonomous driving benchmarks. Both detectors are trained with Casc. R-CNN as a backbone. Method | Training | Testing | Reasonable | Small | Heavy
---|---|---|---|---|---
HRNet | CityPersons | CityPersons | 11.2 | 14.0 | 37.0
Swin-Trans | CityPersons | CityPersons | 9.2 | 10.5 | 36.9
HRNet | ECP | CityPersons | 10.9 | 11.4 | 40.9
Swin-Trans | ECP | CityPersons | 10.2 | 12.9 | 39.6
HRNet | ECP | ECP | 6.9 | 12.6 | 33.1
Swin-Trans | ECP | ECP | 4.5 | 9.6 | 25.2
HRNet | CityPersons | ECP | 17.4 | 40.5 | 49.3
Swin-Trans | CityPersons | ECP | 14.8 | 28.3 | 50.2
TABLE XI: Cross-dataset evaluation of HRNet and Swin-Trans., when sources of training are fixed to Wider Pedestrian and CrowdHuman Method | Training | Testing | Reasonable | Small | Heavy
---|---|---|---|---|---
HRNet | CrowdHuman | Caltech | 3.4 | 11.2 | 32.3
Swin-Trans | CrowdHuman | Caltech | 10.1 | 37.2 | 55.1
HRNet | CrowdHuman | CityPersons | 15.1 | 21.4 | 49.8
Swin-Trans | CrowdHuman | CityPersons | 16.7 | 24.3 | 55.0
HRNet | CrowdHuman | ECP | 17.9 | 36.5 | 56.9
Swin-Trans | CrowdHuman | ECP | 21.1 | 42.8 | 56.0
HRNet | Wider Pedestrian | Caltech | 3.2 | 10.8 | 31.7
Swin-Trans | Wider Pedestrian | Caltech | 9.8 | 30.1 | 55.6
HRNet | Wider Pedestrian | CityPersons | 16.0 | 21.6 | 57.4
Swin-Trans | Wider Pedestrian | CityPersons | 13.9 | 32.5 | 57.7
HRNet | Wider Pedestrian | ECP | 16.1 | 32.8 | 58.0
Swin-Trans | Wider Pedestrian | ECP | 18.1 | 32.5 | 65.8
TABLE XII: Investigating the effect on performance when CrowdHuman, Wider Pedestrian and ECP are merged and both backbones (HRNet and Swin-Trans.) are trained only on the merged dataset. Method | Training | Testing | Reasonable | Small | Heavy
---|---|---|---|---|---
HRNet | CrowdHuman $\rightarrow$ ECP | CP | 10.3 | 12.6 | 40.7
Swin Trans. | CrowdHuman $\rightarrow$ ECP | CP | 11.0 | 12.4 | 43.4
HRNet | Wider Pedestrian $\rightarrow$ ECP | CP | 9.7 | 11.8 | 37.7
Swin Trans. | Wider Pedestrian $\rightarrow$ ECP | CP | 9.5 | 10.8 | 39.7
HRNet | CrowdHuman $\rightarrow$ ECP | Caltech | 2.9 | 11.4 | 30.8
Swin Trans. | CrowdHuman $\rightarrow$ ECP | Caltech | 8.0 | 28.0 | 54.4
HRNet | Wider Pedestrian $\rightarrow$ ECP | Caltech | 2.5 | 9.9 | 31.0
Swin Trans. | Wider Pedestrian $\rightarrow$ ECP | Caltech | 8.8 | 28.1 | 33.9
TABLE XIII: Evaluation of HRNet and Swin-Trans. after fine-tuning on the target set. Method | Training | Testing | Reasonable | Small | Heavy
---|---|---|---|---|---
HRNet | CrowdHuman $\rightarrow$ ECP $\rightarrow$ CP | CP | 8.0 | 8.5 | 27.0
SwinTrans. | CrowdHuman $\rightarrow$ ECP $\rightarrow$ CP | CP | 9.1 | 10.0 | 30.9
HRNet | Wider Pedestrian $\rightarrow$ ECP $\rightarrow$ CP | CP | 7.5 | 8.0 | 28.0
SwinTrans. | Wider Pedestrian $\rightarrow$ ECP $\rightarrow$ CP | CP | 8.9 | 10.4 | 33.8

##  7 Finetuning on the Target Set

Finally, we add the training part of our target set to our progressive
training pipeline as illustrated in Table XIV. Although this paper stresses
the importance of cross-dataset evaluation, for the sake of completeness, we
also include results for target set fine-tuning. Visible improvement across
all splits can be observed for both methods. Interestingly, on CityPersons,
CSP gets better than Cascade RCNN. We attribute this to the fact that CSP’s
design choice is optimized for CityPersons (at the cost of generalization).
Therefore, it benefits more than a general design object detector for target
set fine-tuning on CityPersons. However, both methods are still comparable and
in the case of Caltech, Cascade R-CNN outperforms CSP significantly. Moreover,
the performance on Caltech is nearly in the same order, as that of humans
(0.88). This also brings to the conclusion that Caltech dataset is almost
solved. Next generation pedestrian detectors should use Caltech as a
reference, but focus on more challenging dataset such as ECP and CityPersons.
Importantly, general object detector such as, Cascade RCNN, without fine-
tuning on the target set (cf. Table VIII), can already achieve comparable
results to that with fine-tuning as in Table XIV, making it practical, ready
to use and more suitable to real-world problems.

TABLE XIV: After fine-tuning on the target set. Method | Training | Testing | Reasonable | Small | Heavy
---|---|---|---|---|---
Casc. RCNN | CrowdHuman $\rightarrow$ ECP $\rightarrow$ CP | CP | 8.0 | 8.5 | 27.0
CSP | CrowdHuman $\rightarrow$ ECP $\rightarrow$ CP | CP | 8.1 | 8.9 | 29.3
Casc. RCNN | Wider Pedestrian $\rightarrow$ ECP $\rightarrow$ CP | CP | 7.5 | 8.0 | 28.0
CSP | Wider Pedestrian $\rightarrow$ ECP $\rightarrow$ CP | CP | 7.1 | 7.8 | 26.9
Casc. RCNN | CrowdHuman $\rightarrow$ ECP $\rightarrow$ Caltech | Caltech | 2.2 | 8.1 | 30.7
CSP | CrowdHuman $\rightarrow$ ECP $\rightarrow$ Caltech | Caltech | 3.8 | 9.2 | 31.2
Casc. RCNN | Wider Pedestrian $\rightarrow$ ECP $\rightarrow$ Caltech | Caltech | 1.7 | 7.2 | 25.7
CSP | Wider Pedestrian $\rightarrow$ ECP $\rightarrow$ Caltech | Caltech | 2.4 | 7.5 | 30.3

###  7.1 Quantitative Results on Leaderboard

We further evaluated proposed training pipeline along with ensemble of our two
models, one pre-trained on CrowdHuman [39] and the other one on Wider
Pedestrian [31]. Ensembling is performed by combining the detections followed
by non-maxima suppression, using soft-nms [3]. Final results are evaluated on
the dedicated server333https://github.com/cvgroup-njust/CityPersons
https://eurocity-dataset.tudelft.nl/eval/benchmarks/detection
$\ddagger$: Correspond to our submissions and use of additional training data
(test set annotations are withheld) of CityPersons [52] and ECP [4],
maintained by the benchmark publishers and frequency of submissions are
constraint. Moreover, we have included results only for the published methods
(detailed evaluations of all methods can be seen on the urls provided in the
_footnote_ 3). Results are presented in Table XV and XVI. Our submission
(Cascade RCNN) achieves 1st and 2nd on both leaderboards respectively. These
results serve as a reference for future methods. However, no other method to
the best of our knowledge uses extra training data. Therefore, giving our
submissions an unfair advantage. Finally, as stated above, fine-tuning on
target set is not the goal of the paper and in many cases it is not practical.
In this work, we argue in the favor of cross-dataset evaluation and its
importance.

TABLE XV: Results on the withheld Test Set of CityPersons.

Methods | Reasonable | Small | Heavy
---|---|---|---
MS-CNN [6] | 13.32 | 15.86 | 51.88
FRCNN [52] | 12.97 | 37.24 | 50.47
Cascade MS-CNN [8] | 11.62 | 13.64 | 47.14
Repulsion Loss [45] | 11.48 | 5.67 | 52.59
AdaptiveNMS [25] | 11.40 | 13.64 | 46.99
OR-CNN [53] | 11.32 | 14.19 | 51.43
HBA-RCNN [32] | 11.26 | 15.68 | 43.7
MGAN [37] | 9.29 | 11.38 | 40.97
CrowdHuman $\rightarrow$ECP$\rightarrow$CP $\ddagger$ | 8.35 | 9.85 | 27.87
WiderPerson$\rightarrow$ECP$\rightarrow$CP $\ddagger$ | 8.31 | 10.23 | 28.18
Ensemble $\ddagger$ | 7.69 | 9.16 | 27.08
APD-Pretrain [49] | 7.3 | 10.8 | 28.0

TABLE XVI: Results on the withheld Test Set of ECP.

Training | Reasonable | Small | Heavy
---|---|---|---
R-FCN (with OHEM) [4] | 16.3 | 24.5 | 50.7
SSD [4] | 13.1 | 23.5 | 46.0
Faster R-CNN [4] | 10.1 | 19.6 | 38.1
YOLOv3 [4] | 9.7 | 18.6 | 40.1
CrowdHuman$\rightarrow$ECP $\ddagger$ | 6.6 | 13.6 | 31.3
WiderPerson$\rightarrow$ECP $\ddagger$ | 5.5 | 11.7 | 26.1
APD-Pretrain [49] | 5.3 | 12 .4 | 26.8
Ensemble $\ddagger$ | 5.1 | 11.2 | 25.4
SPNet [22] | 4.2 | 9.5 | 21.6

##  8 Conclusions

In view of the recent developments in the field of pedestrian detection on
existing benchmarks for autonomous driving. We thoroughly investigated and
evaluated existing state of the art pedestrian detectors, moreover, assessed
their practicality and generalization using cross-dataset evaluation as an
evaluation protocol. Through this study, we have pointed out an important, but
often overlooked conclusion, that is, the current pedestrian detectors lack
generalisation and do not handle well even small domain shifts. We have
attributed lack of generalization to the fact that current pedestrian
detectors are tailored towards target datasets, and their architectural design
contains strong biasness towards the target domain, at the cost of
generalisation. Therefore, we have illustrated that general object detectors,
are more robust and are better at handling domain shifts due to generic
design. Besides detectors, we also investigated the role of backbone-network
architectures in generalization. We have compared the generalization ability
of convolutional neural networks and the recent transformer networks, using
the same detection architecture (Cascade R-CNN). We have shown that, although
transformers out perform CNNs in direct dataset evaluation, still, the
generalization ability of CNNs is better than that of transformer networks.
Moreover, CNNs have shown to be better at digesting largescale datasets than
transformers, potentially due to the fact that CNN’s hyper-parameters are
thoroughly optimized over decade of rigorous research, and it is still
relatively early days for transformer networks. Additionally, we have proposed
a progressive-training pipeline in this paper, where we have shown that
benchmarks diverse in scenes and dense in pedestrians, serves to be an
efficient source of pre-training, especially in the context of autonomous
driving, where most benchmarks are monotonous in nature. Finally, findings
presented in this paper can inspire the next generation of pedestrian
detectors, which are more generic, practical and are ready to use.

## References

  * [1] A. Angelova, A. Krizhevsky, V. Vanhoucke, A. Ogale, and D. Ferguson.  Real-time pedestrian detection with deep network cascades.  2015\.
  * [2] B. Benfold and I. Reid.  Stable multi-target tracking in real-time surveillance video.  In CVPR 2011, pages 3457–3464. IEEE, 2011.
  * [3] N. Bodla, B. Singh, R. Chellappa, and L. S. Davis.  Soft-nms–improving object detection with one line of code.  In Proceedings of the IEEE international conference on computer vision, pages 5561–5569, 2017.
  * [4] M. Braun, S. Krebs, F. Flohr, and D. M. Gavrila.  Eurocity persons: A novel benchmark for person detection in traffic scenes.  IEEE transactions on pattern analysis and machine intelligence, 41(8):1844–1861, 2019.
  * [5] G. Brazil, X. Yin, and X. Liu.  Illuminating pedestrians via simultaneous detection & segmentation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4950–4959, 2017.
  * [6] Z. Cai, Q. Fan, R. S. Feris, and N. Vasconcelos.  A unified multi-scale deep convolutional neural network for fast object detection.  In European conference on computer vision, pages 354–370. Springer, 2016.
  * [7] Z. Cai, M. Saberian, and N. Vasconcelos.  Learning complexity-aware cascades for deep pedestrian detection.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3361–3369, 2015.
  * [8] Z. Cai and N. Vasconcelos.  Cascade r-cnn: High quality object detection and instance segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [9] V. Campmany, S. Silva, A. Espinosa, J. C. Moure, D. Vázquez, and A. M. López.  Gpu-based pedestrian detection for autonomous driving.  Procedia Computer Science, 80:2377–2381, 2016.
  * [10] C. Chen, Q. Dou, H. Chen, J. Qin, and P.-A. Heng.  Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 865–872, 2019.
  * [11] N. Dalal and B. Triggs.  Histograms of oriented gradients for human detection.  In international Conference on computer vision & Pattern Recognition (CVPR’05), volume 1, pages 886–893. IEEE Computer Society, 2005\.
  * [12] P. Dollár, R. Appel, S. Belongie, and P. Perona.  Fast feature pyramids for object detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(8):1532–1545, 2014.
  * [13] P. Dollar, C. Wojek, B. Schiele, and P. Perona.  Pedestrian detection: An evaluation of the state of the art.  IEEE transactions on pattern analysis and machine intelligence, 34(4):743–761, 2012.
  * [14] A. Ess, B. Leibe, and L. Van Gool.  Depth and appearance for mobile scene analysis.  In 2007 IEEE 11th international conference on computer vision, pages 1–8. IEEE, 2007.
  * [15] A. Geiger, P. Lenz, and R. Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3354–3361. IEEE, 2012.
  * [16] R. Girshick, J. Donahue, T. Darrell, and J. Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
  * [17] H. Hattori, V. Naresh Boddeti, K. M. Kitani, and T. Kanade.  Learning scene-specific pedestrian detectors without real data.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3819–3827, 2015.
  * [18] A. Hbaieb, J. Rezgui, and L. Chaari.  Pedestrian detection for autonomous driving within cooperative communication system.  In 2019 IEEE Wireless Communications and Networking Conference (WCNC), pages 1–6. IEEE, 2019.
  * [19] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [20] J. Hosang, M. Omran, R. Benenson, and B. Schiele.  Taking a deeper look at pedestrians.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4073–4082, 2015.
  * [21] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [22] C. Jiang, H. Xu, W. Zhang, X. Liang, and Z. Li.  Sp-nas: Serial-to-parallel backbone search for object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11863–11872, 2020.
  * [23] J. Li, S. Liao, H. Jiang, and L. Shao.  Box guided convolution for pedestrian detection.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 1615–1624, 2020.
  * [24] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [25] S. Liu, D. Huang, and Y. Wang.  Adaptive nms: Refining pedestrian detection in a crowd.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6459–6468, 2019.
  * [26] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg.  Ssd: Single shot multibox detector.  In European conference on computer vision, pages 21–37. Springer, 2016.
  * [27] W. Liu, I. Hasan, and S. Liao.  Center and scale prediction: A box-free approach for pedestrian and face detection.  arXiv preprint arXiv:1904.02948, 2019.
  * [28] W. Liu, S. Liao, W. Hu, X. Liang, and X. Chen.  Learning efficient single-stage pedestrian detectors by asymptotic localization fitting.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 618–634, 2018.
  * [29] W. Liu, S. Liao, W. Ren, W. Hu, and Y. Yu.  High-level semantic feature detection: A new perspective for pedestrian detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [30] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  arXiv preprint arXiv:2103.14030, 2021.
  * [31] C. C. Loy, D. Lin, W. Ouyang, Y. Xiong, S. Yang, Q. Huang, D. Zhou, W. Xia, Q. Li, P. Luo, et al.  Wider face and pedestrian challenge 2018: Methods and results.  arXiv preprint arXiv:1902.06854, 2019.
  * [32] R. Lu and H. Ma.  Semantic head enhanced pedestrian detection in a crowd.  arXiv preprint arXiv:1911.11985, 2019.
  * [33] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten.  Exploring the limits of weakly supervised pretraining.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 181–196, 2018.
  * [34] J. Mao, T. Xiao, Y. Jiang, and Z. Cao.  What can help pedestrian detection?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3127–3136, 2017.
  * [35] S. Munder and D. M. Gavrila.  An experimental study on pedestrian classification.  IEEE transactions on pattern analysis and machine intelligence, 28(11):1863–1868, 2006.
  * [36] S. Paisitkriangkrai, C. Shen, and A. Van Den Hengel.  Strengthening the effectiveness of pedestrian detection with spatially pooled features.  In European Conference on Computer Vision, pages 546–561. Springer, 2014.
  * [37] Y. Pang, J. Xie, M. H. Khan, R. M. Anwer, F. S. Khan, and L. Shao.  Mask-guided attention network for occluded pedestrian detection, 2019\.
  * [38] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [39] S. Shao, Z. Zhao, B. Li, T. Xiao, G. Yu, X. Zhang, and J. Sun.  Crowdhuman: A benchmark for detecting human in a crowd.  arXiv preprint arXiv:1805.00123, 2018.
  * [40] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [41] X. Song, K. Zhao, W.-S. C. H. Zhang, and J. Guo.  Progressive refinement network for occluded pedestrian detection.  In Proc. European Conference on Computer Vision, volume 7, page 9, 2020.
  * [42] S. Sun, J. Pang, J. Shi, S. Yi, and W. Ouyang.  Fishnet: A versatile backbone for image, region, and pixel level prediction.  In Advances in Neural Information Processing Systems, pages 760–770, 2018.
  * [43] P. Viola and M. J. Jones.  Robust real-time face detection.  International journal of computer vision, 57(2):137–154, 2004.
  * [44] J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang, et al.  Deep high-resolution representation learning for visual recognition.  arXiv preprint arXiv:1908.07919, 2019.
  * [45] X. Wang, T. Xiao, Y. Jiang, S. Shao, J. Sun, and C. Shen.  Repulsion loss: detecting pedestrians in a crowd.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7774–7783, 2018.
  * [46] C. Wojek, S. Walk, and B. Schiele.  Multi-cue onboard pedestrian detection.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 794–801. IEEE, 2009.
  * [47] B. Wu and R. Nevatia.  Cluster boosted tree classifier for multi-view, multi-pose object detection.  In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8. IEEE, 2007.
  * [48] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [49] J. Zhang, L. Lin, J. Zhu, Y. Li, Y.-c. Chen, Y. Hu, and C. S. Hoi.  Attribute-aware pedestrian detection in a crowd.  IEEE Transactions on Multimedia, 2020.
  * [50] L. Zhang, L. Lin, X. Liang, and K. He.  Is faster r-cnn doing well for pedestrian detection?  In European conference on computer vision, pages 443–457. Springer, 2016.
  * [51] S. Zhang, R. Benenson, M. Omran, J. Hosang, and B. Schiele.  How far are we from solving pedestrian detection?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1259–1267, 2016.
  * [52] S. Zhang, R. Benenson, and B. Schiele.  Citypersons: A diverse dataset for pedestrian detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3213–3221, 2017.
  * [53] S. Zhang, L. Wen, X. Bian, Z. Lei, and S. Z. Li.  Occlusion-aware r-cnn: detecting pedestrians in a crowd.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 637–653, 2018.
  * [54] S. Zhang, Y. Xie, J. Wan, H. Xia, S. Z. Li, and G. Guo.  Widerperson: A diverse dataset for dense pedestrian detection in the wild.  IEEE Transactions on Multimedia, 2019.
  * [55] C. Zhou and J. Yuan.  Bi-box regression for pedestrian detection and occlusion estimation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 135–151, 2018.