Alexandra<EMAIL_ADDRESS>Maor<EMAIL_ADDRESS>Yotam<EMAIL_ADDRESS>Ran<EMAIL_ADDRESS>Tomer
<EMAIL_ADDRESS>Roy J<EMAIL_ADDRESS>Samsung
Israel R&D Center
Tel Aviv
Israel
BLT-net

# You Better Look Twice: a new perspective for designing accurate detectors
with reduced computations

###### Abstract

General object detectors use powerful backbones that uniformly extract
features from images for enabling detection of a vast amount of object types.
However, utilization of such backbones in object detection applications
developed for specific object types can unnecessarily over-process an
extensive amount of background. In addition, they are agnostic to object
scales, thus redundantly process all image regions at the same resolution. In
this work we introduce BLT-net, a new low-computation two-stage object
detection architecture designed to process images with a significant amount of
background and objects of variate scales. BLT-net reduces computations by
separating objects from background using a very lite first-stage. BLT-net then
efficiently merges obtained proposals to further decrease processed background
and then dynamically reduces their resolution to minimize computations.
Resulting image proposals are then processed in the second-stage by a highly
accurate model. We demonstrate our architecture on the pedestrian detection
problem, where objects are of different sizes, images are of high resolution
and object detection is required to run in real-time. We show that our design
reduces computations by a factor of $\times$4-$\times$7 on the Citypersons and
Caltech datasets with respect to leading pedestrian detectors, on account of a
small accuracy degradation. This method can be applied on other object
detection applications in scenes with a considerable amount of background and
variate object sizes to reduce computations.

##  1 Introduction

Object detection is a fundamental computer vision task with an abundant number
of applications. General object detection architectures such as one-stage and
two-stage detectors are designed to enable detection of a vast amount of
object types in an image, without making any prior assumptions regarding
objects’ overall image coverage and amount of background. In such setups, it
is therefore more computations-wise efficient to uniformly extract features
once from the entire image. Over the years, these approaches achieved
impressive detection accuracy results on general datasets such as PASCAL VOC
[Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and Zisserman]
and COCO [Lin et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár,
and Zitnick].

Nonetheless, a variety of real-world applications are developed for detecting
specific object classes, such as pedestrians and vehicles in autonomous
driving [Zhang et al.(2017)Zhang, Benenson, and Schiele, Dollár et
al.(2009)Dollár, Wojek, Schiele, and Perona, Geiger et al.(2012)Geiger, Lenz,
and Urtasun, Braun et al.(2019)Braun, Krebs, Flohr, and Gavrila], buildings
and roads in satellite images [Christie et al.(2018)Christie, Fendley, Wilson,
and Mukherjee], humans and vehicles in aerial images [Xia et al.(2018)Xia,
Bai, Ding, Zhu, Belongie, Luo, Datcu, Pelillo, and Zhang], faces in
surveillance, auto-focus and tagging applications [Jain and Learned-
Miller(2010), Zhu and Ramanan(2012), Yang et al.(2016)Yang, Luo, Loy, and
Tang] to name a few. In such scenarios, a non-negligible amount of image
regions can be potentially referred to as background. These regions, when
detected at a low computational cost and discarded from further processing,
could significantly reduce the number of computations required for rich
features extraction needed for accurate object detection. In addition, low-
computation estimation of objects scale can be utilized to significantly
reduce the resolution of candidate image regions, thus further decreasing the
overall computations of the detector.

In this work we revisit the two-stage architecture to better utilize both
object image coverage and objects scale. To this end, we suggest BLT-net, a
two-stage architecture. BLT-net uses a very lite first-stage architecture to
estimate objects location and their scale and a second-stage that processes
these image proposals for accurate object detection. To further decrease
computations of the second-stage, we aim to dynamically produce an amount of
proposals that is highly similar to the number of relevant objects in the
image, with minimal overlap between proposals. In the second-stage, instead of
directly extracting rich features for accurate object detection on proposals
at full image resolution, we propose downscaling them to the minimum possible
resolution to significantly reduce calculations of this stage.

Figure 1: Accuracy (measured by MR-2, lower is better)/computations of all
pedestrian detectors under 1012 FLOPs for the Citypersons reasonable
validation dataset (2048$\times$1024 px). The dashed blue line depicts the
existing empiric Pareto frontier, green arrows show accuracy/computations
change after detectors were integrated into the BLT-net architecture (red
triangles), creating the new empiric Pareto frontier (dot-dashed red line).

For the second-stage one can develop an accurate detector to process image
crops. General [Cai and Vasconcelos(2018), Tan et al.(2020)Tan, Pang, and Le,
Wang et al.(2020a)Wang, Bochkovskiy, and Liao] and specialized object
detectors [Najibi et al.(2019)Najibi, Singh, and Davis, Wang et al.(2018)Wang,
Xiao, Jiang, Shao, Sun, and Shen, Gao et al.(2018)Gao, Yu, Li, Morariu, and
Davis, Liu et al.(2019b)Liu, Liao, Ren, Hu, and Yu, Uzkent et al.(2020)Uzkent,
Yeh, and Ermon, Yang et al.(2019)Yang, Fan, Chu, Blasch, and Ling] were shown
to achieve high detection accuracy on full-resolution images. We build upon
the success of these detectors and apply them directly on our first-stage
proposals, with no additional fine-tuning.

In this work we demonstrate both empirically and analytically the computations
advantages of utilizing both image coverage and objects scaling estimation
with respect to existing object detection architectures. In addition, this
suggested meta-architecture can also be used to reduce computations of
existing object detectors that are trained to detect objects in scenes with a
significant amount of background, to potentially reduce their computations and
thus shortening their inference time [Reuther et al.(2019)Reuther, Michaleas,
Jones, Gadepally, Samsi, and Kepner] or enabling them to run on weaker
hardware [Ignatov et al.(2019)Ignatov, Timofte, Kulik, Yang, Wang, Baum, Wu,
Xu, and Van Gool].

To demonstrate the effectiveness of our approach, in this work we selected to
focus on detecting pedestrians in driving scenes, a popular use-case that is
highly sensitive to computational load [Lin et al.(2018)Lin, Zhang, Hsu,
Skach, Haque, Tang, and Mars]. This object detection application can benefit
tremendously from computational cost reductions due to its requirement to
process high-resolution input images for detecting objects at various
distances (and thus of various sizes) in real-time [Lin et al.(2018)Lin,
Zhang, Hsu, Skach, Haque, Tang, and Mars]. Our experiments cover the popular
Citypersons [Zhang et al.(2017)Zhang, Benenson, and Schiele] and Caltech
[Dollár et al.(2009)Dollár, Wojek, Schiele, and Perona] datasets. As seen in
Figure 1, our architecture was able to reduce the number of FLOPs111FLOPs
measure mult-add operations  by a factor of $\times$4-$\times$7 with a small
accuracy (MR-2) degradation of 1.4%-2.3$\%$ of several leading pedestrian
detectors.

In summary, the main contributions of our work are as follows:

  * •

A new perspective of the two-stage object detection architecture designed to
better utilize image object coverage and object scales.

  * •

A novel method for decreasing computations of existing object detectors for
scenes with a significant amount of background with objects of possible
variate scales.

  * •

Theoretical and quantitative analysis of computation reduction aspects of our
suggested architecture on the Citypersons and Caltech datasets.

##  2 Related work

Generally, one-stage or two-stage architectures can be applied to solve
various specific object detection use-cases, such as face detection, cars and
humans in aerial images, crops in agricultural images, pedestrians and
vehicles in self-driving cars. To reduce latency and computations and in some
cases to increase detection accuracy, over the years architectures were
modified or redesigned to better utilize characteristics of the scenery and
detected objects, such as foreground/background ratio, objects location and
scale distributions.

For example, background region removal for improved computations and latency
was demonstrated for pedestrian detection on the Caltech dataset using
reinforcement learning techniques [Gao et al.(2018)Gao, Yu, Li, Morariu, and
Davis, Uzkent et al.(2020)Uzkent, Yeh, and Ermon]. In these solutions, the
first-step processed downscaled images by a factor of $\times 2-\times 5$ to
detect candidate regions to be processed in the fine-step at their full
resolution. The same detection architecture was used in both steps. A high
downscaling factor of $\times 5$ resulted in a high number of proposed regions
that were processed by the second stage to compensate for possible misses.
Overall this approach processed $\sim 40\%$ of pixels for both downscaling
factors. In our work we propose an opposite approach: we process in the first-
stage full-resolution images by a very lite detector to enable detection of
even small objects at their full image-resolution and then use objects’
scaling estimation to reduce image resolution of proposed regions to
significantly reduce computations of the accurate detector used in the second-
stage, overall achieving a higher computations reduction factor.

Another approach commonly used for discarding background regions processes
images using a cascade of two or more classical [Viola and Jones(2001)] or CNN
based steps [Li et al.(2015)Li, Lin, Shen, Brandt, and Hua, Yang and
Nevatia(2016), Najibi et al.(2019)Najibi, Singh, and Davis]. This method was
successfully applied for detecting faces [Li et al.(2015)Li, Lin, Shen,
Brandt, and Hua, Yang and Nevatia(2016)] or general objects [Najibi et
al.(2019)Najibi, Singh, and Davis]. To further reduce computations of later
steps, some works united overlapping proposals [Chen et al.(2016)Chen, Hua,
Wen, and Sun, Najibi et al.(2019)Najibi, Singh, and Davis, Yang et
al.(2019)Yang, Fan, Chu, Blasch, and Ling], while others designed their first
step architecture to detect regions containing possible clusters of objects
[Li et al.(2020)Li, Yang, Zhu, Chen, and Guan, Royer and Lampert(2020)].
Nevertheless, these works did not optimize object clustering with respect to
their background regions, which can be further reduced to decrease
computations. In our work we demonstrate that a good scaling estimation of
singular objects in the first-stage combined with a cost-efficient area
merging criterion can significantly discard additional background areas, thus
directly reducing computations of the second-stage detector. In addition, we
show that in scenarios where object sizes significantly vary, it is highly
beneficial to downscale obtained image proposals, to further decrease
computations of the later stage.

In this work we demonstrate our suggested meta-architecture on the
computations-sensitive pedestrian detection problem. The majority of
pedestrian detectors are based on one-stage or two-stage architectures. One
group of works adopted the accurate Faster R-CNN architecture [Ren et
al.(2015)Ren, He, Girshick, and Sun] and introduced various adjustments to the
common scheme. RepLoss [Wang et al.(2018)Wang, Xiao, Jiang, Shao, Sun, and
Shen] altered the training loss, AdaptiveNMS [Liu et al.(2019a)Liu, Huang, and
Wang] refined the NMS step, MGAN [Pang et al.(2019)Pang, Xie, Khan, Anwer,
Khan, and Shao] and PSC-Net [Xie et al.(2020)Xie, Pang, Cholakkal, Anwer,
Khan, and Shao] added additional branches, PedHunter [Chi et al.(2020)Chi,
Zhang, Xing, Lei, Li, Zou, et al.] introduced a masked guided module for
improved detections in crowded scenes, while Pedestron [Hasan et
al.(2020)Hasan, Liao, Li, Akram, and Shao] and CrowdHuman [Shao et
al.(2018)Shao, Zhao, Li, Xiao, Yu, Zhang, and Sun] trained models on extensive
datasets. Other works also adopted single-stage architectures developed for
general object detection such as SSD [Liu et al.(2016)Liu, Anguelov, Erhan,
Szegedy, Reed, Fu, and Berg], YOLO [Redmon et al.(2016)Redmon, Divvala,
Girshick, and Farhadi, Redmon and Farhadi(2017)] and others [Dai et
al.(2016)Dai, Li, He, and Sun, Lin et al.(2017)Lin, Dollár, Girshick, He,
Hariharan, and Belongie] to potentially decrease inference latency and
computations [Huang et al.(2017)Huang, Rathod, Sun, Zhu, Korattikara, Fathi,
Fischer, Wojna, Song, Guadarrama, et al.]. These include ALFNet [Liu et
al.(2018)Liu, Liao, Hu, Liang, and Chen], CSP [Liu et al.(2019b)Liu, Liao,
Ren, Hu, and Yu], APD [Zhang et al.(2019a)Zhang, Lin, Li, Chen, Zhu, Hu, and
Hoi] that use various backbones, loss terms, training dastasets and anchor
based/free approaches. Overall, these detectors use general backbones for
feature extraction that result in an increased amount of computations, as seen
in Figure 1 and Supplementary Material. In this work we will demonstrate how
our proposed architecture enables computation reduction of such existing
designs, significantly improving the Pareto frontier.

##  3 BLT-net

The BLT-net architecture consists of two stages. The first stage is designed
to differentiate objects of interest from the background and estimate their
scale using a very lite CNN applied on full-resolution input images. Since
objects may overlap each other, same image regions (ROIs) could be
unnecessarily processed several times by the second-stage. To this end,
overlapping ROIs are merged into mROIs using a cost-effective area criterion.
mROIs are then adaptively downscaled using objects’ predicted scale to further
reduce computations. Downscaled mROIs are fed into the second-stage that
consists of an object detector that was trained independently to achieve high
detection accuracy. An overview of the suggested architecture and comparison
to Faster R-CNN [Ren et al.(2015)Ren, He, Girshick, and Sun] is described in
Figure 2.

Let us formulate the number of computations of our approach and analyze the
computations reduction factor with respect to an existing architecture
integrated into our second-stage. Specifically, let us denote the number of
FLOPs per pixel of a general object detector by $A$ and the number of
processed pixels in the original image by $N$. Thus the overall computations
of a CNN based model are:

$FLOPs(OD)=A\cdot N$ (1)

Let us denote the number of FLOPs per pixel of a lite CNN first-stage
architecture by $B$ and the total number of pixels in the resulting downscaled
mROIs by $M$. Thus the total number of FLOPs in our suggested architecture
when incorporating an existing object detector in the second stage is:

$FLOPs(BLTnet)=B\cdot N+A\cdot M$ (2)

The FLOPs reduction factor of BLT-net is, therefore:

$1/FLOPs_{reduction\\_factor}=(B\cdot N+A\cdot M)/(A\cdot N)=B/A+M/N$ (3)

Eq. 3 indicates that the number of computations could be decreased by reducing
both the $B/A$ and $M/N$ factors. In this work we suggest a highly efficient
first-stage CNN architecture to reduce the $B/A$ factor and a novel ROI
merging and content-adaptive scaling algorithm to further reduce the $M/N$
factor.

Figure 2: BLT-net vs. the Faster-RCNN two-stage architecture. The first-stage
is replaced by a lite-CNN detector. The ROI pooling module is replaced by the
PCMAD algorithm to merge and downscale image proposals. The second-stage head
is replaced by an accurate detector that independently processes the merged
downscaled proposals (mROIs).

###  3.1 First-stage design

Ideally, we aim at designing a first-stage detector with a high sensitivity
and a minimal false-positive rate to reduce the number of processed pixels by
the second-stage. More formally, let us define two metrics to evaluate the
performance of the first-stage detector:

1\. Sensitivity: the percentage of ground truth bounding boxes in the dataset
for which at least $k$% of their area is covered by mROIs. $k$% should be set
high enough to ensure that adequate image object information is provided to
the second-stage for successful detection.

2\. Relative processed area ($\overline{M}/N$): the average number of pixels
in the dataset processed by the second-stage, relative to the full-resolution
image.

In this work we experimented with two different lite-CNN detection
architectures to meet the criteria, one anchor-free and the other anchor-based
(see Figure 3). First, we denote the anchor-free architecture by Center &
Scale net (C&S). The backbone of this architecture consists of two branches
that process the image at its original resolution (shallower branch) and at
half of the original resolution (deeper branch), a concept previously used [Yu
et al.(2018)Yu, Wang, Peng, Gao, Yu, and Sang, Zhao et al.(2018)Zhao, Qi,
Shen, Shi, and Jia]. The output resolution of the shallow/deep branches was
set to 1/8 and 1/16 respectively. The deep resolution branch output was
upsampled to 1/8 of the original image resolution and fused to the output of
the shallow branch output. The network heads output center-likelihood and
object-scale maps, similarly to Liu _et al_[Liu et al.(2019b)Liu, Liao, Ren,
Hu, and Yu] . The C&S net was trained on the Citypersons dataset with crops of
size 768$\times$768 px and a batch size of 8. We used random scaling in the
range [0.5, 2], random uniform cropping and random mirroring with a
probability of 0.5. The model was trained for 32 epochs with a learning rate
of $10^{-4}$ and fine-tuned for an additional 350 epochs with a learning rate
of $10^{-5}$. While current pedestrian detectors implementations are in the
range of 262-867 GFLOPs for detectors with MR-2<10% (see Supplementary
Material), C&S has 27.5 GFLOPs, achieving a maximum ratio $B/A$ of 0.1.

Second, we developed an anchor-based first-stage detector, named CascadeMV2.
This detector uses a Mobilenet V2 [Sandler et al.(2018)Sandler, Howard, Zhu,
Zhmoginov, and Chen] backbone for feature extraction with a cascade head for
improved accuracy [Cai and Vasconcelos(2018), Liu et al.(2018)Liu, Liao, Hu,
Liang, and Chen]. In this work the detection head was designed using separable
convolutions [Howard et al.(2017)Howard, Zhu, Chen, Kalenichenko, Wang,
Weyand, Andreetto, and Adam] to reduce computations. Losses were set similarly
to anchor-based object detectors [Liu et al.(2018)Liu, Liao, Hu, Liang, and
Chen]. The resulting architecture was trained for 350 epochs on the
Citypersons training dataset with a learning rate of $10^{-4}$ and fine-tuned
for an additional 100 epochs with a learning rate of $10^{-5}$. For the
Caltech dataset, the model trained on the Citypersons dataset was fine-tuned
for an additional 100 epochs with a learning rate of $10^{-5}$. Overall, this
detector has 25 GFLOPs for images with a resolution of 2048$\times$1024 px,
similarly to the C&S net. For comparison purposes, the original Cascade R-CNN
[Cai and Vasconcelos(2018)] has 262 GFLOPs and ALFNet [Liu et al.(2018)Liu,
Liao, Hu, Liang, and Chen] has 171 GFLOPs for the same image resolution.

Our experiments with designing lite CNNs demonstrated that various
architectures could be developed to meet the first-stage criteria, thus we
argue that such architecture is not unique and can be constructed using
multiple variants or fully automated using NAS methods [Elsken et
al.(2019)Elsken, Metzen, Hutter, et al.].

Algorithm 1 PCMAD

Input: $props_{all}$, $heightRanges$. Init: Empty($mROI_{all}$)

for $hRange$ in $heightRanges$ do

$props$ = $props_{all}\in hRange$; $Q,mROIs$=Init($props$)

while not empty($Q$) do

($k$,$j$)=pop($Q$), s.t. min($dist_{x}$($prop_{k}$, $prop_{j}$))

if area($ROI_{kj}$)<area($ROI_{j}$)+area($ROI_{k}$) then

$ROI_{j}$=extended(BBox($prop_{k}$, $prop_{j}$))

Update pairs in $Q$ with updated $ROI_{j}$

if cannot pair $ROI_{j}$ to $ROIs\in Q$ then $ROI_{j}$ $\to$ $mROIs$

$mROIs\to mROI_{all}$

return $mROI_{all}$

procedure Init($props$)

Empty($Q$), Empty($mROIs$)

for each prop $k$,$j$ in $props$ do

$ROI_{k}$,$ROI_{j}$=extended($prop_{k}$),extended($prop_{j}$)

$ROI_{k,j}$=extended(BBox($prop_{k}$, $prop_{j}$))

if area($ROI_{kj}$)<area($ROI_{j}$)+area($ROI_{k}$) then

($k$,$j$) $\to$ $Q$

$\triangleleft$ if $prop_{k}$ is not paired to any other $prop_{j}$

for each $prop_{k}$ in $props$ do

if ($k$, ?) $\notin$ $Q$ then $ROI_{k}$ $\to$ $mROIs$

Return $Q$, $mROIs$

Figure 3: First-stage components. (a) Two possible lite-CNN architectures used
for the first-stage (b) PCMAD pseudo-code.

###  3.2 The PCMAD algorithm

To increase objects’ context and compensate for possible localization and
scaling errors of the first-stage, object proposals were extended by $p$% in
each direction, creating ROIs. To avoid redundant processing of overlapping
ROIs by the second-stage, we developed a proposals content aware merging and
adptive downscaling algorithm (PCMAD) to produce a set of ROIs with minimal
overlapping area. This hierarchical clustering algorithm iteratively merges
overlapping proposals with similar heights if the sum of their ROIs area is
larger than their union area (united proposals after extension). The proposal
pair with the smallest horizontal distance between its centers is selected for
merging first.

The resulting mROIs contain objects of similar heights, all with the original
image resolution. However, high-resolution mROIs may be redundant when
processing large objects for accurate detection, thus to further reduce
computations we conduct mROIs adaptive scaling to a height $h$. This scaling
imposes merging object proposals with heights in a specific range only,
ensuring that mROIs containing smaller objects are downscaled by a smaller
factor than mROIs containing larger objects. In this work height ranges were
set to $[h,1.5h)$, $[2h,2.5h)$, $[2.5h,3h)$ and $[3.5h,4h)$. The $h$ parameter
is determined by the second-stage detector minimum resolution requirements for
achieving the required detection accuracy. PCMAD complexity is $O(P^{3})$ with
respect to the number of proposals $P$. PCMAD pseudocode is shown in Figure 3.

###  3.3 Second-stage architecture

Generally, the second-stage architecture could be developed from scratch to
learn to accurately detect objects from mROIs or one can use existing pre-
trained detectors. As current pedestrian detectors achieve highly accurate
results on the border of saturation for the Citypersons and Caltech datasets,
we opted for integrating such leading pedestrian detectors for the second-
stage. For the Citypersons dataset we integrated Pedestron [Hasan et
al.(2020)Hasan, Liao, Li, Akram, and Shao], APD [Zhang et al.(2019a)Zhang,
Lin, Li, Chen, Zhu, Hu, and Hoi] and ACSP [Wang(2020)] and for the Caltech
dataset we integrated the Pedestron detector. Figure 1 and Supplementary
Material show detection accuracy and computations of these detectors.

The Pedestron detector is based on Faster R-CNN and configured to evaluate
during inference 1000 ROIs for images of size 1024$\times$2048 px. Since we
apply this detector on smaller image crops (mROIs), we scale down to 64 ROIs
per mROI. APD and ACSP are single-stage detectors, thus were applied with
their original configuration. Final detections from all image crops are
processed by a non-maximal-suppression (NMS) algorithm to remove duplicate
detections from overlapping mROIs. The $h$ parameter was set for
Pedestron/ACSP/APD to 256/320/384 px accordingly, to conserve as much as
possible detection accuracy.

##  4 Results

We start by analyzing the accuracy and computations of all published
pedestrian detectors from recent years (see Figure 1 and Supplementary
Material). Pedestrian detection accuracy is measured using the MR-2 metric
[Dollár et al.(2009)Dollár, Wojek, Schiele, and Perona, Zhang et
al.(2017)Zhang, Benenson, and Schiele] on the reasonable setup and all
evaluations were conducted on full images resolution. FLOPs are broadly used
today for estimating computations of backbones [Sandler et al.(2018)Sandler,
Howard, Zhu, Zhmoginov, and Chen, Howard et al.(2019)Howard, Sandler, Chu,
Chen, Chen, Tan, Wang, Zhu, Pang, Vasudevan, et al., Zhang et al.(2018b)Zhang,
Zhou, Lin, and Sun, Tan et al.(2020)Tan, Pang, and Le] and general
architectures [He et al.(2019)He, Zhang, Zhang, Zhang, Xie, and Li, Li and
Ren(2019)]. Although latency and FLOPs are not linearly related [Wang et
al.(2020b)Wang, Wu, Liu, Cai, Zhu, Gan, and Han], previous works [Reuther et
al.(2019)Reuther, Michaleas, Jones, Gadepally, Samsi, and Kepner] showed a
strong correlation between them. In contrast to latency, which is highly
platform dependent, in this work we used FLOPs to allow a direct, platform-
independent comparison between various solutions.

###  4.1 Evaluating the first-stage architecture

The C&S and CascadeMV2 architectures resulted in a B/A factor of
0.046/0.038/0.097 for the Pedestron/ACSP/APD detectors, showing their very
lite implementations. Both architectures also achieved a sensitivity of 99%
when setting $k$ to 85% and $p$ to 10% (see Table 1). Moreover, for
pedestrians higher than 100 px the C&S and CascadeMV2 architectures resulted
in a sensitivity of 98.26%/99.47% respectively, while for pedestrians smaller
than 100 px they obtained a sensitivity of 99.16%/99.08%, indicating detection
robustness to object sizes. These results confirm that a first-stage
architecture is not unique and could be replaced by even lighter designs, as
long as they meet the first-stage criteria. Sensitivity could be further
increased by selecting proposals with lower confidence levels, on account of
processing more proposals in the second-stage, thus enabling to balance
between sensitivity and computations reduction of the next stage, without
additional training.

Since the lite-CNN architecture is not unique, we also studied a more naïve
approach that is sometimes used in coarse-to-fine solutions. In these designs
an off-the-shelf detection architecture is applied on downscaled input images
[Gao et al.(2018)Gao, Yu, Li, Morariu, and Davis, Uzkent et al.(2020)Uzkent,
Yeh, and Ermon]. Similarly, we adopted the single-stage APD detector [Zhang et
al.(2019a)Zhang, Lin, Li, Chen, Zhu, Hu, and Hoi] since it has the lowest
number of FLOPs with MR-2 lower than 10%. To achieve a similar number of FLOPs
as our solution, input images were downscaled by a factor of 3.2 on each axis.
This resulted in a decreased sensitivity of 72.2% (see Table 1), that
increased the overall miss rate to 31.9% (see Table 2). This accuracy
deterioration is expected and impacts most small pedestrians. Specifically,
the sensitivity for pedestrians higher than 100 px was 97.34%, while the
sensitivity of pedestrians smaller than 100 px was 35.58%. Even when
downscaling images by a factor of two on each axis, sensitivity was reduced to
92.1%, resulting in a miss rate of 15.4%, showing that using existing
detectors for direct inference on downscaled input images is not an effective
solution.

| Coarse | Data- | PCMAD? | Avg. | Max. | Sensiti-
---|---|---|---|---|---
detector | set |  | M/N | M/N | vity%
C&S | CP | x | 0.095 | 1.073 | 98.9
| CP | ✓ | 0.091 | 0.38 | 98.9
CascadeMV2 | CP | x | 0.148 | 1.939 | 99.2
| CP | ✓ | 0.096 | 0.56 | 99.3
| CT | x | 0.13 | 1.31 | 98.58
| CT | ✓ | 0.12 | 0.95 | 98.94
APD$\times$1/3.2 | CP | ✓ | 0.095 | 0.55 | 72.2
APD$\times$1/2 | CP | ✓ | 0.12 | 0.64 | 92.1
| Proposals | mROIs |  MR-2 | Avg.
---|---|---|---
extension $p$ | downscaled? |  | M/N
5% | ✓ | 9.88% | 0.085
10% | ✓ | 9.39% | 0.091
15% | ✓ | 9.19% | 0.105
20% | ✓ | 8.8% | 0.127
10% | x | 9.37% | 0.128
Table 1: Gauging PCMAD impact. Left: Relative processed area (M/N) and
sensitivity of C&S and CascadeMV2 architectures calculated for $h$=256 px,
$k$=85% and $p$=10%, evaluated on both Citypersons (CP) and Caltech (CT)
datasets. Right: Ablation study on the effect of $p$ and mROIs downscaling on
MR-2 and M/N, using C&S in the first-stage and Pedestron (HRNet) in the
second-stage on the Citypersons dataset.

###  4.2 Gauging the PCMAD contribution

Although produced ROIs achieved high sensitivity, the maximum relative
processed area ($M/N$) in some images exceeds 100% (see Table 1 and
Supplementary Material). For $h$=265 px the PCMAD algorithm was able to limit
M/N in the worst-case scenario to below 0.56 for the Citypersons dataset, with
an average $M/N$ below 9.6%. Overall, PCMAD significantly reduced the amount
of processed pixels by the second-stage detector, contributing to the overall
BLT-net computations reduction. Downscaling mROIs did not decrease detection
accuracy (see Table 1) for the selected $p$ and $h$ parameters, but
significantly decreased M/N. Detection accuracy was impacted by $p$, where an
increased $p$ improved detection accuracy on account on increasing the M/N
factor. Based on this analysis, we select for all the next analyses $p$=10%,
that balances between a low MR-2 and a low N/M factor.

The number of mROIs divided by the number of pedestrians (Citypersons) when
using the C&S / CascadeMV2 was smaller than 1.26/1.31 respectively for the
different second-stage detectors. These results indicate that the first-stage
was able to optimize the number of mROIs, similarly to the number of ground
truth objects, as opposed to the thousands of ROIs used in general two-stage
detectors [Girshick(2015), Ren et al.(2015)Ren, He, Girshick, and Sun].

###  4.3 BLT-net impact on computations and detection accuracy

Overall, BLT-net significantly reduced the computations of the evaluated
pedestrian detection architectures by a factor of $\times$4-$\times$7 (see
Table 2). Computations reduction was most significant for Pedestron, which had
the highest detection accuracy for small pedestrians, that in turn enabled a
more significant mROI downscaling. The total number of FLOPs when using
Pedestron or APD in BLT-net second-stage was below 83 GFLOPs, well below all
published pedestrian detectors so far. BLT-net increased MR-2 by 1.5%-2.3%
when using C&S and by 1.5%-2.64% when using CascadeMV2 for the Citypersons
dataset with respect to the original architectures, with Pedestron being the
least impacted detector and APD the most impacted detector. Similar results
were obtained for the Caltech dataset.

Although $\sim 1\%$ of misses resulted from the first-stage detector, some
additional misses resulted from applying the fine pedestrian detectors on
mROIs. Such misses are characterized mostly by (a) pedestrians appearing in
crowds/occluded or (b) small and poorly illuminated pedestrians. This
deterioration is also reflected on the MR-2 of the heavy occluded pedestrians,
(see Table 2). All additional misses created by BLT-net’s first and second
stages on the Citypersons validation dataset can be seen in the Supplementary
Material. Accuracy degradation in such scenarios was shown to be improved by
extending proposals margins ($p$) on account of increasing second-stage
computations, as seen in Table 1. Such parameters optimization can be used for
balancing between accuracy and computations reduction, without any additional
training. Alternatively, detection accuracy could be improved by training
second-stage detectors directly on mROIs [Yang et al.(2019)Yang, Fan, Chu,
Blasch, and Ling, Hong et al.(2019)Hong, Kang, and Cho, Li et al.(2020)Li,
Yang, Zhu, Chen, and Guan] for given $p$ and $h$ parameters.

Table 2: Detection accuracy and FLOPs of original pedestrian detection architectures and when integrated into BLT-net on full images. Small/large pedestrians are defined to be lower/higher than 100 px. Reasonable/bare/partial/heavy are as defined by the dataset. Detector | Dataset | BLT-net | GFLOPs | FLOPs |  MR-2 |  MR-2 |  MR-2 |  MR-2 |  MR-2 |  MR-2
---|---|---|---|---|---|---|---|---|---|---
|  | first |  | reduct. | Reas. | Bare | Partial | Heavy | Large | Small
|  | stage |  | factor | % | % | % | % | % | %
Pedestron | CP | - | 596.8 | - | 7.6 | 5.8 | 6.3 | 34.3 | 5.4 | 7.6
(HRNet) | CP | C&S | 82.2 | 7.3 | 9.4 | 7.3 | 8.3 | 38.5 | 7 | 9.5
| CP | CascMv2 | 82.7 | 7.2 | 9.1 | 6.9 | 8.2 | 38.9 | 6.6 | 8.8
Pedestron | CP | - | 262.2 | - | 10.2 | 7.4 | 9.1 | 43.8 | 7.2 | 10.1
(mv2) | CP | C&S | 51.5 | 5.1 | 11.7 | 8.2 | 11.4 | 48.2 | 8.1 | 12.6
| CP | CascMv2 | 50.2 | 5.2 | 11.8 | 8.2 | 10.9 | 47.4 | 8.6 | 10.7
ACSP | CP | - | 730.5 | - | 7.6 | 4.9 | 6.9 | 42.2 | 4.7 | 7.2
CP | CP | C&S | 136.7 | 5.3 | 9.6 | 6.3 | 9.3 | 47.5 | 5.9 | 10.6
| CP | CaMV | 128.2 | 5.7 | 9.2 | 5.9 | 9.0 | 45.4 | 5.6 | 10.1
APD | CP | - | 283.3 | - | 9 | 5.2 | 8.98 | 47.0 | 4.5 | 9.8
| CP | C&S | 75 | 3.8 | 11.3 | 7.8 | 10.1 | 51.3 | 7.8 | 11.6
| CP | CascMv2 | 72.7 | 3.9 | 11.6 | 7.3 | 10.7 | 50.0 | 7.4 | 11.7
| CP |  APD$\times$1/3.2 | 69.5 | 4.1 | 31.9 | 27.7 | 31.9 | 55.9 | 8.2 | 61.5
| CP |  APD$\times$1/2 | 125.5 | 2.3 | 15.4 | 10.7 | 15.2 | 47.2% | 6.8 | 22.2
Pedestron | CT | - | 87.4 | - | 2.22 | - | - | - | - | -
(HRNet) | CT | CascMv2 | 13.87 | 6.3 | 3.63 | - | - | - | - | -

##  5 Conclusions

In this work we revise the two-stage architecture by leveraging object scales
and coverage characteristics to reduce computations. Our suggested design
significantly reduced computations of existing pedestrian detectors while
achieving similar detection accuracy, creating a new empirical Pareto
frontier. This architecture could be applied on other object detection
scenarios with similar data characteristics to benefit from significant
computations reduction.

##  6 Acknowledgments

We would like to thank Lior Talker for the numerous insightful discussions on
the various analyses presented in the paper and Matan Shoef for his
contribution to the development of the C&S net architecture.

## References

  * [Angelova et al.(2015)Angelova, Krizhevsky, Vanhoucke, Ogale, and Ferguson] Anelia Angelova, Alex Krizhevsky, Vincent Vanhoucke, Abhijit Ogale, and Dave Ferguson.  Real-time pedestrian detection with deep network cascades.  2015\.
  * [Braun et al.(2019)Braun, Krebs, Flohr, and Gavrila] Markus Braun, Sebastian Krebs, Fabian Flohr, and Dariu M Gavrila.  Eurocity persons: A novel benchmark for person detection in traffic scenes.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(8):1844–1861, 2019.
  * [Cai and Vasconcelos(2018)] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6154–6162, 2018.
  * [Chen et al.(2020)Chen, Liu, Meng, Xiao, and Ju] Chen Chen, Mengyuan Liu, Xiandong Meng, Wanpeng Xiao, and Qi Ju.  Refinedetlite: A lightweight one-stage object detection framework for cpu-only devices.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 700–701, 2020.
  * [Chen et al.(2016)Chen, Hua, Wen, and Sun] Dong Chen, Gang Hua, Fang Wen, and Jian Sun.  Supervised transformer network for efficient face detection.  In _European Conference on Computer Vision_ , pages 122–138. Springer, 2016.
  * [Cheng et al.(2017)Cheng, Wang, Zhou, and Zhang] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang.  A survey of model compression and acceleration for deep neural networks.  _arXiv preprint arXiv:1710.09282_ , 2017.
  * [Chi et al.(2020)Chi, Zhang, Xing, Lei, Li, Zou, et al.] Cheng Chi, Shifeng Zhang, Junliang Xing, Zhen Lei, Stan Z Li, Xudong Zou, et al.  Pedhunter: Occlusion robust pedestrian detector in crowded scenes.  In _AAAI_ , pages 10639–10646, 2020.
  * [Christie et al.(2018)Christie, Fendley, Wilson, and Mukherjee] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee.  Functional map of the world.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6172–6180, 2018.
  * [Cloutier et al.(2016)Cloutier, Paradis, and Weaver] Michael F Cloutier, Chad Paradis, and Vincent M Weaver.  A raspberry pi cluster instrumented for fine-grained power measurement.  _Electronics_ , 5(4):61, 2016.
  * [Dai et al.(2016)Dai, Li, He, and Sun] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun.  R-fcn: Object detection via region-based fully convolutional networks.  In _Advances in neural information processing systems_ , pages 379–387, 2016.
  * [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * [Dollár et al.(2009)Dollár, Wojek, Schiele, and Perona] Piotr Dollár, Christian Wojek, Bernt Schiele, and Pietro Perona.  Pedestrian detection: A benchmark.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 304–311. IEEE, 2009.
  * [Elsken et al.(2019)Elsken, Metzen, Hutter, et al.] Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al.  Neural architecture search: A survey.  _J. Mach. Learn. Res._ , 20(55):1–21, 2019.
  * [Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and Zisserman] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ , 88(2):303–338, 2010.
  * [Gao et al.(2018)Gao, Yu, Li, Morariu, and Davis] Mingfei Gao, Ruichi Yu, Ang Li, Vlad I Morariu, and Larry S Davis.  Dynamic zoom-in network for fast object detection in large images.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6926–6935, 2018.
  * [Geiger et al.(2012)Geiger, Lenz, and Urtasun] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3354–3361. IEEE, 2012.
  * [Girshick(2015)] Ross Girshick.  Fast r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1440–1448, 2015.
  * [Girshick et al.(2014)Girshick, Donahue, Darrell, and Malik] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 580–587, 2014.
  * [Hasan et al.(2020)Hasan, Liao, Li, Akram, and Shao] Irtiza Hasan, Shengcai Liao, Jinpeng Li, Saad Ullah Akram, and Ling Shao.  Pedestrian detection: The elephant in the room.  _arXiv preprint arXiv:2003.08799_ , 2020.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * [He et al.(2019)He, Zhang, Zhang, Zhang, Xie, and Li] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li.  Bag of tricks for image classification with convolutional neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 558–567, 2019.
  * [Hong et al.(2019)Hong, Kang, and Cho] Sungeun Hong, Sungil Kang, and Donghyeon Cho.  Patch-level augmentation for object detection in aerial images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , pages 0–0, 2019.
  * [Howard et al.(2019)Howard, Sandler, Chu, Chen, Chen, Tan, Wang, Zhu, Pang, Vasudevan, et al.] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1314–1324, 2019.
  * [Howard et al.(2017)Howard, Zhu, Chen, Kalenichenko, Wang, Weyand, Andreetto, and Adam] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ , 2017.
  * [Huang et al.(2017)Huang, Rathod, Sun, Zhu, Korattikara, Fathi, Fischer, Wojna, Song, Guadarrama, et al.] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al.  Speed/accuracy trade-offs for modern convolutional object detectors.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7310–7311, 2017.
  * [Iandola et al.(2016)Iandola, Han, Moskewicz, Ashraf, Dally, and Keutzer] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and$<$0.5 mb model size.  _arXiv preprint arXiv:1602.07360_ , 2016.
  * [Ignatov et al.(2019)Ignatov, Timofte, Kulik, Yang, Wang, Baum, Wu, Xu, and Van Gool] Andrey Ignatov, Radu Timofte, Andrei Kulik, Seungsoo Yang, Ke Wang, Felix Baum, Max Wu, Lirong Xu, and Luc Van Gool.  Ai benchmark: All about deep learning on smartphones in 2019.  In _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , pages 3617–3635. IEEE, 2019.
  * [Jain and Learned-Miller(2010)] Vidit Jain and Erik Learned-Miller.  Fddb: A benchmark for face detection in unconstrained settings.  Technical report, UMass Amherst technical report, 2010.
  * [Kang et al.(2019)Kang, Choi, Park, Park, Lee, Lee, and Lee] Soowon Kang, Hyeonwoo Choi, Sooyoung Park, Chunjong Park, Jemin Lee, Uichin Lee, and Sung-Ju Lee.  Fire in your hands: Understanding thermal behavior of smartphones.  In _The 25th Annual International Conference on Mobile Computing and Networking_ , pages 1–16, 2019.
  * [Lee et al.(2020)Lee, Chung, Lee, Han, and Kwon] MY Lee, J Chung, JH Lee, JH Han, and YS Kwon.  Trends in ai processor technology.  _Electronics and Telecommunications Trends_ , 35(3):66–75, 2020.
  * [Li et al.(2020)Li, Yang, Zhu, Chen, and Guan] Changlin Li, Taojiannan Yang, Sijie Zhu, Chen Chen, and Shanyue Guan.  Density map guided object detection in aerial images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 190–191, 2020.
  * [Li et al.(2015)Li, Lin, Shen, Brandt, and Hua] Haoxiang Li, Zhe Lin, Xiaohui Shen, Jonathan Brandt, and Gang Hua.  A convolutional neural network cascade for face detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5325–5334, 2015.
  * [Li and Ren(2019)] Yixing Li and Fengbo Ren.  Light-weight retinanet for object detection.  _arXiv preprint arXiv:1905.10011_ , 2019.
  * [Lin et al.(2018)Lin, Zhang, Hsu, Skach, Haque, Tang, and Mars] Shih-Chieh Lin, Yunqi Zhang, Chang-Hong Hsu, Matt Skach, Md E Haque, Lingjia Tang, and Jason Mars.  The architectural implications of autonomous driving: Constraints and acceleration.  In _Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems_ , pages 751–766, 2018.
  * [Lin et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer, 2014.
  * [Lin et al.(2017)Lin, Dollár, Girshick, He, Hariharan, and Belongie] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2117–2125, 2017.
  * [Liu et al.(2019a)Liu, Huang, and Wang] Songtao Liu, Di Huang, and Yunhong Wang.  Adaptive nms: Refining pedestrian detection in a crowd.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6459–6468, 2019a.
  * [Liu et al.(2016)Liu, Anguelov, Erhan, Szegedy, Reed, Fu, and Berg] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In _European conference on computer vision_ , pages 21–37. Springer, 2016.
  * [Liu et al.(2018)Liu, Liao, Hu, Liang, and Chen] Wei Liu, Shengcai Liao, Weidong Hu, Xuezhi Liang, and Xiao Chen.  Learning efficient single-stage pedestrian detectors by asymptotic localization fitting.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 618–634, 2018.
  * [Liu et al.(2019b)Liu, Liao, Ren, Hu, and Yu] Wei Liu, Shengcai Liao, Weiqiang Ren, Weidong Hu, and Yinan Yu.  High-level semantic feature detection: A new perspective for pedestrian detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5187–5196, 2019b.
  * [Najibi et al.(2019)Najibi, Singh, and Davis] Mahyar Najibi, Bharat Singh, and Larry S Davis.  Autofocus: Efficient multi-scale inference.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 9745–9755, 2019.
  * [Pang et al.(2019)Pang, Xie, Khan, Anwer, Khan, and Shao] Yanwei Pang, Jin Xie, Muhammad Haris Khan, Rao Muhammad Anwer, Fahad Shahbaz Khan, and Ling Shao.  Mask-guided attention network for occluded pedestrian detection.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 4967–4975, 2019.
  * [Redmon and Farhadi(2017)] Joseph Redmon and Ali Farhadi.  Yolo9000: Better, faster, stronger.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7263–7271, 2017.
  * [Redmon et al.(2016)Redmon, Divvala, Girshick, and Farhadi] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 779–788, 2016.
  * [Ren et al.(2015)Ren, He, Girshick, and Sun] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _Advances in neural information processing systems_ , pages 91–99, 2015.
  * [Reuther et al.(2019)Reuther, Michaleas, Jones, Gadepally, Samsi, and Kepner] Albert Reuther, Peter Michaleas, Michael Jones, Vijay Gadepally, Siddharth Samsi, and Jeremy Kepner.  Survey and benchmarking of machine learning accelerators.  _arXiv preprint arXiv:1908.11348_ , 2019.
  * [Royer and Lampert(2020)] A. Royer and Christoph H. Lampert.  Localizing grouped instances for efficient detection in low-resource scenarios.  _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1716–1725, 2020.
  * [Sandler et al.(2018)Sandler, Howard, Zhu, Zhmoginov, and Chen] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4510–4520, 2018.
  * [Shao et al.(2018)Shao, Zhao, Li, Xiao, Yu, Zhang, and Sun] Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun.  Crowdhuman: A benchmark for detecting human in a crowd.  _arXiv preprint arXiv:1805.00123_ , 2018.
  * [Simonyan and Zisserman(2014)] Karen Simonyan and Andrew Zisserman.  Vgg-16.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * [Song et al.(2018)Song, Liu, Jiang, Wang, Yan, and Leng] Guanglu Song, Yu Liu, Ming Jiang, Yujie Wang, Junjie Yan, and Biao Leng.  Beyond trade-off: Accelerate fcn-based face detector with higher accuracy.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7756–7764, 2018.
  * [Szegedy et al.(2016)Szegedy, Vanhoucke, Ioffe, Shlens, and Wojna] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2818–2826, 2016.
  * [Tan et al.(2020)Tan, Pang, and Le] Mingxing Tan, Ruoming Pang, and Quoc V Le.  Efficientdet: Scalable and efficient object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10781–10790, 2020.
  * [Ujjwal et al.(2020)Ujjwal, Dziri, Leroy, and Bremond] Ujjwal Ujjwal, Aziz Dziri, Bertrand Leroy, and Francois Bremond.  A one-and-half stage pedestrian detector.  In _The IEEE Winter Conference on Applications of Computer Vision_ , pages 776–785, 2020.
  * [Uzkent and Ermon(2020)] Burak Uzkent and Stefano Ermon.  Learning when and where to zoom with deep reinforcement learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12345–12354, 2020.
  * [Uzkent et al.(2020)Uzkent, Yeh, and Ermon] Burak Uzkent, Christopher Yeh, and Stefano Ermon.  Efficient object detection in large images using deep reinforcement learning.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1824–1833, 2020.
  * [Viola and Jones(2001)] Paul Viola and Michael Jones.  Rapid object detection using a boosted cascade of simple features.  In _Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001_ , volume 1, pages I–I. IEEE, 2001.
  * [Wang et al.(2020a)Wang, Bochkovskiy, and Liao] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  Scaled-yolov4: Scaling cross stage partial network.  _arXiv preprint arXiv:2011.08036_ , 2020a.
  * [Wang et al.(2020b)Wang, Wu, Liu, Cai, Zhu, Gan, and Han] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han.  Hat: Hardware-aware transformers for efficient natural language processing.  _arXiv preprint arXiv:2005.14187_ , 2020b.
  * [Wang(2020)] Wenhao Wang.  Adapted center and scale prediction: More stable and more accurate.  _arXiv preprint arXiv:2002.09053_ , 2020.
  * [Wang et al.(2018)Wang, Xiao, Jiang, Shao, Sun, and Shen] Xinlong Wang, Tete Xiao, Yuning Jiang, Shuai Shao, Jian Sun, and Chunhua Shen.  Repulsion loss: Detecting pedestrians in a crowd.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7774–7783, 2018.
  * [Xia et al.(2018)Xia, Bai, Ding, Zhu, Belongie, Luo, Datcu, Pelillo, and Zhang] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang.  Dota: A large-scale dataset for object detection in aerial images.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3974–3983, 2018.
  * [Xie et al.(2020)Xie, Pang, Cholakkal, Anwer, Khan, and Shao] Jin Xie, Yanwei Pang, Hisham Cholakkal, Rao Muhammad Anwer, Fahad Shahbaz Khan, and Ling Shao.  Psc-net: Learning part spatial co-occurence for occluded pedestrian detection.  _arXiv preprint arXiv:2001.09252_ , 2020.
  * [Yang et al.(2019)Yang, Fan, Chu, Blasch, and Ling] Fan Yang, Heng Fan, Peng Chu, Erik Blasch, and Haibin Ling.  Clustered object detection in aerial images.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 8310–8319, 2019.
  * [Yang et al.(2016)Yang, Luo, Loy, and Tang] Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang.  Wider face: A face detection benchmark.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5525–5533, 2016.
  * [Yang and Nevatia(2016)] Zhenheng Yang and Ramakant Nevatia.  A multi-scale cascade fully convolutional network face detector.  In _2016 23rd International Conference on Pattern Recognition (ICPR)_ , pages 633–638. IEEE, 2016.
  * [Yu et al.(2018)Yu, Wang, Peng, Gao, Yu, and Sang] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang.  Bisenet: Bilateral segmentation network for real-time semantic segmentation.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 325–341, 2018.
  * [Zhang et al.(2019a)Zhang, Lin, Li, Chen, Zhu, Hu, and Hoi] Jialiang Zhang, Lixiang Lin, Yang Li, Yun-chen Chen, Jianke Zhu, Yao Hu, and Steven CH Hoi.  Attribute-aware pedestrian detection in a crowd.  _arXiv_ , pages arXiv–1910, 2019a.
  * [Zhang et al.(2017)Zhang, Benenson, and Schiele] Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele.  Citypersons: A diverse dataset for pedestrian detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3213–3221, 2017.
  * [Zhang et al.(2018a)Zhang, Wen, Bian, Lei, and Li] Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, and Stan Z Li.  Occlusion-aware r-cnn: Detecting pedestrians in a crowd.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 637–653, 2018a.
  * [Zhang et al.(2019b)Zhang, Xie, Wan, Xia, Li, and Guo] Shifeng Zhang, Yiliang Xie, Jun Wan, Hansheng Xia, Stan Z Li, and Guodong Guo.  Widerperson: A diverse dataset for dense pedestrian detection in the wild.  _IEEE Transactions on Multimedia_ , 22(2):380–393, 2019b.
  * [Zhang et al.(2018b)Zhang, Zhou, Lin, and Sun] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6848–6856, 2018b.
  * [Zhao et al.(2018)Zhao, Qi, Shen, Shi, and Jia] Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia.  Icnet for real-time semantic segmentation on high-resolution images.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 405–420, 2018.
  * [Zhu and Ramanan(2012)] Xiangxin Zhu and Deva Ramanan.  Face detection, pose estimation, and landmark localization in the wild.  In _2012 IEEE conference on computer vision and pattern recognition_ , pages 2879–2886. IEEE, 2012.