# Towards More Effective PRM-based Crowd Counting via A Multi-resolution
Fusion and Attention Network

Usman Sajid  Guanghui Wang<EMAIL_ADDRESS><EMAIL_ADDRESS>Department of
Electrical Engineering and Computer Science, University of Kansas, Lawrence,
KS, USA, 66045  Department of Computer Science, Ryerson University, Toronto,
ON, Canada M5B 2K3

###### Abstract

The paper focuses on improving the recent plug-and-play patch rescaling module
(PRM) based approaches for crowd counting. In order to make full use of the
PRM potential and obtain more reliable and accurate results for challenging
images with crowd-variation, large perspective, extreme occlusions, and
cluttered background regions, we propose a new PRM based multi-resolution and
multi-task crowd counting network by exploiting the PRM module with more
effectiveness and potency. The proposed model consists of three deep-layered
branches with each branch generating feature maps of different resolutions.
These branches perform a feature-level fusion across each other to build the
vital collective knowledge to be used for the final crowd estimate.
Additionally, early-stage feature maps undergo visual attention to strengthen
the later-stage channel’s understanding of the foreground regions. The
integration of these deep branches with the PRM module and the early-attended
blocks proves to be more effective than the original PRM based schemes through
extensive numerical and visual evaluations on four benchmark datasets. The
proposed approach yields a significant improvement by a margin of $12.6\%$ in
terms of the RMSE evaluation criterion. It also outperforms state-of-the-art
methods in cross-dataset evaluations.

###### keywords:

PRM, crowd counting, multi-resolution, multi-task, visual attention, RMSE.

††journal: Neurocomputing

##  1 Introduction

Deep learning has achieved exceptional success in many computer vision
applications like classification and detection [1, 2, 3]. Crowd counting aims
to estimate the total number of people in a given static image. This is a very
challenging problem in practice since there exists a significant difference in
the crowd number in and across different images, varying images resolution,
large perspective, and severe occlusions [4], as shown in Fig. 1. Accurate
crowd counting can help to effectively organize large crowd gatherings.
Convolutional Neural Networks (CNNs) have proven to be very effective in
dealing with these issues. The state-of-the-art crowd counting techniques are
based on either counting-by-regression or the density-map estimation.

Counting-by-regression [5, 6, 7] schemes learn the mapping of the input image
or patch to its crowd count, whereas the density-map estimation methods [8, 9,
10, 11, 12, 13, 14] yield the crowd-density value per input image pixel that
are summed to get the image final crowd count. In general, counting-by-
regression schemes do not perform reasonably well without any special and
additive mechanism. On the other hand, density-map based methods rely heavily
on the accurate density-map generation for the training images from the
available ground-truth dot-map annotations for the learning process of their
models. In principle, a point-spread function (e.g. Multivariate Gaussian
Kernel) is being deployed to produce pixel-wise crowd-density values. Although
the density-based methods produce reasonable results, the amount of Gaussian
spread challenge limits their overall performance and relatively compromises
their efficacy. Some works [15, 16] also aim to detect people by their head or
body using some well-established CNN-based detectors for the crowd counting
purpose, however, few pixels per head in medium to high-dense crowd images
make it almost impractical to achieve.

Figure 1: Sample crowd images. Estimating the crowd in these images comes up
with many major challenges and problems including huge fluctuation of the
crowd-density in and across different images, different image resolution, far-
reaching wide perspective, and severe occlusions.

Figure 2: The proposed multi-task feature-level fusion based crowd counting
network. The $256\times 256$ input patch ($IP$) passes through the initial
deep layers ($IDL$) to output the initial feature maps ($IFM$) that are fed
into the phase-based multi-branch network. Each of the three branches/columns
maintains its original resolution throughout that branch [17, 18]. The output
channels from the first residual block (RB) of the (Branch-1, Phase-2) are
branched out to make the classification head ($CH$). The $CH$ classifies the
patch ($IP$) 4-way according to its crowd-density level. This labeling process
is being used by the PRM module [7] to decide if appropriate rescaling and
generation of new patches are required or not. The resultant patch(es) then
passes through the concatenation module ($CMod$) to generate the channels
($C$) that are concatenated back into the network after adjusting the number
of channels via the Bottleneck layer ($BL$). The network branches also do the
feature-level fusion regularly to form the model collective knowledge. After
the remaining processing, the Phase-3 outputs three later-stage feature maps
(LFM-B1, LFM-B2, LFM-B3). Each LFM passes through the VACM module along with
their respective branch early-stage feature maps (EFM-B1, EFM-B2, EFM-B3). The
VACM generated final feature maps (FFMs) are eventually inserted into the
concatenation-based Crowd Regression Head ($CRH$) to obtain the final crowd
count (${CC}_{IP}$) for the input Patch ($IP$).

Recently, Sajid et al. [7] proposed a counting-by-regression based method that
uses a lightweight Patch Rescaling Module (PRM) to rescale the input image or
patch accordingly based on its crowd-density level before the crowd
estimation. They also proposed PRM-based schemes, which performed reasonably
better as compared to other state-of-the-art methods. Although these schemes
produce state-of-the-art results, they carry the following key shortcomings:

  * 1.

They maintain a single branch/column architecture with only-one scale focus
that limits their achievable performance and potential effectiveness.

  * 2.

These models only utilize the PRM based input-level multi-resolution rescaling
process, while lacking the beneficial feature-level multi-resolution process.

In this work, we present a new multi-resolution feature-level fusion based
multi-task and visually-attended crowd counting method that aims to address
the major crowd counting challenges as well as further explores and pushes the
boundaries of the PRM module by tackling its shortcomings as mentioned above.
The proposed PRM based scheme, as shown in Fig. 2, incorporates multiple
columns or branches, each with feature maps of different resolutions. Based on
the high-resolution networks [17, 18], these branches perform the fusion or
sharing across each other on a regular basis to form a collective knowledge
that improves the overall network performance. We also perform the visual-
attention process on the early-stage feature maps from each of these branches
to boost later-stage channel’s understanding of foreground and background. On
the other hand, the PRM based rescaling of the input image or patch has been
used to select the appropriate input-level scale based on its crowd-density
level. The experiments (Sec. 5) demonstrate the proposed PRM utilization to be
remarkably more effective as compared to its previous implementations [7, 6].
The key contributions of this paper are listed as follows:

  * 1.

We propose a new multi-resolution and multi-task PRM based visually-attended
crowd counting network that effectively addresses major crowd counting
challenges, including the issues of crowd-like background regions and huge
crowd-variation.

  * 2.

We deploy the plug-and-play PRM module so as to further push its boundaries
and utilize it more effectively as compared to its previous deployments.

  * 3.

We employ the visual attention mechanism in a unique and effective way on
early-stage feature maps that facilitates the later-stage channels to better
understand the foreground regions.

  * 4.

The proposed scheme outperforms the current-best methods in most cases,
including the original PRM based schemes as demonstrated via both numerical
and visual experiments on four benchmark datasets. The proposed model shows an
improvement of up to $12.6\%$ in terms of RMSE evaluation while performing
almost equally best for the MAE evaluation criterion. During the cross-dataset
evaluation, the proposed method outperforms the state-of-the-art.

##  2 Related Work

Crowd counting has many different key challenges including image resolution
and crowd-density fluctuation in and across different images, extreme
occlusions, far-reaching crowd perspective within an image, cluttered no-crowd
regions in the images. Many classical schemes were proposed that generally
aimed to solve this task as either a regression or a detection problem.
Regression-based counting [19, 20, 21] learns the local crowd features mapping
function for the given image to regress the crowd count. But they fail to
reasonably comprehend the key crowd counting challenges. Detection-dependent
[22, 23, 24, 25] schemes detect hand-crafted features to estimate the crowd
count. These methods become impractical and ineffective in the case of high-
dense crowd images because of the very small area per head or person.

Recently, Convolutional Neural Networks (CNNs) based approaches are the
backbone of almost all crowd counting methods. Generally, we can divide them
into three classes: Detection-dependent, Regression-dependent, and density-map
estimation [6]. Well established object detectors (like YOLO et al. [26, 27])
are being employed by the detection-dependent [15, 16] methods to identify the
people in the given image, and then sum all the detections to give the final
crowd count. However, they fail for the images with high-density crowd due to
smaller area per person or head. Shami et al. [15] performed the weighted
average of the CNN-based detections to produce the final people count. Li et
al. [16] employed the CNN based adaptive head detection method that used the
context-based information for the crowd estimation.

Regression-dependent [5, 6, 7] methods map the input image directly to its
crowd estimate. These methods also do not generalize well without any
specialized and supportive mechanism. Wang et. al [5] used the AlexNet [28]
based scheme to directly regress and estimate the people number. Density-map
estimation [8, 9, 10, 11, 13, 14] methods estimate the crowd-density value per
pixel. The pixel-wise density values get added to produce the image’s final
crowd number. These methods first require the conversion of the ground-truth
crowd dot-maps to their respective density-maps using a point-spread function
(e.g. multivariate Gaussian distribution kernel) for the model training.
However, the spread or kernel size remains a major issue that is vital for
their training process quality. MCNN [8], a density-map estimation method,
uses three columns or branches, each with different filter sizes to account
for the respective crowd-scale. These branches are merged together at the end
to generate the final crowd density-map.

Sam et al. [10] designed the Switch-CNN, another density-map method, that
contains a (CNN-based) switch classifier that labels each input image or patch
and routes it accordingly to one of three available specialized crowd
regressors. Sindagi et al. [9] devised a cascaded network that computes the
crowd by first classifying the input 10-way based on its crowd-density level.
Ranjan et al. [11] designed a 2-branch network where the lower-resolution
branch or column helps the higher-resolution for better crowd calculation. Li
et al. [29] presented the CSRNet to obtain the context-based information
through the dilation based deep layers for effective crowd estimation. Idrees
et al. [30] designed the density-map estimation network that performs better
for the crowd counting task by simultaneously solving the crowd localization
as well as for the crowd-density estimation. Xu et al. [14] designed the
density-map based method that used the patch-based density-maps, and grouped
them into several levels based on the crowd-density, followed by the online
learning mechanism for the automatic normalization process with a multi-polar
center loss. Recently, Sajid et al. [6, 7] proposed a simple plug-and-play
patch rescaling module (PRM) that rescales the input image or patch based on
the respective crowd-density label.

Although recent works have produced very promising results, none of them make
full use of their capabilities nor generalize well to hugely varying crowd-
density and other major challenges. In this work, we focus on addressing the
above crowd estimation challenges and issues.

Layer | Output | Filters (F)
---|---|---
Initial Deep Layers (IDL)
$IP$ | $3\times 256\times 256$ |
| $64\times 128\times 128$ |  ($3\times 3$) conv, s = 2, p = 1, $64$F
| $64\times 64\times 64$ |  ($3\times 3$) conv, s = 2, p = 1, $64$F
$IFM$ | $32\times 64\times 64$ |  ($1\times 1$) conv, s = 1, p = 0, $32$F
Classification Head (CH)
| $32\times 64\times 64$ |
| $64\times 32\times 32$ |  ($3\times 3$) conv, s = 2, p = 1, $64$F
| $32\times 16\times 16$ |  ($3\times 3$) conv, s = 2, p = 1, $32$F
| $32\times 8\times 8$ |  ($2\times 2$) Global Avg. Pooling, s = 2
| 1024D FC | -
| 4D FC, Softmax | -
Concatenation Module (CMod)
| $3\times 256\times 256$ |
Conv1 | $64\times 128\times 128$ |  ($3\times 3$) conv, s = 2, p = 1, $64$F
Conv2 | $32\times 64\times 64$ |  ($3\times 3$) conv, s = 2, p = 1, $32$F
RB | $32\times 64\times 64$ | -
Crowd Regression Head (CRH) (Continued from Fig. 5(b))
| $64\times 32\times 32$ |
| $64\times 16\times 16$ |  ($3\times 3$) conv, s = 2, p = 1, $64$F
| $64\times 8\times 8$ |  ($2\times 2$) Avg. Pooling, s = 2
| 1024D, FC | -
| 1D, FC | -
Table 1: Configurations of IDL, CH, CMod, and CRH modules. Each conv operation
denotes the Convolution-BN-ReLU sequence [31]. (s: stride, p: padding, FC:
Fully Connected)

##  3 Proposed Approach

This work aims to address the major crowd counting challenges (e.g., huge
difference in image and scale resolution, severe occlusions, far-reaching
perspective changes, etc.) as well as to exploit the PRM module more
effectively while mitigating its shortcomings. The proposed multi-task
network, as shown in Fig. 2, contains three deep-layered branches with
different resolutions feature maps (channels). The feature-level fusion [17,
18] occurs in between these branches on a regular basis, which helps them to
form a collective knowledge about the input image or patch. As their final
output, each of these branches produces early- (EFM) and later-stage (LFM)
feature maps. The EFMs undergo visual attention process before concatenation
with the LFM channels via the visual attention and concatenation (VACM)
module. The visually-attended EFMs enable the LFMs to clearly distinguish
between foreground and background image regions. The concatenated final
feature maps (FFMs) then pass through the crowd regression head (CRH) to
output the final crowd estimate. The PRM module has been deployed to rescale
the input to its appropriate scale. To start with, we first divide the input
image into fixed-size $256\times 256$ non-overlapping patches. Following that,
we estimate the crowd count for each resultant patch separately using the
proposed scheme. Finally, the image total people count is equal to the
summation of all its patches crowd estimates.

The input patch ($IP$) first proceeds through the initial deep layers (IDL),
as detailed in Table 1, to produce the initial feature maps (IFMs). The IDL
also reduces the input resolution to $1/4$ (from $256\times 256$ to $64\times
64$). Next, these feature maps are routed to Phase-1 of the multi-branch
network. The phase-wise multi-branch network comprises of five key components:
Multi-resolution branches and phases, Multi-resolution fusion, PRM module
deployment, VACM module, and Crowd Regression Head (CRH). They are detailed as
follows.

###  3.1 Multi-resolution Branches and Phases

The network comprises of three phases (Phase-1, 2, and 3). The initially
generated feature maps (IFMs) or channels pass through the phase-wise
organized multi-branch deep layers, starting from Phase-1. Each phase contains
the total number of multi-resolution branches equal to its phase number.
Consequently, Phase-1, 2, and 3 contain one, two, and three deep branches,
respectively. Each branch also maintains its channels resolution throughout
that branch [17, 18]. These branches also perform feature-level fusion across
each other on a regular basis to form a collective knowledge-based learning
process, as detailed in the next subsection 3.2. The channel resolution and
the total number of channels in a specific branch depend on the highest-
resolution branch configuration (Branch-1). The channel resolution decreases
by half in each subsequent lower-resolution branch. However, the total number
of channels increases $2\times$ times as we move from higher to lower-
resolution branches. Thus, Branch-1, 2, 3 contain ($32\times 64\times 64$),
($64\times 32\times 32$), ($128\times 16\times 16$) channels respectively,
where it is denoted by ($Channels\times Width\times Height$).

Figure 3: 3-Layered Residual Unit [32] being used in the Residual Block (RB).
Each RB is composed of four such units.

Each phase also contains the Residual Blocks (RB). Each RB comprises of four
residual units [32] that are 3-layer based residual modules as shown in Fig.
3. Batch Normalization (BN) [33] and the non-linear ReLU activation [34]
follow each convolution operation. The number of such RB modules remains the
same in each branch of a specific phase.

Eventually, the phase-based three-branch structure outputs six separate blocks
of feature maps; three early- (EFM) and three later-stage (LFM) feature maps.
The phase-3 output blocks from the respective branch Bx (x=1,2, or 3) serve as
the LFM-Bx channels. To obtain the early-stage EFM-Bx channels, we take the
very first channels being produced in that specific branch Bx. These channels
proceed forward for further processing.

###  3.2 Repeating Multi-resolution Fusion

The branches in a phase regularly share their channels across each other via
the summation-based fusion. This sharing process helps in learning and
building the collective information and knowledge from all branches that
naturally enhances the generalization potential of the proposed network
towards huge crowd-density and scale variation. The higher-branch channels are
fused into the lower-resolution channels using the $(3\times
3,stride-2,padding-1)$ convolution(s) to down-size the resolution accordingly
[17, 18]. To fuse the Branch-1 channels into the Branch-2, Branch-1 channels
are down-sampled by using one such $3\times 3$ convolution operation.
Similarly, the Branch-1 fusion into Branch-3 requires this convolution
operation twice. To fuse the lower-resolution channels into the higher-level
branch, bilinear upsampling has been applied to lower-resolution features to
up-size them accordingly before the fusion process [17, 18].

###  3.3 PRM Module Deployment

The purpose of the Patch Rescaling Module (PRM) [7] is to rescale the input
patch ($IP$) based on its crowd-density level. As defined in [7], we first
require the 4-way crowd-density classification (No-Crowd (NCP), low-Crowd
(LCP), Medium-Crowd (MCP), High-Crowd Patch (HCP)) for input ($IP$) before
using the PRM module. Thus, we branch-out the output channels from the first
RB module of (Branch-1, Phase-2). These channels then proceed through the
crowd-density Classification Head ($CH$) that performs the required 4-way
classification (NCP, LCP, MCP, HCP). The ground-truth class-label
($CL_{IP(gt)}$) for the input patch $IP$ have been defined as follows [7] for
the training and evaluation purposes:

$CL_{IP(gt)}=\begin{cases}NCP&\text{$cc_{gt}$ = 0}\\\ LCP&\text{0 \textless\
$cc_{gt}$ $\leq$\ 0.05 * $cc_{max}$}\\\ MCP&\text{0.05 * $cc_{max}$ \textless\
$cc_{gt}$ $\leq$\ 0.2 * $cc_{max}$}\\\ HCP&\text{0.2 * $cc_{max}$ \textless\
$cc_{gt}$}\\\ \end{cases}$ (1)

where $cc_{gt}$ and $cc_{max}$ denote the actual people count in that patch
and maximum possible crowd number in any $256\times 256$ patch in the given
dataset respectively. Patch with zero ground-truth crowd count is naturally
categorized with the $NCP$ label. Crowd patches with at most $5\%$ count of
$cc_{max}$ are labeled as $LCP$ class. Similarly, for the patch ($IP$) with
ground-truth crowd count in between $5\%$ to $20\%$ of the maximum value (with
$20\%$ inclusive) is considered as the $MCP$ category. Whereas, containing
more than $20\%$ crowd count of $cc_{max}$ value makes the input patch fall
into the $HCP$ class label. Depending on the designated class label
(${Class}_{IP}$), the PRM rescales the input patch ($IP$) accordingly as given
in [7]. Consequently, it generates one or more new $256\times 256$ size
patches. For (${Class}_{IP}=NCP,LCP,MCP$), the PRM generates ($1,1,1$) new
patches respectively. In case of (${Class}_{IP}=HCP$), the PRM divides the
input patch ($IP$) into four new $128\times 128$ patches, then upscales each
by $2\times$ to output the final $256\times 256$ size patches [7]. The PRM
generated patches then separately go through the concatenation module (CMod)
to generate the initial feature maps. Eventually, these features go through
the concatenation process with the (Branch-1, Phase-2) second RB-block output,
followed by the Bottleneck layer ($BL$) to adjust the number of channels
before proceeding further.

The $CH$ configuration is shown in Table 1. It utilizes the softmax based
4-way classification activation with the cross-entropy loss, given as follows:

$Loss_{CH}=-\sum_{i=1}^{4}y_{i}log(\hat{y_{i}})$ (2)

where $y_{i}$ denotes the actual class (1 or 0) and $\hat{y_{i}}$ indicates
the predicted class label. Similarly, the concatenation module (CMod)
configuration is shown in Table 1, where it consists of several deep layers to
eventually yield the final ($32\times 64\times 64$) channels to be used next
for the concatenation. It is also worth mentioning that the input patch
($IP$), classified as the NCP label during the test time, will be
automatically discarded without any further processing. This is very effective
especially in the case of discarding cluttered background regions in the
images (e.g. tree leaves) that look very similar to the dense-crowd region.

Figure 4: Visual Attention and Concatenation module (VACM). The early-stage
feature maps (EFM-Bx) from the Branch Bx (=1,2, or 3) get visual attention and
output the segmentation map (SM) that undergoes an element-wise multiplication
with the original EFM-Bx channels to generate the visually-attended feature
maps (VAFM). These VAFM maps concatenate with the later-stage feature maps
(LFM-Bx) of the same branch before passing through the channel-adjusting deep
layer to output the final feature maps (FFM-Bx) for the specific branch Bx.

Figure 5: Concatenation-based Crowd Regression Head (CRH) concatenates the
lower-level branches with the highest-level (Branch-1), followed by several
deep layers to form the regression head.

###  3.4 Visual Attention and Concatenation Module (VACM)

The purpose of the VACM module is to visually attend the EFM feature maps and
share them with the LFM channels for better foreground vs background
understanding. The VACM module visually attends the early-stage feature maps
(EFM-B1, EFM-B2, EFM-B3) and concatenates them separately with their
respective branch later-stage channels (LFM-B1, LFM-B2, LFM-B3). The resultant
final feature maps (FFM-B1, FFM-B2, FFM-B3) then proceed through the crowd
regression head (CRH) for the final crowd estimate. As shown in Fig. 4, the
input EFM-Bx block passes through three deep convolution layers to produce the
attention-based segmentation map (SM $\in[0,1]_{W\times H}$). The SM undergoes
element-wise multiplication with the original EFM-Bx block to produce visually
attended feature maps (VAFM). These VAFMs are then concatenated with the input
LFM-Bx block and channel adjusted to output final feature maps (FFM-Bx) for
the branch Bx.

This early-attention mechanism helps in strengthening the early-stage feature
maps of each branch towards better understanding and distinguishing the
spatial foreground in comparison to the background. More importantly, this
information becomes an integral part of the later-stage channels collective
knowledge by concatenation-based sharing with them. Consequently, later-stage
channels use this information to boost areas of interest and neglect
background pixels. The SM weights are trained using the cross-entropy error
($Loss_{SM}$) between the SM and the ground-truth map. To compute the ground-
truth map, we use each person’s localization information already available in
the benchmark datasets. This attention process significantly improves the
network performance as shown in the ablation study in Sec. 5.5.

###  3.5 Crowd Regression Head ($CRH$)

The VACM module outputs three separate sets of final feature maps (FFB-B1,
FFB-B2, FFB-B3), each for the respective branch. These channels are then
routed to the Crowd Regression Head (CRH). The CRH concatenates the FFB-B2 and
FFB-B3 outputs with the FFB-B1 output channels using the bilinear upsampling
(BU) as shown in Fig. 5. Following that, it passes through various deep
convolution and Fully Connected (FC) layers and eventually through the final
1-dimensional single neuron (1D, FC) to regress the final crowd count
(${CC}_{IP}$) for the input patch ($IP$) as shown in Table 1. When class label
${Class}_{IP}=HCP$, then the crowd count for the input patch ($IP$) is the sum
of all four PRM generated and rescaled ($256\times 256$ size) patches, given
as follows:

$\begin{split}CC_{IP}=\sum_{i=1}^{4}cc_{p(i)},\ (if\
Class_{IP}=HCP)\end{split}$ (3)

where $cc_{p(i)}$ with ($i=1,2,3\ and\ 4$) denotes the four rescaled patches
created by the PRM for the input patch ($IP$) being classified with the $HCP$
label. The regressor utilizes Mean Square Error (MSE) as its training loss
function, defined as follows:

$Loss_{Regressor}=\frac{1}{T}\sum_{s=1}^{T}(F(x_{s},\Theta)-y_{s})^{2}$ (4)

where $T$ represents the total training samples per batch, $y_{s}$ indicates
the actual crowd number for the input patch $x_{s}$, and $F(.)$ represents the
mapping or transformation function with the learnable weights parameters
$\Theta$ that learns to regress the crowd number for the input patch. Finally,
the total network loss ($Loss_{total}$) is the sum of 4-way classification,
segmentation map (SM), and the crowd regression losses:

$Loss_{total}=Loss_{Regressor}\ +\ Loss_{CH}+\ Loss_{SM}$ (5)

##  4 Evaluation and Training Details

We utilize the commonly used crowd counting metrics for numerical evaluation:
Mean Absolute Error ($MAE$) and Root Mean Squared Error ($RMSE$), as defined
below:

$MAE=\frac{1}{TI}\sum_{h=1}^{TI}|CC_{h}-\hat{CC_{h}}|$ (6)

$RMSE=\sqrt[]{\frac{1}{TI}\sum_{h=1}^{TI}(CC_{h}-\hat{CC_{h}})^{2}}$ (7)

where $TI$ indicates the total number of test images, and $CC_{h}\ and\
\hat{CC_{h}}$ denote the true and the predicted crowd counts respectively for
the test image $h$.

Network Training Details. To train the network, we randomly take out $75,000$
patches of $128\times 128$, $256\times 256$, and $512\times 512$ sizes from
the predefined training images. The resultant patches with mixed crowd numbers
are resized to $256\times 256$ size as per required. We double the training
patches number by performing the horizontal-flip based data augmentation on
each patch. The proposed network has been trained for 120 epochs with a batch
size of 16. Stochastic gradient descent (SGD) has been used as the optimizer
with a weight decay of 0.0001 and the Nesterov Momentum equal to 0.9. We also
employed the multi-step learning rate ($\alpha$) that starts with the value of
0.001 and decreases to 1/2 after every 30 epochs. Also, $10\%$ training data
has been separated for the network validation purpose during the model
training.

##  5 Experiments

This section presents both numerical and visual results based on different
experiments conducted on four extensively used crowd counting benchmarks: UCF-
QNRF [30], ShanghaiTech [8], AHU-Crowd [35], and the WorldExpo’10 [36]
dataset. These datasets are totally different from each other as they vary
remarkably in terms of image resolution, average people count per image,
maximum/minimum people per image, background regions, total images, and
different lighting conditions. First, we compare our scheme numerically with
the state-of-the-art models on these benchmarks. During these quantitative
experiments, we also compare with the original HRNet [17, 18] model for its
two independent versions: HRNet-DM and HRNet-DR. The HRNet-DM network outputs
a density-map (DM) by first concatenating all branches outputs at Branch-1,
followed by up-sampling it to the size of $256\times 256$. This approach is
similar to the original HRNet for semantic segmentation. The final crowd-count
for the input patch ($IP$) will then be the sum of all DM pixel values.
Whereas, the HRNet-DR version directly regresses the final single-value crowd
count, and can be considered similar to the proposed network without the VACM
and PRM mechanisms, but having more residual blocks and parameters as given in
the original HRNet [17, 18]. Similarly, we also report results for the
proposed scheme base network (HRNet-base) without the PRM and VACM modules.
Next, we present the ablation experiments and the classification head $CH$
performance analysis. Followed by the cross-dataset evaluation. Finally, the
visual analysis has been presented to discuss qualitative performance.

| ShanghaiTech-A | ShanghaiTech-B | UCF-QNRF
---|---|---|---
Method | MAE | RMSE | MAE | RMSE | MAE | RMSE
MCNN [8] | 110.2 | 173.2 | 26.4 | 41.3 | 277 | 426.0
CMTL [9] | 101.3 | 152.4 | 20.0 | 31.1 | 252 | 514.0
Switch-CNN [10] | 90.4 | 135.0 | 21.6 | 33.4 | 228 | 445.0
SaCNN [37] | 86.8 | 139.2 | 16.2 | 25.8 | - | -
IG-CNN [38] | 72.5 | 118.2 | 13.6 | 21.1 | - | -
ACSCP [39] | 75.7 | 102.7 | 17.2 | 27.4 | - | -
CSRNet [29] | 68.2 | 115.0 | 10.6 | 16.0 | - | -
CL[30] | - | - | - | - | 132 | 191.0
CFF [40] | 65.2 | 109.4 | 7.2 | 12.2 | 93.8 | 146.5
RRSP [13] | 63.1 | 96.2 | 8.7 | 13.6 | - | -
CAN [41] | 62.3 | 100.0 | 7.8 | 12.2 | 107 | 183.0
L2SM [14] | 64.2 | 98.4 | 7.2 | 11.1 | 104.7 | 173.6
BL [42] | 62.8 | 101.8 | 7.7 | 12.7 | 88.7 | 154.8
RRP [43] | 63.2 | 105.7 | 9.4 | 13.9 | 93 | 156.0
HA-CCN [44] | 62.9 | 94.9 | 8.1 | 13.4 | 118.1 | 180.4
ADSCNet [45] | 55.4 | 97.7 | 6.4 | 11.3 | 71.3 | 132.5
RPNet [46] | 61.2 | 96.9 | 8.1 | 11.6 | - | -
ZoomCount [6] | 66.6 | 94.5 | - | - | 128 | 201.0
PRM-based[7] | 67.8 | 86.2 | 8.6 | 11.0 | 94.5 | 141.9
HRNet-DM [17] | 90.5 | 111.9 | 18.7 | 29.2 | 173 | 285.4
HRNet-DR [17] | 88.8 | 110.3 | 17.6 | 27.6 | 171 | 277.2
HRNet-Base | 76.3 | 110.0 | 14.2 | 21.8 | 122.5 | 219.0
Ours | 56.1 | 79.8 | 6.6 | 9.8 | 71.3 | 120.7
Ours w/o PRM | 72.1 | 109.8 | 12.1 | 20.9 | 98.1 | 137.5
Table 2: Numerical experiments on the UCF-QNRF [30] and the ShanghaiTech [8]
benchmarks. Our proposed scheme outperforms the state-of-the-art models
(including the original PRM-based) under the RMSE standard criterion, while
indicating closer or equal to the best results for the MAE evaluation metric.
The proposed network errors increase in all cases without the PRM module usage
(last row).

###  5.1 UCF-QNRF Dataset Numerical Evaluation

The UCF-QNRF [30] dataset contains a total of $1,535$ images with a pre-
established training/testing division of 1201/334 respectively. The images
contain a wide range of crowd-density and vary greatly in image resolution and
background setting. The total people annotations in the dataset equal to
$1,251,642$, while the images resolution varies between ($300\times 377$) and
($6666\times 9999$). We compare the proposed approach with the state-of-the-
art models as reported in Table 2. The evaluation demonstrates that our model
performs the best in comparison to the state-of-the-art for the RMSE
evaluation criterion with $8.9\%$ performance boost (from $132.5$ to $120.7$)
amid performing equally best for the MAE metric. It may also be noted that the
proposed model performs better for both metrics as compared to the original
PRM based scheme (CC-2P) [7]. Without the PRM module (last row), the proposed
scheme performance degrades hugely by $27.3\%$ and $12.2\%$ in terms of the
MAE and RMSE, respectively. This result demonstrates the PRM significance in
the proposed model. Moreover, the HRNet-DM and HRNet-DR models prove
insufficient as shown in Table 2.

###  5.2 ShanghaiTech Dataset Numerical Evaluation

The ShanghaiTech [8] dataset is pre-divided into two independent parts. Part-A
comprises of a total $482$ images with diverse and dense crowd range, image
resolution, and varying lighting conditions. These images are already divided
into $300$ training and $182$ testing images. The average image resolution is
$589\times 868$ with a total of $241,677$ human annotations. Total people per
image is approximately $501$ on average. Part-B contains $716$ images
($400/316$ train/test split respectively) with relatively sparse crowd range
and a total of $88,488$ people annotations. The mean image resolution for this
part is $768\times 1024$ with an average of around $124$ people per image. We
analyze the proposed scheme on this benchmark and also compare it with the
state-of-the-art (including the original PRM-based [7]) models. The results
are shown in Table 2, from which we can see that our model yields the best
results for the RMSE criterion with an improvement of $7.4\%$ (from $86.2$ to
$79.8$) and $10.9\%$ (from $11.0$ to $9.8$) on both benchmark parts
respectively, and also performs reasonably well in terms of the MAE evaluation
metric. For both dataset parts, our model subjects to a huge error increase
(MAE: $22.2\%$, RMSE: $27.3\%$ for Part-A, and MAE: $45.4\%$, RMSE: $53.1\%$
for Part-B) without the PRM module deployment.

Method | MAE | RMSE
---|---|---
Haar Wavelet [47] | 409.0 | -
DPM [48] | 395.4 | -
BOW–SVM [49] | 218.8 | -
Ridge Regression [20] | 207.4 | -
Hu et al. [35] | 137 | -
DSRM [50] | 81 | 129
ZoomCount [6] | 74.9 | 111
CC-2P (PRM-based)[7] | 66.6 | 101.9
HRNet-DM [17] | 80.8 | 125.8
HRNet-DR [17] | 80.0 | 121.1
HRNet-Base | 78.9 | 119.3
Ours | 57.5 | 89.0
Ours w/o PRM | 76.1 | 115.3
Table 3: AHU-Crowd [35] benchmark dataset based experiments indicate that the
proposed scheme appears to be the best for both the evaluation metrics in
contrast to the state-of-the-arts including the original PRM-based model. PRM-
less version of our network subjects to a huge error increase in all settings
as given in last row.

###  5.3 Numerical Experiments on AHU-Crowd Dataset

AHU-Crowd dataset [35] poses a great challenge with totally different
statistics as compared to the ShanghaiTech and the UCF-QNRF datasets. It only
contains $107$ images with $58$ to $2,201$ ground-truth people count per
image. The dataset also contains a total of $45,807$ people annotations. Based
on the standard literature practice, we carried out the 5-fold cross-
validation for the (MAE, RMSE) based numerical evaluation. In each fold, $96$
images were selected for training, and the remaining $11$ images for the
testing purpose. We report the numerical evaluation and comparison results in
Table 3. These findings indicate that the proposed method outperforms other
state-of-the-art methods (including the original PRM based scheme) with
significant improvement of $13.7\%$ (from $66.6$ to $57.5$), $12.6\%$ (from
$101.9$ to $89.0$) for the MAE and RMSE metrics respectively. Moreover, our
model without the PRM module, HRNet-DM, and HRNet-DR networks subject to a
huge error increase in each case as shown in Table 3.

| S1 | S2 | S3 | S4 | S5 | Average
---|---|---|---|---|---|---
Zhang et al. [36] | 9.8 | 14.1 | 14.3 | 22.2 | 3.7 | 12.9
MCNN [8] | 3.4 | 20.6 | 12.9 | 13.0 | 8.1 | 11.6
Switch-CNN [10] | 4.4 | 15.7 | 10.0 | 11.0 | 5.9 | 9.4
CP-CNN [51] | 2.9 | 14.7 | 10.5 | 10.4 | 5.8 | 8.9
IG-CNN [38] | 2.6 | 16.1 | 10.15 | 20.2 | 7.6 | 11.3
IC-CNN [11] | 17.0 | 12.3 | 9.2 | 8.1 | 4.7 | 10.3
CAN [41] | 2.4 | 9.4 | 8.8 | 11.2 | 4.0 | 7.2
ZoomCount [6] | 2.1 | 15.3 | 9.0 | 10.3 | 4.5 | 8.3
M-SFANet [52] | 1.88 | 13.24 | 10.07 | 7.5 | 3.87 | 7.32
DSSINet [53] | 1.57 | 9.5 | 9.46 | 10.35 | 2.49 | 6.67
ASNet [54] | 2.22 | 10.11 | 8.89 | 7.14 | 4.84 | 6.64
PRM-based[7] | 1.8 | 10.7 | 9.2 | 8.8 | 4.3 | 6.96
HRNet-DM [17] | 3.5 | 15.2 | 11.3 | 10.3 | 5.2 | 9.10
HRNet-DR [17] | 3.5 | 14.9 | 10.8 | 10.1 | 4.9 | 8.84
HRNet-Base | 2.95 | 13.1 | 10.2 | 9.6 | 4.8 | 8.13
Ours | 1.53 | 9.04 | 8.03 | 7.10 | 2.22 | 5.58
Ours w/o PRM | 2.19 | 10.98 | 9.75 | 9.11 | 5.08 | 7.42
Table 4: Quantitative evaluation and comparison on the WorldExpo’10 dataset
[36] demonstrate that the proposed scheme appears as the best by yields the
least MAE error on all five test scenes (S1-S5) as well as their average
value. Our PRM-less network (last row) subjects to significant performance
degradation, thus, emphasizing its effectiveness.

###  5.4 WorldExpo’10 Benchmark Numerical Experiments

The WorldExpo’10 [36] dataset contains $3,980$ annotated frames, collected
from $1,132$ video sequences of $108$ different scenes. Out of these frames,
$3,380$ images from $103$ different scenes have been used for the training
purpose. The remaining $600$ frames from $5$ different scenes work as the test
split. As per the standard practice, we only utilize the pre-defined Region of
Interest (RoI) as given in both training and testing images. Also, we evaluate
and compare with the state-of-the-art on five test scenes using the MAE
evaluation metric. As shown in Table 4, the proposed approach yields the best
performance on all scene images with the $16.0\%$ (from $6.64$ to $5.58$)
average MAE error decrease. The model subjects to $24.8\%$ average MAE
increase (from $5.58$ to $7.42$) without the PRM module deployment, thus,
signifying its importance in our network.

###  5.5 Ablation Experiments Study

In this section, we present six ablation studies on the ShanghaiTech dataset
[8] to investigate the effect of different components of the proposed scheme.

Total Branches Choice
---
Total Branches | MAE | RMSE
1 | 91.6 | 131.4
2 | 73.6 | 109.7
3 (our default choice) | 56.1 | 79.8
4 | 69.6 | 97.5
Residual Units (RUs) per RB Block Quantity Effect
RU units per RB block: 2 | 77.9 | 101.1
3 | 71.5 | 98.3
4 (our default choice) | 56.1 | 79.8
5 | 71.7 | 97.2
6 | 73.6 | 102.8
2-Layered vs 3-Layered Residual Unit Choice
2-Layered | 70.6 | 96.9
3-Layered (our default choice) | 56.1 | 79.8
Branching-out Positioning in the Network
From RB (of Phase1) | 71.0 | 97.9
1st RB (of Branch-1, Phase2) | 56.1 | 79.8
2nd RB (of Branch-1, Phase2) | 69.1 | 94.8
1st RB (of Branch-1, Phase3) | 70.9 | 96.1
Visual Attention (VACM) Effect
w/o VACM | 63.9 | 82.4
w VACM (our default choice) | 56.1 | 79.8
EFM-Bx and LFM-Bx choices for SM, VACM Generation and Concatenation
Features used for SM & VAFM Generation | For Concatenation | MAE | RMSE
EFM-Bx (our default) | LFM-Bx (our default) | 56.1 | 79.8
LFM-Bx | LFM-Bx | 59.4 | 81.0
LFM-Bx | EFM-Bx | 58.8 | 80.5
Table 5: Six different sets of ablation experiments validate our selection of
the few vital hyper-parameters for the proposed network.

Figure 6: Four-branch based version of the proposed network for the branch
quantity ablation study. In this setting, a new branch (Branch-4) has been
added into the proposed (three-branch) model that naturally results in a new
phase (Phase-4) as well. Also, one additional EFM (EFM-B4) block emerges from
Branch-4 that passes through VACM along with its respective LFM block
(LFM-B4). The outputs from Phase-4 now serve as the LFM feature maps for the
remaining crowd estimation process.

Effect of the total number of Columns/Branches. First ablation study discusses
the consequences related to the quantity of multi-resolution branches (or
columns) being used in the proposed network. We explore this critical hyper-
parameter by experimenting separately with different quantities of such
branches. The study results are shown in Table 5. Using only one branch
($Branch-1$) produces the worst results, since it only contains a single
column and thus lacks any fusion or information-sharing. Two-Branch
($Branch-1,Branch-2$) network does not contain the $Branch-3$ column, but
performs better than one-branch based model. The proposed network, with three
multi-resolution branches, gives the best performance in comparison to the
above configurations as well as the model with four multi-resolution branches
as indicated in the same table. Due to this ablation study outcome, the
proposed model has been designed with three multi-resolution branches. In the
four-branch network experiment, we deployed an additional branch ($Branch-4$)
with $2\times$ down-scaled resolution ($8\times 8$) and double the channels
($256$) than the $Branch-3$ as shown in Fig. 6. The four-branch based model
naturally contains an additional phase ($Phase4$) to cover the fusion process
for $Branch-4$ with other branches. Naturally, it also contains one more
early-stage block (EFM-B4) emerging from Branch-4. The EFM-B4 is routed to the
VACM module along with its respective later-stage block (LFM-B4).
Additionally, the $Phase4$ outputs now serve as the LFM blocks.

Effect of the number of residual units in RB blocks. Here, we investigate the
effect of using four 3-layered residual units (RU) per RB block as compared to
deploying other potential quantities ($2,3,5,$ or $6$ RU units per RB block).
As shown in the ablation experiments results in Table 5, 4 RU units per RB
block yield the best results with the lowest MAE and RMSE errors. Thus, it
acts as our default and preferred choice in the proposed network.

Effect of using 3-layered vs 2-layered residual unit. We have two major
choices for the Residual Unit: 2-layered or 3-layered deep residual unit as
given in [32]. Results for both choices are shown in Table 5. It is evident
that the 3-layered RU performs much better than the 2-layered residual unit.

Effect of branching-out location in the network. As shown in the proposed
network in Fig. 2, we branch-out the output features of the first RB block in
the (Branch-1, Phase-2) to feed into the classification head. Here, we
investigate the effect of the location of this branching-out by re-positioning
it to other Branch-1 RB blocks output. As shown in Table 5, our default choice
of branching-out from 1st RB of (Branch-1, Phase-2) gives the lowest MAE and
RMSE error. Additionally, in each ablation experiment setting, the
concatenation back into the network happens with the RB block of the Branch-1
that is next and subsequent to the RB block responsible for the branching-out.

VACM Module Effect. Visual attention on the early-stage channels helps in
enriching the later-stage feature maps. Consequently, the VACM module should
improve the overall network performance. As shown in Table 5, the visual
attention process boosts the network effectiveness by ($12.2\%$, $3.2\%$) in
terms of (MAE, RMSE) respectively.

EFM-Bx and LFM-Bx usage choices effect in the VACM module. Here, we
investigate the effect of deploying the EFM-Bx and LFM-Bx features in
different combinations for SM and VAFM features generation and the subsequent
concatenation process in the VACM module. The results are shown in Table 5,
where the EFM-Bx and LFM-Bx features have been used in different settings for
SM and VAFM generation as well as the concatenation process. As shown, the
proposed combination of deploying EFM-Bx features for SM and VAFM generation,
followed by concatenating them into the LFM-Bx features empirically proves to
be the most effective choice.

Figure 7: The classification head ($CH$) effectiveness and PRM label-wise usage analysis on the ShanghaiTech-A dataset test image patches. The left graph indicates that the $CH$ head performs better than the state-of-the-art CC-2P [7] model classifier on all four labels (NCP,LCP,MCP,HCP). It also demonstrates that a huge percentage of patches ($55.9\%$) are designated with either $LCP$ or $HCP$, thus, emphasizing frequent use of the subsequent PRM rescaling operation. The $CH$ also performs reasonably well in terms of label-wise precision (right block).  Method | MAE | RMSE
---|---|---
Switch-CNN [10] | 301 | 457
Cascaded-mtl [9] | 308 | 478
CC-2P (PRM based) [7] | 219 | 305
Ours | 197 | 271
Table 6: Based on the comparison with the state-of-the-art methods during the
cross-dataset evaluation, our approach outperforms them under both evaluation
criteria.

###  5.6 Cross-Dataset Evaluation

To further assess the proposed model, we conducted the cross-dataset
evaluation. ShanghaiTech [8] benchmark has been used for all models training,
while the testing has been conducted using the UCF-QNRF dataset. Table 6
reports the proposed approach cross-dataset performance and also compares it
to other state-of-the-art schemes. It is clear from the results that the
proposed model outperforms other methods including the original PRM-based
scheme. These findings also indicate the better generalization potential of
our scheme towards unseen images with different dynamics and crowd diversity.

###  5.7 Classification Head ($CH$) performance and PRM class-wise usage
analysis

The classification head ($CH$) plays a pivotal role in the success of the
subsequent PRM process. Here, we investigate and compare our classification
head ($CH$) effectiveness with the state-of-the-art PRM-based CC-2P model [7]
classifier on the ShanghaiTech-A [8] benchmark test images patches. The
results for all four crowd-labels ($NCP,LCP,MCP,HCP$) are shown in Fig. 7,
where we can see that our $CH$ classifier yields more accurate and closer to
the ground-truth results for each class-label as compared to the CC-2P
classifier. On the other hand, this analysis also gives insight into the
percentage of image patches utilizing any specific PRM re-scaling operation.
As shown in Fig. 7, around $41.9\%$ and $14\%$ of test image patches have been
classified as $LCP$ and $HCP$ respectively, and consequently require
appropriate PRM-based re-scaling. Similarly, $\sim 13\%$ of the test patches
have been discarded as being classified with the $NCP$ label that may have
resulted in huge crowd over-estimation otherwise. Thus, the $CH$ head proves
more effective in rightly categorizing all crowd-density labels. The PRM’s
main re-scaling operations (Up- and Down-scaling) are required in most cases
($55.9\%$), therefore, emphasizing its great importance in the proposed
scheme. The $CH$ head also demonstrates reasonable performance in terms of
label-wise precision (with minimum $94\%$ precision) as shown on the right in
Fig. 7. This is partly due to better training of Phase-1 RB block with the aid
of inter-branch fusion and the VACM process.

GT=1443, PRM=1131

Ours=1427, DME=388

GT=1103, PRM=908

Ours=1089, DME=842

GT=556, PRM=478

Ours=555, DME=236

GT=3653, PRM=2992

Ours=3609, DME=1692

GT=2472, PRM=2017

Ours=2488, DME=1370

GT=207, PRM=130

Ours=204, DME=109

Figure 8: Visual evaluation. The proposed scheme generates better crowd counts than the state-of-the-art PRM-based [7] and the density-map estimation (DME) [30] based methods when compared in terms of the ground truth (GT). For each test image, the VACM generated segmentation-map (SM) also indicates reasonable foreground information capture that is being utilized by subsequent spatial visual attention process.  Method |  $T_{total}$ (secs) |  $T_{avg}$ (secs) |  $Parameters$ (millions) | MAE | RMSE
---|---|---|---|---|---
CSRNet [29] | 60.2 | 0.33 | 16.3 | 68.2 | 115.0
CP-CNN [51] | 122.8 | 0.68 | 68.4 | 73.6 | 106.4
ZoomCount [6] | 85.4 | 0.47 | 14.0 | 66.6 | 94.5
CC-2P [7] | 55.4 | 0.30 | 5.1 | 67.8 | 86.2
Ours | 49.7 | 0.27 | 25.6 | 56.1 | 79.8
Table 7: Inference speed and total model parameters analysis on ShanghaiTech
Part-A dataset.

###  5.8 Inference Speed Analysis

In this experiment, we perform inference speed analysis and comparison with
four state-of-the-art schemes (CSRNet [29], CP-CNN [51], ZoomCount [6], PRM-
based CC-2P [7]) as given in Table 7. $T_{total}$ and $T_{avg}$ denote total
and average time taken by the models on $182$ test images of ShanghaiTech
Part-A benchmark. NVIDIA Titan Xp GPU has been used in all settings. As shown
in this result, the proposed scheme appears as the most efficient and
effective one with the least $T_{total}$, $T_{avg}$, MAE, and RMSE values.
Better efficiency is achieved due to the parallel structure of the proposed
network. Moreover, in the high-density case (${Class}_{IP}=HCP$), all four
PRM-generated independent patches are also processed in parallel in the next
steps, thus, increasing the efficiency. We also report the total learnable
network parameters in the same table. The PRM-based CC-2P network contains the
least parameters. The proposed scheme contains relatively much more
parameters, but yields better efficiency and speed.

###  5.9 Visual Analysis

In this section, we show a few visual results. Six original test images are
shown in Fig. 8, where we analyze our scheme against the state-of-the-art PRM
[7] and the density-map estimation (DME) [30] based methods. For each test
image, we show the original image as well as the segmentation map (SM) being
generated in the VACM module. These images contain hugely varying crowd-
density and scale with fluctuating lighting conditions and background. As
shown in Fig. 8, the proposed scheme yields the best results that are closer
to the ground-truth in comparison to the two competing models. For the
visualization purpose, each SM is thresholded at 0.5 to make it a binary
segmentation map. As it can be seen qualitatively, the early-stage feature
maps generate these highly accurate maps and help in foreground information
capture that is being utilized by subsequent spatial visual attention process.

##  6 Conclusion

We have proposed a new multi-resolution fusion and multi-task based crowd
counting network with visual attention in this paper by further exploring and
more effectively utilizing the PRM module. The proposed method relies on the
PRM module and builds the collective knowledge using the feature-level fusion
across the multi-resolution branches as well as visually attending the early-
stage channels to boost the foreground vs background understanding of later-
stage channels. This integration technique outperforms the state-of-the-art
(including the original PRM based) methods as demonstrated through extensive
standard numerical and visual experiments and comparisons. The proposed scheme
also demonstrates better generalization ability during the cross-dataset
evaluations.

## Acknowledgement

This work was partly supported in part by the Natural Sciences and Engineering
Research Council of Canada (NSERC) under grant no. RGPIN-2021-04244, and the
United States Department of Agriculture (USDA) under Grant no.
2019-67021-28996. We would also like to thank Xi Mo for proofreading the
paper.

## References

  * [1] F. Cen, X. Zhao, W. Li, G. Wang, Deep feature augmentation for occluded image classification, Pattern Recognition 111 (2021) 107737.
  * [2] K. Li, N. Y. Wang, Y. Yang, G. Wang, Sgnet: A super-class guided network for image classification and object detection, arXiv preprint arXiv:2104.12898.
  * [3] W. Ma, K. Li, G. Wang, Location-aware box reasoning for anchor-based single-shot object detection, IEEE Access 8 (2020) 129300–129309.
  * [4] U. Sajid, X. Chen, H. Sajid, T. Kim, G. Wang, Audio-visual transformer based crowd counting, arXiv preprint arXiv:2109.01926.
  * [5] C. Wang, H. Zhang, L. Yang, S. Liu, X. Cao, Deep people counting in extremely dense crowds, in: Proceedings of the 23rd ACM international conference on Multimedia, 2015, pp. 1299–1302.
  * [6] U. Sajid, H. Sajid, H. Wang, G. Wang, Zoomcount: A zooming mechanism for crowd counting in static images, IEEE Transactions on Circuits and Systems for Video Technology, 30 (10) (2020) 3499–3512.
  * [7] U. Sajid, G. Wang, Plug-and-play rescaling based crowd counting in static images, in: The IEEE Winter Conference on Applications of Computer Vision, 2020, pp. 2287–2296.
  * [8] Y. Zhang, D. Zhou, S. Chen, S. Gao, Y. Ma, Single-image crowd counting via multi-column convolutional neural network, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 589–597.
  * [9] V. A. Sindagi, V. M. Patel, CNN-based cascaded multi-task learning of high-level prior and density estimation for crowd counting, in: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2017, pp. 1–6.
  * [10] D. B. Sam, S. Surya, R. V. Babu, Switching convolutional neural network for crowd counting, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2017, pp. 4031–4039.
  * [11] V. Ranjan, H. Le, M. Hoai, Iterative crowd counting, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 270–285.
  * [12] U. Sajid, W. Ma, G. Wang, Multi-resolution fusion and multi-scale input priors based crowd counting, arXiv preprint arXiv:2010.01664.
  * [13] J. Wan, W. Luo, B. Wu, A. B. Chan, W. Liu, Residual regression with semantic prior for crowd counting, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 4036–4045.
  * [14] C. Xu, K. Qiu, J. Fu, S. Bai, Y. Xu, X. Bai, Learn to scale: Generating multipolar normalized density map for crowd counting, arXiv preprint arXiv:1907.12428.
  * [15] M. Shami, S. Maqbool, H. Sajid, Y. Ayaz, S.-C. S. Cheung, People counting in dense crowd images using sparse head detections, IEEE Transactions on Circuits and Systems for Video Technology 29 (9) (2018) 2627–2636.
  * [16] W. Li, H. Li, Q. Wu, F. Meng, L. Xu, K. N. Ngan, Headnet: An end-to-end adaptive relational network for head detection, IEEE Transactions on Circuits and Systems for Video Technology 30 (2) (2019) 482–494.
  * [17] K. Sun, B. Xiao, D. Liu, J. Wang, Deep high-resolution representation learning for human pose estimation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2019, pp. 5693–5703.
  * [18] J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang, et al., Deep high-resolution representation learning for visual recognition, IEEE transactions on pattern analysis and machine intelligence.
  * [19] A. B. Chan, N. Vasconcelos, Bayesian poisson regression for crowd counting, in: IEEE International Conference on Computer Vision, 2009, pp. 545–551.
  * [20] K. Chen, C. C. Loy, S. Gong, T. Xiang, Feature mining for localised crowd counting., in: BMVC, Vol. 1, 2012, p. 3.
  * [21] D. Ryan, S. Denman, C. Fookes, S. Sridharan, Crowd counting using multiple local features, in: Digital Image Computing: Techniques and Applications, DICTA, 2009, pp. 81–88.
  * [22] W. Ge, R. T. Collins, Marked point processes for crowd counting, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2009, pp. 2913–2920.
  * [23] M. Li, Z. Zhang, K. Huang, T. Tan, Estimating the number of people in crowded scenes by mid based foreground segmentation and head-shoulder detection, in: 2008 19th International Conference on Pattern Recognition, IEEE, 2008, pp. 1–4.
  * [24] M. Wang, X. Wang, Automatic adaptation of a generic pedestrian detector to a specific traffic scene, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 3401–3408.
  * [25] B. Wu, R. Nevatia, Detection of multiple, partially occluded humans in a single image by bayesian combination of edgelet part detectors, in: IEEE International Conference on Computer Vision, 2005, pp. 90–97.
  * [26] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once: Unified, real-time object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 779–788.
  * [27] W. Ma, Y. Wu, F. Cen, G. Wang, Mdfn: Multi-scale deep feature learning network for object detection, Pattern Recognition 100 (2020) 107149.
  * [28] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, 2012, pp. 1097–1105.
  * [29] Y. Li, X. Zhang, D. Chen, CSRNet: Dilated convolutional neural networks for understanding the highly congested scenes, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1091–1100.
  * [30] H. Idrees, M. Tayyab, K. Athrey, D. Zhang, S. Al-Maadeed, N. Rajpoot, M. Shah, Composition loss for counting, density map estimation and localization in dense crowds, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 532–546.
  * [31] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger, Densely connected convolutional networks., in: CVPR, Vol. 1, 2017, p. 3.
  * [32] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [33] S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv:1502.03167.
  * [34] V. Nair, G. E. Hinton, Rectified linear units improve restricted boltzmann machines, in: ICML, 2010.
  * [35] Y. Hu, H. Chang, F. Nian, Y. Wang, T. Li, Dense crowd counting from still images with convolutional neural networks, Journal of Visual Communication and Image Representation 38 (2016) 530–539.
  * [36] C. Zhang, H. Li, X. Wang, X. Yang, Cross-scene crowd counting via deep convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 833–841.
  * [37] L. Zhang, M. Shi, Q. Chen, Crowd counting via scale-adaptive convolutional neural network, in: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, 2018, pp. 1113–1121.
  * [38] D. Babu Sam, N. N. Sajjan, R. Venkatesh Babu, M. Srinivasan, Divide and Grow: Capturing Huge Diversity in Crowd Images With Incrementally Growing CNN, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3618–3626.
  * [39] Z. Shen, Y. Xu, B. Ni, M. Wang, J. Hu, X. Yang, Crowd counting via adversarial cross-scale consistency pursuit, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5245–5254.
  * [40] Z. Shi, P. Mettes, C. G. Snoek, Counting with focus for free, arXiv preprint arXiv:1903.12206.
  * [41] W. Liu, M. Salzmann, P. Fua, Context-aware crowd counting, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 5099–5108.
  * [42] Z. Ma, X. Wei, X. Hong, Y. Gong, Bayesian loss for crowd count estimation with point supervision, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 6142–6151.
  * [43] X. Chen, Y. Bin, C. Gao, N. Sang, H. Tang, Relevant region prediction for crowd counting, Neurocomputing 407 (2020) 399–408.
  * [44] V. A. Sindagi, V. M. Patel, HA-CCN: Hierarchical attention-based crowd counting network, IEEE Transactions on Image Processing 29 (2019) 323–335.
  * [45] S. Bai, Z. He, Y. Qiao, H. Hu, W. Wu, J. Yan, Adaptive dilated network with self-correction supervision for counting, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4594–4603.
  * [46] Y. Yang, G. Li, Z. Wu, L. Su, Q. Huang, N. Sebe, Reverse perspective network for perspective-aware object counting, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4374–4383.
  * [47] M. Oren, C. Papageorgiou, P. Sinha, E. Osuna, T. Poggio, Pedestrian detection using wavelet templates, in: CVPR, Vol. 97, 1997, pp. 193–199.
  * [48] P. Felzenszwalb, D. McAllester, D. Ramanan, A discriminatively trained, multiscale, deformable part model, in: Computer Vision and Pattern Recognition, CVPR, 2008, pp. 1–8.
  * [49] G. Csurka, C. Dance, L. Fan, J. Willamowski, C. Bray, Visual categorization with bags of keypoints, in: Workshop on statistical learning in computer vision, ECCV, Vol. 1, 2004, pp. 1–2.
  * [50] H. Yao, K. Han, W. Wan, L. Hou, Deep spatial regression model for image crowd counting, arXiv preprint arXiv:1710.09757.
  * [51] V. A. Sindagi, V. M. Patel, Generating high-quality crowd density maps using contextual pyramid CNNs, in: 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 1879–1888.
  * [52] P. Thanasutives, K.-i. Fukui, M. Numao, B. Kijsirikul, Encoder-decoder based convolutional neural networks with multi-scale-aware modules for crowd counting, arXiv preprint arXiv:2003.05586.
  * [53] L. Liu, Z. Qiu, G. Li, S. Liu, W. Ouyang, L. Lin, Crowd counting with deep structured scale integration network, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1774–1783.
  * [54] X. Jiang, L. Zhang, M. Xu, T. Zhang, P. Lv, B. Zhou, X. Yang, Y. Pang, Attention scaling for crowd counting, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4706–4715.