©2022 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

# CrowdFormer: Weakly-supervised Crowd counting with Improved Generalizability

###### Abstract

Convolutional neural networks (CNNs) have dominated the field of computer
vision for nearly a decade due to their strong ability to learn local
features. However, due to their limited receptive field, CNNs fail to model
the global context. On the other hand, transformer, an attention-based
architecture can model the global context easily. Despite this, there are
limited studies that investigate the effectiveness of transformers in crowd
counting. In addition, the majority of the existing crowd counting methods are
based on the regression of density maps which requires point-level annotation
of each person present in the scene. This annotation task is laborious and
also error-prone. This has led to increased focus on weakly-supervised crowd
counting methods which require only the count-level annotations. In this
paper, we propose a weakly-supervised method for crowd counting using a
pyramid vision transformer. We have conducted extensive evaluations to
validate the effectiveness of the proposed method. Our method is comparable to
the state-of-the-art on the benchmark crowd datasets. More importantly, it
shows remarkable generalizability.

Index Terms—  Crowd counting, vision transformers, weakly-supervised method.

##  1 Introduction

Analysis of crowd scenes is an important task in video surveillance.
Specifically, the estimation of the crowd motion, crowd behaviour, and
population density are essential to enhancing public safety. A sub-task of
crowd analysis is crowd counting which provides an estimate of the number of
people in a crowd scene. In the last decade, CNN-based methods have proven to
be effective for many computer vision tasks including crowd counting. Taking
advantage of CNN’s ability to learn powerful image representations,
researchers have succeeded in generating comparatively better density maps
than those generated using traditional hand-crafted features.

Transformers, introduced by Vaswani et al. [1] have achieved remarkable
results for natural language processing (NLP) tasks. A transformer adopts a
self-attention mechanism by differentially weighting the significance of each
part of the input data. Motivated by the success of transformers in NLP, many
researchers have explored transformers for computer vision tasks. Unlike the
widely-used CNNs for computer vision tasks, transformers effectively capture
long-range dependencies and have global receptive fields. Dosovitskiy et al.
[2] introduced a vision transformer named ViT for image classification. The
ViT first splits input images into patches and then projects these patches
into 1-D sequences of linear embeddings. These embeddings are then fed into a
transformer encoder. However, the output feature maps of ViT are of low
resolution and single scale. More recently, hierarchical transformer
architectures [3], [4], and [5] have been introduced. These architectures
provide multi-scale feature maps at the output, thereby overcoming the
limitations of ViT.

The main contributions of our work are as follows:

  * •

We introduce a simple end-to-end crowd counting pipeline $-$ CrowdFormer that
utilizes a pyramid structured vision transformer to extract multi-scale
features with a global context.

  * •

We propose an effective feature aggregation module to combine features from
different stages of the transformer and a simple regression head to estimate
the crowd number.

  * •

We conduct extensive experiments on four benchmark crowd counting datasets to
demonstrate the effectiveness of the proposed method. Our method achieves
state-of-the-art counting performance on two of the datasets. On the remaining
two datasets, our results are comparable to the state-of-the-art. Also, our
approach has greater generalizability and achieves state-of-the-art
performance in cross-dataset evaluations, outperforming even most of the
fully-supervised methods.

Fig. 1: The pipeline of the proposed CrowdFormer

##  2 Related Works

In recent years, significant progress has been made with typical crowd
counting methods that regress a density map. These density-map-regression-
based methods employ a feature extraction module that extracts high-level
features from the input images, followed by a regression head for pixel-level
density map regression based on the extracted features. The drawback of these
methods is that they require point-level annotations to generate the ground
truth density maps. Usually, the process of annotation involves placing a dot
at the centre of the head of each person present in the image. This requires a
lot of human effort and becomes challenging especially when the crowd density
is high. In addition, human errors in labeling are inevitable. Furthermore,
these annotations are not required at the time of evaluating the models.
Therefore, it is desirable to develop weakly-supervised crowd counting
methods, which require only the total number of people rather than the
location of each person in the image.

In the literature, weakly-supervised crowd counting models have been developed
without point-level annotations of the data or with limited amount of point-
level annotations. The MATT [6] is trained predominantly using count-level
annotations, along with a small amount of point-level annotations. Similarly,
Sam et al. [7] optimized most of their model parameters with unlabeled data.
The remaining few parameters were optimized using point-level annotations. The
weakly-supervised solution in [8] is based on the Gaussian process for
estimating the crowd density. Yang et al. [9] proposed a soft-label sorting
network that directly regresses the crowd count without any location
supervision.

Transformer-based architectures have been explored in BCCT [10], TransCrowd
[11] and CCTrans [12]. Among these, BCCT employs a ViT [2] based model for
fully supervised crowd counting with various attention mechanisms. TransCrowd
also utilizes a ViT based transformer encoder, followed by a regression head
for weakly-supervised crowd counting. On the other hand, CCTrans [12] uses
Twins-SVT [3] backbone and a sophisticated decoder head. It can perform both
fully as well as weakly-supervised crowd counting.

##  3 Proposed Method

The proposed pipeline for crowd counting is shown in Fig.1. It employs a pre-
trained transformer backbone to extract multi-scale features from the input
image. The features from different stages are combined by a feature
aggregation module and fed into a regression head which gives an estimate of
the crowd count.

###  3.1 Transformer Backbone

The proposed CrowdFormer uses a pyramid vision transformer PVTv2 [13] as a
feature extractor backbone. In particular, we use the ‘pvt_v2_b5’ version of
the PVTv2. It has four stages and each stage generates feature maps of a
different scale. The architecture of each stage consists of an overlapping
patch embedding layer and $L_{i}$ number of transformer encoder layers, i.e.,
$L_{i}$ encoder layers in $i^{th}$ stage.

PVTv2 utilizes overlapping patch embeddings to tokenize images. While
generating image patches, the adjacent windows are overlapped by half of its
area. The overlapping patch embedding is implemented by applying a convolution
with zero-padding and a suitable stride. Specifically, for an input of size
$W\times H\times C$, a convolution layer with kernel size $2S-1$, zero-padding
of $S-1$, stride $S$, and $C^{\prime}$ number of kernels is used to generate
an output of size $\frac{W}{S}\times\frac{H}{S}\times C^{\prime}$. The stride
in convolution for generating the patches is $S=4$ for the first stage and
$S=2$ for the remaining stages. Thus, we obtain a set of feature maps from the
$i^{th}$ stage, with dimensions scaled-down by $2^{(i+1)}$ as compared to the
size of the input image.

At the beginning of stage $i$, the input is evenly divided into overlapping
patches of equal size, then each patch is flattened and projected into $C_{i}$
dimensional embeddings. The embedding dimensions in PVTv2 are 64, 128, 320 and
512 for stage 1, 2, 3, and 4, respectively. These patch embeddings are then
passed through the transformer encoders. Each encoder consists of a self-
attention mechanism and a feed-forward neural network. The position encoding
is done in the feed-forward neural network. The transformer requires this
positional information to determine the sequence order.

In CrowdFormer, the input image size is $384\times 384\times 3$ pixels and the
patch sizes are $7\times 7\times 3$ for the first stage and $3\times 3\times
C_{i}$, where $C_{i}$ is the embedding dimension of the $i^{th}$ stage. As
mentioned previously, $C_{2}=64$, $C_{3}=128$, and $C_{4}=320$. Therefore, we
get the output features (from individual stages) of dimension $96\times
96\times 64$, $48\times 48\times 128$, $24\times 24\times 320$, and $12\times
12\times 512$.

###  3.2 Feature Aggregation and Regression Head

We have designed a novel feature aggregation module for effectively utilizing
the outputs of the feature extractor backbone. We take the feature maps from
each stage, perform global average pooling (GAP) to obtain 1-D sequences of
dimensions 64, 128, 320, and 512. Each of these sequences is then projected
into a 1-D sequence of length 6912. Further, these equal length sequences are
concatenated to get a single sequence of length 27648. The resultant sequence
is then fed into a regression head. We use a simple regression head consisting
of only a single linear layer. It transforms the concatenated sequence into a
scalar value representing the estimated crowd count.

###  3.3 Loss Function

Since the crowd count varies significantly from one image to another, we have
used smooth $L_{1}$ loss rather than $L_{1}$ loss, which is sensitive to
outliers [12]. The smooth $L_{1}$ loss is defined as follows:

$L_{1}\;\mbox{(smooth)}=\begin{cases}\frac{0.5\times(x-y)^{2}}{\beta},&\quad\text{if}\;|x-y|\leq\beta\\\
|x-y|-0.5\times\beta,&\quad\text{otherwise}\\\ \end{cases}$ (1)

The above loss function works as $L_{1}$-loss and $L_{2}$-loss when
$|x-y|>\beta$ and $|x-y|\leq\beta$, respectively. Here, $\beta$ is a hyper-
parameter, $x$ and $y$ are the predicted and the ground truth crowd count,
respectively.

##  4 Experiments

###  4.1 Training Setting and Hyper-parameters

The proposed CrowdFormer uses a pre-trained transformer backbone for feature
extraction. We have initialized the transformer encoder using the weights of
the model pre-trained on the ImageNet-1K dataset [14] and followed the data
preparation strategy adopted in [11]. During training, we use data
augmentation techniques such as random horizontal flipping and grayscaling.
Since image size varies within and across different datasets, all images are
resized to $1152\times 768$. Each resized image is partitioned into six non-
overlapping images of size $384\times 384$.

We use AdamW [15] optimizer. The loss function parameter $\beta$ is set to 1
for SHA and QNRF, 7 for SHB, and 15 for UCF_CC_50 datasets. The learning rate
and weight decay are set to 1e-5 and 1e-5, respectively. The batch size is set
to 1. All our experiments are performed using the PyTorch framework on Google
Colab with a single 16 GB P100 GPU.

Table 1: Comparison with state-of-the-art weakly-supervised methods on SHA [16], SHB [16], UCF_CC_50 [17], and QNRF [18] datasets. The best and second best results are shown in bold and underline, respectively.  Method | UCF_CC_50 | QNRF | SHA | SHB
---|---|---|---|---
MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE
Yang et al. [9] | - | - | - | - | 104.6 | 145.2 | 12.3 | 21.2
MATT [6] | 355.0 | 550.2 | - | - | 80.1 | 129.4 | 11.7 | 17.5
TransCrowd [11] | - | - | 97.2 | 168.5 | 66.1 | 105.1 | 9.3 | 16.1
CCTrans [12] | 245.0 | 343.6 | 92.1 | 158.9 | 64.4 | 95.4 | 7.0 | 11.5
ours | 229.6 | 360.3 | 93.3 | 160.9 | 62.1 | 94.8 | 8.5 | 13.6
Table 2: Performance comparison of different methods in cross-dataset settings. The best and second best results are shown in bold and underline, respectively. (# - fully-supervised, * - weakly-supervised).  Method | SHB $\rightarrow$ SHA | SHA $\rightarrow$ SHB | QNRF $\rightarrow$ SHA | QNRF $\rightarrow$ SHB | SHA $\rightarrow$ QNRF | SHB $\rightarrow$ QNRF
---|---|---|---|---|---|---
MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE
MCNN [16]# | 221.4 | 357.8 | 85.2 | 142.3 | - | - | - | - | - | - | - | -
D-ConvNet [19] # | 140.4 | 226.1 | 49.1 | 99.2 | - | - | - | - | - | - | - | -
RRSP [20] # | - | - | 40.0 | 68.5 | - | - | - | - | - | - | - | -
BL [21] # | - | - | - | - | 69.8 | 123.8 | 15.3 | 26.5 | - | - | - | -
TransCrowd [11]* | 141.3 | 258.9 | 18.9 | 31.1 | 78.7 | 122.5 | 13.5 | 21.9 | - | - | - | -
ours* | 121.6 | 208.8 | 16.0 | 26.0 | 70.8 | 114.9 | 10.3 | 16.0 | 147.6 | 300.8 | 304.4 | 586.1

###  4.2 Datasets and Evaluation Metrics

We have evaluated the proposed CrowdFormer on four benchmark datasets. The
ShanghaiTech [16] dataset is a collection of 1,198 crowd images having a total
of 330,165 annotations. This dataset is divided into two parts $-$
ShanghaiTech Part-A (SHA) containing 300 training and 182 testing images, and
Part-B (SHB) containing 400 training and 316 testing images.

The UCF-QNRF [18] (QNRF) dataset contains 1,535 images captured from various
crowd scenes with approximately a million annotations. The count range varies
from 49 to 12,865, with an average count of 815.4. The training and the
testing set contain 1,201 and 334 images, respectively.

The UCF_CC_50 [17] is a very challenging dataset with severe perspective
distortion. It is a collection of 50 gray-scale images collected from the
Internet. It has a total of 63,974 annotations with an average people count of
1280. Since this dataset does not have a separate training and test set, we
have performed 5-fold cross-validation by following the standard protocol
discussed in [17].

As with previous works, we choose mean absolute error (MAE) and mean squared
error (MSE) to evaluate the counting performance.

$\mbox{MAE}=\frac{1}{N}\sum_{i=1}^{N}|P_{i}-G_{i}|$ (2)
$\mbox{MSE}=\sqrt{\frac{1}{N}\sum_{i=1}^{N}|P_{i}-G_{i}|^{2}}$ (3)

where $N$ is the number of test images, $P_{i}$ and $G_{i}$ are the predicted
and the ground truth count for the $i^{th}$ image, respectively.

##  5 Results and Discussion

###  5.1 Performance on Popular Benchmark Datasets

To evaluate the performance of the proposed method, we conduct a set of
experiments on four popular benchmark datasets. The results are presented in
Table 1. Among the weakly-supervised methods, the proposed CrowdFormer clearly
outperforms TransCrowd on all the datasets and achieves the state-of-the-art
performance on SHA (in terms of both MAE and MSE) and UCF_CC_50 (in terms of
MAE). Specifically, CrowdFormer provides an improvement of 3.6% in MAE and
0.63% in MSE on SHA, 6.3% in MAE on UCF_CC_50 compared with CCTrans [12]. On
the other datasets, our method achieves the second best results.

The most plausible reason for superior performance of CrowdFormer compared
with state-of-the-art CNN-based crowd counting methods is its ability to
capture the global context. As can be observed, CrowdFormer also outperforms
another transformer-based method TransCrowd [11] consistently. This is because
TransCrowd uses only single-scale features, while our method learns multi-
scale image features. Although CCTrans [12] also employs a four-stage pyramid
structured transformer to extract multi-scale features, it uses only features
from the last three stages. In addition, it performs bilinear upsampling in
its complex feature aggregation scheme to increase the resolution of the
feature maps from the latter stages. This may have an impact on the quality of
the feature maps. On the other hand, CrowdFormer uses features from all the
four stages of the transformer and performs learnable projections during
feature aggregation. Our method performs sequence to count mapping, which does
not require summation of a 2-D density map to get the crowd count, as has been
done in [12].

###  5.2 Cross-dataset Performance

We perform a set of cross-dataset evaluations on the four datasets to assess
the generalizability of the proposed CrowdFormer. In these evaluations, we
train the model on a source dataset and test it on a target dataset without
any fine-tuning. As can be observed in Table 2, CrowdFormer achieves state-of-
the-art results for weakly-supervised crowd counting. Our method performs
better than even fully supervised methods except for QNRF $\rightarrow$ SHA
for which BL [21] achieves slightly lower MAE. Note that CCTrans [12] has not
reported its performance in any cross-dataset scenario. The results presented
in Table 2 indicate that the models trained on relatively more challenging
dataset such as SHA or QNRF perform better on less challenging datasets such
as SHB. As expected, the models trained on SHB do not perform very well on SHA
and QNRF.

The CrowdFormer shows remarkable generalizability in cross-dataset evaluations
due to its self-attention-like architecture that models short and long-range
spatial dependencies [22]. A model having good generalizability is expected to
perform well when deployed in the real-world and faced with unseen data with
diverse characteristics arising from changes in crowd density, perspective,
background, and scale.

##  6 Conclusion

We have developed a pyramid vision transformer-based pipeline for crowd
counting in a weakly-supervised setting. The transformer backbone extracts
multi-scale features with global context and effectively captures the semantic
crowd information. The feature aggregation module combines all the features
followed by a simple regression head that regresses the crowd count. Extensive
evaluations on four benchmark crowd datasets demonstrate the effectiveness of
the proposed method in crowd counting and its generalizability.

## References

  * [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 5998–6008.
  * [2] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021.
  * [3] X. Chu, Z. Tian, Y. Wang, B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen, “Twins: Revisiting the design of spatial attention in vision transformers,” in _NeurIPS_ , 2021.
  * [4] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in _IEEE/CVF International Conference on Computer Vision (CVPR)_ , 2021, pp. 568–578.
  * [5] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” _International Conference on Computer Vision (ICCV)_ , 2021.
  * [6] Y. Lei, Y. Liu, P. Zhang, and L. Liu, “Towards using count-level weak supervision for crowd counting,” _Pattern Recognition_ , vol. 109, p. 107616, 2020.
  * [7] D. Sam, N. Sajjan, H. Maurya, and R. Babu, “Almost unsupervised learning for dense crowd counting,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, pp. 8868–8875, 2019.
  * [8] M. Borstel, M. Kandemir, P. Schmidt, M. Rao, K. Rajamani, and F. Hamprecht, “Gaussian process density counting from weak supervision,” in _2016 European Conference on Computer Vision (ECCV)_ , 2016, pp. 365–380.
  * [9] Y. Yang, G. Li, Z. Wu, L. Su, and N. Sebe, “Weakly-supervised crowd counting learns from sorting rather than locations,” in _2020 European Conference on Computer Vision (ECCV)_ , 11 2020, pp. 1–17.
  * [10] G. Sun, Y. Liu, T. Probst, D. Pani Paudel, N. Popovic, and L. Van Gool, “Boosting crowd counting with transformers,” _arXiv e-prints_ , pp. arXiv–2105, 2021.
  * [11] D. Liang, X. Chen, W. Xu, Y. Zhou, and X. Bai, “TransCrowd: Weakly-supervised crowd counting with transformer,” _arXiv preprint arXiv:2104.09116_ , 2021.
  * [12] Y. Tian, X. Chu, and H. Wang, “CCTrans: Simplifying and improving crowd counting with transformer,” _arXiv preprint arXiv:2109.14483_ , 2021\.
  * [13] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “PVTv2: Improved baselines with pyramid vision transformer,” _Computational Visual Media_ , vol. 8, no. 3, pp. 1–10, 2022.
  * [14] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [15] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _International Conference on Learning Representations, ICLR_ , 2019.
  * [16] Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma, “Single-image crowd counting via multi-column convolutional neural network,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 589–597.
  * [17] H. Idrees, I. Saleemi, C. Seibert, and M. Shah, “Multi-source multi-scale counting in extremely dense crowd images,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2013, pp. 2547–2554.
  * [18] H. Idrees, M. Tayyab, K. Athrey, D. Zhang, S. Al-Maadeed, N. Rajpoot, and M. Shah, “Composition loss for counting, density map estimation and localization in dense crowds,” in _European Conference on Computer Vision (ECCV)_ , 2018, pp. 532–546.
  * [19] Z. Shi, L. Zhang, Y. Liu, X. Cao, Y. Ye, M.-M. Cheng, and G. Zheng, “Crowd counting with deep negative correlation learning,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 5382–5390.
  * [20] J. Wan, W. Luo, B. Wu, A. B. Chan, and W. Liu, “Residual regression with semantic prior for crowd counting,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 4031–4040.
  * [21] Z. Ma, X. Wei, X. Hong, and Y. Gong, “Bayesian loss for crowd count estimation with point supervision,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 6141–6150.
  * [22] Y. Bai, J. Mei, A. L. Yuille, and C. Xie, “Are transformers more robust than CNNs?” _CoRR_ , vol. abs/2111.05464, 2021. [Online]. Available: https://arxiv.org/abs/2111.05464