Citation

Y. Logan, M. Prabhushankar and G. AlRegib, ”DECAL: DEployable Clinical Active
Learning,” ICML 2022 Workshop on Adaptive Experimental Design and Active
Learning in the Real World, July 2022

Review

Date of publication: 22 July 2022

Codes

https://github.com/olivesgatech/Patient-Aware-Active-Learning

Bib

@ARTICLE{Logan2022_ICML_ReALML,
author={Y. Logan, M. Prabhushankar and G. AlRegib},
journal={ICML 2022 Workshop on Adaptive Experimental Design and Active
Learning in the Real World},
title={DECAL: DEployable Clinical Active Learning},
year={2022}

Contact

<EMAIL_ADDRESS>OR<EMAIL_ADDRESS>
http://ghassanalregib.info/

# DECAL: DEployable Clinical Active Learning

Yash-yee Logan<EMAIL_ADDRESS>
Mohit Prabhushankar<EMAIL_ADDRESS>
Ghassan AlRegib<EMAIL_ADDRESS>
OLIVES Lab, Georgia Institute of Technology
Atlanta, GA 30332, USA

###### Abstract

Conventional machine learning systems that operate on natural images assume
the presence of attributes within the images that lead to some decision.
However, decisions in medical domain are a resultant of attributes within
medical diagnostic scans and electronic medical records (EMR). Hence, active
learning techniques that are developed for natural images are insufficient for
handling medical data. We focus on reducing this insufficiency by designing a
deployable clinical active learning (DECAL) framework within a bi-modal
interface so as to add practicality to the paradigm. Our approach is a _plug-
in_ method that makes natural image based active learning algorithms
generalize better and faster. We find that on two medical datasets on three
architectures and five learning strategies, DECAL increases generalization
across 20 rounds by approximately 4.81%. DECAL leads to a 5.59% and 7.02%
increase in average accuracy as an initialization strategy for optical
coherence tomography (OCT) and X-Ray respectively. Our active learning results
were achieved using 3000 (5%) and 2000 (38%) samples of OCT and X-Ray data
respectively.

Keywords: Active Learning, Clinical Context, Real-World Deployment

##  1 Introduction and Related Work

Active learning aims to find the optimal subset of samples from a dataset for
a machine learning model to learn a task well (Settles (2009); Dasgupta
(2011)). It is studied because of its ability to reduce the costly and
laborious burden on experts to provide data annotations. Typical setups focus
on acquisition functions that measure the informativeness of samples using
constructs from ensemble learning (Beluch et al. (2018)), probabilistic
uncertainty (Hanneke et al. (2014); Gal et al. (2017)) and data representation
(Sener and Savarese (2017); Geifman and El-Yaniv (2017)). These works were
originally developed for the natural image domain and although several studies
have adapted these and other techniques to medical imagery (Shi et al. (2019);
Nath et al. (2020); Melendez et al. (2016); Otálora et al. (2017); Logan et
al. (2022)), they have not been adopted or utilized in real clinical settings.

One reason for this non-adoption is that conventional active learning does not
follow the diagnostic process. This is because of the experimental settings in
natural images that aided the development of existing active learning
algorithms (Ash et al. (2019); Sener and Savarese (2017); Hsu and Lin (2015)).
Natural images typically contain homogeneous class attributes that can be
extracted from the images themselves. Also, these attributes are usually
enough to distinguish between classes. However, in medicine, pathologies
manifest themselves in visually diverse formats across multiple patients. For
example, the characteristics of an aged healthy person are visually different
from a young healthy person. So how do doctors overcome this? They include
clinical data from EMR to assist with their arrival at a diagnostic decision
(Brush Jr et al. (2017); Brundin-Mather et al. (2018)). EMR consists of
patient ID, demographics, diagnostic imaging and test results that allow a
clinician to make a diagnosis. We recommend that active learning frameworks
for medical image classification be designed within a bi-modal interface so as
to add practicality to the paradigm. With this in mind, we design and evaluate
a DECAL framework that integrates EMR data. We show that DECAL aids existing
active learning algorithms in finding the best subset for labeling as well as
initializing the active learning framework. As such, DECAL is a _plug-in_
approach on top of existing active learning based methods.

##  2 Assessing Active Learning Framework for Medical Domain

We conduct a set of controlled experiments to evaluate the effectiveness of a
DECAL framework relative to conventional frameworks.

###  2.1 Dataset Descriptions

We use images and EMR data from the OCT dataset by Kermany et al. (2018). The
dataset consists of grayscale, cross-sectional, foveal scans having varying
sizes. We use images from 3 retinal diseases: 10488 choroidal
neovascularization (CNV), 36345 diabetic macular edema (DME) and 7756 Drusen
annotated at the image level. Samples in training and oracle sets are from
1852 unique patients. The test set consists of 250 images from each diseased
class from 486 unique patients.

We also use images and EMR data from the X-Ray dataset also by Kermany et al.
(2018). The X-rays are grayscale, cross-sectional chest scans from children
belonging to a healthy class and 2 types of pneumonia: viral and bacterial
annotated at the image level. We use 1349 healthy, 1345 viral and 2538
bacterial samples in the combined training and oracle sets from 2650 unique
patients. The test set consists of 234 healthy, 148 viral and 242 bacterial
images from 431 unique patients. There is also no overlap in patients or
imagery in train or test sets for both datasets. This means the imagery in the
train and test sets come from different patient cohorts. EMR data used for our
analysis was patient identity from both datasets.

Figure 1: Sample imagery of visual characteristics of disease states across patient cohorts. Table 1: Information about dataset size and the number of samples added after each training round. Details | Dataset
---|---
OCT | X-Ray
Oracle + Training Set Images | 54589 | 5232
Images in Initial Training Set | 128 | 128
Images Queried per Iteration | 128 | 128
Unique Patients per Iteration | 128 | 128
Total Unique Patients | 2338 | 3081

###  2.2 Active Learning with EMR Data

Figure 1 shows sample scans from each dataset with patients having the same
disease. The visual characteristics across patients are noticeably different.
This intra-class diversity is a typical occurrence in medical datasets.
Existing active learning paradigms fail to properly account for disease
manifestations from whom there is less data. This oversight is very dangerous
for safety critical domains like medicine. Thus, we posit that EMR data, in
the form of patient identity, be leveraged to account for the intra-class
diversity present in medical datasets. We use patient identity as a _plug-in_
constraint that can be applied prior to sample selection with any query
acquisition function. The next batch of informative samples will have unique
patient identity from the unlabeled pool and be appended to the training set.
This process is repeated to determine the minimum number of labeled samples
needed to maximize model performance.

###  2.3 Experiments

##### Implementation Details

We assess our active learning framework on Resnet-18, Resnet-50 and
Densenet-121 (He et al. (2016); Huang et al. (2017)). We do not use pre-
trained models in any of our analysis. We use the Adam optimizer with a
learning rate of 1.5e-4. Hyper-parameters are tuned based on the OCT dataset
and then the same parameters are used for the X-Ray dataset. For each round,
the Resnet and Densenet models are trained until 98% and 94% accuracy is
achieved on the training set respectively. Following each round, the model’s
weights are reset and randomly initialized. This is repeated with three
different random seeds. We aggregate and report average accuracy and standard
deviation. All images are resized to $128\times 128$ and OCT scans are
normalized with $\mu=0.1987$ and $\sigma=0.0786$ while X-Rays with
$\mu=0.4823$ and $\sigma=0.0379$. Table 1 shows more implementation details
for each dataset.

####  2.3.1 Initializing Active Learning with EMR Data

Existing frameworks typically start active learning by randomly selecting a
small amount of samples to train the initial model. Subsequently, they apply
methods of ranking sample informativeness. By doing this they naively assume
that the data distribution is even, which is hardly the case in medical
datasets as shown in Figure 1. Randomly selecting from an unbalanced
distribution is not guaranteed to gather a representative sample of the
classes present (Zhu et al. (2008)). Therefore, we recommend the integration
of EMR data from the outset to circumvent this.

To do this, we first compute the distribution of patients throughout the
unlabeled pool. Then, we select a fixed number of images from unique patients
IDs and pair it with its annotation for the initial training set. The
intuition behind this strategy is for the first training samples to have
maximally dissimilar images. These samples are then used to start our DECAL
paradigm. We present two experimental modalities in the initialization phase
depending upon the availability of data.

##### Large Initial Training Set

We select 1000 samples at random from the unlabeled pool and train a model for
each architecture and dataset for the first round only as our baseline. Then,
we perform DECAL initialization by selecting one image from a 1000 unique
patients in the unlabeled pool. We then train a model for each architecture
and dataset for the first round only and compare it to the baseline by
reporting the average accuracy and standard deviation on the test set. The
results are presented in Section 3 Table 2.

##### Small Initial Training Set

We select 128 samples with DECAL initialization then start both conventional
active learning and DECAL methods and record the earliest round where average
accuracy is greater than random chance (33%). Next we compute the percentage
increase/decrease that DECAL achieves relative to the corresponding baseline.
The results are presented in Section 3 Table 3.

####  2.3.2 Baseline Sample Acquisition Algorithms

We apply patient ID as a modular ”plug-in” constraint prior to sample
selection with each of these baseline algorithms to make our framework
clinically deployable. The first baseline is standard random sampling, the
next three are margin, least confidence and entropy uncertainty based sampling
(Settles (2009)) and the last is an amalgamation of diversity and uncertainty-
based sampling approaches known as BADGE (Ash et al. (2019)).

##  3 Results

##### Initializing Active Learning with EMR Data

First, we show results to validate the importance of integrating patient ID
from the onset. When a large initial train set is used, Table 2 shows in
yellow that DECAL initialization always leads to higher accuracy regardless of
architecture or dataset type. DECAL initialization lead to a 5.59% and 7.02%
increase in average accuracy for OCT and X-Ray data respectively. However,
evaluating the impact of DECAL initialization with a small initial training
set mandates a different approach. Since we use non-pre-trained models,
training with a small set will unsurprisingly result in over-fitting. This
means generalization on the test set will be that of random chance until the
training pool becomes large enough. Table 3 highlights in yellow the instances
when DECAL initialization lead to a percentage increase in average accuracy
during early rounds of training. From these results we see DECAL
initialization leads to better generalization for at least 5 of the 10 query
strategies across all architectures and datasets.

##### Active Learning with EMR Data

We use learning curves to evaluate how DECAL can better characterize disease
states. Due to space constraints, we show learning curves for only Resnet-18
and Densenet-121 architectures in Figure 2 and Figure 3. In these plots x-axis
corresponds to the number of samples in the train set and y-axis corresponds
to the performance accuracy on the test set. Each colored curve is the average
of five trials, with standard errors being shown by the shaded regions. We see
that DECAL consistently matches or surpasses the baseline algorithms.
Furthermore, we see DECAL often having an edge over baseline algorithms from
the early rounds of training onward like in Figures 2(b), 3(a). This is an
indicator that DECAL methods not only generalize better but also generalize
faster than the baselines.

Table 2: DECAL vs random initialization using large train set.

OCT Dataset Initialization Resnet-18 Resnet-50 Densenet-121 Random 61.38 $\pm$
4.53 64.4 $\pm$ 6.75 76.93 $\pm$ 4.64 DECAL yellow 64.53 $\pm$ 9.83 yellow
69.28 $\pm$ 3.41 yellow 80.16 $\pm$ 5.34 X-Ray Dataset Initialization
Resnet-18 Resnet-50 Densenet-121 Random 52.24 $\pm$ 9.47 66.95 $\pm$ 3.98
70.12 $\pm$ 7.44 DECAL yellow 58.2 $\pm$ 9.42 yellow 71.08 $\pm$ 4.17 yellow
72.82 $\pm$ 7.44

Table 3: DECAL vs random initialization during early rounds of training. +/-
numbers show the percentage increase/decrease in average accuracy relative to
random.

Dataset OCT X-Ray Round 4 2 1 2 2 1 Query Resnet-18 Resnet-50 Densenet-121
Resnet-18 Resnet-50 Densenet-121 Random Initialization Random 54.37 46.18
42.66 63.91 69.64 46.37 Entropy 53.62 46.88 49.97 55.88 51.18 46.18 BADGE
60.53 48.69 35.3 57.50 65.8 40.32 Margin 55.57 47.01 47.52 56.28 68.34 54.03
Least Conf 49.49 49.46 46.85 55.03 57.30 47.46 DECAL Random 61.44 54.5 48.48
64.16 63.42 46.37 DECAL Entropy 64.48 50.72 52.66 51.92 67.27 51.57 DECAL
BADGE 63.97 53.44 46.9 64.8 69.55 47.08 DECAL Margin 58.88 48.64 50.4 57.14
62.5 47.03 DECAL Least Confidence 60.48 49.30 49.78 60.06 64.42 40.99 DECAL
Initialization Random yellow+9.21% yellow+3.76% yellow+5.95% yellow+2.03%
-5.11% -0.75% Entropy -6.95% yellow+3.45% -10.74% yellow+11.90% yellow+24.73%
yellow+25.52% BADGE yellow+5.73% yellow+8.64% yellow+21.38% yellow+9.91%
yellow1.70% -2.48% Margin -1.87% yellow+6.53% -16.45% -3.02% -10.43%
yellow+2.17% Least Confidence yellow+17.61% -6.08% -12.23% yellow+5.23%
yellow+9.68% -6.27% DECAL Random -1.17% -6.01% yellow+14.35% yellow+4.09%
yellow+8.04% yellow+22.88% DECAL Entropy yellow+4.67% -0.53% -16.71%
yellow+4.31% -5.90% yellow+3.83% DECAL BADGE yellow+3.25% -7.99% yellow+0.80%
yellow+4.55% yellow+1.01% yellow+0.89% DECAL Margin yellow+0.66% yellow+5.53%
yellow+5.55% yellow+9.87% yellow+2.91% yellow+20.69% DECAL Least Confidence
-4.19% -2.86% -3.37% yellow+3.15% -0.24% yellow+10.85%

(a)

(b)

(c)

(d)

Figure 2: Test accuracy vs sample count during training for the OCT on
Resnet-18.

(a)

(b)

(c)

(d)

Figure 3: Test accuracy vs sample count during training for the X-Ray on
Densenet-121.

##  4 Conclusion and Next Steps

In this work we motivate the design of a DECAL framework for medical
applications. Augmenting active learning paradigms with EMR data creates the
perfect setting for which active learning can be of true utility within the
medical domain. In this study we have demonstrated this by designing an active
learning framework constrained on a medically grounded prior gleaned from
clinical EMR data about patient identity.

Determining what other forms of EMR best serve active learning paradigms
remains an open research question. Our next steps include investigating
additional EMR data to inject into our framework for a more holistic analysis.
Towards these efforts, we have collaborated alongside Retina Consultants of
Texas (Houston, TX, USA) to create our own dataset (Logan* et al. (2022 Under
Review)) that contains a plethora of EMR specific to ophthalmology. The
clinical information consists of patient identity, general demographics,
ocular disease state (Best Corrected Visual Acuity, Central Sub-field
Thickness) and detailed ocular imaging in the form of spectral domain OCT,
fundus photography and fluorescein angiography collected per the protocol.
These were measured and recorded during routine visits to the clinic. In
addition to DECAL, clinical context also aids research in supervised
contrastive learning (Kokilepersaud et al. (2022)).

Acknowledgments

This material is based upon work supported by the National Science Foundation
Graduate Research Fellowship under Grant No. DGE-1650044.

## References

  * Ash et al. (2019) Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal.  Deep batch active learning by diverse, uncertain gradient lower bounds.  _arXiv preprint arXiv:1906.03671_ , 2019.
  * Beluch et al. (2018) William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler.  The power of ensembles for active learning in image classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 9368–9377, 2018.
  * Brundin-Mather et al. (2018) Rebecca Brundin-Mather, Andrea Soo, Danny J Zuege, Daniel J Niven, Kirsten Fiest, Christopher J Doig, David Zygun, Jamie M Boyd, Jeanna Parsons Leigh, Sean M Bagshaw, et al.  Secondary emr data for quality improvement and research: a comparison of manual and electronic data collection from an integrated critical care electronic medical record system.  _Journal of critical care_ , 47:295–301, 2018.
  * Brush Jr et al. (2017) John E Brush Jr, Jonathan Sherbino, and Geoffrey R Norman.  How expert clinicians intuitively recognize a medical diagnosis.  _The American journal of medicine_ , 130(6):629–634, 2017.
  * Dasgupta (2011) Sanjoy Dasgupta.  Two faces of active learning.  _Theoretical computer science_ , 412(19):1767–1781, 2011.
  * Gal et al. (2017) Yarin Gal, Riashat Islam, and Zoubin Ghahramani.  Deep bayesian active learning with image data.  In _International Conference on Machine Learning_ , pages 1183–1192. PMLR, 2017.
  * Geifman and El-Yaniv (2017) Yonatan Geifman and Ran El-Yaniv.  Deep active learning over the long tail.  _arXiv preprint arXiv:1711.00941_ , 2017.
  * Hanneke et al. (2014) Steve Hanneke et al.  Theory of disagreement-based active learning.  _Foundations and Trends® in Machine Learning_ , 7(2-3):131–309, 2014.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hsu and Lin (2015) Wei-Ning Hsu and Hsuan-Tien Lin.  Active learning by learning.  In _Twenty-Ninth AAAI conference on artificial intelligence_ , 2015\.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * Kermany et al. (2018) Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al.  Identifying medical diagnoses and treatable diseases by image-based deep learning.  _Cell_ , 172(5):1122–1131, 2018.
  * Kokilepersaud et al. (2022) Kiran Kokilepersaud, Mohit Prabhushankar, Ghassan AlRegib, Stephanie Trejo Corona, and Charles Wykoff.  Gradient-based severity labeling for biomarker classification in oct.  In _International Conference on Image Processing (ICIP)_. IEEE, 2022\.
  * Logan et al. (2022) Yash-yee Logan, Ryan Benkert, Ahmad Mustafa, and Ghassan AlRegib.  Patient aware active learning for fine-grained oct classification.  In _International Conference on Image Processing (ICIP)_. IEEE, 2022\.
  * Logan* et al. (2022 Under Review) Yash-yee Logan*, Kiran Kokilepersaud*, Stephanie Trejo Corona, Mohit Prabhushankar, Ghassan AlRegib, and Charles Wykoff.  Olives: Optical labels for investigating visual eye semantics.  In _Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks_. IEEE, 2022 Under Review.
  * Melendez et al. (2016) Jaime Melendez, Bram van Ginneken, Pragnya Maduskar, Rick H. H. M. Philipsen, Helen Ayles, and Clara I. Sánchez.  On combining multiple-instance learning and active learning for computer-aided detection of tuberculosis.  _IEEE Transactions on Medical Imaging_ , 35(4):1013–1024, 2016.  doi: 10.1109/TMI.2015.2505672.
  * Nath et al. (2020) Vishwesh Nath, Dong Yang, Bennett A Landman, Daguang Xu, and Holger R Roth.  Diminishing uncertainty within the training pool: Active learning for medical image segmentation.  _IEEE Transactions on Medical Imaging_ , 40(10):2534–2547, 2020.
  * Otálora et al. (2017) Sebastian Otálora, Oscar Perdomo, Fabio González, and Henning Müller.  Training deep convolutional neural networks with active learning for exudate classification in eye fundus images.  In _Intravascular imaging and computer assisted stenting, and large-scale annotation of biomedical data and expert label synthesis_ , pages 146–154. Springer, 2017.
  * Sener and Savarese (2017) Ozan Sener and Silvio Savarese.  Active learning for convolutional neural networks: A core-set approach.  _arXiv preprint arXiv:1708.00489_ , 2017.
  * Settles (2009) Burr Settles.  Active learning literature survey.  2009\.
  * Shi et al. (2019) Xueying Shi, Qi Dou, Cheng Xue, Jing Qin, Hao Chen, and Pheng-Ann Heng.  An active learning approach for reducing annotation cost in skin lesion analysis.  In _International Workshop on Machine Learning in Medical Imaging_ , pages 628–636. Springer, 2019.
  * Zhu et al. (2008) Jingbo Zhu, Huizhen Wang, Tianshun Yao, and Benjamin K Tsou.  Active learning with sampling by uncertainty and density for word sense disambiguation and text classification.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_ , pages 1137–1144, 2008.


# VMambaCC: A Visual State Space Model for Crowd Counting

Hao-Yuan Ma<EMAIL_ADDRESS>School of Computer Science and
Technology, Soochow UniversityNo. 333, Ganjungdong
RoadSuzhouJiangsuchina215000 ,  Li Zhang<EMAIL_ADDRESS>School of
Computer Science and Technology, Soochow UniversityNo. 333, Ganjingdong
RoadSuzhouJiangsuchina215000 and  Shuai Shi<EMAIL_ADDRESS>School of Computer Science and Technology, Soochow UniversityNo. 333,
Ganjungdong RoadSuzhouJiangsuchina215000

###### Abstract.

As a deep learning model, Visual Mamba (VMamba) has a low computational
complexity and a global receptive field, which has been successful applied to
image classification and detection. To extend its applications, we apply
VMamba to crowd counting and propose a novel VMambaCC (VMamba Crowd Counting)
model. Naturally, VMambaCC inherits the merits of VMamba, or global modeling
for images and low computational cost. Additionally, we design a Multi-head
High-level Feature (MHF) attention mechanism for VMambaCC. MHF is a new
attention mechanism that leverages high-level semantic features to augment
low-level semantic features, thereby enhancing spatial feature representation
with greater precision. Building upon MHF, we further present a High-level
Semantic Supervised Feature Pyramid Network (HS2PFN) that progressively
integrates and enhances high-level semantic information with low-level
semantic information. Extensive experimental results on five public datasets
validate the efficacy of our approach. For example, our method achieves a mean
absolute error of 51.87 and a mean squared error of 81.3 on the
ShangHaiTech_PartA dataset. Our code is coming soon.

Computer vision, Crowd counting, Deep learning, Visual state space model

##  1\. Introduction

Crowd counting refers to the task of quantifying the number of individuals in
images or videos using computer vision techniques. This task is of significant
importance to predicting crowd trends and optimizing resource allocation, and
can provide crucial information for many practical applications, such as urban
planning (Song et al., 2021), security surveillance (Cheng et al., 2022), and
retail analysis (Ma et al., 2024).

For crowd counting tasks, there has been challenges including high density,
occlusion, and small targets. To meet these challenges, current crowd counting
models are mainly constructed based on two architectures: Convolutional Neural
Networks (CNNs) (Zhang et al., 2016; Cheng et al., 2022; Ma et al., 2024;
Cheng et al., 2019; Li et al., 2018) and Transformer (Liang et al., 2022; Tran
et al., [n. d.]).

CNN is a kind of deep learning model specifically designed for processing data
like images. There are many classic CNN models for crowd counting, such as
Multi-Column CNN (MCNN) (Zhang et al., 2016), Congested Scene Recognition
Network (CSRNet) (Li et al., 2018), Context-Aware Feature Network (CANNet)
(Liu et al., 2019), Attention Scaling NetWork (ASNet) (Jiang et al., 2020),
Shallow Feature Based Dense Attention Network (SDANet) (Miao et al., 2020),
and Fine-Grained Extraction Network (FGENet) (Ma et al., 2024). These models
leverage CNN to extract features and fuse multi-scale features for counting
individuals. However, these methods are sensitive to changes in position and
scale. In addition, as Kolesnikov et al. (Kolesnikov et al., 2021) point out
that CNN-based models struggle to obtain a global receptive field; thus, these
models are restricted when dealing with multi-scale individuals.

Transformer is a kind of deep learning model originally designed for
processing time sequence data. By dividing spatial positions and image patches
into sequences, Kolesnikov et al. (Kolesnikov et al., 2021) proposed a Visual
Transformer (ViT) based on Transformer to capture relationships between pixels
in images. Transformer-based models mainly use ViT to extract features from
images. Typical models are Crowd Localization TRansformer (CLTR) (Liang et
al., 2022) and Local features by Vision Transformer for Crowd counting
(LoViTCrowd) (Tran et al., [n. d.]). These methods possess a global receptive
field. However, the computational complexity of Transformer-based models
exponentially increases as the size of the input images, leading to reduced
efficiency in tasks with large computational costs (Gu and Dao, 2023).

Recently, Gu and Dao (Gu and Dao, 2023) proposed a Mamba model, which is a
state space model, for natural language processing. Mamba enables each element
in a sequence to interact with any previously scanned element, thereby
reducing the quadratic complexity of the attention mechanism in Transformer to
the linear complexity. Inspired by the Mamba, Liu et al. (Liu et al., 2024)
proposed a Visual Mamba (VMamba) model. VMamba designs a Cross-Scan Module
(CSM) that adopts a four-way scanning strategy, starting from the four corners
of the image. In doing so, VMamba can have a global attention and maintain the
linear computational complexity.

Figure 1. Comparison curves of flops vs. image size obtained by four methods.

Building upon VMamba, we propose a versatile VMambaCC model that is capable of
crowd counting and crowd localization. VMambaCC not only captures global
information but also tackles the multi-scale variations caused by the
perspective effect. Our network stands out from popular Transformer-based or
CNN architectures by providing linear scaling of feature sizes. Fig. 1 plots
the curves of computational complexity (flops) vs. image size obtained by our
method and other three mainstream methods. These curves imply that our method
is capable of handling the processing more efficiently, with the computational
complexity consistently maintained at a lower level. Our main contributions
are as follows:

  * •

A novel VMambaCC model is proposed, which is the first visual state space
model for crowd counting that is a high-density task. VMambaCC has not only a
global receptive field but also a fast learning and inference speed.

  * •

VMambaCC contains a Multi-head High-level Feature (MHF) enhancement mechanism
that is newly designed for enhancing low-level semantic information by
reinforcing high-level semantic features.

  * •

On the basis of the MHF attention mechanism, this study presents a feature
fusion network, or the High-level Semantic Supervised Feature Pyramid Network
(HS2FPN). This network gradually integrates information from different scales
and partially preserves high-level semantic features.

The subsequent sections of this paper are structured as follows. In Section 2,
we provide a concise review of the relevant literature pertaining to crowd
counting tasks. Section 3 delves into a detailed exposition of the VMambaCC
model, encompassing its architecture, design philosophy, and the innovative
components including the MHF attention mechanism and the HS2FPN feature fusion
network. Section 4 presents the results of our counting, localization, and
ablation studies, offering a comprehensive evaluation and analysis of our
model. Finally, Section 5 summarizes this paper.

##  2\. Related work

Crowd counting has attracted widespread attention in the field of computer
vision. For decades, methods for crowd counting include detection-based (Zhao
and Nevatia, 2003; Dong et al., 2007; Subburaman et al., 2012), density map-
based (Zhang et al., 2016; Li et al., 2018; Liu et al., 2019; Cheng et al.,
2022; Wan et al., 2021; Tran et al., [n. d.]; Lin et al., 2022; Cheng et al.,
2019), and point-based approaches (Ma et al., 2024; Song et al., 2021; Liang
et al., 2022).

Detection-based methods transform the crowd counting task into an object
detection problem by detecting individuals in the image for counting. These
methods typically utilize object detection algorithms such as faster region-
CNN (Ren et al., 2017) and YOLO (Ge et al., 2021) to detect individuals in an
image and count them based on the detected instances. Although detection-based
methods can provide more accurate localization and counting of individuals,
their performance may be affected when dealing with high-density and heavily
occluded scenarios.

On the basis of the principle of density estimation, density map-based methods
generate a grayscale image, where each pixel represents the estimated density
of individuals at corresponding locations. Typically, density estimation is
performed using some schemes, such as Gaussian kernel functions (Ahonen et
al., 2009), where each annotated point in the original image is convolved with
a Gaussian kernel to obtain the density estimation for each pixel and form a
density map. Popular density map-based methods include MCNN (Zhang et al.,
2016), CSRNet (Li et al., 2018), and others. However, density maps, as an
estimate of the overall distribution of the crowd, lose the individual
location information, leading to limitations in individual tracking and
recognition. In addition, the superposition of Gaussian kernels can introduce
additional noise to the data (Cheng et al., 2022).

Point-based methods employ an effective approach to crowd counting by
representing individuals as points and estimating the total number of
individuals in the image (Song et al., 2021). Compared to other methods, the
point-based framework offer better robustness and computational efficiency as
it only requires detecting the positions of individuals rather than complete
body detection boxes, reducing computational complexity. This kind of
approaches are commonly used in crowded scenes such as stations and squares
for real-time monitoring and analysis of crowd behavior, with broad
application prospects. Some classic point methods include P2PNet (Song et al.,
2021), FGENet (Ma et al., 2024), and CLTR (Liang et al., 2022). So does our
model.

##  3\. Proposed model

This section first introduces the framework of VMambaCC, then discusses the
newly proposed the MHF enhancement mechanism, and finally describes the fusion
network, HS2FPN.

First of all, we clarify some notations. The set of annotated points in an
image is denoted as $S_{a}=\\{(x_{i},y_{i})|i=1,\cdots,N\\}$, where $N$ is the
number of points, and $x_{i}$ and $y_{i}$ represent the horizontal and
vertical coordinates of the $i$-th individual in the image, respectively. Let
$S_{p}=\\{(\hat{x}_{j},\hat{y}_{j},\hat{c}_{j})|j=1,\cdots,M\\}$ be the set of
predicted points, where $\hat{x}_{j}$ and $\hat{y}_{j}$ represent the
predicted horizontal and vertical coordinates of the $j$-th individual in the
image, respectively, $\hat{c}_{j}$ is the confidence score that the $j$-th
predicted point is an individual, and $M$ is the number of predicted points.

Figure 2. Architecture of VMambaCC model.

###  3.1. Framework of VMambaCC

VMambaCC is the first Mamba-based model specifically tailored for crowd
counting tasks. Because VMambaCC has the advantages of VMamba, it addresses
the issue of Transformer’s quadratic growth in computational cost with image
block size and retains the global receptive field.

As shown in Fig. 2, VMambaCC consists of three stages: feature extraction,
feature fusion, and prediction. The input of this model is an image, and the
output is a set of predicted points with confidence scores. Therefore,
VMambaCC can be taken as a kind of point-based counting model. For a given
crowd image $\mathbf{X}$, VMamba is first used to extract its feature maps
with three scales, denoted as $\mathbf{F}_{s}$, $s=1,2,3$. Subsequently, these
feature maps pass through the feature fusion stage. By progressively fusing
features at three scales, HS2FPN preserves high-level feature supervision and
captures individual information at different resolutions. Finally, our model
outputs both positions and confidence scores of individuals in the image
$\mathbf{X}$ after passing through the prediction stage.

For training VMambaCC, we utilize a Three-Task Combination (TTC) loss function
that involves matching annotated points with predicted points. We treat this
as a bipartite graph matching problem and use the Hungarian algorithm to match
points in $S_{a}$ with those in $S_{p}$. Without loss of generality, the first
$N$ predicted points in $S_{p}$ are matched successfully with annotated points
in $S_{a}$ in order.

In VMambaCC, we design a novel MHF attention mechanism and then propose a
fusion network (NS2PFN) based on this new attention mechanism. By effectively
combining the semantic information from high-level features with the detailed
information from low-level features, MHF can help our model improve its
performance when handling multi-scale data. In the following, we discuss them
further.

Figure 3. Structure of MHF attention mechanism.

###  3.2. MHF attention mechanism

Fig. 3 shows the structure of MHF attention. Roughly, the MHF attention can be
divided into three parts: the Channel Enhancement Module (CEM), the Multi-head
Spatial Enhancement Module (MSEM), and the High-level CEM (HCEM).

####  3.2.1. CEM

The purpose of CEM is to assign attention weights to channels, allowing the
model to better select and emphasize important features. In this module, the
high-level feature maps are passed through max-pooling and average-pooling
layers.

Let $\mathbf{F}_{h}$, $\mathbf{F}_{m}$ and $\mathbf{F}_{a}$ be the high-level,
max-pooled and average-pooled features, respectively. That is,

(1) $\mathbf{F}_{m}=MaxPool(\mathbf{F}_{h})$

and

(2) $\mathbf{F}_{a}=AvgPool(\mathbf{F}_{h})$

where $MaxPool(\cdot)$ and $AvgPool(\cdot)$ are the operations of max-pooling
and average-pooling, respectively.

Let $\mathbf{F}_{out_{1}}$ be the output of CEM. Then, $\mathbf{F}_{out_{1}}$
can be obtained by

(3) $\mathbf{F}_{out_{1}}(c)=W(c)\mathbf{F}_{h}(c)$

where $\mathbf{F}_{out_{1}}(c)$ and $\mathbf{F}_{h}(c)$ are the feature maps
in the $c$-th channel of $\mathbf{F}_{out_{1}}$ and $\mathbf{F}_{h}$,
respectively, $\mathbf{W}=[W(1),\cdots,W(c)]^{T}$ is the channel weight vector
that is defined as

(4)
$\mathbf{W}=\text{Sigmoid}\left(Conv_{s}(\mathbf{F}_{m})+Conv_{s}(\mathbf{F}_{a})\right)$

$Conv_{s}(\cdot)$ represents a sequence of convolutional layers with shared
weights, and $Sigmoid(\cdot)$ denotes the Sigmoid activation function.

####  3.2.2. MSEM

The role of MSEM is to make the model focus on useful information, which could
improve the robustness of the model. If we extract only the maximum and
average values in feature maps among all channels, then it is highly possible
to lose lots of grained information. Therefore, we divide feature maps into
multiple groups and design multi-head spatial features to focus on grained
information from different groups. After obtain the output of CEM, we divide
$\mathbf{F}_{out_{1}}$ into 4 groups according to channels. That is,

(5) $\mathbf{F}_{g_{i}}=Group(\mathbf{F}_{out_{1}}),i=1,2,3,4$

where $Group(\cdot)$ is the operation of dividing feature maps into groups.

Then, the output of MSEM can be expressed by

(6)
$\mathbf{F}_{out_{2}}=Concat\left(\mathbf{G}_{1},\cdots,\mathbf{G}_{4}\right)$

where $\mathbf{G}_{i}=Conv(Max(\mathbf{F}_{g_{i}}),Avg(\mathbf{F}_{g_{i}}))$,
$Concat(\cdot)$ is the feature map concatenation, $Conv(\cdot)$ filters the
extracted feature maps, $Max(\cdot)$ and $Avg(\cdot)$ are functions of taking
the maximum and average values among the channels, respectively.

####  3.2.3. HCEM

The main purpose of this module is to enhance important channels and upsample
feature maps. After MSEM, there may be some redundant information. In order to
filter out the redundant information, we need to enhance
$\mathbf{F}_{out_{2}}$ in terms of channels, then fuse the feature maps and
multiply them onto the low-level features.

(8) $\displaystyle\mathbf{F}_{out3}$ $\displaystyle=$ $\displaystyle
Upsample(Conv(Conv_{s}(MaxPool(\mathbf{F}_{out_{2}}))$
$\displaystyle+Conv_{s}(AvgPool(\mathbf{F}_{out_{2}}))))$

where $Upsample(\cdot)$ stands for upsampling.

Table 1. Statistics of crowd counting datasets. Dataset | #Number of images | #Training/Validation/Test | Total | Min | Average | Max
---|---|---|---|---|---|---
SHT_A | 482 | 300/0/182 | 241,677 | 33 | 501.4 | 3139
SHT_B | 716 | 400/0/316 | 88,488 | 9 | 123.6 | 578
UCF_QNRF | 1,535 | 1201/0/334 | 1,251,642 | 49 | 815 | 12,865
JHU-CROWD | 4250 | 3888/0/1062 | 1,114,785 | - | 262 | 7286
UCF_CC_50 | 50 | - | 63,974 | 94 | 1,280 | 4,543

###  3.3. HS2FPN

We make a thorough analysis on existing crowd counting models and find that
existing feature fuse networks have a limitation when dealing with high-
density and complex scenes. The main constraints lie in the insufficient
global perception capability of these networks, which fails to effectively
capture the overall information in images, and the inadequate extraction and
representation of key features in scenes, especially when dealing with
occlusion and varying scales of crowds. Therefore, HS2FPN aims to address
these issues, thereby improving the accuracy and robustness of crowd counting.

HS2FPN mainly consists of multiple Visual State Space (VSS) and MHF attention
blocks. VSS, a module from VMamba (Liu et al., 2024), can effectively enhance
the model’s perception of global information in images through a four-way
scanning strategy, enabling a more comprehensive understanding of scenes and
improving the capability to capture distribution features of crowds in complex
scenes. This strategy provides the model with a unique perspective to observe
and process image data, significantly reducing computational complexity while
maintaining global perception capability.

Figure 4. MHF Attention block connection.

MHF attention blocks enhance the model’s ability to capture and represent
important features by integrating high-level features when handling occlusion
and multi-scale crowds. Additionally, we also design the enhancement scheme
for enhancing low-level features using high-level features, as shown in Fig.
4. Let $\mathbf{F}_{l}$ be low-level features. Then, our enhancement scheme
can be described as

(9) $\mathbf{F}_{out_{4}}=\mathbf{F}_{l}\odot\mathbf{F}_{out_{3}}$

where $\odot$ is the element-wise multiplication operator,
$\mathbf{F}_{out_{3}}$ is the output of MHF attention block, which is obtained
from high-level features $\mathbf{F}_{h}$. Ultimately, HS2FPN achieves the
fusion of crucial feature extraction and representation capabilities for
multi-scale features.

Table 2. Performance comparison of mainstream crowd counting models on three datasets. Method | Localization | Architecture | SHT_A | SHT_B | UCF_QNRF | JHU-Crowd
---|---|---|---|---|---|---
MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE
MCNN(Zhang et al., 2016) | $\times$ | CNN | 110.2 | – | 26.4 | – | – | – | 160.6 | 377.7
CSRNet(Li et al., 2018) | $\times$ | CNN | 68.2 | 115.0 | 10.6 | 16 | 120.3 | 208.5 | 72.2 | 249.9
CANNet(Liu et al., 2019) | $\times$ | CNN | 62.3 | 100.0 | 7.8 | 12.2 | 107.0 | 183 | 100.1 | 314.0
ASNet(Jiang et al., 2020) | $\times$ | CNN | 57.78 | 90.13 | – | – | 91.59 | 159.71 | – | –
SDANet(Miao et al., 2020) | $\times$ | CNN | 63.6 | 101.8 | 7.8 | 10.2 | – | – | 59.3 | 248.9
SCARNet(Gao et al., 2019) | $\times$ | CNN | 66.3 | 144.1 | 9.5 | 15.2 | – | – | – | –
GauNet(Cheng et al., 2022) | $\times$ | CNN | 54.8 | 89.1 | 6.2 | 9.9 | 81.6 | 153.7 | 69.4 | 262.4
P2PNet(Song et al., 2021) | $\surd$ | CNN | 52.7 | 85.1 | 6.3 | 9.9 | 85.32 | 154.5 | – | –
FGENet(Ma et al., 2024) | $\surd$ | CNN | 51.66 | 85.00 | 6.34 | 10.53 | 85.2 | 158.76 | – | –
CLTR(Liang et al., 2022) | $\surd$ | Transformer | 56.9 | 95.2 | 6.5 | 10.6 | 85.8 | 141.3 | 59.5 | 240.6
LoViTCrowd(Tran et al., [n. d.]) | $\times$ | Transformer | 54.8 | 80.9 | 8.6 | 13.8 | 87 | 141.9 | – | –
VMambaCC | $\surd$ | Mamba | 51.87 | 81.3 | 7.48 | 12.47 | 88.42 | 144.73 | 54.41 | 201.93

##  4\. Experiments

To validate the effectiveness of VMambaCC, we performed experiments on four
public crowd datasets and compared it with state-of-the-art models.

###  4.1. Experimental Setups

Datasets. Four challenging datasets were used here: Shanghai Tech (Zhang et
al., 2016), UCF_QNRF (Idrees et al., 2018), JHU-Crowd (Sindagi et al., 2022)
and UCF_CC_50 (Idrees et al., 2013). The detailed information of these dataset
is shown in Table 1, where both SHT_A and SHT_B belong to the Shanghai Tech
dataset.

  * •

Shanghai Tech is a large-scale crowd counting dataset consisting of 1198
images and 330,165 annotations. This dataset covers various scene types and
densities. There is some imbalance in the number of images with different
densities. SHT_A and SHT_B are its two subsets, where SHT_A has significantly
higher density than SHT_B.

  * •

UCF-QNRF is a dataset containing 1535 challenging images and 1,251,642
annotations, with diverse image scenes, angles, densities, and lighting
variations.

  * •

JHU-Crowd is a large-scale dataset with 4250 images and 1,114,785 annotations,
averaging 262 individuals per image. Additionally, this dataset provides rich
annotations, including point-level, image-level, and head-level annotations.

  * •

UCF_CC_50 is a challenging dataset containing 50 images with an average of
1280 annotations per image. This dataset contains a variety of scenes and
perspectives, such as concerts, protests, stadiums and marathons.

Data Augmentation. Some data augmentation strategies were adopted here,
including random scaling with a probability of 0.5 (scaling factors ranging
from 0.7 to 1.3), random horizontal flipping with a probability of 0.5, color
adjustments, Gaussian noise addition, and random cropping to the size of
$128\times 128$. For the QNRF dataset, we constrained the maximum size to not
exceed 1408 and maintained the aspect ratio of the original images.

Hyperparameter Settings. The Adam (Kingma and Ba, 2015) optimizer was applied
to our model with a learning rate of $10^{-4}$. For each image in all datasets
except QNRF, we set a reference point every two pixels. For the QNRF dataset,
we generated a reference point every two pixels vertically and horizontally.

Evaluation Metrics. For crowd counting tasks, Mean Absolute Error (MAE) and
Mean Squared Error (MSE) are two common evaluation metrics. MAE represents the
average of the absolute errors between predicted and true values, while MSE is
the average of the squared errors between predicted and true values. These two
metrics help us assess the accuracy and precision of the model in crowd
counting tasks, which are defined as

$\displaystyle\text{MAE}=\frac{1}{NUM}\sum_{j=1}^{NUM}|N_{j}-M_{j}|$

and

$\displaystyle\text{MSE}=\frac{1}{NUM}\sqrt{\sum_{j=1}^{NUM}(N_{j}-M_{j})^{2}}$

where $N_{j}$ and $M_{j}$ are the numbers of annotation and predicted points
in the $j$-th test image, and $NUM$ is the number of test images. Lower values
of the both metrics indicate better performance of a model.

For localization tasks, Precision (P), Recall (R), and F1-score (F1) are the
evaluation metrics. Following (Liang et al., 2022), we define
$\sqrt{d_{w}^{2}+d_{h}^{2}}/2$ as the distance between annotated points and
matched predicted points, where $d_{w}$ is the horizontal distance between the
predicted and true points, and $d_{h}$ is the vertical distance between the
matched predicted point and the annotated point. The annotated points are
treated as positive samples. Now, we need to judge which matched predicted
points are positive or negative. Let $\sigma$ be the judgment threshold.
Specifically, a matched predicted point is positive if the distance between it
and the corresponding annotated point is less than $\sigma$; otherwise, it is
negative. In this case, these metrics can be described as

$P=\frac{TP}{N}$

and

$R=\frac{TP}{M}$

where $TP$ is the number of positive matched predicted points, $N$ is the
number of annotated points, and $M$ is the number of predicted points. Because
F1-score is the harmonic mean of precision and recall, it provides a single
score that balances both of these measures, which is defined as

$F1=2\times\frac{P\times R}{P+R}$

Figure 5. Demonstration of the predictive effect of our model.

###  4.2. Counting Experiments

For crowd counting tasks, VMambaCC was compared with 11 mainstream methods,
including 9 CNN-based models (MCNN (Zhang et al., 2016), CSRNet (Li et al.,
2018), CANNet (Liu et al., 2019), ASNet (Jiang et al., 2020), SDANet (Miao et
al., 2020), Spatial-Channel-wise Attention Regression (SCARNet) (Gao et al.,
2019), Gaussian kernels Network (GauNet) (Cheng et al., 2022), P2PNet (Song et
al., 2021), and FGENet (Ma et al., 2024)) as well as 2 Transformer-based
models (CLTR (Liang et al., 2022) and LoViTCrowd (Tran et al., [n. d.])).

Experimental results on Shanghai Tech, UCF_QNRF, and JHU-Crowd datasets are
shown in Table 2, where the best values are in bold font, and the second best
are underlined. From Table 2, we can see that in terms of MSE, VMambaCC
achieves the second best result on SHT_A, which is 0.4 points lower compared
to the best method LoVitCrowd. In terms of MAE, our method differs from FGENet
by 0.21 points. We consider that there is some annotation inaccuracy in SHT_A.

Additionally, GauNet (a CNN-based model) performs the best on SHT_B and our
model (a VMamba-based model) is not so good, which is possibly due to
insufficient samples and fewer annotated points in SHT_B. In this case, there
has a certain gap between VMambaCC and existing methods.

On the UCF_QNRF dataset, there is also a certain gap between VMambaCC and the
current best method. Some samples in UCF_QNRF have tens of thousands of
annotations, which increases the computational effort of the Hungarian
matching algorithm. Owing to experimental equipment limitations, our method
cannot use an appropriate batch_size in the training process; thus, VMambaCC
could not achieve its best performance.

VMambaCC achieves the best results on the JHU-Crowd dataset, with an
improvement of 4.89 in MAE compared to the second-ranked SDANet method and an
improvement of 38.67 in MSE compared to the second-ranked CLTR method.

On the UCF_CC_50 dataset, five-fold cross-validation was employed. Results are
presented in Table 3. The comparison with other methods is listed in Table 4.
Observation on Table 4 indicates that our method achieves significant
improvements over other approaches on the UCF_CC_50 dataset. Compared to the
second-ranked FGENet, VMambaCC has an improvement of 78.22 points in MAE and
137.16 points in MSE. Obviously, Our method has reached the state-of-the-art
level on this datasets.

Table 3. Five-fold cross-validation of VMambaCC on UCF_CC_50. Fold | MAE | MSE
---|---|---
1 | 84.00 | 103.96
2 | 37.70 | 47.04
3 | 53.90 | 66.84
4 | 68.20 | 78.97
5 | 77.90 | 96.75
Mean | 64.34 | 78.712
Table 4. Performance comparison of mainstream crowd counting models on UCF_CC_50. Method | MAE | MSE
---|---|---
MCNN (Zhang et al., 2016) | 377.6 | –
CSRNet (Li et al., 2018) | 266.1 | 397.5
CANNet (Liu et al., 2019) | 212.2 | 243.7
ASNet (Jiang et al., 2020) | 174.84 | 251.63
SDANet (Miao et al., 2020) | 227.6 | 316.4
SCARNet (Gao et al., 2019) | 227.6 | 316.4
GauNet (Cheng et al., 2022) | 186.3 | 256.5
P2PNet (Song et al., 2021) | 172.7 | 256.1
FGENet (Ma et al., 2024) | 142.56 | 215.87
VMambaCC | 64.34 | 78.71

###  4.3. Localization Experiments

To compare VMambaCC with other methods, we conducted localization experiments
on SHT_A because this dataset has a large scale variation. Comparison methods
are Localization-based Fully CNN (LCFCN) (Laradji et al., 2018),Method in
(Ribera et al., 2019), Locate-Size and Count CNN (LSC-CNN) (Sam et al., 2021),
Topological constraints Count (TopoCount) (Abousamra et al., 2021), and CLTR
(Liang et al., 2022). Here, we set $\sigma=4$ and $\sigma=8$.

Experimental results are shown in Table 5. We can see that VMambaCC achieves
the best localization performance. When $\sigma=4$, VMambaCC outperforms the
second-best method by 16.2% in precision, 17.6% in recall, and 16.85% in
F1-score. If $sigma=8$, our method surpasses the second-best method by 3.16%
in precision, 5.21% in recall, and 4.18% in F1-score. Findings demonstrate the
effectiveness of VMamba in localization tasks.

To further validate the localization performance of VMambaCC, we also
performed experiments on SHT_B, JHU-Crowd, and UCF_QNRF datasets. The results
are shown in Table 6. It is evident that our approach has achieved commendable
performance across these three datasets. Moreover, the outcomes provide a
benchmark for comparison that will be instrumental for subsequent research
endeavors.

Table 5. Localization performance comparison obtained by six methods on SHT_A.

Method $\sigma$=4 $\sigma$=8 P(%) R(%) F(%) P(%) R(%) F1(%) LCFCN (Laradji et
al., 2018) 43.30% 26.00% 32.50% 75.10% 45.10% 56.30% Method in (Ribera et al.,
2019) 34.90% 20.70% 25.90% 67.70% 44.80% 53.90% LSC-CNN (Sam et al., 2021)
33.40% 31.90% 32.60% 63.90% 61.00% 62.40% TopoCount(Abousamra et al., 2021)
41.70% 40.60% 41.10% 74.60% 72.70% 73.60% CLTR (Liang et al., 2022) 43.60%
42.70% 43.20% 74.90% 73.50% 74.20% VMamba 59.80% 60.3% 60.05% 78.06% 78.71%
78.38%

Table 6. Localization experiments of VMambaCC on different datasets. Dataset | $\sigma$ | P(%) | R(%) | F(%)
---|---|---|---|---
SHT_B | 4 | 63.77% | 63.21% | 63.49%
SHT_B | 8 | 80.77% | 80.06% | 80.41%
JHU-Crowd | 4 | 47.89% | 46.18% | 47.02%
JHU-Crowd | 8 | 64.63% | 62.32% | 63.45%
UCF_QNRF | 4 | 43.59% | 42.91% | 43.25%
UCF_QNRF | 8 | 69.75% | 68.67% | 69.21%

###  4.4. Ablation experiments

MHF. Simple ablation experiments about the MHF attention mechanism were
conducted on the SHT_A dataset . The possible ablation components are CEM,
MSEM, HLCEM, the number of heads, and connection position. Thus, we ablated
them separately, as shown in Table 7.

From Table 7, it can be seen that the baseline model without MHF (the first
row in this table) obtained an MAE of 54.75 and an MSE of 88.46 on the SHT_A
dataset. When adding CEM alone (the second row in Table 7), the model
performance deteriorates compared to the baseline because some noise may be
introduced through directly multiplying channels of high-level features with
low-level features. However, when adding both CEM and MSEM with one head (the
third row in Table 7), both MAE and MSE of this model are improved because
this operation retains spatial information in high-level features.
Additionally, setting Head Nums to 4, the fusion effect of multiple sets of
weights extracted by MSEM is not good, resulting in worse results compared to
no multi-head operation. To determine the correct position for element-wise
multiplication, we compared feature enhancement before and after feature
fusion. The results show that performing element-wise multiplication after
feature fusion leads to crossover effects, which may introduce noise. Thus, it
is better to perform the operation of element-wise multiplication before
feature fusion.

Table 7. Ablation Experiments on MHF

CEM MSEM HLCEM #Head Connection position SHT_A MAE MSE $\times$ $\times$
$\times$ – – 54.75 88.46 $\checkmark$ $\times$ $\times$ – before 61.71 101.31
$\checkmark$ $\checkmark$ $\times$ 1 before 52.16 83.92 $\checkmark$
$\checkmark$ $\times$ 1 post 52.21 87.85 $\checkmark$ $\checkmark$ $\times$ 4
before 54.3 90.83 $\checkmark$ $\checkmark$ $\checkmark$ 4 post 63.88 97.52
$\checkmark$ $\checkmark$ $\checkmark$ 4 before 51.87 81.30

Table 8. Ablation experiments on stage of feature fusion. Feature fusion | SHT_A
---|---
MAE | MSE
FPN (Lin et al., 2017) | 60.53 | 105.36
FGFP (Ma et al., 2024) | 56.28 | 90.83
HS2FPN (Without MHF) | 54.75 | 88.46
HS2FPN (ours) | 51.87 | 81.30

Feature Fusion Comparative experiments on the feature fusion part were also
conducted. Our feature fusion network could be replaced by other networks,
such as FPN (Lin et al., 2017) and FGFP (Ma et al., 2024). In addition, our
HS2FPN could delete the MHF attention mechanism. Thus, four feature fusion
networks were compared, as shown in Table 8.

When using FPN in the feature fusion stage, our model obtains 60.53 MAE and
105.36 MSE. We believe that FPN simply adds different features may lead to the
loss of many important semantic information. VMambaCC with FGFP obtains 56.28
MAE and 90.83 MSE, which is better than VMambaCC with FPN. The improvement
reason lies in the fact that FGFP weights the spatial and channel dimensions,
allowing the model to focus more on important information. VMambaCC with
HS2FPN has the best results in both MAE and MSE. Compared to FGFP, HS2FPN
shows improvements of 4.41 MAE and 9.53 MSE. This advantage is mainly due to
our reasonable utilization of high-level semantic features to supervise low-
level semantic features.

###  4.5. Visualization

The comprehensive set of experiments has substantiated the efficacy of our
approach in the domain of counting and localization. These validations not
only affirm the dependability of our method but also lay down a solid
theoretical groundwork for the advancement and refinement of future models.
The results of our predictions are shown in Figure 5. We are confident that
these achievements will significantly propel research in the field and lay the
foundation for subsequent technological innovations.

##  5\. Conclusion

In this study, we introduce VMamba and propose an innovative crowd counting
model, VMambaCC. This counting model has a global receptive field and the
linearly escalating computational complexity in the context of crowd counting.
A new attention mechanism, MHF, is design in this study. MHF employs high-
level semantics to guide the refinement of lower-level semantic details; thus,
it can avoid the loss of high-level semantic information during feature
fusion. By leveraging the MHF mechanism as a foundation, we have further
developed the feature fusion network, HS2FPN. Extensive experimental
validation confirms the pronounced superiority of our model in confronting
contemporary challenges in crowd counting and localization. Therefore,
VMambaCC contributes significant insights and a reference framework for
further research and practical applications within crowd counting domain.

## References

  * (1)
  * Abousamra et al. (2021) Shahira Abousamra, Minh Hoai, Dimitris Samaras, and Chao Chen. 2021.  Localization in the Crowd with Topological Constraints. In _AAAI_. 872–881.
  * Ahonen et al. (2009) Timo Ahonen, Jiří Matas, Chu He, and Matti Pietikäinen. 2009.  Rotation invariant image description with local binary pattern histogram fourier features. In _SCIA_. 61–70.
  * Cheng et al. (2022) Zhi-Qi Cheng, Qi Dai, Hong Li, Jingkuan Song, Xiao Wu, and Alexander G Hauptmann. 2022.  Rethinking Spatial Invariance of Convolutional Networks for Object Counting. In _CVPR_. 19638–19648.
  * Cheng et al. (2019) Zhi Qi Cheng, Jun Xiu Li, Qi Dai, Xiao Wu, Jun Yan He, and Alexander Hauptmann. 2019.  Improving the Learning of Multi-column Convolutional Neural Network for Crowd Counting.  (2019).
  * Dong et al. (2007) Lan Dong, Vasu Parameswaran, Visvanathan Ramesh, and Imad Zoghlami. 2007.  Fast crowd segmentation using shape indexing. In _ICCV_. 1–8.
  * Gao et al. (2019) Junyu Gao, Qi Wang, and Yuan Yuan. 2019.  SCAR: Spatial-/channel-wise attention regression networks for crowd counting.  _Neurocomputing_ (2019).
  * Ge et al. (2021) Zheng Ge, Songtao Liu, Feng02 Wang, Zeming Li, and Jian Sun. 2021.  YOLOX: Exceeding YOLO Series in 2021.  (2021).
  * Gu and Dao (2023) Albert Gu and Tri Dao. 2023.  Mamba: Linear-Time Sequence Modeling with Selective State Spaces.  (2023).  https://doi.org/10.48550/arXiv.2312.00752
  * Idrees et al. (2013) Haroon Idrees, Imran Saleemi, Cody Seibert, and Mubarak Shah. 2013.  Multi-source Multi-scale Counting in Extremely Dense Crowd Images. In _CVPR_. 2547–2554.
  * Idrees et al. (2018) Haroon Idrees, Muhmmad Tayyab, Kishan Athrey, Dong Zhang, Somaya Al-Maadeed, Nasir Rajpoot, and Mubarak Shah. 2018.  Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds. In _ECCV_. 532–546.
  * Jiang et al. (2020) Xiaoheng Jiang, Li Zhang, Mingliang Xu, Tianzhu Zhang, Pei Lv, Bing Zhou, Xin Yang, and Yanwei Pang. 2020.  Attention Scaling for Crowd Counting. In _CVPR_. 4706–4715.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Lei Ba. 2015.  Adam: A Method for Stochastic Optimization. In _ICLR_.
  * Kolesnikov et al. (2021) Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In _ICLR_.
  * Laradji et al. (2018) Issam H. Laradji, Negar Rostamzadeh, Pedro O. Pinheiro, David Vázquez, and Mark Schmidt. 2018.  Where Are the Blobs: Counting by Localization with Point Supervision. In _ECCV_. 560–576.
  * Li et al. (2018) Yuhong Li, Xiaofan Zhang, and Deming Chen. 2018.  CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes. In _CVPR_. 1091–1100.
  * Liang et al. (2022) Dingkang Liang, Wei Xu, and Xiang Bai. 2022.  An End-to-End Transformer Model for Crowd Localization. In _ECCV_. 38–54.
  * Lin et al. (2022) Hui Lin, Zhiheng Ma, Rongrong Ji, Yaowei Wang, and Xiaopeng Hong. 2022.  Boosting Crowd Counting via Multifaceted Attention. In _CVPR_. 19628–19637.
  * Lin et al. (2017) Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. 2017.  Feature Pyramid Networks for Object Detection. In _CVPR_. 936–944.
  * Liu et al. (2019) Weizhe Liu, Mathieu Salzmann, and Pascal Fua. 2019.  Context-Aware Crowd Counting. In _CVPR_. 5099–5108.
  * Liu et al. (2024) Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. 2024.  VMamba: Visual State Space Model.  (2024).  https://doi.org/10.48550/arXiv.2401.10166
  * Ma et al. (2024) Hao-Yuan Ma, Li Zhang, and Xiang-Yi Wei. 2024.  FGENet: Fine-Grained Extraction Network for Congested Crowd Counting. In _MultiMedia Modeling - 30th International Conference (MMM)_.
  * Miao et al. (2020) Yunqi Miao, Zijia Lin, Guiguang Ding, and Jungong Han. 2020.  Shallow Feature Based Dense Attention Network for Crowd Counting. In _AAAI_. 11765–11772.
  * Ren et al. (2017) Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2017.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  _IEEE Trans. Pattern Anal. Mach. Intell._ (2017).
  * Ribera et al. (2019) Javier Ribera, David Guera, Yuhao Chen, and Edward J. Delp. 2019.  Locating Objects Without Bounding Boxes. In _CVPR_. 6479–6489.
  * Sam et al. (2021) Deepak Babu Sam, Skand Vishwanath Peri, Mukuntha Narayanan Sundararaman, Amogh Kamath, and R. Venkatesh Babu. 2021.  Locate, Size, and Count: Accurately Resolving People in Dense Crowds via Detection.  _TPAMI_ (2021), 2739–2751.
  * Sindagi et al. (2022) Vishwanath A. Sindagi, Rajeev Yasarla, and Vishal M. Patel. 2022.  JHU-CROWD++: Large-Scale Crowd Counting Dataset and A Benchmark Method.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2022), 2594–2609.
  * Song et al. (2021) Qingyu Song, Changan Wang, Zhengkai Jiang, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yang Wu. 2021.  Rethinking Counting and Localization in Crowds:A Purely Point-Based Framework. In _ICCV_. 3365–3374.
  * Subburaman et al. (2012) Venkatesh Bala Subburaman, Adrien Descamps, and Cyril Carincotte. 2012.  Counting people in the crowd using a generic head detector. In _AVSS_. 470–475.
  * Tran et al. ([n. d.]) Nguyen Hoang Tran, Ta Duc Huy, Soan Thi Minh Duong, Nguyen Phan, Dao Huu Hung, Chanh D. Tr. Nguyen, Trung H. Bui, and Steven Quoc Hung Truong. [n. d.].  Improving Local Features with Relevant Spatial Information by Vision Transformer for Crowd Counting. In _33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022_.
  * Wan et al. (2021) Jia Wan, Ziquan Liu, and Antoni B Chan. 2021.  A Generalized Loss Function for Crowd Counting and Localization. In _CVPR_. 1974–1983.
  * Zhang et al. (2016) Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. 2016.  Single-Image Crowd Counting via Multi-Column Convolutional Neural Network. In _CVPR_. 589–597.
  * Zhao and Nevatia (2003) Tao Zhao and Ramakant Nevatia. 2003.  Bayesian human segmentation in crowded situations. In _CVPR_. II–459.