# Optimizing Skin Lesion Classification via Multimodal Data and Auxiliary Task
Integration

###### Abstract

The rising global prevalence of skin conditions, some of which can escalate to
life-threatening stages if not timely diagnosed and treated, presents a
significant healthcare challenge. This issue is particularly acute in remote
areas where limited access to healthcare often results in delayed treatment,
allowing skin diseases to advance to more critical stages. One of the primary
challenges in diagnosing skin diseases is their low inter-class variations, as
many exhibit similar visual characteristics, making accurate classification
challenging. This research introduces a novel multimodal method for
classifying skin lesions, integrating smartphone-captured images with
essential clinical and demographic information. This approach mimics the
diagnostic process employed by medical professionals. A distinctive aspect of
this method is the integration of an auxiliary task focused on super-
resolution image prediction. This component plays a crucial role in refining
visual details and enhancing feature extraction, leading to improved
differentiation between classes and, consequently, elevating the overall
effectiveness of the model. The experimental evaluations have been conducted
using the PAD-UFES20 dataset, applying various deep-learning architectures.
The results of these experiments not only demonstrate the effectiveness of the
proposed method but also its potential applicability under-resourced
healthcare environments.

Index Terms—  Auxiliary Learning, Biomedical Imaging, Computer Vision, Deep
Learning, Skin Lesion Classification,

##  1 Introduction

Skin cancer, the most prevalent form of cancer globally, has shown that early
detection can significantly reduce mortality rates. Dermatologists typically
use dermoscopy imaging and the ABCD rule—assessing Asymmetry, Boundary, Color,
and Diameter—to diagnose skin lesions [1]. However, this traditional method
encounters challenges such as varying imaging qualities and differences in
medical expertise, highlighting the necessity for computer-aided diagnostic
tools. Research in automated skin lesion classification using dermoscopic
images has progressed [2, 3]. However, as shown in Figure 1, these systems
often struggle with high similarity across different classes and low
similarity within the same class [4]. To enhance accuracy, recent studies are
incorporating metadata like clinical and demographic information alongside
images [5, 6]. This approach mimics the diagnostic process of dermatologists,
considering more than just visual attributes, which improves the performance
of automated systems. Additionally, integrating auxiliary task learning, where
an additional task is learned alongside the main objective, has shown to be
beneficial for generalizability [7, 8].

The challenge of low inter-class similarity remains, even when using
multimodal approaches that combine images with metadata. The proposed method
addresses this by incorporating an auxiliary task into the classification
pipeline. This involves using image features to predict super-resolution
images, refining the visual details of the original images. This innovation
has shown notable improvements in skin lesion classification, surpassing
existing methods. This approach is particularly valuable in rural and
underserved areas, where delays in accessing medical expertise and technology
can lead to advanced disease progression. By utilizing smartphone-captured
images and metadata, our method can be used as an effective early screening
tool for these regions, enhancing the prospects of timely and accurate
diagnosis.

Fig. 1: Examples of skin lesion images [9]. Fig. 2: A schematic diagram
illustrating the training and testing process of the proposed approach. The
auxiliary task learning guides the visual feature extractor to refine the
extracted features.

##  2 Proposed Approach

This section describes the methodology of the proposed approach. The central
concept underlying this research work is auxiliary task supervision, which
enhances the extraction of visual features. The refined visual features are
combined with metadata features to classify skin lesions. Figure 2 illustrates
the steps involved in the proposed method.
Problem Formulation: Given $X_{img}$, a set of skin lesion images with
corresponding metadata $X_{meta}$ containing various demographic and clinical
features, along with diagnostic labels, y $\in$ [1…$N_{class}$], where N
represents the total number of classes. The objective is to train a model that
predicts y for test data ($X_{img-test}$, $X_{meta-test}$) during the
inference phase.

The images are captured using smartphones under varying lighting conditions,
thus incorporating some colour variations as shown in Figure 1. To handle such
variations, a gray color constancy algorithm [10] is used to ensure colour
uniformity, followed by contrast-limited adaptive histogram equalization
(CLAHE) [11] to enhance the contrast of the images [12]. The metadata includes
features but they have missing values. A neural network based imputation [13]
is applied to complete the missing information.

###  2.1 Feature Extraction

The proposed multimodal approach consist of visual and textual feature
extractors explained as follows.
Visual Feature Extractor: Images contain maximum information and are pivotal
in enhancing the overall classification performance. To extract visual
information, various Convolutional Neural Networks (CNN) are used, each
varying in breadth and depth. The lower layers of the model learn features
such as edges and texture, and the focus is shifted towards the shapes and
patterns of the lesions. Let $\phi_{visual}$ denote the visual feature
extractor, wherein the final feature maps of the selected model are flattened
to create a feature vector for the input images. Textual Feature Extractor: To
encode the metadata information, a four-layer fully connected network is
employed. Each of these fully connected layers is followed by a ReLU
activation function, except for the final layer. The network processes the
diverse metadata features such as age, gender, bleed and cancer history, to
learn the interaction within them. The visual and textual feature vector for
image ximg and xmeta can be represented as equation 1.

$FV_{Image}=\phi_{visual}(x_{img}),FV_{Meta}=\phi_{meta}(x_{meta})$ (1)

###  2.2 Auxiliary Task Learning

Various skin diseases share similar visual characteristics, leading to
confusion in their classification. Hence, more distinctive features are
required to enhance the classification process. These distinctive features
enable the model to distinguish subtle variations that might be overlooked,
thereby improving the accuracy of the classification.

The proposed approach implemented an auxiliary task to improve and refine the
visual features. The process involves creating a super-resolution (SR) image
from a given image by employing either an image processing technique such as
bilinear or bicubic interpolation or a pre-trained deep learning model.
Integrating the task helps to focus on finer image details for predicting the
SR image. Since there is a shared encoder between the auxiliary task and the
visual feature extractor, the visual features also get refined for the
classification task. The generated SR image is a reference for comparison with
the predicted SR image. The last feature maps generated by $\phi_{visual}$ are
passed to the decoder (6-layered CNN) to predict the SR image, and the model
is jointly optimized for auxiliary and classification tasks.

###  2.3 Feature-level Fusion and Classification

To integrate features from both modalities, a feature-level fusion technique
based on multiplication is employed, where the textual features weigh the
visual features. An adaptive convolution layer before the flattening layer is
incorporated within $\phi_{visual}$ to ensure the element-wise multiplication
$\odot$ of $FV_{Image}$ and $FV_{Meta}$, resulting in final feature vector,
$FV_{Final}$, for the classification.

$FV_{Final}=FV_{Image}\odot FV_{Meta}$ (2)

The FVFinal is fed to the classifier consisting of two fully connected layers
to give the final predicted class.

Loss Function The training loss function consists of two components - weighted
cross entropy loss and SR loss. The weighted cross-entropy loss, as shown in
equation 3, between true label ,y, and predicted label, $\hat{y}$.

$L_{wce}=-\sum_{i=1}^{N}(w_{i}\cdot
y_{i}\cdot\log(\hat{y}_{i})+(1-y_{i})\cdot\log(1-\hat{y}_{i}))$ (3)

The weights ’$w_{i}$’ are computed based on the number of samples in each
class. The SR loss in equation 4 gives the loss between the generated SR image
and the predicted SR image.

$L_{SR}=\frac{1}{n}\sum_{i=1}^{n}(SR_{i}-\hat{SR}_{i})^{2}$ (4)

The final loss is the weighted combination of $L_{wce}$ and $L_{SR}$.

$L_{final}=\alpha L_{wce}+\beta L_{SR}$ (5)

Here, $\alpha=0.5$ and $\beta=1.0$ are adjustable factors used to bring the
overall loss function in a similar range.

##  3 Experimental Setup

Dataset Details: This research aims to propose an approach for classifying
smartphone-based skin lesions. This work utilized a publicly available
dataset, PAD-UFES20 [9], which comprises 2298 smartphone-captured images
classified into six classes: actinic keratosis (ACK) with 730 images, basal
cell carcinoma (BCC) with 845 images, melanoma (MEL) with 52 images, nevus
(NEV) with 244 images, squamous cell carcinoma (SCC) with 192 images, and
seborrheic keratosis (SEK) with 235 images, and their corresponding metadata.
The metadata has 26 features, encompassing 21 patient clinical features (such
as age, gender, anatomical location, and cancer history), a diagnostic label,
and four identifiers, such as image ID and patient ID. This work followed the
train-val-test split of 80:10:10 as given in [14].

Implementation details: For visual feature extraction, various deep neural
network architectures, including VGG-13 [15], ResNet50 [16], MobileNet-V2
[17], EfficientNet-B4 [18], and DenseNet-121 [19] are used, varying in depth
and width and pretrained on the ImageNet [20]. The models were fine-tuned on
the PAD-UFES20 dataset. For textual feature extraction, a 4-layered, fully
connected network having dimensions as [64, 128, 256, 512] is used. The
auxiliary task in the proposed approach is evaluated using image processing
and deep learning techniques. The models were trained for 70 epochs with a
batch size of 32 and utilized the SGD optimizer with an initial learning rate
of 0.01. We incorporated a stepLR scheduler with a step size of 15, reducing
the learning rate by 0.1 at each step. All images were resized to 224x224x3.
The super-resolution block upscale the images by a factor of 2, resulting in
dimensions of 448x448x3. For classification, a weighted cross-entropy loss is
used to address class imbalance and Mean Squared Error (MSE) loss is used for
the auxiliary task. Various data augmentation techniques including horizontal
and vertical flips, image scaling, brightness adjustments, contrast,
saturation, and the addition of random noise are applied during training, to
introduce variability in the data. All the experiments were implemented using
the PyTorch framework on an Nvidia DGX station.

##  4 Results and Analysis

This section presents the quantitative and qualitative results of the proposed
method. The performance of the proposed approach is evaluated using balanced
accuracy (BACC), classification accuracy (ACC), and area under the curve (AUC)
across five distinct CNN architectures. Table 1 gives a comparative analysis
of the approach against other existing methods. The proposed approach achieved
a BACC of 0.832, greater than the state-of-the-art (SOTA) result of 0.806 on
the PAD-UFES20 dataset. The proposed approach resulted in a BACC of 0.811 for
the best-performing model in SOTA results, ResNet50, greater than the 0.806
reported in [14]. Table 1 also presents the performance of MetaNet [21], where
authors have used element-wise multiplication without an auxiliary task. It is
evident that the proposed approach also outperforms their performance. This is
because more discriminating features are extracted, which results in improved
performance. To support this, a visual representation of the embedding space
is presented, wherein samples from each class are visibly closer than the SOTA
method, as demonstrated in Figure 5. The presence of well-distinguished
clusters in the visualization substantiates the performance improvement
achieved by the proposed approach. The consistent improvement of performance
metrics across all models demonstrates its superiority across all existing
approaches. Additional evaluation metrics critical for assessing the medical
diagnosis model, such as precision, recall, sensitivity, specificity, and
classwise accuracy (CA) for the best-performing model, VGG13, are also
computed as given in Table 2. The specificity is greater than 90% for all
classes, indicating the accuracy of the model to identify truly negative
classes. Also, the sensitivity values for each class except the SCC exceed
80%, emphasizing the model’s ability to identify true positives, which is
critical for assuring reliable diagnoses.

Table 1: Performance comparison with existing approaches.

Model | ACC | BACC | AUC
---|---|---|---
No Metadata [22] | VGG13 | 0.709 | 0.654 | 0.901
ResNet50 | 0.616 | 0.651 | 0.901
EfficientNet-B4 | 0.656 | 0.640 | 0.911
MobileNet-V2 | 0.655 | 0.637 | 0.898
DenseNet-121 | 0.636 | 0.640 | 0.893
Concat [23] | VGG13 | 0.712 | 0.72 | 0.929
ResNet50 | 0.741 | 0.728 | 0.929
EfficientNet-B4 | 0.765 | 0.758 | 0.945
MobileNet-V2 | 0.738 | 0.741 | 0.927
DenseNet-121 | 0.742 | 0.747 | 0.932
MetaNet [21] | VGG13 | 0.749 | 0.754 | 0.937
ResNet50 | 0.732 | 0.742 | 0.936
EfficientNet-B4 | 0.744 | 0.737 | 0.931
MobileNet-V2 | 0.700 | 0.717 | 0.922
DenseNet-121 | 0.745 | 0.745 | 0.932
MetaBlock [22] | VGG13 | 0.728 | 0.736 | 0.933
ResNet50 | 0.735 | 0.765 | 0.935
EfficientNet-B4 | 0.748 | 0.77 | 0.944
MobileNet-V2 | 0.724 | 0.754 | 0.938
DenseNet-121 | 0.723 | 0.746 | 0.931
Visual [14] | VGG13 | 0.807 | 0.77 | 0.952
ResNet50 | 0.812 | 0.806 | 0.953
EfficientNet-B4 | 0.772 | 0.784 | 0.953
MobileNet-V2 | 0.806 | 0.789 | 0.954
DenseNet-121 | 0.799 | 0.779 | 0.950
Proposed | VGG13 | 0.849 | 0.832 | 0.960
ResNet50 | 0.848 | 0.811 | 0.967
EfficientNet-B4 | 0.833 | 0.797 | 0.967
MobileNet-V2 | 0.836 | 0.811 | 0.953
DenseNet-121 | 0.822 | 0.794 | 0.962

Table 2: Evaluation metrics of the proposed approach.

Metrics | ACK | BCC | MEL | NEV | SCC | SEK
---|---|---|---|---|---|---
Recall | 0.88 | 0.85 | 0.89 | 0.95 | 0.50 | 0.92
Precision | 0.90 | 0.88 | 0.89 | 0.93 | 0.40 | 0.95
F1 Score | 0.89 | 0.87 | 0.89 | 0.94 | 0.44 | 0.94
Sensitivity | 0.88 | 0.85 | 0.89 | 0.95 | 0.50 | 0.92
Specificity | 0.95 | 0.93 | 0.99 | 0.99 | 0.93 | 0.99
| CA
---
87.70 | 85.11 | 88.88 | 95.00 | 50.00 | 92.31

Fig. 3: Confusion matrix for the best-performing model

To showcase the misclassifications between different classes, the confusion
matrix is shown in Figure 3, supported by the Area Under Curve (AUC) as
presented in Figure 4. It is evident that most of the confusion can be seen in
BCC and SCC classes. This is because these classes have minute visual
differences; both are pigmented and also share a similar set of clinical
features. However, it is important to note that this misclassification does
not impact patient care because both classes require a further biopsy for a
proper diagnosis. For further evaluation of the role of auxiliary task, we
have replaced the bilinear SR technique with other techniques as reported in
Table 3. The reported results also show consistent improved results for skin
lesion classification.

Since the proposed approach is mainly feature-based, there is scope for
improvement in extracting more distinctive features that could further solve
the problem of differentiating deadly skin diseases such as MEL from minor
skin diseases like ACK. Based on the reported results, the proposed method
improves the skin lesion classification task and outperforms existing SOTA
results for all the models.

Fig. 4: AUC for the best-performing model Fig. 5: tSNE plots of the proposed and SOTA model. Table 3: Proposed model with varying SR techniques Model | ACC | BACC | AUC
---|---|---|---
Bicubic | 0.849 | 0.828 | 0.965
NinaSR [24] | 0.838 | 0.815 | 0.963
EDSR [25] | 0.843 | 0.821 | 0.959

##  5 Conclusion

This research presents a novel multimodal method for skin lesion
classification that significantly enhances diagnostic accuracy by
incorporating an auxiliary super-resolution image prediction task. Leveraging
smartphone-captured images along with diverse clinical and demographic data,
this approach significantly refines visual features, leading to enhanced
performance. A thorough comparative analysis with existing methodologies
reveals consistent enhancements across several key performance metrics. The
distinctiveness of different lesion classes is further illustrated through a
tSNE plot, providing a clear visualization of the embedding space. In future,
we will extend this auxiliary task-learning technique to address different
medical conditions, incorporating various modalities such as X-rays, MRI, CT
scans and other related auxiliary tasks to further enhance the feature
representation.

##  6 COMPLIANCE WITH ETHICAL STANDARDS

This research adheres to ethical standards, and all experiments conducted
using the the publicly available dataset found at PAD-UFES20 strictly comply
with the terms of use set forth by the dataset creators. The dataset utilized
in this study is openly available for research purposes, and we affirm that
our research does not involve any attempts at deanonymization of patients’
data. Our study prioritizes the privacy and ethical considerations associated
with medical data, and we have taken measures to ensure the responsible and
ethical use of the dataset in accordance with established guidelines.

## References

  * [1] Camacho et al.,  “Multi-class skin lesion classification using prism-and segmentation-based fractal signatures,”  vol. 197, pp. 116671, 2022.
  * [2] Maria Rita Verdelho and Catarina Barata,  “On the impact of self-supervised learning in skin cancer diagnosis,”  ISBI, pp. 1–5, 2022.
  * [3] Mahapara Khurshid, Mayank Vatsa, and Richa Singh,  “Multi-task explainable skin lesion classification,”  arXiv preprint arXiv:2310.07209, 2023.
  * [4] Wang et al.,  “Intra-class consistency and inter-class discrimination feature learning for automatic skin lesion classification,”  Medical Image Analysis, vol. 85, pp. 102746, 2023.
  * [5] Dong et al.,  “Learning from dermoscopic images in association with clinical metadata for skin lesion segmentation and classification,”  CBM, vol. 152, pp. 106321, 2023.
  * [6] Iscen et al.,  “Improving image recognition by retrieving from web-scale image-text data,”  CVPR, pp. 19295–19304, 2023.
  * [7] He et al.,  “Dme-net: Diabetic macular edema grading by auxiliary task learning,”  MICCAI, pp. 788–796, 2019.
  * [8] Chen et al.,  “Auxiliary learning with joint task and data scheduling,”  ICML, pp. 3634–3647, 2022.
  * [9] Pacheco et al.,  “Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones,”  Data in brief, vol. 32, pp. 106221, 2020.
  * [10] Graham D Finlayson and Elisabetta Trezzi,  “Shades of gray and colour constancy,”  Color and Imaging Conference, vol. 2004, no. 1, pp. 37–41, 2004.
  * [11] Pizer et al.,  “Adaptive histogram equalization and its variations,”  Computer Vision, Graphics, and Image Processing, vol. 39, no. 3, pp. 355–368, 1987.
  * [12] Jaisakthi et al.,  “Automated skin lesion segmentation of dermoscopic images using grabcut and k-means algorithms,”  IET Computer Vision, vol. 12, no. 8, pp. 1088–1095, 2018.
  * [13] Pundhir et al.,  “Towards improved skin lesion classification using metadata supervision,”  ICPR, pp. 4313–4320, 2022.
  * [14] Pundhir et al.,  “Visually aware metadata-guided supervision for improved skin lesion classification using deep learning,”  Workshop on the Ethical and Philosophical Issues in Medical Imaging, pp. 65–76, 2022.
  * [15] Simonyan et al.,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [16] He et al.,  “Deep residual learning for image recognition,”  CVPR, pp. 770–778, 2016.
  * [17] Sandler et al.,  “Mobilenetv2: Inverted residuals and linear bottlenecks,”  CVPR, pp. 4510–4520, 2018.
  * [18] Mingxing Tan and Quoc Le,  “Efficientnet: Rethinking model scaling for convolutional neural networks,”  ICML, pp. 6105–6114, 2019.
  * [19] Huang et al.,  “Densely connected convolutional networks,”  CVPR, pp. 4700–4708, 2017.
  * [20] Deng et al.,  “Imagenet: A large-scale hierarchical image database,”  CVPR, pp. 248–255, 2009.
  * [21] Li et al.,  “Fusing metadata and dermoscopy images for skin disease diagnosis,”  ISBI, pp. 1996–2000, 2020.
  * [22] Pacheco et al.,  “An attention-based mechanism to combine images and metadata in deep learning models applied to skin cancer classification,”  IEEE Journal BHI, vol. 25, pp. 3554–3563, 2021.
  * [23] Pacheco et al.,  “The impact of patient clinical information on automated skin cancer detection,”  CBM, vol. 116, pp. 103545, 2020.
  * [24] Gabriel Gouvine,  “Ninasr: Efficient small and large convnets for super-resolution,” https://github.com/Coloquinte/torchSR/blob/main/doc/NinaSR.md, 2021.
  * [25] Lim et al.,  “Enhanced deep residual networks for single image super-resolution,”  CVPR Workshops, pp. 136–144, 2017.


# Enhancing and Dissecting Crowd Counting By Synthetic Data

###### Abstract

In this article, we propose a simulated crowd counting dataset CrowdX, which
has a large scale, accurate labeling, parameterized realization, and high
fidelity. The experimental results of using this dataset as data enhancement
show that the performance of the proposed streamlined and efficient benchmark
network ESA-Net can be improved by 8.4%. The other two classic heterogeneous
architectures MCNN and CSRNet pre-trained on CrowdX also show significant
performance improvements. Considering many influencing factors determine
performance, such as background, camera angle, human density, and resolution.
Although these factors are important, there is still a lack of research on how
they affect crowd counting. Thanks to the CrowdX dataset with rich annotation
information, we conduct a large number of data-driven comparative experiments
to analyze these factors. Our research provides a reference for a deeper
understanding of the crowd counting problem and puts forward some useful
suggestions in the actual deployment of the algorithm.

Index Terms—  Crowd Counting, Crowd Density Estimation, Synthetic Data, Domain
Transfer

##  1 Introduction

The application of crowd counting is very wide, from the construction of smart
cities to the statistics of passenger flow in front of shops, all of which
need the help of crowd counting algorithms.

A lot of work has been proposed to improve the performance of detection
algorithms. These studies either focus on proposing more advanced network
structures (for example multi-column network [1, 2], scale aggregation module
[3, 4] and scale adaptive module [5, 6, 7]), or focus on designing more
suitable loss functions [8]. These two focus points have greatly improved the
performance of existing algorithms. In addition to the work on the algorithm
side, on the dataset side, some open-source crowd counting datasets have also
been proposed. The mainstream ones are UCFCC50 [9], ShanghaiTech A (ST_A) [1],
UCSD [10], ShanghaiTech B (ST_B) [1], and WorldExpo’10 [11]. Among them, the
number of people in UCFCC50 [9] and ShanghaiTech A (ST_A) [1] is very crowded,
and some even reach 3000 people in a single picture. These two datasets are
collected from the Internet, and there are no unified scene and shooting
characteristics. The other three datasets are captured from surveillance
cameras with large differences in viewing angles, so the number of people is
generally not crowded.

Fig. 1: The left column shows a real image and a CrowdX image. The ground
truth and the estimation results are shown in middle and right column. Fig. 2:
The proposed CrowdX dataset.

These studies and datasets have improved the accuracy of crowd counting
algorithms. However, these methods often have poor and unexplainable
performance when they are migrated to the real environment. In summary, there
are currently two issues that need to be resolved. 1) As shown in the table 1,
the existing real-world datasets are generally not large, and insufficient
training data will cause data-driven methods to be unable to learn diverse,
complex, and complete features. To solve this problem, a simple method is to
annotate large amounts of data. However, manually annotating them is very
laborious and expensive. 2) Scene background, camera angle of view, and crowd
density have a great influence on crowd counting algorithms. There is
currently a lack of research on how they affect performance. For the first
question, [12] proposed a simulation dataset GCC, which is constructed by game
scenes and can be automatically labeled, but this method cannot control the
key influence variables. For the second question, [1] proposed a method for
head size adaptation, [13] proposed an architecture for head resolution
adaptation, and [14] proposes a training data selection strategy to improve
algorithm performance. Although these methods have improved some performance,
they only consider a single variable.

In our work, we refer to the application of simulated datasets [15, 16, 17,
18, 19] in other fields, and construct a crowd simulation dataset that
parameterizes the influencing factors to solve the first problem. To solve the
second problem, we design a data-driven strategy to analyze the impact of key
factors utilizing comparative experiments. Specifically, we propose a dataset
generator based on the Unity3D[20], which can parametrically control the
influencing factors and generate a very realistic simulation dataset. We call
the generated dataset CrowdX, which contains 24,000 automatically annotated
crowd images. Compared with existing real-world datasets, CrowdX is easy to
collect and annotate. Compared with the existing simulation dataset [12],
CrowdX is more realistic and can control influencing factors.

In the experimental stage, to evaluate the effectiveness of using CrowdX for
data enhancement, we first propose a streamlined and efficient benchmark
network ESA-Net, then pre-train the network on CrowdX, and finally fine-tune
the network with real-world datasets. The results show that the accuracy of
the benchmark network on the ShanghaiTech B dataset has increased by 8.4% and
the accuracy has reached the state-of-the-art. In addition, the accuracy of
the other two methods MCNN [1] and CSRNet [21] has also been significantly
improved, which verifies the effectiveness of using CrowdX as a data
augmentation method. To evaluate the impact of the key factors in the
performance, we have done many comparative experiments and find some
interesting conclusions. Based on these findings, we provide some suggestions
for selecting training datasets and deploying surveillance cameras.

Table 1: Statistics of some widely used crowd datasets. PC means partially
parameterizd and FC means fully parameterizd.

| Number | Resolution | Average | Type
---|---|---|---|---
UCF_CC_50 [9] | 50 | 2101$\times$2888 | 1279 | Real
ST_A [1] | 482 | 589$\times$868 | 501 | Real
ST_B [1] | 716 | 768$\times$1024 | 123 | Real
WorldExpo [11] | 3980 | 576$\times$720 | 50 | Real
UCSD[10] | 2000 | 158$\times$238 | 25 | Real
UCF_QNRF [22] | 1525 | 2013$\times$2902 | 815 | Real
GCC [12] | 15212 | 1080$\times$1920 | 501 | Syn(PC)
CrowdX | 24000 | 768$\times$1024 | 500 | Syn(FC)

Fig. 3: The proposed efficient self-attention network ESA-Net.

##  2 Proposed Synsetic Dataset: CrowdX

###  2.1 Assets and Tools Description

Based on the work of [23], we choose the PersonX dataset originally used for
person re-identification to construct CrowdX. PersonX contains 1,266 3D
pedestrian models with diversified heights, weights, skin colors, hairstyles,
and clothing. In addition to static properties, the dynamic features are also
editable (such as standing, walking, and running), which makes the generated
simulation dataset highly realistic. In terms of the simulation engine, we
chose to use Unity3D to build the dataset. With its excellent rendering
capabilities, we can simulate scenes, pedestrians, cameras, lights, etc.
Considering that the crowd counting algorithm tends to be used in urban
scenes, we select three urban scenes from the Unity3D asset store, all of
which contain urban components such as buildings, urban roads, and traffic
lights. In addition to these virtual backgrounds, we also use five solid color
backgrounds to build CrowdX. The generated crowd image is shown in 2.

###  2.2 Proposed Generation Method

For the virtual background, we manually select some sampling points from the
three simulated cities to place pedestrians. These sampling points include
pedestrian streets, parks, motor vehicles, etc., with strong diversity and
high fidelity. For the solid color background, we directly sample 5 colors as
the background. For each scene, we set the camera pitch angle to 30°, 50°,
70°, and 90°to take photos. To be consistent with the actual application and
simplify the process, we set the roll value of the camera’s Euler angle to 0
and fix the distance from the camera to the center of the scene. We sample a
random number from 1 to 1000 and add the corresponding number of pedestrians
to the scene with 100 as the step. The standing position is obtained by
randomly sampling the visible area of the scene, and the standing direction is
sampled according to a Gaussian distribution. The mean value of this Gaussian
distribution is randomly generated, which can be considered as the main
direction of the flow of people

###  2.3 Properties of CrowdX

As shown in the figure 2, CrowdX is composed of crowd images with a simulated
city background and a solid color background, with a total of 24,000
automatically labeled simulated images. The scale of this dataset exceeds
other existing datasets and provides richer annotation information. Pedestrian
annotation parameters include each pedestrian’s id, 3D position, 2D camera
plane position, height, and standing direction. The annotation parameters of
the scene include the ID and type of the scene. The camera’s annotation
parameters include position, rotation, and rendering resolution.

##  3 Experiments

###  3.1 Efficient benchmark Method: ESA-Net

The proposed ESA-Net architecture is shown in Fig 3, which consists of four
parts, including 1) VGG16 [24] based front-end for feature representation; 2)
Two consecutive self-attention blocks for feature refining; 3) Two consecutive
dilated convolutional layers used to expand the receptive field; 4) a $1\times
1$ convolutional layer at the end of the network, used to predict the density
map. The network is streamlined and efficient enough, that is suitable to use
in a production environment, compared with [25]. We use L2 loss to train the
benchmark network, and use mean absolute error (MAE) and mean square error
(MSE) to evaluate performance, which is defined as:

$MAE=\frac{1}{N}\sum_{i=1}^{N}{|C_{i}-C_{i}^{GT}|}\ \ \
MSE=\sqrt{\frac{1}{N}\sum_{i=1}^{N}{|C_{i}-C_{i}^{GT}|}^{2}}$ (1)

where $C_{i}$ is the estiamted number of people, and $C^{GT}_{i}$ is the
groundtruth number. We pre-train the front-end on ImageNet [26], and then
train the benchmark network on the crowd datasets. We use a stochastic
gradient descent (SGD) optimizer with a batch size of 50, set Nesterov
momentum to 0.9, and set the weight decay to $5\times 10^{-4}$. The learning
rate is initially set to $10^{-6}$, and then divided by 10 every 3 epochs.

Table 2: The effect of fine-tuning the models pre-trained on CrowdX on the
real ShanghaiTech B dataset(MAE/MSE).

Method | None/ImageNet | GCC | CrowdX
---|---|---|---
MCNN | 26.4/41.3 | 18.8/28.2 |  16.9/24.4($\downarrow$36.0%)
CSRNet | 10.6/16.0 | 10.1/15.7 |  9.7/14.6($\downarrow$8.5%)
ESA-NET | 8.3/12.9 | - |  7.6/11.8($\downarrow$8.4%)

###  3.2 Performence on CrowdX

We conduct two experiments to study the effectiveness of using CrowdX for data
enhancement, including 1) training the benchmark model only on the
ShanghaiTech B dataset, 2) pre-training on CrowdX, and then fine-tuning the
model on the ShanghaiTech B dataset. The results show that the performance of
the model pre-trained by CrowdX is improved from 8.3 to 7.6, and the accuracy
is improved by about 8.4%. We also test it on the ShanghaiTech B dataset, and
the absolute error of the first 100 test samples shows that in almost all
samples, the pre-trained model is significantly better than the non-pre-
trained model, which further proves the effectiveness of the proposed CrowdX
dataset in data enhancement.

Furthermore, we conduct experiments on the other two latest models, MCNN [1]
and CSRNet [21]. Table 2 shows that we achieve improvements on both
architectures, with MCNN improved by 36% and CSRNet by 8.5%. Our method is
better than GCC [12], probably because the generation process of CrowdX has
more comprehensive parameter control, and most of the background of CrowdX is
an urban scene, which is consistent with the ShanghaiTech B.

Table 3: The effect of the background scenes on experimental results. The
value in row i and column j represents CMAE [dataset(i), dataset(j)], and the
same below.

|  | Solid Color Scene
---
| Synthetic Scene
---
| Real-world Scene
---
| Solid Color
---
11.6(15.4) | 61.2(109.8) | 33.7(53.0)
| Synthetic
---
33.3(41.8) | 14.9(23.1) | 37.8(68.6)
| Real-world
---
75.7(91.6) | 158.0(200.6) | 15.9(23.1)

Table 4: The effect of the camera perspective on experimental results. CP(30)
is a dataset whose data is collected by controlling the camera perspective
angle to equal 30°. CP(30,50) is the union set of CP(30) and CP(50).

| CP(30) | CP(50) | CP(70) | CP(90) | CP(30,50) | ST_B
---|---|---|---|---|---|---
CP(30) | 20.1(28.4) | 16.5(26.5) | 38.4(47.9) | 65.3(74.4) | 18.3(27.5) | 39.4(67.1)
CP(50) | 41.7(60.7) | 15.2(26.3) | 21.6(26.8) | 38.7(44.7) | 39.3(67.7) | 39.3(67.7)
CP(70) | 134.5(167.3) | 52.1(69.0) | 13.3(22.3) | 12.6(18.0) | 93.4(67.7) | 57.2(94.0)
CP(90) | 179.5(270.5) | 98.2(124.0) | 21.7(36.0) | 12.8(19.1) | 138.9(178.0) | 67.8(107.3)
CP(30,50) | 17.6(23.9) | 12.3(22.4) | 11.7(17.9) | 29.2(35.3) | 14.9(23.1) | 37.8(68.6)

###  3.3 Analysis of Influential Factors

In this section, we firstly define an operator CMAE $(\psi_{1},\psi_{2})$ to
represent the result obtained by training the benchmark on the training set of
$\psi_{1}$ and then evaluatingon the test set of $\psi_{2}$.

How does background affect performance? From the diagonal of Table 3, we can
know that the simpler the background of the dataset, the better the
performance of the trained model. Besides, compared with the transfer from the
simulated scene to the real scene, the benchmark model has better performance
in the transfer from the solid color scene to the real scene, which indicates
that the complex background hinders the feature transfer.

How does camera perspective affect performance? Intuitively, the diagonal in
Table 4 should all be the best performance, but many exceptions can be found.
Firstly, we find that CMAE [CP (30), CP (30)] $>$ CMAE [CP (30), CP (50)]. It
can be inferred that the pitch value of the camera in CP (30) is too small,
which increases the occlusion between people. The loss caused by occlusion is
greater than the loss caused by the feature difference. In addition, we find
that CMAE [CP(50), CP(ST B)] is smaller than CMAE [CP(70), CP(ST B)] and CMAE
[CP(90), CP(ST B)]. We speculate that when the pitch value difference between
the test set and the training set increases, the error will also increase.
Furthermore, we find that the model trained on CP(30,50) is better than the
model trained on a single CP(30) or CP(50). One possible reason is that
although the camera pitch value is 30 degrees, there are still many people at
a 50-degree angle.

How does person density affect performance? We combine the data of CP (30) and
CP (50) to form a dataset named PN (all). Then, we divide PN (all) into three
sub-sets PN (0-200), PN (200-400), and PN (400+). PN (200-400) means that the
number of people is between 200 and 400. The experimental results are shown in
Table 5. We find that all models achieve the best performance on PN (0-200),
which shows that the fewer people, the higher the accuracy. This is because
the occlusion will be reduced when there are few people. In addition, we find
that CMAE [PN(0-200), PN(400+)] is much larger than CMAE [PN(400+),
PN(0-200)]. It seems that a model trained in a high-density scene is more
versatile than a model trained in a low-density scene. We think this may be
because the model trained on high-density datasets can learn occlusion
features.

Table 5: The effect of the number of people on experimental results.

|  | PN(0-200)
---
| PN(200-400)
---
| PN(400+)
---
| PN(all)
---
| PN(0-200)
---
9.4(12.5) | 49.6(64.4) | 154.8(173.6) | 62.8(97.7)
| PN(200-400)
---
13.4(17.7) | 17.6(22.1) | 55.3(71.3) | 26.6(41.8)
| PN(400+)
---
27.8(31.2) | 34.9(40.7) | 33.2(44.9) | 31.5(38.5)
| PN(all)
---
8.8(12.0) | 14.3(17.8) | 23.2(34.9) | 14.9(23.1)

How dose image resolution affect counting performance?

We use two image resolutions ($1024\times 768$) and ($512\times 384$) to
explore the effect of image resolution on the algorithm performance. To ignore
the influence of factors such as background and perspective, we only use solid
color background and camera perspective 30°in CrowdX. The results are shown in
Table 6. It can be inferred from CMAE (Low resolution, Low resolution)$<$CMAE
(High resolution, High resolution) that large resolution is not necessarily
helpful for improving the performance. From CMAE (Low resolution, High
resolution) is much larger than CMAE (High resolution, Low resolution), we can
see that models trained on low-resolution datasets are more versatile than
models trained on high-resolution datasets.

Table 6: The effect of image resolution on experimental results.

|  | Low Resolution
---
| High Resolution
---
| Low Resolution
---
15.1(20.7) | 18.3(25.5)
| High Resolution
---
32.7(43.2) | 19.8(27.1)

##  4 Conclusion

In this article, we propose a synthetic crowd dataset CrowdX to enhance the
level of crowd counting algorithms. With this dataset, the main influencing
factors are analyzed and application suggestions are given. The conclusions of
these experiments can improve the industry’s understanding of the algorithm.

## References

  * [1] Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Ma Yi,  “Single-image crowd counting via multi-column convolutional neural network,”  in Computer Vision & Pattern Recognition, 2016.
  * [2] Lokesh Boominathan, Srinivas SS Kruthiventi, and R Venkatesh Babu,  “Crowdnet: A deep convolutional network for dense crowd counting,”  in Proceedings of the 24th ACM international conference on Multimedia. ACM, 2016, pp. 640–644.
  * [3] Xinkun Cao, Zhipeng Wang, Yanyun Zhao, and Fei Su,  “Scale aggregation network for accurate and efficient crowd counting,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 734–750.
  * [4] Lingke Zeng, Xiangmin Xu, Bolun Cai, Suo Qiu, and Tong Zhang,  “Multi-scale convolutional neural networks for crowd counting,”  in 2017 IEEE International Conference on Image Processing (ICIP). IEEE, 2017, pp. 465–469.
  * [5] Weizhe Liu, Krzysztof Lis, Mathieu Salzmann, and Pascal Fua,  “Geometric and physical constraints for head plane crowd density estimation in videos,”  arXiv preprint arXiv:1803.08805, 2018.
  * [6] Ze Wang, Zehao Xiao, Kai Xie, Qiang Qiu, Xiantong Zhen, and Xianbin Cao,  “In defense of single-column networks for crowd counting,”  arXiv preprint arXiv:1808.06133, 2018.
  * [7] Siyu Huang, Xi Li, Zhi-Qi Cheng, Zhongfei Zhang, and Alexander Hauptmann,  “Stacked pooling: Improving crowd counting by boosting scale invariance,”  arXiv preprint arXiv:1808.07456, 2018.
  * [8] Zhiheng Ma, Xing Wei, Xiaopeng Hong, and Yihong Gong,  “Bayesian loss for crowd count estimation with point supervision,”  in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 6142–6151.
  * [9] Haroon Idrees, Imran Saleemi, Cody Seibert, and Mubarak Shah,  “Multi-source multi-scale counting in extremely dense crowd images,”  in Computer Vision & Pattern Recognition, 2013.
  * [10] A. B. Chan, Z. S. Liang, and N. Vasconcelos,  “Privacy preserving crowd monitoring: Counting people without people models or tracking,”  in IEEE Conference on Computer Vision & Pattern Recognition, 2008\.
  * [11] Cong Zhang, Kai Kang, Hongsheng Li, Xiaogang Wang, Rong Xie, and Xiaokang Yang,  “Data-driven crowd understanding: A baseline for a large-scale crowd dataset,”  IEEE Transactions on Multimedia, vol. 18, no. 6, pp. 1048–1061, 2016.
  * [12] Qi Wang, Junyu Gao, Wei Lin, and Yuan Yuan,  “Learning from synthetic data for crowd counting in the wild,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 8198–8207.
  * [13] Youmei Zhang, Chunluan Zhou, Faliang Chang, and Alex C Kot,  “Multi-resolution attention convolutional neural network for crowd counting,”  Neurocomputing, vol. 329, pp. 144–152, 2019.
  * [14] Deepak Babu Sam, Shiv Surya, and R Venkatesh Babu,  “Switching convolutional neural network for crowd counting,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017, pp. 4031–4039.
  * [15] Swami Sankaranarayanan, Yogesh Balaji, Arpit Jain, Ser Nam Lim, and Rama Chellappa,  “Learning from synthetic data: Addressing domain shift for semantic segmentation,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3752–3761.
  * [16] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig,  “Virtual worlds as proxy for multi-object tracking analysis,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4340–4349.
  * [17] Xuan Li, Kunfeng Wang, Yonglin Tian, Lan Yan, Fang Deng, and Fei-Yue Wang,  “The paralleleye dataset: A large collection of virtual images for traffic vision research,”  IEEE Transactions on Intelligent Transportation Systems, vol. 20, no. 6, pp. 2072–2084, 2018.
  * [18] Igor Barros Barbosa, Marco Cristani, Barbara Caputo, Aleksander Rognhaugen, and Theoharis Theoharis,  “Looking beyond appearances: Synthetic training data for deep cnns in re-identification,”  Computer Vision and Image Understanding, vol. 167, pp. 50–62, 2018\.
  * [19] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez,  “The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3234–3243.
  * [20] “Unity3d engine.,” https://unity3d.com/.
  * [21] Yuhong Li, Xiaofan Zhang, and Deming Chen,  “Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1091–1100.
  * [22] Haroon Idrees, Muhmmad Tayyab, Kishan Athrey, Dong Zhang, Somaya Al-Maadeed, Nasir Rajpoot, and Mubarak Shah,  “Composition loss for counting, density map estimation and localization in dense crowds,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 532–546.
  * [23] Xiaoxiao Sun and Liang Zheng,  “Dissecting person re-identification from the viewpoint of viewpoint,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 608–617.
  * [24] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  .
  * [25] Yi Hou, Chengyang Li, Fan Yang, Cong Ma, Liping Zhu, Yuan Li, Huizhu Jia, and Xiaodong Xie,  “Bba-net: A bi-branch attention network for crowd counting,”  in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 4072–4076.
  * [26] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei,  “Imagenet: A large-scale hierarchical image database,”  in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255.