# Visual Crowd Analysis: Open Research Problems

Muhammad Asif Khan, Hamid Menouar
Qatar Mobility Innovations Center
Qatar University
Doha, Qatar
{muhammada<EMAIL_ADDRESS>
&Ridha Hamila
Department of Electrical Engineering
Qatar University
Doha, Qatar.
<EMAIL_ADDRESS>
Corresponding author: Alternate email<EMAIL_ADDRESS>is a trade name of
Qatar University-QSTP-B, www.qmic.com

Over the last decade, there has been a remarkable surge in interest in
automated crowd monitoring within the computer vision community. Modern deep-
learning approaches have made it possible to develop fully-automated vision-
based crowd-monitoring applications. However, despite the magnitude of the
issue at hand, the significant technological advancements, and the consistent
interest of the research community, there are still numerous challenges that
need to be overcome. In this article, we delve into six major areas of visual
crowd analysis, emphasizing the key developments in each of these areas. We
outline the crucial unresolved issues that must be tackled in future works, in
order to ensure that the field of automated crowd monitoring continues to
progress and thrive. Several surveys related to this topic have been conducted
in the past. Nonetheless, this article thoroughly examines and presents a more
intuitive categorization of works, while also depicting the latest
breakthroughs within the field, incorporating more recent studies carried out
within the last few years in a concise manner. By carefully choosing prominent
works with significant contributions in terms of novelty or performance gains,
this paper presents a more comprehensive exposition of advancements in the
current state-of-the-art.

##  1 Introduction

The increasing population in urban areas often leads to crowded situations in
densely populated areas which pose several challenges and threats to public
safety. To ensure the safety of the people, crowd management strategies
require efficient crowd analysis. While manual analysis of the crowd is a
tedious task, automatic crowd analysis is desired in many situations.

Automatic crowd monitoring using visual analysis is a hot topic in computer
vision research Khan et al. (2023c) with many interesting applications in city
surveillance, social distancing, transportation, sports, wildlife monitoring,
etc. Thirumalaisamy et al. (2022); Zhang et al. (2022). Over the last decade,
advances in deep learning have brought new possibilities to achieve state-of-
the-art performances in various visual crowd analysis problems such as crowd
counting, object detection, activity recognition, anomaly detection, motion
analysis, etc. Although numerous research efforts have achieved remarkable
performance in visual crowd analysis, there remain several challenges and
problems yet to address. The reason for many of the unsolved problems lies in
the underlying complexity and challenges related to crowd scenes as compared
to other computer vision tasks e.g., severe occlusions, clutters, scale
variations, unpredictable motion patterns, complex crown behaviors, the
unknown context of crowd activities, etc. Thus, crowd analysis is often
considered more challenging than other computer vision tasks. The complex
nature of the problem impacts the development of the crowd visual analysis
system and requires more sophisticated algorithms and models, collection of
diverse and large-scale datasets, hardware resources for real-time
performance, and system-level integration of such algorithms with cameras,
sensors, and data storage systems. Algorithmic innovation and data
availability are particularly of paramount importance for computer vision
researchers.

To understand these challenges, we thoroughly investigated the literature on
crowd analysis using computer vision and mostly using deep learning. We also
studied different traditional approaches, to understand how the innovations
progressed over time and how they created an impact. We greatly focused on
research works published in major scientific venues and chose those works that
have significantly contributed to the body of knowledge in terms of
identifying a real problem, innovation in methodology, or claiming significant
performance gains against previous methods. We found existing surveys
categorizing works in different ways, however, when studying the common trends
and distinct approaches in these works, we found a more intuitive and
meaningful categorization of crowd analysis. Our approach is to categorize
these works into different types of analysis tasks where each task is
completely different and requires different methods to accomplish the task
objective. As illustrated in Fig. 1, these works on visual crowd analysis are
categorized into six major areas i.e., crowd counting, object detection and
tracking, motion analysis, behavior recognition, anomaly detection, and crowd
prediction.

Figure 1: Six areas or applications of crowd visual analysis.

Crowd counting is to estimate the density of people or the total count in a
particular geographical area. Objection detection aims to detect and localize
particular objects of interest in the crowd e.g., detecting women only,
detecting people holding banners or sticks, etc. Motion analysis refers to the
collective mobility state of the crowd e.g., if the crowd is stationary/moving
along with other motion statistics such as direction, speed, flux analysis,
etc. Behavior analysis determines the collective attribute of the crowd
focusing on the activities performed by crowd members to extract contextual
behavior information e.g. if the crowd is calm, violent, etc. Anomaly
detection focuses on finding unusual and abnormal events and activities at
both individual and group levels. Crowd prediction refers to the prediction of
proactive accumulation (assembly) or influx and efflux of people in/from a
particular region that can lead to a crowd.  Generally, there is a logical
sequence in the implementation of these applications. For instance, the first
information which might interest a user or agency monitoring a crowd is the
estimated crowd density, followed by other aspects of crowd members such as
age, gender, etc., and any detecting objects such as banners, posters, and
sticks held by the members. The next information one might be interested in is
continuously following up on the crowd movement to know whether the crowd is
stationary or moving in a specific direction. The aforementioned aspects are
covered in the first three areas of crowd analysis. Then, a sophisticated
system may provide a more detailed analysis such as crowd behavior (mood,
specific activities performed by crowd members), and detecting specific
abnormal events or objects. Lastly, the prediction about crowd formation or
dispersion in a specific region or area at a specific time can be a piece of
very useful information that one would desire to obtain.

While traditionally, many semi-automated computer vision methods have been
proposed Davies et al. (1995); Silveira Jacques Junior et al. (2010b; a), the
recent advancements in modern deep learning have revolutionized the
development of fully automated vision-based crowd-monitoring applications. By
leveraging the power of deep neural networks the accuracy, efficiency, and
overall performance of such applications have been significantly improved.
Deep learning methods automatically learn and extract meaningful patterns and
features from large-scale crowd visual data such as images and videos, thus
enabling better crowd analysis. Also, the use of transfer learning allows
deploying models trained on one dataset in a different scenario after fine-
tuning, speeding up the training process.

###  1.1 Similar and Related Studies

Several studies have been conducted in the past which typically focus on
individual areas of crowd analysis. For example, Sindagi & Patel (2018);
Cenggoro (2019); Ilyas et al. (2020); Gao et al. (2020); Luo et al. (2020);
Gouiaa et al. (2021); Fan et al. (2022); Khan et al. (2023c) covers crowd
counting research and mainly discusses the advancements in model
architectures, benchmarking, and datasets. Hu et al. (2004b); Luca et al.
(2020); Kumar (2021) focus on crowd motion analysis, discussing crowd motion
predictions, flow classification, and behavior analysis using motion patterns.
Joshi & Patel (2021); Modi & Parikh (2022); Sharif et al. (2022) studies
anomaly detection with focus on methods, datasets and comparisons of results.
Unlike the aforementioned studies, few studies also cover the multiple aspects
of crowd analysis. For instance, Li et al. (2015) covers motion analysis,
behavior recognition, and anomaly detection but do not cover counting, density
estimation, object detection, crowd prediction etc. H.Y. et al. (2017) covers
density estimation, motion detection, and behavior recognition, but do not
cover object detection, and anomaly detection. Similarly, Zhang et al. (2018)
covers only on physics-inspired methods for crowd analysis and focuses on
motion analysis in videos. Table 1 presents a list of recent survey articles
on the topic.

Although the studies discussed earlier provide a thorough review of the
literature with useful taxonomy of works, our study presents a more
comprehensive review consisting of the six intuitive areas of crowd analysis.
Our survey is unique from the previous surveys as it does not provide a
thorough review covering each and every related work published, but focuses on
the most prominent works in each application area. Thus this review is more
concise, allowing even non-experts and novice researchers in the area of crowd
analysis to quickly understand the state-of-the-art in research on the
respective topic. Due to the fast-paced research in computer vision, several
related works published a few years before may not serve the purpose and yet
another new study with fresh perspectives and insights is highly commendable.

Table 1: Related studies on crowd visual analysis. Ref |  Topic |  Brief Scope
---|---|---
Sindagi & Patel (2018) |  Crowd counting |  Methods, model architectures, datasets, future insights
Ilyas et al. (2020) |  Crowd counting |  Methods, model architectures, performance evaluation.
Gao et al. (2020) |  Crowd counting |  Model architectures and categorization, datasets, evaluation metrics, challenges, and insights.
Gouiaa et al. (2021) |  Crowd counting |  Methods, the taxonomy of CNN models, datasets, and applications.
Fan et al. (2022) |  Crowd counting |  Methods, applications, and models.
Khan et al. (2023c) |  Crowd counting |  Datasets, model architecture, loss functions, evaluation metrics, insights
Hu et al. (2004b) |  Motion analysis |  Motion analysis, behavior analysis, moving object classification and identification.
Luca et al. (2020) |  Motion analysis |  Flow prediction, next-location prediction, flow generation, and trajectory generation.
Kumar (2021) |  Behavior analysis |  Methodologies for the organized crowd and non-organized crowd, datasets.
Modi & Parikh (2022) |  Anomaly detection |  Methods for anomaly detection with merits and demerits.
Sharif et al. (2022) |  Anomaly detection |  Methods, algorithms, datasets, and metrics with details analysis and comparisons.
Li et al. (2015) |  Crowd analysis |  Models, datasets, algorithms, and evaluation of crowd motion analysis, behavior recognition, and anomaly detection.
H.Y. et al. (2017) |  Crowd analysis |  Density estimation, motion detection and tracking, and behavior recognition.
Zhang et al. (2018) |  Crowd analysis |  Physics-inspired methods for crowd video analysis.

###  1.2 Contribution and Paper Organization

Although several studies exist as stated in the previous section, discussing
individual areas of crowd-monitoring applications, we aim to provide a concise
study on the topic targeted to a broader audience from various disciplines. We
intentionally omitted more technical details and focused more on the crux of
the problem for a smooth flow for the reader. For instance, there have been
more than a hundred crowd-counting models published in the literature, we
chose only those with significant contributions in terms of novelty in model
architecture, performance gains, or other aspects of design and evaluation.
Similarly, we carefully chose original research works with major contributions
(despite if the work is overlooked previously) in each application domain and
included them in our survey to summarise the SOTA in each of the six areas of
crowd analysis.

This article discusses the six intuitive areas of crowd visual analysis with a
concise description of each, provides a brief overview of the state-of-the-art
methods, and highlights the open problems in each domain. Each subsequent
section describes one of the six areas. The last section draws the conclusion
and presents future insights and research directions to advance the SOTA in
the respective area.

##  2 Counting and Density Estimation

One of the first aspects of crowd monitoring is to estimate the headcount (a
scalar value for the whole image) or density across different parts of the
scene. As people in the crowd are usually clumped together into groups,
density estimates provide more information than just the total count.
Headcount or density estimation can provide good situational awareness for the
monitoring entities like law enforcement agencies and event managers.

###  2.1 Methods and State-of-the-art

The de-facto method for counting people in an image or video frame is density
estimation. A convolution neural network (CNN) based model is trained to
estimate the crowd density. In the density estimation method, each head is
detected using a Gaussian blob around the center of the head. This is a pixel-
level regression problem and the commonly used Euclidean loss function (or
mean squared error MSE) is used to train the model. Fig. 2 presents the ground
truth density map for a crowd image used by the CNN model.

Figure 2: Crowd counting using density estimation. The predicted density map
shows crowd density across the image scene. The sum of all pixel values of the
density map equals the predicted count in the image.

The first CNN-based crowd density estimation model was CrowdCNN Zhang et al.
(2015). The CrowdCNN is a single-column CNN architecture that outperforms
traditional non-CNN methods however, still lacks the capability to adapt to
the scale variations in head sizes. Scale variation can arise from several
reasons such as distance from the camera, camera perspective effects, image
resolution, etc. To overcome scale variations, multi-column CNN networks were
proposed Zhang et al. (2016); Sindagi & Patel (2017). Multi-column models can
capture scale variations to some extent (i.e., more columns are needed for
images with larger scale variations which may increase the model size
significantly.). However, more efficient architectures were proposed to
replace multi-column architectures e.g., encoder-decoder networks Jiang et al.
(2019); Song et al. (2021); Gao et al. (2019), networks with multi-scale
modules Zeng et al. (2017); Wang & Breckon (2022) etc. Encoder-decoder models
allow hierarchical feature extraction and aggregation at multiple stages in
the encoder and decoder modules, respectively. These models are good at
preserving the spatial resolution of the predicted density maps using
downsampling and upsampling operations, however, take longer times to train
and converge. Recently, non-CNN models using vision transformers are also
proposed Liang et al. (2022); Tian et al. (2021). Transformer-based models
apply self-attention mechanisms to model global context and capture long-range
dependencies. Their disadvantage is that these models require more
computational resources compared to CNNs due to the larger number of
parameters and self-attention operations.

Crowd counting is the relatively most rigorously investigated area after
object detection with the availability of a large number of datasets (Mall
Chen et al. (2012), ShanghaiTech Zhang et al. (2016), UCF-QNRF Idrees et al.
(2018), JHU-Crowd Sindagi et al. (2020), NWPU-Crowd Wang et al. (2021),
DroneRGBT Peng et al. (2020) etc.) and a large variety of models with
reasonably good performance. The performance gains in the counting accuracy
have been high initially when single-column shallow models have been replaced
by multi-column CNN, and then pyramids structure models using transfer
learning and multi-scale modules. However, over the last 1-2 years, the gain
in accuracy has been incremental despite major architectural changes and novel
loss function. Fig. 3 compared some well-known crowd counting models showing
the mean absolute error over a benchmark dataset and the size of the model
parameters. Furthermore, a summary of model architectures, the learning (loss)
functions, evaluation metrics, and the training methods (supervised/weakly
supervised, etc.) is presented in Table 3.

Table 2: A summary of research efforts in crowd counting, categorized in distinct categories i.e., model architecture, loss functions, metrics, and training methods. Research Area |  Summary of the SOTA
---|---
Model Architectures |  Single-column Zhang et al. (2015), multi-column Zeng et al. (2017), Encoder-decoder Jiang et al. (2019), pyramid, Vision transformers Liang et al. (2022)
Loss functions |  Euclidean loss Zhang et al. (2016), Combination loss Zhang et al. (2015), Curriculum Loss Wang & Breckon (2022), Composite loss Idrees et al. (2018), AP Loss Jiang et al. (2020), PRA Loss Jiang et al. (2020), SCL Loss Jiang et al. (2019), OT Wang et al. (2020)
Evaluation metrics |  MAE, MSE, GAME Guerrero-Gómez-Olmedo et al. (2015), SSIM Li et al. (2018), PSNR Li et al. (2018), PAME , PMSE , MPAE
Training methods |  Supervised/weakly-supervised Yang et al. (2020); Liang et al. (2022); Lei et al. (2021), curriculum learning Khan et al. (2023b)
Figure 3: Performance of well-known crowd counting models over years.

###  2.2 Challenges and Open Problems

The accuracy performance (measured using mean absolute error (MAE) metric) of
crowd counting models has significantly improved over time on benchmark
datasets. For instance, the MAE over ShanghaiTech Part A dataset for MCNN
Zhang et al. (2016) was 110.2 which is reduced to 66.1 by TransCrowd Liang et
al. (2022). A similar improvement has been achieved on other benchmarking
datasets Khan et al. (2023c) as well. Despite the significant improvements in
counting accuracy, there still remain several challenges.

  * •

The problem of scale variations caused due to perspective effects have been
overcome by multi-scale architectures, other issues such as occlusions and
complex backgrounds are still a major challenge in many complex scenes.
Although some methods propose background segmentation as a preprocessing step,
that can increase the complexity of the task.

  * •

Second, there is still more room for further improvement in counting accuracy
on new benchmark datasets such as UCF-QNRF, JHUCrowd++, and NWPU Crowd
datasets due to more challenging scenes e.g., extremely dense crowds, extreme
weather and low light conditions, camera blur effects, etc. These conditions
are common in the real world and the results typically achieved over previous
datasets would not be attainable on these new datasets and thus in the real
world.

  * •

Third, the commonly used metric in almost all studies is MAE provides an
average performance of crowd-counting models. In practice, a model may produce
less accurate predictions on difficult examples (e.g., extremely dense and
blurry images), but the MAE (being an average over the entire test set) is
compensated to be low due to better predictions on easy examples posing a low
value of MAE (e.g., less dense and clear images).

  * •

Fourth, as the benchmark datasets become larger and more challenging over
time, the resulting models to achieve better performance over these benchmarks
become deeper. This means more complexity to run these models in real-time,
especially on resource-limited edge devices. This will create potential
bottlenecks in edge-based crowd-monitoring solutions. Lightweight models are
being developed Khan et al. (2023a; b), but these provide limited accuracy in
very dense crowds.

We urge the requirement of more scenario-specific datasets to build reliable
models for production. Models trained on generic datasets typically require a
long training time and yet lack generalization capabilities.

##  3 Object Detection and Tracking

There are many situations in which one is interested to detect an object of
interest in a video frame and then track it over time in consecutive frames.
Detecting a particular object class (e.g., pedestrians, vehicles, men, women,
etc.) in a single image or video frame is called object detection, whereas
identifying an individual object or a set of objects in consecutive video
frames from a single camera or frames from multiple cameras is called object
tracking.

###  3.1 Methods and State-of-the-art

Object detection is a well-researched problem that gained significant
attention in computer vision and several prominent models are developed over
time that achieved state-of-the-art performance. These models are typically
divided into two categories i.e., anchor-based methods and anchor-free models.
Anchor-based methods use a predefined set of anchor boxes placed over the
entire image and predict the final set of boxes around the detected objects.
These models provide better accuracy and are widely deployed e.g., Faster-RCNN
Girshick (2015), SSD Liu et al. (2015), and YOLO (v3) Redmon & Farhadi (2018).
Anchor-based methods adapt well to various scales and aspect ratios and work
well in complex scenes. However, their performance is greatly affected by
various factors such as sizes, aspect ratios, the number of anchor boxes, and
shape variations Tian et al. (2019). The predefined anchor boxes require
manual design and careful calibration, which can be time-consuming and
computationally expensive. However, anchor-based methods have been widely
adopted.

Anchor-free methods are relatively new and more efficient than anchor-based
methods. They use the keypoint detection approach. For example in CornerNet
Law & Deng (2018), the model predicts the top-left and bottom-right corners
around objects to draw the final bounding box. In CenterNet Duan et al.
(2019), a single point at the center of the object is detected to draw the
bounding box.

A fully convolutional one-stage (FCOS) detector is another single-stage
anchor-free method proposed in Tian et al. (2019) that detects objects using
per-pixel prediction that achieve comparable performance to anchor-based
methods (Faster-RCNN, YOLOv2, etc.) and outperforms previous anchor-free
methods (CornerNet, etc.). Anchor-free methods are simpler in design and
implementation, and computationally efficient. However, these models perform
poorly in accurately localizing small objects or objects with complex shapes,
especially in dense and overlapping instances. They also require more training
data to sufficiently train.

More recently, non-CNN methods based are getting attention in detection tasks.
The vision transformer (ViT) model is proposed in Chen et al. (2022) as an
alternative to CNN-based models. The original ViT and other Transformer-based
models have shown comparative performance in many tasks compared to several
CNN-based models. However, CNNs are still considered as defacto-methods for
detection tasks due to their fast learning capabilities (training and fine-
tuning) as compared to the transformer models. Fig. 4 depicts a sample output
of an object detection model showing bounding boxes around the detected
objects (i.e., persons). Table LABEL:tab-detection shows a list of objection
detection models in the three categories.

Figure 4: Object detection sample from EuroCity Persons dataset Braun et al.
(2019).

There have been several publicly available datasets for object detection tasks
however, the most popular datasets used for benchmarking are Pascal VOC
Everingham et al. (2012) and MS-COCO Lin et al. (2014). All mainstream object
detection models (e.g., YOLO family, FCOS, CornerNet, etc.) are evaluated over
these datasets, making the fair benchmarking of model accuracy.

Table 3: A summary of research efforts in crowd counting, categorized in distinct categories i.e., model architecture, loss functions, metrics, and training methods. Research Area |  Summary of the SOTA
---|---
Model Architectures |  Single-column Zhang et al. (2015), multi-column Zeng et al. (2017), Encoder-decoder Jiang et al. (2019), pyramid, Vision transformers Liang et al. (2022)
Loss functions |  Euclidean loss Zhang et al. (2016), Combination loss Zhang et al. (2015), Curriculum Loss Wang & Breckon (2022), Composite loss Idrees et al. (2018), AP Loss Jiang et al. (2020), PRA Loss Jiang et al. (2020), SCL Loss Jiang et al. (2019), OT Wang et al. (2020)
Evaluation metrics |  MAE, MSE, GAME Guerrero-Gómez-Olmedo et al. (2015), SSIM Li et al. (2018), PSNR Li et al. (2018), PAME , PMSE , MPAE
Training methods |  Supervised/weakly-supervised Yang et al. (2020); Liang et al. (2022); Lei et al. (2021), curriculum learning Khan et al. (2023b)

Fig. 5 shows a performance comparison of various object detection models
discussed in this section.

Figure 5: Average precision (AP50) of well-known object detection models on MS
COCO dataset (except YOLO which is evaluated on Pascal VOC dataset).

###  3.2 Challenges and Open Problems

Both anchor-based and anchor-free methods have achieved significant
performance gains (in production) in the past few years, but both methods
still face some intrinsic limitations in the context of crowd scenes. While
some of these challenges are generic to any computer vision tasks, some are
more severe in crowded scenes given as follows:

  * •

Viewpoint variation - when an object looks different when captured from
different angles.

  * •

Object deformation - when an object appears in different shapes in the same
frame or in consequent frames of a video (e.g., a person bending down).

  * •

Severe occlusions - when one or more objects are partially not visible in an
image due to overlapping with other objects in front of them.

  * •

Illuminations - when there are large variations of brightness values of pixels
in images.

  * •

Clutters - when an image contains many or large objects other than objects of
interest.

The aforementioned issues are very likely to be encountered in many real-world
scenarios. Although augmentation techniques play a role to improve the models’
performance in learning such challenging environments, there is still no
standard method that solves these problems in all scenarios. It is worthy
noting that the existing models for object detection are not well-suited for
crowd environments and hence despite finetuning produce poor results. This is
a serious concern that hinders the adoption of serious surveillance
applications (e.g., that can be used by law enforcement agencies).

##  4 Motion Analysis

Understanding crowd dynamics can provide more useful information in addition
to the crowd count and density estimates. For instance, one may be interested
to know whether the crowd is stationary or moving. For a moving crowd, it will
be interesting to understand the crowd flux and other patterns related to the
crowd movement including trajectory, direction, velocity, etc. It also
includes detecting stationary groups in the crowd. Motion analysis has many
interesting applications e.g., access control, human identification,
congestion analysis, and multi-camera interactive surveillance Hu et al.
(2004a). Motion analysis may refer to recognizing the movement of body parts
of a person (e.g., gestures, actions, etc.), but in the context of the crowd,
it is often referred to as the coherent motion of a group of individuals.
Crowd motion analysis is of great interest in understanding crowd behavior
analysis and scene understanding e.g., categorizing a crowd as a stationary or
moving crowd, crowd trajectory predictions, crowd flux analysis, and crowd
motion patterns analysis. Motion analysis may also include studying abnormal
motion behavior Gupta et al. (2019).

###  4.1 Methods and State-of-the-art

Crowd motion analysis methods aim to detect, track, and analyze motion
patterns (e.g., in Fig. 6) to infer important insights about the dynamic
behavior of a crowd. Technically, motion analysis includes tempo-spatial
analysis of the crowd. There have been several manual and end-to-end automated
methods and mathematical models for crowd movement statistics.  Traditionally,
motion analysis used methods such as motion segmentation (pixel-wise
separation of moving objects from the background), temporal differencing
(pixel-wise differences consecutive frames), and optical flow (using flow
vectors of moving objects over time) Hu et al. (2004b).  For instance, a
running count of people’s trajectories passing through user-defined lines in a
scene is used to measure crowd flows. Several flows can be further integrated
over multiple spatial and temporal windows. Crowd motion is represented in
different ways e.g., optical flow (movement of each pixel from one frame to
another), particle flow (moving grid of particles with optical flow through
numerical integration), streaklines flows (traces left as line upon injection
of colored material in the flow.), spatiotemporal features, tracklets (a
fragment of trajectory obtains by a tracker with a short period of time), etc
Saqib (2019).

Motion segmentation-based methods can be easily implemented using background
subtraction or may use methods such as clustering or graph-based approaches.
These methods are more sensitive to noise and variations in lighting
conditions and poorly perform in the presence of occlusions and recognizing
complex motion patterns. Optical flow-based methods capture the motion of
every pixel. The algorithms are computationally efficient allowing real-time
performance. However, these methods suffer from inaccuracies to capture small
motions between frames, frames with repetitive patterns, and significant scale
variation.  Authors in Rabaud & Belongie (2006) focus on counting moving
objects in a crowd by detecting independent motions. Similarly Lin et al.
(2009); Gupta et al. (2019) studies the global motion patterns in crowds. Ali
& Shah (2008) studies the segmentation of crowd flows whereas Hu et al. (2008)
uses optical flows to learn the crowd motion patterns.

###  4.2 Challenges and Open Problems

  * •

Most research on crowd motion analysis is based on the representation of
pictorial information such as color, texture, etc. However, these methods only
provide some very basic analysis of motion patterns and do not sufficiently
provide semantic information on crowd motion. Automated motion analysis not
only requires large-scale video datasets but also efficient methods to extract
both micro and macro statistics of crowd motion. The existing methods
seriously lack the sophistication required for real-world implementations.

  * •

To extract more high-level and intuitive motion information, models need to
learn automatic semantic features and relationships between low-level
pictorial features and high-level semantic features.

Figure 6: Crowd motion patterns in Crowd 11 dataset Dupont et al. (2017).

##  5 Behavior, Activity and Context Recognition

Behavior analysis refers to the analysis of the crowd as a whole or a portion
of the crowd on a longer time scale (i.e., minutes to hours). It may involve
simpler tasks such as identifying the state of the crowd’s behavior e.g.,
calm, active, violent crowd, or may involve complex activity recognition.
Activity recognition refers to detecting various grouped activities of crowd
members such as protesting, dancing, fighting, etc. usually on a shorter time
scale (i.e., seconds to minutes). It can also refer to actions of an
individual object e.g., pose estimation. Some examples of group activities are
depicted in Fig. 7.

The individuals in a crowd may interact with each other and engage with each
other in different activities. While activity recognition detects crowd
activities using shape, pose, or motion features, context analysis studies the
social interaction among group members using time, location, and other
contextual information and their relation to the crowd. Context analysis of
crowd activity is a more complex problem than activity recognition due to many
other factors e.g., environment and other factors related to psychology and
sociology.

###  5.1 Methods and State-of-the-art

Activity recognition involves a framework for defining individual and grouped
activities and then assigning unique descriptors to each activity. The next
step is then to accurately learn the spatial and temporal profiles of each
activity either using traditional methods such as mathematical models or
machine learning or to apply end-to-end learning using deep learning models.
Behavior analysis is typically considered as a macroscopic crowd analysis,
whereas activity recognition may refer to as microscopic crowd analysis. In
Kok et al. (2016), the authors present common attributes of a crowd (i.e.,
decentralized, collective motion, emergency behavior) and map these attributes
to biological and physical entities.

The research on contextual analysis of crowd activities analysis is still very
limited due to the inherited complexity of the problem. Authors in Benetka et
al. (2019) present a qualitative analysis of the context in human activity
recognition using several attributes such as time and location to establish a
spatiotemporal context in the human activity prediction system. Several works
propose the use of low-level feature-based methods such as optical flow.In
Tran et al. (2015), authors studied various aspects of the crowd context
analysis. First, discovering meaningful groups in a crowd is modeled as a
dominant set clustering algorithm. Second, it uses group context activity
(GCA) descriptors of a target person and its semantic neighbors and applies
conditional random field (CRF) and support vector machines (SVM). The authors
use two datasets (i) the Collective Activity dataset that involves simple
activities i.e., crossing, waiting, queuing, walking, and talking (with two
additional activities i.e., dancing and jogging), and (ii) UCLA Courtyard
dataset having 10 human activities (Riding, Skateboard, Riding Bike, Riding
Scooter, Driving Car, Walking, Talking, Waiting, Reading, Eating, and
Sitting).

###  5.2 Challenges and Open Problems

  * •

Activity recognition and behavior analysis are generally more complex due to
the semantic relationships between the detected activities to the human
habits. A single behavior can be generally mapped to multiple semantic
concepts. Thus, to infer a meaningful semantic behavior, an accurate
relationship between the low-level features and the semantic behavior must be
established apriori.

  * •

It is generally very hard to detect and track individual persons in different
crowded scenes Shao et al. (2017).

  * •

Some key problems in behavior analysis include applying background knowledge
and reasoning theory to correctly define natural language descriptors to
semantic behaviors and then learning these behaviors from the transformations
of the object in a scene at different levels.

  * •

It is generally desired to use multiple cameras in crowd surveillance.
However, it is also more challenging to apply data fusion from multiple
sources (e.g., cameras or other sensors) which involves automatically
inferring human activities and behaviors from multiple features (rather than
images or frames). Feature extraction and fusion from multiple camera sources
also require significant hardware resources and large-scale adoption of such
systems can be slower.

  * •

Context recognition in crowd analysis is of major importance in a fully
automatic crowd surveillance application. It mostly involves multi-modal data
from several sources including camera outputs (images or videos) as well as
data from sources such as social media (tweets, Facebook posts, comments,
etc.), and real-time check-in/checkout records from venues such as airports,
metro stations, event venues, etc. The rich information obtained from these
diverse sources can provide more accurate contextual information on crowd
activities and more accurate semantic descriptors. However, the research in
this area is still very limited, and existing works only touch the surface of
this broad and deep area in crowd analysis.

Figure 7: Activity recognition with several crowd activities Wang et al.
(2022).

##  6 Anomaly Detection

Anomaly detection refers to finding anomalous (aka abnormal) events and has
many significant applications e.g., crime detection, traffic violations,
abandoned objects detection, weapons detection, etc. In the context of crowd
monitoring, anomalies are found using spatiotemporal feature analysis of the
video frames as well as in a single image. Fig. 8 shows example anomalies in
an outdoor scene.

###  6.1 Methods and State-of-the-art

Crowd anomaly detection is a challenging task mainly due to the rare
occurrence of such events and thus lack of sufficient data on anomalous
events. The definition of anomaly is subjective and the same event can be
classified as normal and abnormal at different times and places. Anomalies can
be segregated into (i) point anomaly (when a single entity or person looks or
behaves differently than other entities in the scene), (ii) contextual anomaly
(when an entity or object is treated as abnormal in a specific contextual
situation or environment), and (iii) collective anomalies (when a group of
instances behaves abnormally than the rest of the entities in the scene).
Anomaly detection is also classified as a global anomaly (does the scene/frame
has anomaly) versus local anomaly (the localization of the anomaly in the
frame or video).

The most basic and commonly used approach to detect anomalies is to train a
one-class classifier (OCC) trained with data containing normal examples. The
OCC model trained with a sufficiently large number of normal training
instances can predict abnormal events. However, even collecting data with all
normal events is not easy.

Anomaly detection in crowded scenes has been addressed in numerous ways using
a variety of algorithms including classical machine learning schemes e.g.,
k-means and SVMs Yang et al. (2019), GMM, etc., as well as deep learning
methods e.g., CNNs Joshi & Patel (2021); Pang et al. (2020); Wu et al. (2020),
LSTM Esan et al. (2020), GANs Luo et al. (2017a); Chen et al. (2021), and
autoencoders (AEs) Simonyan & Zisserman (2014); Pawar & Attar (2021), bag-of-
words (BOW) method, and physics-inspired approaches Wu et al. (2010), etc.
These methods are often inter-related and the exact taxonomy is difficult. In
BOW method, spatiotemporal visual features are extracted to detect abnormal
events Javan Roshtkhari & Levine (2013). BOW method for anomaly detection may
not capture complex anomalies and can be used to train on specific anomalies.
Datasets for anomaly detection include UCSD Mahadevan et al. (2010a),
PETS2009, UMN Bird et al. (2006), CUHK-Avenue Lu et al. (2013), and
ShanghatTech Campus Luo et al. (2017b).

###  6.2 Challenges and Open Problems

Anomaly detection is very challenging due to several reasons including data
availability, compute power requirement, fairness, and generalization. Some of
the open problems and challenges are listed as follows.

  * •

First, there is no universal definition of an abnormal event i.e., an event
that is considered abnormal may be considered abnormal in a different context.
For example, a person carrying a gun is abnormal but becomes normal when the
person carrying the gun is a police constable. Thus, the context is always
significant in anomaly detection which makes the anomaly detection problem
very challenging. This is a serious challenge that must be tackled first to
enhance the outcomes of research on anomaly detection. Contribution from
government and private law enforcement agencies can play significant in data
acquisition and annotation. Without sufficient data and standard definitions
of crowd anomalies, the research outcomes will be significantly limited.

  * •

Second, there is a lack of good datasets for anomaly detection. The existing
datasets cover only a small number of anomalies. The methods to label data in
these datasets also vary (frame-level anomalies, video-level anomalies,
segment-level anomalies, etc.).

  * •

Third, it is not easy to create large datasets with diverse anomalies because
several abnormal events can not be obtained beforehand. Video datasets for
anomalies are sparse, and not typically accessible publicly. The publicly
available datasets are mostly captured in the same location and capture very
few anomalies.

  * •

Fourth, most real-world anomalies can be better captured in video sequences,
thus the anomaly detection datasets are typically of very huge size (as
compared to other computer vision tasks such as object classification or
detection). As a result, huge computational resources are required to train
deep-learning models over tons of videos.

Figure 8: Anomaly examples in pedestrian spaces e.g., wheelchair, skater,
biker, and cart. Mahadevan et al. (2010b).

##  7 Crowd Prediction

Crowd prediction refers to predicting in advance crowd accumulation in a
particular region. Predicting crowds ahead of time has key significance in
various scenarios and applications such as events, detecting incidents,
tourism attractions, etc. Crowd prediction typically is complex and often
leverages multi-modal data from various sources, such as surveillance cameras,
social media, mobile phone signals, sensors, etc. Machine learning and
statistical modeling techniques are employed to learn and make predictions
about future crowds. The process may relate to specific crowd events or
gatherings, to estimate the duration of the event, forecast peak crowd hours,
or anticipate the popularity of certain areas or attractions within the event.
A crowd in specified administrative areas can emerge typically in different
ways e.g., Fountainhead: crowd emerges from a single direction and spread over
different directions just like a fountain, Bottleneck: crowd emerges from
different directions and assembles at a single point, and Lane: crowd emerges
at a uniform rate and moves in a single direction. Crowd prediction involves
forecasting the aggregated flows (incoming and outgoing) in a specified
region. Further information such as the origin and destination of crowd flows
are also studied as part of the crowd prediction problem.

###  7.1 Methods and State-of-the-art

Crowd prediction can be simply modeled as a time series forecasting problem to
regress an increase in crowd density over time starting from a no-crowd
condition or can also use spatial statistics of crowds to predict crowds in
spatial regions. Traditionally, it uses moving averaging methods such as
Autoregressive Integrated Moving Average (ARIMA) Liu et al. (2021). However,
these methods fail to capture the complex temporal and spatial dependencies
despite several feature engineering techniques. To cope with complex
dependencies, deep learning methods including CNN Song et al. (2020), LSTM Li
et al. (2019), and graph neural networks (GNNs) Li et al. (2022) have been
proposed.  Although multi-modal data can be used for predicting crowds, a
sufficient quantity of such datasets does not exist. The use of LSTM and CNNs
together has been a more promising method to predict crowds over shorter time
periods. However, over longer time spans, forecasting methods (e.g., ARIMA)
are applied.

###  7.2 Challenges and Open Problems

  * •

Crowd prediction can either be logically concluded from the analysis of crowd
motion patterns or predicted directly using time series prediction models.
However, the research in this area is very limited, and very few works are
found often using deep learning methods such as CNN and LSTM.

  * •

Crowd prediction has both spatial and temporal dependencies and generally, it
is quite difficult to predict crowd accumulation in a region over long
periods.

  * •

The prediction in fountainhead and bottleneck is more difficult than a lane
pattern.

  * •

Due to the spatial dependency, crowd flow prediction in irregular-shaped
regions can be more challenging than in regular-shaped regions.

Lastly, Table 4 provides a summary of the aforementioned six crowd analysis
areas, some common examples of tasks in each area, and open research
challenges.

Table 4: Crowd analysis tasks and Open research problems. Crowd Analysis |  Example Tasks |  Open Challenges
---|---|---
Crowd Counting |  Find the total headcount in a frame.
crowd localization in a large region.
|  Handling severe occlusions, scale variations in highly dense crowd images,
perspective distortion, multi-view counting.
Object Detection |  Detect and localize objects of interest.
Track objects in a video. |  Viewpoint variations, Object deformation, severe occlusions, clutter, illumination, etc.
Motion Analysis |  Observe crowd collectiveness.
Is the crowd slow/fast-moving?
Where is the crowd heading? at which speed?
Find trajectories and motion patterns.
|  Complex semantic features, complex relationship between low-level pictorial
features with high-level semantic features.
Behavior/Activity Analysis |  Is the crowd calm or active?
Are people protesting, dancing, fighting, etc? |  Complex mapping of low-level features to activities, semantic relationship and mapping of detected activities to human behavior, standard activity definitions to semantic behavior.
Anomaly Detection |  Detect abandoned objects.
Detect any weapons or crimes?
Detect intrusion. |  Lacks of a unified definition of anomalies, lack of realistic datasets, the huge size of video datasets 111An extensive list of challenges can be found in Sharif et al. (2022).
Crowd prediction |  Detect if a crowd is expected in a region?
How crowd accumulates over time? |  Difficult to capture dual (spatial and temporal) dependencies, complex motion patterns.

##  8 Conclusion and Future Insights

This article defines six major areas of visual crowd analysis which together
form a full-fledged automated crowd-monitoring system. Each of these six areas
involves a different level of complexity, and thus the state-of-the-art
greatly varies in these. For instance, crowd counting and detection are two
areas with significantly improved results achieved recently over large
benchmark datasets. Typically, existing real-world crowd-monitoring
implementations cover these tasks. The benchmarking in counting and detection
is also very clear and can be easily compared. However, there are still areas
requiring significant attention. In counting and density estimation, the
common regression loss function (Euclidean loss or MSE) has been used.
However, over time, several other loss functions (e.g., OT loss, AP Loss, PRA
Loss, etc.) have been proposed with reportedly improved performance. However,
these loss functions could not be continued in the subsequent works and many
recent works consistently used the original MSE loss function. It is
encouraged to evaluate these loss functions over different datasets and
several mounting models to conclude their potential benefits in this domain.
We also encourage testing the potential performance gain of using curriculum
learning (CL) in crowd-counting tasks with appropriate curriculum strategies.

Motion analysis that focuses on crowd-level mobility statistics often uses
flow-based and spatiotemporal features. Flow-field models are relatively more
studied. However, clustering models are gaining attention due to their
performance in more crowded scenes. Motion analysis tasks vary such as
detecting crowd source/sink, trajectory finding, speed, etc. The benchmarking
in this application domain is less coherent due to the nature of the task and
multiple objectives being considered in existing studies.  Behavior analysis
and anomaly detection which are sometimes overlapping are the most complex
tasks and the progress in these tasks is still very limited and scattered in
terms of methods, approaches, assumptions, and objectives despite their major
importance in several use cases. The lack of definition of anomalies,
activities, and behavior causes researchers to use different objectives and
evaluation metrics which makes benchmarking unfair. We suggest that future
research should focus on developing common definitions of activities and
anomalies considering context and enhancing existing datasets as well as
creating larger and balanced datasets. We also believe that in many scenarios,
local anomalies would be required rather than global anomalies which shall
make the task easier to learn; however, the variations in environment and
context will make the cross-domain transfer-learning challenging. Physics-
inspired approaches (e.g., energy models) are interesting directions to
implement anomaly detection.

We envision major advances in the near future in the under-explored areas due
to the recent developments in generative AI which will be helpful to cope with
the need for more training data. Furthermore, crowd analysis in real-time
typically requires fast inference. In CCTV-based surveillance, it is more
convenient to perform all processing and inference tasks on a local server due
to the high-speed wired connectivity option however, in aerial surveillance
(using drones) on-device processing and inference may be more convenient in
some cases. Thus, lightweight crowd analysis models will be preferred.

#### Acknowledgments

This publication was made possible by the PDRA award PDRA7-0606-21012 from the
Qatar National Research Fund (a member of The Qatar Foundation). The
statements made herein are solely the responsibility of the authors.

## References

  * Ali & Shah (2008) Saad Ali and Mubarak Shah.  Floor fields for tracking in high density crowd scenes.  In _European Conference on Computer Vision_ , 2008.
  * Benetka et al. (2019) Jan R. Benetka, John Krumm, and Paul N. Bennett.  Understanding context for tasks and activities.  In _Proceedings of the 2019 Conference on Human Information Interaction and Retrieval_ , CHIIR ’19, pp. 133–142, New York, NY, USA, 2019\. Association for Computing Machinery.  ISBN 9781450360258.
  * Bird et al. (2006) N. Bird, S. Atev, N. Caramelli, R. Martin, O. Masoud, and N. Papanikolopoulos.  Real-time, online detection of abandoned objects in public areas.  In _Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006._ , pp. 3775–3780, 2006.  doi: 10.1109/ROBOT.2006.1642279.
  * Braun et al. (2019) Markus Braun, Sebastian Krebs, Fabian B. Flohr, and Dariu M. Gavrila.  Eurocity persons: A novel benchmark for person detection in traffic scenes.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2019.  ISSN 0162-8828.  doi: 10.1109/TPAMI.2019.2897684.
  * Cenggoro (2019) Tjeng Wawan Cenggoro.  Deep learning for crowd counting: A survey.  _Engineering, MAthematics and Computer Science (EMACS) Journal_ , 2019\.
  * Chen et al. (2021) Dongyue Chen, Lingyi Yue, Xingya Chang, Ming Xu, and Tong Jia.  Nm-gan: Noise-modulated generative adversarial network for video anomaly detection.  _Pattern Recognit._ , 116:107969, 2021.
  * Chen et al. (2012) Ke Chen, Chen Change Loy, Shaogang Gong, and Tony Xiang.  Feature mining for localised crowd counting.  In _Proceedings of the British Machine Vision Conference_ , pp. 21.1–21.11. BMVA Press, 2012.
  * Chen et al. (2022) Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Y. Qiao.  Vision transformer adapter for dense predictions.  _ArXiv_ , abs/2205.08534, 2022.
  * Davies et al. (1995) Anthony C. Davies, J. H. Yin, and Sergio A. Velastín.  Crowd monitoring using image processing.  _Electronics & Communication Engineering Journal_, 7:37–47, 1995.
  * Duan et al. (2019) Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 6568–6577, 2019.
  * Dupont et al. (2017) Camille Dupont, Luis Tobías, and Bertrand Luvison.  Crowd-11: A dataset for fine grained crowd behaviour analysis.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pp. 2184–2191, 2017.  doi: 10.1109/CVPRW.2017.271.
  * Esan et al. (2020) Dorcas Oladayo Esan, Pius Adewale Owolawi, and Chuling Tu.  Anomalous detection system in crowded environment using deep learning.  _2020 International Conference on Computational Science and Computational Intelligence (CSCI)_ , pp. 29–35, 2020.
  * Everingham et al. (2012) M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results.  http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html, 2012\.
  * Fan et al. (2022) Zizhu Fan, Hong Zhang, Zheng Zhang, Guangming Lu, Yudong Zhang, and Yaowei Wang.  A survey of crowd counting and density estimation based on convolutional neural network.  _Neurocomputing_ , 472:224–251, 2022.
  * Gao et al. (2019) Chenyu Gao, Peng Wang, and Ye Gao.  Mobilecount: An efficient encoder-decoder framework for real-time crowd counting.  In _Pattern Recognition and Computer Vision: Second Chinese Conference, PRCV 2019, Xi’an, China, November 8–11, 2019, Proceedings, Part II_ , pp. 582–595. Springer-Verlag, 2019.
  * Gao et al. (2020) Guangshuai Gao, Junyu Gao, Qingjie Liu, Qi Wang, and Yunhong Wang.  Cnn-based density estimation and crowd counting: A survey.  _ArXiv_ , abs/2003.12783, 2020.
  * Girshick (2015) Ross B. Girshick.  Fast r-cnn.  _2015 IEEE International Conference on Computer Vision (ICCV)_ , pp. 1440–1448, 2015.
  * Gouiaa et al. (2021) Rafik Gouiaa, Moulay A. Akhloufi, and Mozhdeh Shahbazi.  Advances in convolution neural networks based crowd counting and density estimation.  _Big Data Cogn. Comput._ , 5:50, 2021.
  * Guerrero-Gómez-Olmedo et al. (2015) Ricardo Guerrero-Gómez-Olmedo, Beatriz Torre-Jiménez, Roberto Javier López-Sastre, Saturnino Maldonado-Bascón, and Daniel Oñoro-Rubio.  Extremely overlapping vehicle counting.  In _IbPRIA_ , 2015.
  * Gupta et al. (2019) Tanu Gupta, Vimala Nunavath, and Sudip Roy.  Crowdvas-net: A deep-cnn based framework to detect abnormal crowd-motion behavior in videos for predicting crowd disaster.  In _2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)_ , pp. 2877–2882, 2019.  doi: 10.1109/SMC.2019.8914152.
  * Hu et al. (2008) Min Hu, Saad Ali, and Mubarak Shah.  Learning motion patterns in crowded scenes using motion flow field.  In _2008 19th International Conference on Pattern Recognition_ , pp. 1–5, 2008.  doi: 10.1109/ICPR.2008.4761183.
  * Hu et al. (2004a) Weiming Hu, Tieniu Tan, Liang Wang, and S. Maybank.  A survey on visual surveillance of object motion and behaviors.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , 34(3):334–352, 2004a.  doi: 10.1109/TSMCC.2004.829274.
  * Hu et al. (2004b) Weiming Hu, Tieniu Tan, Liang Wang, and Stephen J. Maybank.  A survey on visual surveillance of object motion and behaviors.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , 34:334–352, 2004b.
  * H.Y. et al. (2017) Swathi H.Y., G. Shivakumar, and H. S. Mohana.  Crowd behavior analysis: A survey.  _2017 International Conference on Recent Advances in Electronics and Communication Technology (ICRAECT)_ , pp. 169–178, 2017.
  * Idrees et al. (2018) Haroon Idrees, Muhmmad Tayyab, Kishan Athrey, Dong Zhang, Somaya Ali Al-Maadeed, Nasir M. Rajpoot, and Mubarak Shah.  Composition loss for counting, density map estimation and localization in dense crowds.  _ArXiv_ , abs/1808.01050, 2018.
  * Ilyas et al. (2020) Naveed Ilyas, Ahsan Shahzad, and Kiseon Kim.  Convolutional-neural network-based image crowd counting: Review, categorization, analysis, and performance evaluation.  _Sensors (Basel, Switzerland)_ , 20, 2020.
  * Javan Roshtkhari & Levine (2013) Mehrsan Javan Roshtkhari and Martin D. Levine.  An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions.  _Comput. Vis. Image Underst._ , 117(10):1436–1452, oct 2013.  ISSN 1077-3142.
  * Jiang et al. (2020) Xiaoheng Jiang, Li Zhang, Mingliang Xu, Tianzhu Zhang, Pei Lv, Bing Zhou, Xin Yang, and Yanwei Pang.  Attention scaling for crowd counting.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 4705–4714, 2020.
  * Jiang et al. (2019) Xiaolong Jiang, Zehao Xiao, Baochang Zhang, Xiantong Zhen, Xianbin Cao, David S. Doermann, and Ling Shao.  Crowd counting and density estimation by trellis encoder-decoder networks.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 6126–6135, 2019.
  * Joshi & Patel (2021) Kinjal Vishnuprasad Joshi and Narendra M. Patel.  A cnn based approach for crowd anomaly detection.  _Int. J. Next Gener. Comput._ , 12, 2021.
  * Khan et al. (2023a) Muhammad Asif Khan, Hamid Menouar, and Ridha Hamila.  Dronenet: Crowd density estimation using self-onns for drones.  In _2023 IEEE 20th Consumer Communications & Networking Conference (CCNC)_, pp. 455–460, 2023a.  doi: 10.1109/CCNC51644.2023.10059904.
  * Khan et al. (2023b) Muhammad Asif Khan, Hamid Menouar, and Ridha Hamila.  Lcdnet: a lightweight crowd density estimation model for real-time video surveillance.  _Journal of Real-Time Image Processing_ , 20, 2023b.
  * Khan et al. (2023c) Muhammad Asif Khan, Hamid Menouar, and Ridha Hamila.  Revisiting crowd counting: State-of-the-art, trends, and future perspectives.  _Image Vision Comput._ , 129(C), jan 2023c.  ISSN 0262-8856.  doi: 10.1016/j.imavis.2022.104597.  URL https://doi.org/10.1016/j.imavis.2022.104597.
  * Kok et al. (2016) Ven Jyn Kok, Mei Kuan Lim, and Chee Seng Chan.  Crowd behavior analysis: A review where physics meets biology.  _Neurocomputing_ , 177:342–362, 2016.
  * Kumar (2021) Ajay Kumar.  Crowd behavior monitoring and analysis in surveillance applications: A survey.  2021\.
  * Law & Deng (2018) Hei Law and Jia Deng.  Cornernet: Detecting objects as paired keypoints.  _International Journal of Computer Vision_ , 128:642–656, 2018.
  * Lei et al. (2021) Yinjie Lei, Yan Liu, Pingping Zhang, and Lingqiao Liu.  Towards using count-level weak supervision for crowd counting.  _Pattern Recognit._ , 109:107616, 2021.
  * Li et al. (2022) Fuxian Li, Jie Feng, Huan Yan, Depeng Jin, and Yong Li.  Crowd flow prediction for irregular regions with semantic graph attention network.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 13:1 – 14, 2022.
  * Li et al. (2015) Teng Li, Huan Chang, Meng Wang, Bingbing Ni, Richang Hong, and Shuicheng Yan.  Crowded scene analysis: A survey.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 25:367–386, 2015.
  * Li et al. (2019) Wei Li, Wei Tao, Junyang Qiu, Xin Liu, Xingyu Zhou, and Zhisong Pan.  Densely connected convolutional networks with attention lstm for crowd flows prediction.  _IEEE Access_ , 7:140488–140498, 2019.
  * Li et al. (2018) Yuhong Li, Xiaofan Zhang, and Deming Chen.  Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1091–1100, 2018.
  * Liang et al. (2022) Dingkang Liang, Xiwu Chen, Wei Xu, Yu Zhou, and Xiang Bai.  Transcrowd: weakly-supervised crowd counting with transformers.  _Science China Information Sciences_ , 2022.
  * Lin et al. (2009) Dahua Lin, Eric Grimson, and John Fisher.  Learning visual flows: A lie algebraic approach.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 747–754, 2009.  doi: 10.1109/CVPR.2009.5206660.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár.  Microsoft coco: Common objects in context, 2014.  URL http://arxiv.org/abs/1405.0312.  cite arxiv:1405.0312Comment: 1) updated annotation pipeline description and figures; 2) added new section describing datasets splits; 3) updated author list.
  * Liu et al. (2021) Menghang Liu, Luning Li, Qiang Li, Yu Bai, and Cheng Hu.  Pedestrian flow prediction in open public places using graph convolutional network.  _ISPRS Int. J. Geo Inf._ , 10:455, 2021.
  * Liu et al. (2015) W. Liu, Dragomir Anguelov, D. Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg.  Ssd: Single shot multibox detector.  In _European Conference on Computer Vision_ , 2015.
  * Lu et al. (2013) Cewu Lu, Jianping Shi, and Jiaya Jia.  Abnormal event detection at 150 fps in matlab.  In _2013 IEEE International Conference on Computer Vision_ , pp. 2720–2727, 2013.  doi: 10.1109/ICCV.2013.338.
  * Luca et al. (2020) Massimiliano Luca, Gianni Barlacchi, Bruno Lepri, and Luca Pappalardo.  A survey on deep learning for human mobility.  _ACM Computing Surveys (CSUR)_ , 55:1 – 44, 2020.
  * Luo et al. (2017a) Weixin Luo, Wen Liu, and Shenghua Gao.  A revisit of sparse coding based anomaly detection in stacked rnn framework.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , pp. 341–349, 2017a.
  * Luo et al. (2017b) Weixin Luo, Wen Liu, and Shenghua Gao.  A revisit of sparse coding based anomaly detection in stacked rnn framework.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pp. 341–349, 2017b.  doi: 10.1109/ICCV.2017.45.
  * Luo et al. (2020) Ying Luo, Jinhu Lu, and Baochang Zhang.  Crowd counting for static images: A survey of methodology.  _2020 39th Chinese Control Conference (CCC)_ , pp. 6602–6607, 2020\.
  * Mahadevan et al. (2010a) Vijay Mahadevan, Wei-Xin LI, Viral Bhalodia, and Nuno Vasconcelos.  Anomaly detection in crowded scenes.  In _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1975–1981, 2010a.
  * Mahadevan et al. (2010b) Vijay Mahadevan, Wei-Xin LI, Viral Bhalodia, and Nuno Vasconcelos.  Anomaly detection in crowded scenes.  In _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1975–1981, 2010b.
  * Modi & Parikh (2022) Harshad S. Modi and Dhaval A. Parikh.  A survey on crowd anomaly detection.  _International Journal of Computing and Digital Systems_ , 2022.
  * Pang et al. (2020) Guansong Pang, Cheng Yan, Chunhua Shen, Anton van den Hengel, and Xiao Bai.  Self-trained deep ordinal regression for end-to-end video anomaly detection.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 12170–12179, 2020.
  * Pawar & Attar (2021) Karishma Pawar and Vahida Z. Attar.  Application of deep learning for crowd anomaly detection from surveillance videos.  _2021 11th International Conference on Cloud Computing, Data Science & Engineering (Confluence)_, pp. 506–511, 2021.
  * Peng et al. (2020) Tao Peng, Qing Li, and Pengfei Zhu.  Rgb-t crowd counting from drone: A benchmark and mmccn network.  In _Computer Vision – ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part VI_ , pp. 497–513, Berlin, Heidelberg, 2020. Springer-Verlag.
  * Rabaud & Belongie (2006) V. Rabaud and S. Belongie.  Counting crowded moving objects.  In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , volume 1, pp. 705–711, 2006.  doi: 10.1109/CVPR.2006.92.
  * Redmon & Farhadi (2018) Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  _ArXiv_ , abs/1804.02767, 2018.
  * Saqib (2019) Muhammad Saqib.  Automatic analysis of crowd dynamics using computer vision and machine learning approaches.  2019\.
  * Shao et al. (2017) Jing Shao, Chen Change Loy, and Xiaogang Wang.  Learning scene-independent group descriptors for crowd understanding.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 27(6):1290–1303, 2017.  doi: 10.1109/TCSVT.2016.2539878.
  * Sharif et al. (2022) Md. Haidar Sharif, Lei Jiao, and Christian Walter Peter Omlin.  Deep crowd anomaly detection: State-of-the-art, challenges, and future research directions.  _ArXiv_ , abs/2210.13927, 2022.
  * Silveira Jacques Junior et al. (2010a) Julio Cezar Silveira Jacques Junior, Soraia Raupp Musse, and Claudio Rosito Jung.  Crowd analysis using computer vision techniques.  _IEEE Signal Processing Magazine_ , 27(5):66–77, 2010a.  doi: 10.1109/MSP.2010.937394.
  * Silveira Jacques Junior et al. (2010b) Julio Cezar Silveira Jacques Junior, Soraia Raupp Musse, and Claudio Rosito Jung.  Crowd analysis using computer vision techniques.  _IEEE Signal Processing Magazine_ , 27(5):66–77, 2010b.  doi: 10.1109/MSP.2010.937394.
  * Simonyan & Zisserman (2014) Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In _NIPS_ , 2014.
  * Sindagi & Patel (2017) Vishwanath A. Sindagi and Vishal M. Patel.  Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting.  _2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_ , pp. 1–6, 2017.
  * Sindagi & Patel (2018) Vishwanath A. Sindagi and Vishal M. Patel.  A survey of recent advances in cnn-based single image crowd counting and density estimation.  _Pattern Recognit. Lett._ , 107:3–16, 2018.
  * Sindagi et al. (2020) Vishwanath A Sindagi, Rajeev Yasarla, and Vishal M Patel.  Jhu-crowd++: Large-scale crowd counting dataset and a benchmark method.  _Technical Report_ , 2020.
  * Song et al. (2021) Qingyu Song, Changan Wang, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Jian Wu, and Jiayi Ma.  To choose or to fuse? scale selection for crowd counting.  In _AAAI Conference on Artificial Intelligence_ , 2021.
  * Song et al. (2020) Xiao Song, Kai Chen, Xu Li, Jinghan Sun, Baocun Hou, Yong Cui, Baochang Zhang, Gang Xiong, and Zilie Wang.  Pedestrian trajectory prediction based on deep convolutional lstm network.  _IEEE Transactions on Intelligent Transportation Systems_ , 22:3285–3302, 2020.
  * Thirumalaisamy et al. (2022) Manikandan Thirumalaisamy, Shajahan Basheer, Shitharth Selvarajan, Sara A. Althubiti, Fayadh Alenezi, Gautam Srivastava, and Jerry Chun-Wei Lin.  Interaction of secure cloud network and crowd computing for smart city data obfuscation.  _Sensors_ , 22(19), 2022.  ISSN 1424-8220.  doi: 10.3390/s22197169.  URL https://www.mdpi.com/1424-8220/22/19/7169.
  * Tian et al. (2021) Ye Tian, Xiangxiang Chu, and Hongpeng Wang.  Cctrans: Simplifying and improving crowd counting with transformer.  _ArXiv_ , abs/2109.14483, 2021.
  * Tian et al. (2019) Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 9626–9635, 2019.
  * Tran et al. (2015) Khai N. Tran, Xu Yan, Ioannis A. Kakadiaris, and Shishir K. Shah.  A group contextual model for activity recognition in crowded scenes.  In _VISAPP 2015 - Proceedings of the 10th International Conference on Computer Vision Theory and Applications, Volume 2, Berlin, Germany, 11-14 March, 2015_ , pp. 5–12. SciTePress, 2015.
  * Wang et al. (2020) Boyu Wang, Huidong Liu, Dimitris Samaras, and Minh Hoai.  Distribution matching for crowd counting.  _ArXiv_ , abs/2009.13077, 2020.
  * Wang et al. (2022) Hao Wang, Junchao Liao, Tianheng Cheng, Zewen Gao, Hao Liu, Bo Ren, Xiang Bai, and Wenyu Liu.  Knowledge mining with scene text for fine-grained recognition.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 4614–4623. IEEE, 2022.  doi: 10.1109/CVPR52688.2022.00458.  URL https://doi.org/10.1109/CVPR52688.2022.00458.
  * Wang et al. (2021) Qi Wang, Junyu Gao, Wei Lin, and Xuelong Li.  Nwpu-crowd: A large-scale benchmark for crowd counting and localization.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43:2141–2149, 2021.
  * Wang & Breckon (2022) Qian Wang and T. Breckon.  Crowd counting via segmentation guided attention networks and curriculum loss.  _IEEE Transactions on Intelligent Transportation Systems_ , 2022.
  * Wu et al. (2020) Chongke Wu, Sicong Shao, Cihan Tunc, and Salim Hariri.  Video anomaly detection using pre-trained deep convolutional neural nets and context mining.  _2020 IEEE/ACS 17th International Conference on Computer Systems and Applications (AICCSA)_ , pp. 1–8, 2020.
  * Wu et al. (2010) Shandong Wu, Brian E. Moore, and Mubarak Shah.  Chaotic invariants of lagrangian particle trajectories for anomaly detection in crowded scenes.  In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pp. 2054–2060, 2010.  doi: 10.1109/CVPR.2010.5539882.
  * Yang et al. (2019) Meng Yang, Sutharshan Rajasegarar, Sarah Monazam Erfani, and Christopher Leckie.  Deep learning and one-class svm based anomalous crowd detection.  _2019 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–8, 2019.
  * Yang et al. (2020) Yifan Yang, Guorong Li, Zhe Wu, Li Su, Qingming Huang, and Nicu Sebe.  Weakly-supervised crowd counting learns from sorting rather than locations.  In _Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII_ , pp. 1–17, Berlin, Heidelberg, 2020. Springer-Verlag.
  * Zeng et al. (2017) Lingke Zeng, Xiangmin Xu, Bolun Cai, Suo Qiu, and Tong Zhang.  Multi-scale convolutional neural networks for crowd counting.  _2017 IEEE International Conference on Image Processing (ICIP)_ , pp. 465–469, 2017.
  * Zhang et al. (2015) Cong Zhang, Hongsheng Li, Xiaogang Wang, and Xiaokang Yang.  Cross-scene crowd counting via deep convolutional neural networks.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 833–841, 2015.
  * Zhang et al. (2022) Junwei Zhang, Zhuzhu Wang, Dandan Wang, Xinglong Zhang, Brij B. Gupta, Ximeng Liu, and Jianfeng Ma.  A secure decentralized spatial crowdsourcing scheme for 6g-enabled network in box.  _IEEE Transactions on Industrial Informatics_ , 18(9):6160–6170, 2022.  doi: 10.1109/TII.2021.3081416.
  * Zhang et al. (2018) Xuguang Zhang, Qinan Yu, and Hui Yu.  Physics inspired methods for crowd video surveillance and analysis: A survey.  _IEEE Access_ , 6:66816–66830, 2018.
  * Zhang et al. (2016) Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma.  Single-image crowd counting via multi-column convolutional neural network.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 589–597, 2016.  doi: 10.1109/CVPR.2016.70.


# Logarithmic Sobolev Inequalities of Fractional Order on Noncommutative Tori

Gihyun Lee  Gihyun Lee
Department of Mathematics: Analysis, Logic and Discrete Mathematics, Ghent
University, Krijgslaan 281, Building S8, B 9000 Ghent, Belgium
<EMAIL_ADDRESS>
###### Abstract.

In this paper, we prove a version of the logarithmic Sobolev inequality of
fractional order on noncommutative $n$-tori for any dimension $n\geq 2$.

###### Key words and phrases:

Logarithmic Sobolev inequality, noncommutative tori

###### 2020 Mathematics Subject Classification:

35A23, 46E35, 46L52, 58B34

##  1\. Introduction

Among all noncommutative spaces studied in Alain Connes’ noncommutative
geometry program [11] noncommutative tori are the most extensively studied
ones. One of the reasons behind the extensive research conducted on
noncommutative tori is the presence of counterparts to various mathematical
tools employed in the study of analysis and geometry. For example,
counterparts to the notions such as vector bundles, Fourier series, spaces of
smooth functions, Sobolev spaces and pseudodifferential calculus are
established and available in the setting of noncommutative tori (see, e.g.,
[8, 10, 17, 18, 24, 34] and the references therein). Moreover, noncommutative
tori are utilized in the mathematical modeling of physical phenomena, such as
the quantum Hall effect [2], topological insulators [4, 25] and string theory
[12, 29].

Let us now briefly review the classical Sobolev inequalities on Euclidean
spaces. The classical Sobolev inequality states that if a function $f$ defined
on $\mathbb{R}^{n}$, along with its first derivatives, belongs to
$L_{p}(\mathbb{R}^{n})$ and if $q=(\frac{1}{p}-\frac{1}{n})^{-1}$ is finite,
then $f$ is in $L_{q}(\mathbb{R}^{n})$. This inequality has broad applications
in various fields of analysis, including the study of partial differential
equations. However, the classical Sobolev inequality has the following
limitations. As evident from the value of $q$ we see that, as the dimension
$n$ increases, the difference between $q$ and $p$ narrows. Consequently, as
$n$ grows, the improvement in summability obtained from the differentiability
decreases. For this reason, in the infinite dimensional setting such as
quantum field theory we cannot expect to obtain an inequality precisely
corresponds to the classical Sobolev inequality. Motivated by this observation
Gross [15] introduced and proved the following logarithmic Sobolev inequality
on Euclidean spaces.

$\int_{\mathbb{R}^{n}}|f(x)|^{2}\log{|f(x)|}\,d\nu(x)\leq\int_{\mathbb{R}^{n}}|\nabla
f(x)|^{2}\,d\nu(x)+\|f\|_{2}^{2}\log{\|f\|_{2}}.$

Here $\nu$ denotes the Gaussian measure on $\mathbb{R}^{n}$ and
$\|\cdot\|_{2}$ denotes the $L_{2}$-norm associated with $\nu$. As mentioned
in [15] this inequality can be utilized in the infinite dimensional setting,
because the coefficients in the inequality do not depend on the dimension $n$.

Since the introduction of Gross’ logarithmic Sobolev inequality, various
methods have been employed to establish logarithmic Sobolev inequalities in
different settings. Given the vast number of papers on this topic, it is not
feasible to encompass all the references here. However, for a partial
overview, let us list a few results which can be found in the literature.
Rosen presented and proved a logarithmic Sobolev inequality on weighted
$\mathbb{R}^{n}$ [27]. Weissler proved a logarithmic Sobolev inequality on the
circle [33]. Gross [16] and Chatzakou-Kassymov-Ruzhansky [6] investigated
logarithmic Sobolev inequalities on Lie groups. Stroock-Zegarlinski [31] and
Bodineau-Helffer [3] obtained logarithmic Sobolev inequalities for spin
systems. Brannan-Gao-Junge derived a version of logarithmic Sobolev inequality
by utilizing lower bound of the Ricci curvature of a compact Riemannian
manifold [5].

In the case of noncommutative tori, as mentioned above, there exist
counterparts to the tools used in Fourier theory for ordinary tori (see [8]).
Therefore, the arguments employed on ordinary tori can often be adapted to the
setting of noncommutative tori. By using this harmonic analysis technique of
noncommutative tori and the theory of operator algebras Xiong-Xu-Yin provided
a detailed account on the construction of Sobolev, Besov and Triebel-Lizorkin
spaces on noncommutative tori ([34]; see also [18, 28, 30] for Sobolev spaces
on noncommutative tori). The embedding theorems between these spaces are also
proved in [34]. In addition, in [20, 21] McDonald-Ponge proved versions of
Sobolev inequalities on noncommutative tori as consequences of the Lieb-
Thirring inequalities on (curved) noncommutative tori.

However, to the best of the author’s knowledge, logarithmic Sobolev
inequalities in the setting of noncommutative tori haven’t been studied much
in the literature. The logarithmic Sobolev inequality on noncommutative
$2$-tori by Khalkhali-Sadeghi [19] is the only existing result on this topic.
They attempted to establish logarithmic Sobolev inequality on noncommutative
$2$-tori by adopting Weissler’s proof of the logarithmic Sobolev inequality on
the circle [33]. However, due to technical issues arising from the
noncommutativity, they were only able to obtain the following form of
logarithmic Sobolev inequality for strictly positive elements of the form
$x=\sum_{k\in\mathbb{Z}}x_{k}U_{1}^{k}U_{2}^{kl}$, where $0\neq
l\in\mathbb{Z}$.

$\tau\big{(}x^{2}\log{x}\big{)}\leq\sum_{k\in\mathbb{Z}}(1+|l|)\,|k|\,|x_{k}|^{2}+\|x\|_{L_{2}}^{2}\log{\|x\|_{L_{2}}}.$

We refer to Section 2 for notations and background material on noncommutative
tori.

In this paper, we prove the following logarithmic Sobolev inequalities of
fractional order on noncommutative tori for any dimension $n\geq 2$.

######  Theorem 1.1.

Let $0<x\in C^{\infty}(\mathbb{T}_{\theta}^{n})$, $a>0$ and $0<s<\frac{n}{2}$.
Then there is a constant $C(n,s,a)>0$ depends only on $n,s$ and $a$ such that

$\tau\left[x^{2}\log\left(\frac{x^{2}}{\|x\|_{L_{2}}^{2}}\right)\right]\leq
C(n,s,a)\,\|x\|_{W_{2}^{s}}^{2}-\frac{n}{s}(\log{a}+1)\,\|x\|_{L_{2}}^{2}.$
(1.1)

Our proof of this theorem is based on the short proof of fractional order
logarithmic Sobolev inequality on $\mathbb{R}^{n}$ presented in a recent
article by Chatzakou-Ruzhansky [7]. In the setting of noncommutative tori,
there are tools correspond to the key tools used in the proof by Chatzakou-
Ruzhansky [7], which enables us to apply their approach immediately to
noncommutative tori. Instead of Jensen’s inequality for concave functions and
probability measures used in the proof by Chatzakou-Ruzhansky, we utilize the
Jensen’s operator inequality known in the operator algebraic setting ([9, 14];
see also [23]) and the embedding theorem between Sobolev spaces proved in [34]
to prove Theorem 1.1.

Although the main result of this paper, Theorem 1.1, is the first result on
logarithmic Sobolev inequality of _fractional order_ on noncommutative tori,
it still has certain limitations and room for improvement. First, the Sobolev
norm used on the right-hand side of (1.1) needs to be replaced by the
homogeneous Sobolev norm,

$\|x\|_{\dot{W}_{2}^{s}(\mathbb{T}_{\theta}^{n})}^{2}:=\sum_{k\in\mathbb{Z}^{n}}|k|^{2s}\,|x_{k}|^{2},\qquad
s>0.$

However, an inequality between homogeneous Sobolev norms on noncommutative
tori which can be applied to the arguments in this paper is missing in the
literature. Although McDonald-Ponge’s Sobolev inequalities [20, 21] deal with
homogeneous Sobolev norms, these results cannot be directly applied to the
arguments of this paper. In [20, 21] Sobolev inequalities are proven only for
zero mean value elements, i.e., elements $x$ with $\tau(x)=0$. But in order
for the logarithm defined by holomorphic functional calculus appearing in
(1.1) to make sense, our focus should be restricted to strictly positive
elements, and strictly positive elements cannot have a zero mean. Another
issue is the lack of a result on the sharpness of Sobolev inequalities on
noncommutative tori. In [7] Chatzakou-Ruzhansky utilized the sharp constant of
the Sobolev inequality on $\mathbb{R}^{n}$ obtained in [13] to get an explicit
expression of the constant appearing in their logarithmic Sobolev inequality
and study its behavior. Similarly, if we could determine the sharpness of
Sobolev inequalities on noncommutative tori, we could expect to improve the
constant on the right-hand side of (1.1) or get an explicit expression of it.
In particular, the sharpness of Sobolev inequalities would enable us to verify
the conjecture on the logarithmic Sobolev inequality on noncommutative
$2$-tori stated in [19] by setting $a=\frac{1}{e}$ in (1.1).

This paper is organized as follows. In Section 2, We gather some background
material on noncommutative tori, Sobolev spaces and Jensen’s operator
inequality used in this paper. In Section 3, we prove the logarithmic Sobolev
inequalities of fractional order on noncommutative tori, the main result of
this paper (Theorem 1.1).

##  2\. Preliminaries

In this section, we gather background material on noncommutative tori, Sobolev
spaces and Jensen’s operator inequality used in the proof of the main theorem
in Section 3.

###  2.1. Noncommutative tori

We refer to [11, 17, 26] for a more detailed account on noncommutative tori.

Let $\theta$ be a skew-symmetric $n\times n$ matrix over $\mathbb{R}$. The
noncommutative torus associated with $\theta$, denoted by
$\mathbb{T}_{\theta}^{n}$, is a noncommutative space in the sense of Alain
Connes’ noncommutative geometry [11]. The $C^{*}$-algebra
$C(\mathbb{T}_{\theta}^{n})$ and the von Neumann algebra
$L_{\infty}(\mathbb{T}_{\theta}^{n})$ of $\mathbb{T}_{\theta}^{n}$ are
generated by the unitaries $U_{1},\ldots,U_{n}$ subject to the relations,

$U_{k}U_{j}=e^{2\pi i\theta_{jk}}U_{j}U_{k},\qquad 1\leq j,k\leq n.$

These unitaries $U_{1},\ldots,U_{n}$ can be concretely realized as unitary
operators on $L_{2}(\mathbb{T}^{n})$ (see, e.g., [17]), and hence both
$C(\mathbb{T}_{\theta}^{n})$ and $L_{\infty}(\mathbb{T}_{\theta}^{n})$ are
$*$-subalgebras of $\mathscr{L}(L_{2}(\mathbb{T}^{n}))$, the $C^{*}$-algebra
of bounded linear operators on $L_{2}(\mathbb{T}^{n})$. If $\theta=0$, then we
recover the spaces of continuous linear functions $C(\mathbb{T}^{n})$ and
essentially bounded functions $L_{\infty}(\mathbb{T}^{n})$ on the ordinary
$n$-torus $\mathbb{T}^{n}=\mathbb{R}^{n}/(2\pi\mathbb{Z})^{n}$.

In what follows, for any $k=(k_{1},\ldots,k_{n})\in\mathbb{Z}^{n}$, we shall
denote $U_{1}^{k_{1}}\cdots U_{n}^{k_{n}}$ by $U^{k}$. Given any
$T\in\mathscr{L}(L_{2}(\mathbb{T}^{n}))$, we define

$\tau(T)={\left\langle
T1|1\right\rangle}_{L_{2}(\mathbb{T}^{n})}=(2\pi)^{-n}\int_{\mathbb{T}^{n}}(T1)(x)\,dx.$

This defines a tracial state on both $C(\mathbb{T}_{\theta}^{n})$ and
$L_{\infty}(\mathbb{T}_{\theta}^{n})$. By a direct computation it can be shown
that $\tau(U^{k})=0$ for $0\neq k\in\mathbb{Z}^{n}$ and $\tau(1)=1$. For
$x,y\in C(\mathbb{T}_{\theta}^{n})$ we define

${\left\langle
x|y\right\rangle}_{L_{2}(\mathbb{T}_{\theta}^{n})}=\tau(xy^{*}).$

Let us denote by $L_{2}(\mathbb{T}_{\theta}^{n})$ the Hilbert space completion
of $C(\mathbb{T}_{\theta}^{n})$ with respect to this inner product. Then the
family $\\{U^{k};\,k\in\mathbb{Z}^{n}\\}$ forms an orthonormal basis for
$L_{2}(\mathbb{T}_{\theta}^{n})$. This orthonormal basis plays the role of the
standard orthonormal basis for $L_{2}(\mathbb{T}^{n})$ utilized in Fourier
analysis, i.e., every $x\in L_{2}(\mathbb{T}_{\theta}^{n})$ can be uniquely
written as follows.

$x=\sum_{k\in\mathbb{Z}^{n}}x_{k}U^{k},\qquad x_{k}:={\left\langle
x|U^{k}\right\rangle}_{L_{2}(\mathbb{T}_{\theta}^{n})}.$ (2.1)

Furthermore, the GNS construction (see, e.g., [1]) associated with $\tau$
gives rise to the $*$-representations of $C(\mathbb{T}_{\theta}^{n})$ and
$L_{\infty}(\mathbb{T}_{\theta}^{n})$ on the Hilbert space
$L_{2}(\mathbb{T}_{\theta}^{n})$.

The $C^{*}$-algebra $C(\mathbb{T}_{\theta}^{n})$ admits a strongly continuous
action of $\mathbb{R}^{n}$, denoted by $\alpha_{s}(x)$ for
$s\in\mathbb{R}^{n}$ and $x\in C(\mathbb{T}_{\theta}^{n})$. Hence the triple
$(C(\mathbb{T}_{\theta}^{n}),\mathbb{R}^{n},\alpha)$ forms a $C^{*}$-dynamical
system. For the elements $U^{k}$, $k\in\mathbb{Z}^{n}$, this action is given
by

$\alpha_{s}(U^{k})=e^{is\cdot k}U^{k},\qquad s\in\mathbb{R}^{n}.$

The $C^{*}$-dynamical system structure on $C(\mathbb{T}_{\theta}^{n})$ enables
us to define the dense subalgebra $C^{\infty}(\mathbb{T}_{\theta}^{n})$ of
smooth elements of the action $\alpha$, i.e., we define

$C^{\infty}(\mathbb{T}_{\theta}^{n}):=\left\\{x\in
C(\mathbb{T}_{\theta}^{n});\,\text{$\mathbb{R}^{n}\ni s\mapsto\alpha_{s}(x)\in
C(\mathbb{T}_{\theta}^{n})$ is a $C^{\infty}$-map}\right\\}.$

We also have the following characterization of the smooth elements in
$C(\mathbb{T}_{\theta}^{n})$ in terms of the Fourier series expansion given in
(2.1).

$C^{\infty}(\mathbb{T}_{\theta}^{n})=\Big{\\{}x=\sum_{k\in\mathbb{Z}^{n}}x_{k}U^{k};\,(x_{k})_{k\in\mathbb{Z}^{n}}\in\mathscr{S}(\mathbb{Z}^{n})\Big{\\}}.$

Here $\mathscr{S}(\mathbb{Z}^{n})$ denotes the space of rapidly decaying
sequences indexed by $\mathbb{Z}^{n}$ with entries in $\mathbb{C}$.
Furthermore, the action $\mathbb{R}^{n}\ni s\mapsto\alpha_{s}(x)\in
C^{\infty}(\mathbb{T}_{\theta}^{n})$ also enables us to define the derivations
$\partial_{1},\ldots,\partial_{n}$ on $C^{\infty}(\mathbb{T}_{\theta}^{n})$.
For $j=1,\ldots,n$, we define
$\partial_{j}:C^{\infty}(\mathbb{T}_{\theta}^{n})\rightarrow
C^{\infty}(\mathbb{T}_{\theta}^{n})$ by letting

$\partial_{j}(x)=\partial_{s_{j}}\alpha_{s}(x)\big{|}_{s=0},\qquad x\in
C^{\infty}(\mathbb{T}_{\theta}^{n}).$

In particular, for the elements $U^{k}$,
$k=(k_{1},\ldots,k_{n})\in\mathbb{Z}^{n}$, we obtain
$\partial_{j}(U^{k})=ik_{j}(U^{k})$, $1\leq j\leq n$.

###  2.2. Sobolev and $L_{p}$-spaces

A detailed account on Sobolev spaces on noncommutative tori can be found in
[18, 28, 30, 34].

Let us denote the $L_{2}$-version of Sobolev space of order $s\geq 0$ on
$\mathbb{T}_{\theta}^{n}$ by $W_{2}^{s}(\mathbb{T}_{\theta}^{n})$. This space
consists of elements $x=\sum_{k\in\mathbb{Z}^{n}}x_{k}U^{k}$ in
$L_{2}(\mathbb{T}_{\theta}^{n})$ such that

$\big{(}(1+|k|^{2})^{\frac{s}{2}}x_{k}\big{)}_{k\in\mathbb{Z}^{n}}\in\ell_{2}(\mathbb{Z}^{n}).$

The space $W_{2}^{s}(\mathbb{T}_{\theta}^{n})$ is a Hilbert space with the
inner product,

${\left\langle
x|y\right\rangle}_{W_{2}^{s}(\mathbb{T}_{\theta}^{n})}:=\sum_{k\in\mathbb{Z}^{n}}(1+|k|^{2})^{s}x_{k}\overline{y_{k}},\qquad
x=\sum_{k\in\mathbb{Z}^{n}}x_{k}U^{k},y=\sum_{k\in\mathbb{Z}^{n}}y_{k}U^{k}\in
W_{2}^{s}(\mathbb{T}_{\theta}^{n}).$

We shall denote the norm associated with this inner product by
$\|\cdot\|_{W_{2}^{s}(\mathbb{T}_{\theta}^{n})}$.

We apply the theory of noncommutative $L_{p}$-spaces (see, e.g., [22, 32]) to
the von Neumann algebra $L_{\infty}(\mathbb{T}_{\theta}^{n})$ to construct the
$L_{p}$-spaces of $\mathbb{T}_{\theta}^{n}$. Recall that the elements of
$L_{\infty}(\mathbb{T}_{\theta}^{n})$ are represented as bounded linear
operators on the Hilbert space $L_{2}(\mathbb{T}^{n})$. We say that a closed
and densely defined operator on $L_{2}(\mathbb{T}^{n})$ is affiliated with
$L_{\infty}(\mathbb{T}_{\theta}^{n})$ if it commutes with the commutant of
$L_{\infty}(\mathbb{T}_{\theta}^{n})$ in $\mathscr{L}(L_{2}(\mathbb{T}^{n}))$.
Given a positive operator $x$ on $L_{2}(\mathbb{T}^{n})$ affiliated with
$L_{\infty}(\mathbb{T}_{\theta}^{n})$ let
$x=\int_{0}^{\infty}\lambda\,dE(\lambda)$ be its spectral representation and
set

$\tau(x):=\int_{0}^{\infty}\lambda\,d\tau(E(\lambda)).$

For $1\leq p<\infty$, the $L_{p}$-space of $\mathbb{T}_{\theta}^{n}$, denoted
by $L_{p}(\mathbb{T}_{\theta}^{n})$, consists of all elements $x$ in the
$*$-algebra of $L_{\infty}(\mathbb{T}_{\theta}^{n})$-affiliated operators on
$L_{2}(\mathbb{T}^{n})$ such that

$\|x\|_{L_{p}(\mathbb{T}_{\theta}^{n})}:=\tau\big{(}|x|^{p}\big{)}^{\frac{1}{p}}<\infty.$

The space $L_{p}(\mathbb{T}_{\theta}^{n})$ is a Banach space with the norm
$\|\cdot\|_{L_{p}(\mathbb{T}_{\theta}^{n})}$.

Furthermore, we have the following inclusion of Sobolev spaces into
$L_{p}$-spaces. This is a particular case of the embedding theorem for Sobolev
spaces on noncommutative tori stated in [34].

######  Proposition 2.1 (see [34, Theorem 6.6]).

Let $p>2$ and set $s=n(\frac{1}{2}-\frac{1}{p})$. Then there is a continuous
embedding,

$W_{2}^{s}(\mathbb{T}_{\theta}^{n})\subset L_{p}(\mathbb{T}_{\theta}^{n}).$

###  2.3. Jensen’s operator inequality

######  Definition 2.2.

Let $f(t)$ be a function on an interval $I\subset\mathbb{R}$. We say that
$f(t)$ is _operator convex_ if, for all $\lambda\in[0,1]$ and for every
selfadjoint operator $A$ and $B$ on a Hilbert space $\mathscr{H}$ such that
$\operatorname{Sp}{A}\subset I$ and $\operatorname{Sp}{B}\subset I$, we have

$f((1-\lambda)A+\lambda B)\leq(1-\lambda)f(A)+\lambda f(B).$

In contrast, $f(t)$ is called _operator concave_ if $-f(t)$ is operator
convex.

######  Proposition 2.3 ([9, 14]; see also [23, Theorem 1.20]).

Let $\mathscr{H}$ and $\mathscr{H}^{\prime}$ be Hilbert spaces. Suppose that
$f(t)$ is an operator convex function on the interval $I$,
$x\in\mathscr{L}(\mathscr{H})$ and $\operatorname{Sp}{x}\subset I$. Then we
have

$\pi(f(x))\geq f(\pi(x)),$ (2.2)

for any positive normalized linear map
$\pi:\mathscr{L}(\mathscr{H})\rightarrow\mathscr{L}(\mathscr{H}^{\prime})$.

######  Remark 2.4.

The inequality (2.2) is of course reversed if we employ an operator concave
function instead of an operator convex function.

##  3\. The Proof of Theorem 1.1

In this section, we provide the proof of Theorem 1.1, the main result of this
paper.

###### Proof of Theorem 1.1.

Let $0<x\in C^{\infty}(\mathbb{T}_{\theta}^{n})$. Then, for any
$\varepsilon>0$, we have

$\tau\left[x^{2}\log\left(\frac{x^{2}}{\|x\|_{L_{2}}^{2}}\right)\right]=\frac{1}{\varepsilon}\,\tau\left[x^{2}\log\left(\frac{x^{2}}{\|x\|_{L_{2}}^{2}}\right)^{\varepsilon}\right]=\frac{\|x\|_{L_{2}}^{2}}{\varepsilon}\,\tau\left[\frac{x^{2}}{\|x\|_{L_{2}}^{2}}\log\left(\frac{x^{2\varepsilon}}{\|x\|_{L_{2}}^{2\varepsilon}}\right)\right].$
(3.1)

Note that the map $\mathscr{L}(L_{2}(\mathbb{T}^{n}))\ni
u\mapsto\tau\left[\frac{|x|^{2}}{\|x\|_{L^{2}}^{2}}u\right]\in\mathbb{C}$ is a
positive normalized linear map. Moreover, we know by [23, Example 1.7] that
the logarithmic function $\log{t}$ is operator concave on $(0,\infty)$.
Therefore, as we have
$\operatorname{Sp}\big{(}x^{2\varepsilon}/\|x\|_{L_{2}}^{2\varepsilon}\big{)}\subset(0,\infty)$,
it follows from Jensen’s operator inequality (Proposition 2.3 and Remark 2.4)
that we have

$\displaystyle\tau\left[\frac{x^{2}}{\|x\|_{L_{2}}^{2}}\log\left(\frac{x^{2\varepsilon}}{\|x\|_{L_{2}}^{2\varepsilon}}\right)\right]$
$\displaystyle\leq\log\tau\left(\frac{x^{2\varepsilon+2}}{\|x\|_{L_{2}}^{2\varepsilon+2}}\right)$
$\displaystyle=(\varepsilon+1)\log\left\|\frac{x}{\|x\|_{L_{2}}}\right\|_{L_{2\varepsilon+2}}^{2}$
$\displaystyle=(\varepsilon+1)\log\left(\frac{\|x\|_{L_{2\varepsilon+2}}^{2}}{\|x\|_{L_{2}}^{2}}\right).$
(3.2)

For all $0<b,t\in\mathbb{R}$, we have

$\log{t}\leq bt-\log{b}-1.$

Combining this with the estimates (3.1) and (3.2) we obtain

$\displaystyle\tau\left[x^{2}\log\left(\frac{x^{2}}{\|x\|_{L_{2}}^{2}}\right)\right]$
$\displaystyle\leq\frac{\|x\|_{L_{2}}^{2}(\varepsilon+1)}{\varepsilon}\,\log\left(\frac{\|x\|_{L_{2\varepsilon+2}}^{2}}{\|x\|_{L_{2}}^{2}}\right)$
$\displaystyle\leq\frac{\|x\|_{L_{2}}^{2}(\varepsilon+1)}{\varepsilon}\left(b\frac{\|x\|_{L_{2\varepsilon+2}}^{2}}{\|x\|_{L_{2}}^{2}}-\log{b}-1\right)$
$\displaystyle=\frac{\varepsilon+1}{\varepsilon}\left(b\,\|x\|_{L_{2\varepsilon+2}}^{2}-[\log{b}+1]\,\|x\|_{L_{2}}^{2}\right).$
(3.3)

Let $\varepsilon>0$ be such that $2\varepsilon+2=\frac{2n}{n-2s}$ for
$0<s<\frac{n}{2}$ and set $b=ea^{2}$ for $a>0$. We know by Proposition 2.1
that there is $C(n,s)>0$ depending only on $n$ and $s$ such that

$\|x\|_{L_{2\varepsilon+2}}^{2}\leq C(n,s)\,\|x\|_{W_{2}^{s}}^{2}.$

It then follows from this and (3.3) that

$\displaystyle\tau\left[x^{2}\log\left(\frac{x^{2}}{\|x\|_{L_{2}}^{2}}\right)\right]$
$\displaystyle\leq\frac{\frac{n}{n-2s}}{\frac{2s}{n-2s}}\left(ea^{2}\,\|x\|_{L_{2\varepsilon+2}}^{2}-[\log(ea^{2})+1]\,\|x\|_{L_{2}}^{2}\right)$
$\displaystyle\leq\frac{n}{2s}\left(ea^{2}C(n,s)\,\|x\|_{W_{2}^{s}}^{2}-2(\log{a}+1)\,\|x\|_{L_{2}}^{2}\right)$
$\displaystyle=\frac{nea^{2}}{2s}C(n,s)\,\|x\|_{W_{2}^{s}}^{2}-\frac{n}{s}(\log{a}+1)\,\|x\|_{L_{2}}^{2}.$

This completes the proof. ∎

## Acknowledgements

The author wishes to thank Michael Ruzhansky for helpful discussions related
to the topic of this paper. The research for this article is financially
supported by the FWO Odysseus 1 grant G.0H94.18N: Analysis and Partial
Differential Equations and by the Methusalem programme of the Ghent University
Special Research Fund (BOF) (Grant number 01M01021).

## References

  * [1] Arveson, W.: _An invitation to $C$*-algebras_. Springer-Verlag, 1981.
  * [2] Bellissard, J.; van Elst, A.; Schulz-Baldes, H.: _The noncommutative geometry of the quantum Hall effect_. J. Math. Phys. 35 (1994), 5373–5451.
  * [3] Bodineau, T.; Helffer, B.: _The log-Sobolev inequalities for unbounded spin systems_. J. Funct. Anal. 166 (1999), no. 1, 168–178.
  * [4] Bourne, C.; Carey, A.L.; Rennie, A.: _A non-commutative framework to topological insulators_. Rev. Math. Phys. 28 (2016), no. 2, 1650004, 51 pp..
  * [5] Brannan, M.; Gao, L.; Junge, M.: _Complete logarithmic Sobolev inequalities via Ricci curvature bounded below_. Adv. Math. 394 (2022), 108129 (60 pages).
  * [6] Chatzakou, M.; Kassymov, A.; Ruzhansky, M.: _Logarithmic Sobolev inequalities on Lie groups_. Preprint arXiv:2106.15652, 35 pages.
  * [7] Chatzakou, M.; Ruzhansky, M.: _Revised logarithmic Sobolev inequalities of fractional order_. Preprint arXiv:2302.05126, 8 pages.
  * [8] Chen, Z.; Xu, Q.; Yin, Z.: _Harmonic analysis on quantum tori_. Comm. Math. Phys. 322 (2013), no. 3, 755–805.
  * [9] Choi, M.D.: _A Schwarz inequality for positive linear maps on $C^{*}$-algebras_. Illinois J. Math. 18 (1974), 565–574.
  * [10] Connes, A.: _C*-algèbres et géométrie differentielle_. C. R. Acad. Sc. Paris, sér. A, 290 (1980), 599–604.
  * [11] Connes, A.: _Noncommutative geometry_. Academic Press, Inc., San Diego, CA, 1994.
  * [12] Connes, A.; Douglas, M.R.; Schwarz, A.: _Noncommutative geometry and matrix theory: compactification on tori_. J. High Energy Phys. (1998), no. 2, Paper 3, 35 pp..
  * [13] Cotsiolis, A.; Tavoularis, N.: _Best constants for Sobolev inequalities for higher order fractional derivatives_. J. Math. Anal. Appl. 295 (2004), no. 1, 225-236.
  * [14] Davis, C.: _A Schwartz inequality for convex operator functions_. Proc. Amer. Math. Soc. 8 (1957), 42–44.
  * [15] Gross, L.: _Logarithmic Sobolev inequalities_. Am. J. Math. 97 (1975), no. 4, 1061–1083.
  * [16] Gross, L.: _Logarithmic Sobolev inequalities on Lie groups_. Illinois J. Math. 36 (1992), no. 3, 447–490.
  * [17] Ha, H.; Lee, G.; Ponge, R.: _Pseudodifferential calculus on noncommutative tori, I. Oscillating integrals_. Int. J. Math. 30 (2019), 1950033 (74 pages).
  * [18] Ha, H.; Lee, G.; Ponge, R.: _Pseudodifferential calculus on noncommutative tori, II. Main properties_. Int. J. Math. 30 (2019), 1950034 (73 pages).
  * [19] Khalkhali, M.; Sadeghi, S.: _On logarithmic Sobolev inequality for the noncommutative two torus_. J. Pseudo-Differ. Oper. Appl. 8 (2017), 453–484.
  * [20] McDonald, E.; Ponge, R.: _Cwikel estimates and negative eigenvalues of Schrödinger operators on noncommutative tori_. J. Math. Phys. 61 (2022), 043503 (37 pages).
  * [21] McDonald, E.; Ponge, R.: _Dixmier trace formulas and negative eigenvalues of Schrödinger operators on curved noncommutative tori_. Adv. Math. 412 (2023), 108815 (57 pages).
  * [22] Nelson, E.: _Notes on non-commutative integration_. J. Funct. Anal. 15 (1974), 103–116.
  * [23] Pečarić, J.; Furuta, T.; Mićić Hot J.; Seo, Y.: _Mond-Pečarić method in operator inequalities: inequalities for bounded selfadjoint operators on a Hilbert space_. Monographs in Inequalities. Element, Zagreb, 2005.
  * [24] Polishchuk, A.; Schwarz, A.: _Categories of holomorphic vector bundles on noncommutative two-tori_. Comm. Math. Phys. 236 (2003), no. 1, 135–159.
  * [25] Prodan, E.; Schulz-Baldes, H.: _Bulk and boundary invariants for complex topological insulators: from K-theory to physics_. Mathematical Physics Studies. Springer, 2016.
  * [26] Rieffel, M.: _Noncommutative tori-A case study of non-commutative differentiable manifolds_. Contemp. Math., 105, Amer. Math. Soc.,̇ Providence, RI, 1990, pp. 191–211.
  * [27] Rosen, J.: _Sobolev inequalities for weighted spaces and supercontractivity_. Trans. Amer. Math. Soc. 222 (1976), 367–376.
  * [28] Rosenberg, J.: _Noncommutative variations on Laplace’s equation_. Anal. PDE 1 (2008), 95–114.
  * [29] Seiberg, N.; Witten, E.: _String theory and noncommutative geometry_. J. High Energy Phys. (1999), no. 9, Paper 32, 93 pp..
  * [30] Spera, M.: _Sobolev theory for noncommutative tori_. Rend. Sem. Mat. Univ. Padova 86 (1992), 143–156.
  * [31] Stroock, D.W.; Zegarlinski, B.: _The logarithmic Sobolev inequality for continuous spin systems on a lattice_. J. Funct. Anal. 104 (1992), no. 2, 299-326.
  * [32] Terp, M.: _$L^{p}$ -spaces associated with von Neumann algebras_. Copenhagen University, 1981.
  * [33] Weissler, F.: _Logarithmic Sobolev inequalities and hypercontractivity estimates on the circle_. J. Funct. Anal. 37 (1980), 218–234.
  * [34] Xiong, X.; Xu, Q.; Yin, Z.: _Sobolev, Besov and Triebel-Lizorkin spaces on quantum tori_. Mem. Amer. Math. Soc. 252 (2018) no. 1203, 86 pp..