# Multimedia Datasets for Anomaly Detection: A Survey

###### Abstract

Audio or video anomaly datasets play a crucial role in automated surveillance.
The range of applications expand from outlier object/ situation detection to
the detection of life-threatening events. This research area has been active
for more than 1.5 decades, and consequently, more and more datasets dedicated
to anomalous actions and object detection have been created. Making use of
these public anomaly datasets enable researchers to compare various anomaly
detection frameworks with the same input data. This survey aims to address the
lack of a comprehensive comparison and analysis of public datasets for audio,
video, and audio-visual based anomaly detection. It also assists in selecting
the best dataset for bench-marking frameworks. Additionally, we discuss gaps
in the existing dataset and future direction insights toward developing
multimodal anomaly detection datasets.

Index Terms—  dataset survey, video anomaly detection, audio-visual anomaly
detection, surveillance

##  1 introduction

Anomaly detection corresponds to finding unexpected behavior in data [1].
Detection of anomalies can offer important insights into a large number of
monitoring, safety-critical, commercially, or scientifically significant real-
world applications such as extreme climate event detection, disaster forecast,
mechanical fault detection, disease outbreak detection, fire/ blast detection,
fraud detection, network intrusion detection, etc. The topic has been studied
in a variety of research areas and application domains due to its broad
applicability. Additionally, due to wide availability of cheap microphones and
CCTV cameras and fast audio/ video processing power, applications relying on
these modalities have seen a tremendous growth. In recent years, we have
experienced a significant rise in automated audio/ video surveillance
applications such as biometric-based person identification, alarm-based CCTV
scene monitoring, automatic detection of traffic rule violations, audio based
machine fault detection, audio-video based anomalous behaviour recognition,
etc. Manual monitoring of scenes, deployed with hundreds of CCTV cameras or
microphones, by human personnel is not scalable as well as error-prone due to
its own human limitations. Therefore, automated scene analysis and alarm
raising have been in demand for the detection of anomalous events/ objects and
consequently avoiding any hazardous situations. Consequently, the topic has
been under investigation since long, and many frameworks for anomaly detection
have been proposed. To evaluate and compare with other frameworks, common
dataset/s are used by the researchers. Towards this end, many anomaly datasets
have been created.Datasets have been created in literature either based on a
specific type of anomaly, or for generic anomaly detection applications. In
case of audio, datasets are mainly developed for specific anomaly detection.
In the case of video, some are specifically focused on certain types of
anomaly, while others are more generic. Scene monitoring using video has
gained more attention compared to audio since spatial information, motion
information, appearance information, etc., are available in video to analyze
the situation/ object. However, relying solely on visual data implies a number
of assumptions about the CCTV cameras’ uninterrupted and robust performance.
Besides this, some anomalous situations may go undetected if we consider only
audio or video from the target scene. Therefore, using audio and video
together as complementary information has been in focus in a variety of
applications recently [2].

In this survey, we discuss the existing video, audio, and audio-visual
datasets for anomaly detection and facilitate a detailed comparison of these
datasets based on various parameters, which will help researchers to choose
suitable datasets for specific constraints and applications. To the best of
our knowledge, there exists only one short survey paper [3] on a video dataset
published in 2016. We present video datasets from diverse applications, audio
datasets, audio-visual datasets in a structured manner. Additionally, there
are more than fifteen new datasets contributed to the video anomaly detection
domain after the year 2016, which generates the need to explain and compare
those datasets.

The rest of the paper is organized as follows. The public video datasets for
anomaly detection are discussed in the next section. In Section 3, we present
a discussion on video datasets. Section 4 covers the public audio datasets. A
discussion about the audio datasets have been provided in Section 5 . We
present public audio-visual datasets in Section 6. Further comparison of
audio-visual dataset is presented in Section 7. Section 8 has a discussion on
future directions. Finally, we conclude the paper in Section 9.

##  2 Video anomaly datasets

In this section, we discuss the publicly available video datasets that are
being used or can be used for anomaly detection. There are more than 30
datasets for the purpose. We can categorize the video datasets into three
major categories based on the nature of anomalies present in them. They are
heterogeneous anomaly datasets, specific, and others. Datasets that do not
contain some specific kind of anomalies but are diverse in nature are
categorised as heterogeneous. On the other hand, some datasets that contain
specific kind of anomalous events or objects are categorized as specific
anomaly datasets. Further, specific anomaly datasets are divided in two sub-
categories; the first one consists of datasets that involve people in the
majority and events/actions are around them, the other sub-category has
datasets that are built around traffic related events. There are some other
datasets that are not primarily developed for anomaly detection but other sub-
tasks of scene monitoring such as detecting group formation, tracking people,
counting people, human interaction/action classification, etc. We keep such
public datasets in other datasets categories. We discuss them in this paper as
they can also be utilized for generic scene surveillance or can be useful in
creating some giant anomaly datasets. The above-discussed taxonomy is shown in
Fig. 1.

Fig. 1: Taxonomy followed for video anomaly datasets

In the below sub-sections, we describe the datasets by following the taxonomy
in Fig. 1. Within a category, they are discussed in increasing order of their
release year so that readers may also comprehend a picture of development in
video datasets over the year. Also, for each dataset, we describe some common
set of attributes like release year, how the dataset is collected, their main
goal, type of anomaly, duration, frame rate, resolution, merit/demerit if any,
type of ground truth, etc. Additionally, a figure illustrating some
representative video frames is provided for each dataset. This
characterization is expanded to include additional technical or specific types
of attributes important for dataset selection, and the resulting information
is assembled in multiple tables in Section 3. This enables a more direct
comparison of all the datasets discussed in this article.

For each dataset, we provide the web link from where it can be downloaded in
footnotes. Take note that some of the references in this work are URLs.
Because online addresses are not always permanent and are subject to
modification, we have verified the working of all the cited URLs.

###  2.1 Heterogeneous Anomaly datasets

The datasets discussed here posses diverse kind of anomalies including rare
objects as well as events events. Some examples of anomalies in these datasets
are presence of cart/bicycle/motor-car on pedestrian area, person throwing
papers, fight, run, etc. There are only 8 datasets in this category. Majority
of the datasets were collected from the institute of the authors who created
the dataset. Now, we will be discussing the public video datasets having
heterogeneous anomalies.

Fig. 2: Representative frames for the UCSD Peds1 and Peds2 dataset. First row
shows normal frames and second row shows abnormal frames. First two columns
are from Peds1 and last column is from Peds2 sequences.

####  2.1.1 UCSD dataset

The UCSD dataset [4] is a widely used and non-staged anomaly dataset. It was
recorded at the University of California, San Diego, in the year 2010. The
dataset was collected by deploying a stationary camera mounted at an elevation
overlooking pedestrian walkways. The dataset records two different scenes:
Peds1 and Peds2. For each of the scenes, the train-test is clearly specified,
where the train has only normal samples, and the test has normal as well as
abnormal samples. Peds1 has 34 short clips for training, and 36 clips for
testing, whereas Peds2 has 16 training and 12 testing clips. Frames in each of
the datasets are of the same size and almost the same number of frames (200
frames in each clip of Peds1 and 120-180 frames each clip of Peds2). Along
with frame-level annotations, pixel-level annotations (10 test clips for Peds1
and 12 test clips for Peds2) are also provided for anomaly localization in the
frame. The number of people on the pathways varied from low to crowded in this
dataset. Video segments containing the only pedestrian are regarded as normal
events. The presence of wheelchairs, bikers, skaters, cycles, small carts,
people in wheelchairs, and people walking across a walkway or grass were some
common anomalies observed in the footage. Some normal and abnormal frames of
this dataset are shown in Fig. 2. Despite its wide popularity, the dataset has
some shortcomings. Relatively small data in terms of frame count, total
abnormalities, variety of anomalies, and presence of appearance-based
anomalies only, etc., are some of the deficiencies.

Fig. 3: Representative frames of the AVENUE dataset. First row shows normal
frames and second row shows abnormal frames.

####  2.1.2 Avenue dataset

The Avenue dataset [5] is another widely used dataset, captured in CUHK campus
avenue and published in 2013. It consists of short video clips recorded in an
outdoor environment. The camera is placed such that it looks at the side of a
building with a pedestrian walkway in front of it. Anomalies are mostly staged
and consist of actions such as a person throwing papers or a backpack into the
air, running, and loitering. Some normal and abnormal frames of this dataset
are shown in Fig. 3. A slight camera shake is also present in some of the
clips. The training volume has a total of 16 clips and consists of only normal
events. The testing volume has 21 clips with both normal and abnormal events.
The clips are recorded at 25 Frame Per Second (FPS) and have a resolution of
640 × 360. Clips’ duration varies in 1-60 seconds. Both pixel-level and frame-
level ground truth is available; hence frame-level, as well as pixel-level
evaluation, can be done. Like UCSD Peds1, Peds2, this dataset also has few and
fewer variable anomalies.

Fig. 4: Sample frames from the four cameras in ARENA dataset (image source
[6]).

####  2.1.3 ARENA dataset

The ARENA dataset [6] was published as a part of the PETS2014 challenge. ARENA
stands for ‘Architecture for the REcognition of threats to mobile assets using
Networks of multiple Affordable sensors’. It was recorded within the framework
of the EU project ARENA. The dataset was recorded by installing four cameras
(Resolution: 1280 x 960 pixels, frame rate: 30 FPS) with no overlap at the
crossing path and car park in the University of Reading. The samples frames
from the four cameras are shown in Fig. 4. It offers 22 acted scenarios of
abnormal behavior around the parked vehicle. They are categorized into three
categories which are (1) Something is wrong: Abnormal behavior that however
cannot be considered as a real threat, (2) Potentially criminal: The security
of the driver, the vehicle or any contained asset is in danger, and (3)
Criminal behavior: The security/safety of the driver, the vehicle or any
contained asset has been breached.

Fig. 5: Representative frames for the ShanghaiTech dataset. First row shows
normal frames and second row shows abnormal frames.

####  2.1.4 ShanghaiTech Campus dataset

The ShanghaiTech Campus dataset [7], released in 2017, is one of the largest
video anomaly datasets, which is recorded by deploying CCTV cameras at
multiple places on the ShanghaiTech campus. It covers a diversity of 13
different scenes with different illumination, camera angle, and area of
surveillance scenes. There are 330 training videos (normal events) and 107
testing videos (normal and abnormal events), each with a resolution of 480 ×
856 pixels. The dataset has a total of 317398 frames. It contains 130 abnormal
events with frame-level and pixel-level annotation. Anomalies are mainly
staged, which includes fighting, robbing, jumping, fighting, and vehicle
(skaters, bicycle) in pedestrian areas, etc. Some example anomalies can be
seen in Fig. 5. The dataset does not adhere to the single scene formulation
since it is designed to be used to learn a single model for anomaly detection.
One can assume it to be 13 different datasets, each having one scene, but then
the dataset will be too small, and sometimes negligible anomalies present.

Fig. 6: The Sample frames with abnormality in bounding boxes in some frames
from LV dataset (image source [8]).

####  2.1.5 LV dataset

The Live Videos (LV) dataset [8] published in the year 2017 is a rich
collection of realistic videos captured by surveillance cameras in challenging
environmental conditions such as changing illumination and camera motion. It
has 30 video clips representing 30 different scenarios which are collected
from online sources. The total duration of this dataset is 3.93 hours. The
frame rate and resolutions of the clips are variable. Minimum and maximum
resolution are (176 × 144) and (1280 × 720), whereas the frame rate is in
range (7.5 - 30). The dataset has a total of 34 anomalous events (68989
frames) from 14 unique categories. The anomalous event is Fighting, people
clashing, arm robberies, thefts, car accidents, hit and runs, fires, panic,
vandalism, kidnapping, homicide, cars in the wrong-way, people falling,
loitering, prohibited u-turns and trespassing. Some frames from different
clips with anomaly localization as bounding boxes are shown in Fig. 6. Each
clip has some normal and some abnormal frames. Authors provide frame-level
annotation along with bounding boxes for the events of interest. The dataset
is useful for modeling security threat based anomaly events

Fig. 7: Representative frames for the UCF Crime dataset. First row shows
normal frames and second row shows abnormal frames.

####  2.1.6 UCF-Crime dataset

The UCF-Crime dataset [9], released in 2018, is a large collection of internet
videos taken from hundreds of different cameras resulting in a diversity of
scenes. It includes 1900 untrimmed surveillance footage, out of which 950
footage have normal events, whereas the remaining 950 belongs to predefined 13
categories of criminal activities. It includes events like Abuse, Arrest,
Arson, Assault, Accident, Burglary, Explosion, Fighting, Robbery, Shooting,
Stealing, Shoplifting, and Vandalism. Some of the frames from normal and
abnormal clips are shown in Fig. 7. The dataset also specifies a predefined
train-test split. It has 800 normal and 810 anomalous videos for training and
150 normal and 140 anomalous videos for testing purposes. Each video has a
frame rate of 30 and resolution 320×240. For evaluation, video-level, i.e.,
frame-level annotation, is only available for this dataset. The dataset spans
over multiple scenes; hence it may not be useful for the applications or model
which deals with a single scene. Also, the dataset is more inclined to
activity detection.

Fig. 8: Representative frames for the Street Scene dataset. First row shows
normal frames and second row shows frames with anomalies.

####  2.1.7 Street Scene dataset

Street Scene [10] is another large volume dataset (203257 frames in total)
that was recently released (the year 2020). It consists of 46 training (normal
events) and 35 testings (normal and abnormal) video sequences captured from a
static USB camera looking down on a two-lane street with bike lanes and
pedestrian sidewalks. Videos were collected at various times during two
consecutive summers and mainly during the daytime. It captured a variety of
activities like cars driving, turning, stopping, and parking; pedestrians
walking, jogging, and pushing strollers; and bikers riding in bike lanes.
Additionally, the videos have moving shadows and non-stationary backgrounds
such as movement in tree leaves, flag due to wind, etc. There are a total of
56,847 and 146,410 frames for training and testing, respectively, each of
resolution 1280 x 720 pixels. The test videos contain natural anomalies such
as pedestrians jaywalking across the road, pedestrians loitering on the
sidewalk, bikers on the sidewalk, bikers outside a bike lane, cars making
u-turns, cars parked illegally, cars outside a car lane, etc. These test
videos reported a total of 205 anomalous events spread over 17 categories. For
each testing video, ground truth annotations are provided in the form of
bounding boxes around each anomalous event in each frame. Representative
frames from this dataset are shown in Fig. 8.

Fig. 9: Representative frames for the ADOC dataset. First row shows normal
frames and second row shows frames with anomalies.

####  2.1.8 ADOC dataset

The ADOC [11] is the longest untrimmed dataset for video anomaly detection
that was added in the year 2020. Moreover, it is recorded at a single scene (a
university campus area) over consecutive 24 hours. It captured a variable
number of people flow and the presence of rare objects in the scene. The
versatile range of illumination change, shadow size, and sparse range of
people count in the scene make the dataset more challenging. The video turns
to grey image mode from RGB during the night, and spotting anomalies under
such low illumination will be difficult. Fig. 9 shows some of the anomalies in
the night scene. The second and third column of the second row shows the
presence of cart (left-top corner of the second column) and a person with
stature (bottom-middle of the third column). The video is recorded a 3 FPS and
at a resolution of 2048 × 1536. The 24-hours video is divided into four
6-hours chunks. Authors have also suggested some initial split of train-test;
however, one may formulate different experimental setup as well. Events or
objects occurring at rare frequencies are regarded as anomalies in this
dataset. Authors report a total of 27 categories of anomalous events such as a
person walking with balloons/umbrella/dog/suitcase, a person on a vehicle,
crowd gathering, a person holding a signboard, a bag left behind, truck on a
walkway, etc. Walking of people occurs in largest frequency, so it is treated
normal event in this dataset. Frame-level and pixel-level annotations are both
available. As a continuous 24 hours of uncut footage, the dataset is ideal for
developing a scene monitoring application.

###  2.2 Specific Anomaly datasets: Crowd Surveillance based

The datasets discussed here have few specific anomalies in crowd scenarios.
These datasets can be used for various tasks such as crowd counting, tracking
people, person re-identification, group dispersion/formation identification,
crowd behavior analysis, etc. Some of the datasets were gathered from publicly
accessible sources such as Getty Images, Pond5, YouTube, and Flickr or through
public cameras. Others were collected from the institutes with the same name
as the dataset. Now, we will be discussing the public datasets with crowd-
related specific anomalies in increasing order of their release year.

Fig. 10: Representative frames of the Canoe dataset. The left frames is a
normal frames and right one is an abnormal frame.

####  2.2.1 Canoe dataset

The Canoe dataset [12] is a 34-second sequence recorded near a river. It was
released in 2008. The clip shows a flow of water and the appearance of a canoe
after 22 seconds in the clip. The video is recorded at 30 Hz and 320 × 240
resolution. The presence of canoe is treated abnormal object here. The dataset
has limited use due to its very short duration and only one example of
anomaly.

Fig. 11: Representative frames of the Subway dataset. First row shows normal
frames and second row shows frames with anomalies.

####  2.2.2 Subway dataset111Available upon request

The Subway dataset [13], released in 2008, comprises two sub-datasets, ‘Subway
Exit’ and ‘Subway Entrance’, captured with a CCTV at a subway exit and
entrance location, respectively. The videos are recorded at a frame rate of 25
FPS and 512 × 384 resolution. These datasets were the largest one-chunk video
(Subway Exit: 43 minutes 16 seconds, Subway Entrance: 1 hour 36 minutes 9
seconds) anomaly datasets till the year 2019. Anomalies in these videos are
identified as what would interest a security guard in the subway entry and
exit area. However, only some fixed classes of anomalies are considered and
annotated here. Walking in wrong direction, entry without payment, loitering
near the exit were some staged anomalies in these videos. Only frame-level
sparse annotations are available; pixel-wise annotations are missing. There is
no explicit train-test separation. Usually, the first 5-20 minutes of video,
which are from normal class, is used for training and the rest for testing by
the researchers. Some of the normal and abnormal frames are shown in Fig. 11.

Fig. 12: Representative frames of the UMN dataset. First row shows normal
frames and second row shows frames with anomalies.

####  2.2.3 UMN dataset

The UMN dataset [14] released in 2009 includes panic-driven crowd videos from
the University of Minnesota. The videos are recorded at 30 FPS and 320 × 240
resolution. The dataset has 11 short clips of 3 scenes of people roaming
around an outdoor field (total 48 seconds), an outdoor courtyard (total 77
seconds), and an indoor hallway (total 138 seconds). Each clip starts with
people walking in different directions and ends in a sudden evacuation
scenario. Sample normal and abnormal frames are shown in Fig. 12. The dataset
is staged and has only one type as well as a single anomalous event per clip.
Also, the dataset does not specify any train-test split and does not provide
spatial annotations.

Fig. 13: Representative frames of the Web dataset. First row shows normal
frames and second row shows frames with anomalies.

####  2.2.4 Web dataset

The Web dataset [14] was published along with UMN dataset in 2009 to study
abnormal crowd behavior. The dataset is build of 12 normal and 8 abnormal
crowd videos taken from the sites like Getty Images and ThoughtEquity.com. The
videos have different resolutions and FPS. The normal crowd videos span 4-102
seconds, and abnormal crowd videos span 2-29 seconds. The normal crowd
behavior includes Pedestrian walking, marathon running, etc., and scenes of
panic-escape, protesters clashing, and crowd fighting are some abnormal crowd
behavior in this dataset. Each video has a single type of crowd movement.
Overall this small dataset has less variability and complexity and hence a
less challenging crowd movement dataset. Fig. 13 shows sample frames of the
normal and abnormal sequences.

Fig. 14: Representative frames of the PETS2009 dataset. First row shows normal
frames and second row shows frames with anomalies.

####  2.2.5 PETS2009 dataset

The PETS2009 dataset [15] released in 2009 includes normal and abnormal
sequences of a crowd walking and evacuating scenes. The sequences are recorded
at 7 FPS with two resolutions, 786×576 and 720×576. The dataset is not
primarily meant for anomaly detection but different tasks such as analysis of
group activities, counting people, tracking individuals, estimation of local
dispersion, crowd formation, and splitting, etc. Estimation of the start and
end of such activities can also be another task. People running in one
direction or evacuation can be anomalies to be detected in these datasets.
Overall, this dataset is more suitable for crowd analysis than anomaly
detection. Some of the normal frames and abnormal frames are shown in Fig. 14.

Fig. 15: Representative frames for different clips in Anomalous Behavior
datasets. Frames in the first and third rows are normal whereas frames in
second and fourth rows are abnormal.

####  2.2.6 Anomalous Behavior datasets

In [16], authors released 6 video sequences for anomalous object or behavior
detection in the year 2010. The video sequences are from a different scenario
with illumination effects, scene clutter, variable target appearance, rapid
motion, and camera jitters. All 6 sequences are manually annotated by creating
a grid on the frames. These video sequences are briefly discussed below.
Boat-Sea: It is a video sequence capturing a sea view. The flow of water is
normal behavior, and the appearance of a boat is treated abnormal here. This
clip is recorded at 19 FPS with resolution 720×576 and has a total duration of
1 minute 56 seconds.
Boat-River: It is similar to the Boat-Sea sequence. It records a river view.
The appearance of a boat is treated abnormal here too. The total length of
this clip is 1 minute 8 seconds. It is shot at 5 FPS and 704× 576-pixel
resolution.
Caouflage: This sequence is acted by one person. The presence of the person
has treated anomaly here. It is of 54-second duration shot at 29.97 FPS with
320× 240-pixel resolution.
Airport-WrongDir: This sequence has footage of people passing from a gate for
a security check. The camera is mounted at the top, and hence the top view of
people is visible. People going in the opposite direction are treated anomaly
here. It is of 1 minute 39-second duration with 300×300 resolution and a frame
rate of 25 Hz.
Belleview: This is a gray-scale video showing a traffic scene at different
times of the day. Cars entering the intersection from the left or right is
considered illegal turn here and hence anomaly. The video is of 4 minutes 51
seconds duration and has a resolution of 320 × 240, recorded at 10 FPS. It has
2918 frames, out of which the first 200 frames, which are normal, are used for
training.
Train: This is most challenging compared to all the above sequences. This clip
is shot inside a moving train with camera jitters and rapidly varying
illumination as the train passes inside a tunnel. The total length of this
clip is 12 minutes. It is shot at 25 FPS and resolution of 288×386. The
movement of people inside the train is regarded anomaly here.

In all the above datasets, the rare phenomena or appearance of a rare object
is treated anomaly. Multiple video sequences are useful for anomaly detection
in different scenarios. However, a very constrained amount of anomaly, event,
and duration of clip limits the utility of this dataset. Sample frames from
the normal and abnormal class for each of these clips are shown in Fig. 15.

Fig. 16: Sample frames from Rodriguez’s dataset (image source [17]). Fig. 17:
Example of rare event in Rodriguez’s dataset (image source [17]).

####  2.2.7 Rodriguez’s dataset

Rodriguez’s dataset [17] for crowd analysis purposes was released in 2011. It
was created using video crawling from search engines and stock footage sources
(e.g., Getty Images and YouTube) (Fig. 16). It has 520 video footage of dense
crowds. Along with the enormous number of crowd recordings, the dataset
contains ground truth trajectories for 100 individuals chosen at random from
the population of all moving persons. The dataset can also be used for
anomalous trajectory detection. Authors [17] used this dataset for tracking
rare and abrupt event detection too. This refers to the detection of motion of
an individual not following the global pattern of motion in the crowd (shown
in Fig. 17).

Fig. 18: Representative frames for Hockey dataset (image source [18]).

####  2.2.8 Hockey dataset333Available upon request

The Hockey dataset [18] is a violence detection video dataset published in
2011. The dataset has activities with violence and non-violence from a hockey
game in the same scene setting. Specifically, the authors collect 1000 clips
of action from hockey games of the National Hockey League (NHL)
(representative frames in Fig. 18). Each clip has 50 frames with a resolution
of 720×576 pixels. Annotation is done on clip level with the binary label as
fight and non-fight. The dataset is available on request to authors.

Fig. 19: Representative frames for Movie dataset (image source [18]).

####  2.2.9 Movie dataset555Available upon request

The Movie dataset [18] is also designed specifically for violence detection
along with the Hockey dataset described above. To generalize the fight scene,
authors collect fight scenes from various movies. Specifically, the authors
collect 100 clips with fights and 100 clips with non-fight scenes. Since the
clips are from different sources, resolutions are multiple.

Fig. 20: Scenes from 15 real video sequences along with identified crowd
behaviors in UCF Crowd dataset (image source [19]).

####  2.2.10 UCF Crowd dataset

The UCF dataset [19] for crowd analysis was released in 2012 by the ‘Center
for Research in Computer Vision’ group at UCF institute. It was acquired from
the web and PETS2009 challenge. It has a total of 38 videos comprising outdoor
crowds, palaces, and traffic scenes. It is also mainly designed for a few
crowd behaviors type categorization like merging crowd, circulating crowd,
blocking crowd, etc. Some of these crowd behavior types are shown in Fig. 20.
Although, it has few anomalies such as rare motion pattern of an
individual/group (similar to Rodriguez’s dataset), scene evacuation (from
PETS2009 dataset), etc

Fig. 21: Sample frames from Voilent-Flows dataset (image source [20]).

####  2.2.11 Violent-Flows dataset

The Violent-Flows dataset [20] is a crowd dataset released in 2012. It
consists of 246 videos (average length of 3.60 seconds) collected from the
web. All the videos are resized to 320×240 resolution. The collection has a
wide range of scene types, video qualities, and surveillance scenarios.
However, from an anomalous standpoint, it comprises only a few specific types
of anomalies, all of which fall under the category of crowd violence. Some
representative frames are pasted in Fig. 21.

Fig. 22: Representative frame with trajectories of crowd in Grand central
station dataset (image source [21]).

####  2.2.12 Grand Central Station dataset

The Grand Central Station dataset [21] is a continuous video sequence captured
inside the New York Grand Central Station (Fig. 22). It was published in the
year 2012. The footage has mainly crowd movements. It can be used for person
counting, tracking, crowd behavior analysis, and public surveillance. The
camera is mounted at a high elevation. The total length of the video is 33
minutes and 20 seconds. It has a resolution of 720×480 and a frame rate of 25.
The dataset also provides the KLT keypoint trajectories from the video. The
dataset can also be used for detecting an anomalous or rare walking pattern.

Fig. 23: Representative frames showing different scenes from the AGORASET
dataset.

####  2.2.13 AGORASET dataset

The AGORASET dataset [22] is a synthetic video dataset published in 2012.
Twenty-six avatars with variable shapes and colors were generated to create
the synthetic scenes. Some of the scenarios are shown in Fig. 23. A total of 7
categories of scenes were developed using crowd models proposed by Helbing et
al. [23], exhibiting different patterns of crowd movement. The different scene
includes humans in a free environment, two environments with obstacles, an
evacuation through a door, a dispersion, a rotation similar to the Mecca crowd
scene, and an unstructured crowd. The Mental Ray Physical Sky model [24] is
used for the scene rendering process. Approximately 23 videos with 1 minute of
average duration were created. They have a resolution of 640×480 pixels and 30
FPS. Along with the synthetic videos, the authors provide the associated
pedestrian information (position of pedestrians, associated velocity) and its
continuous analog quantities (such as density) and the pedestrian segmentation
masks. This dataset has few anomaly samples, such as evacuation and dispersion
scenarios. However, the synthetic scene is too simple (plain background and
constant illumination) to be a realistic scene.

Fig. 24: Representative frames for Meta-Tracking for Video Scene Understanding
dataset (source [25]).

####  2.2.14 Meta-Tracking for Video Scene Understanding dataset

The Meta-Tracking for Video Scene Understanding dataset [25] was developed in
2013. It is mainly collected from online video repositories and is best suited
for holistic crowd movement analysis. It is largely collected from UCF data,
the change detection.net database, YouTube, and a personal database. The
videos are from 12 different scenes, with each having 300-8000 frames. Some of
the scenes include a marathon, a religious pilgrimage, an Indian market,
pedestrian traffic, etc. Representative frames are shown in Fig. 24. The
resolution of videos is in the range 320×240 to 720×404 with variable frame
rate too. The clips have varying crowd densities. The dataset can be used for
crowd-based surveillance, behavior analysis, counting, tracking, group
detection, etc.

Fig. 25: A representative frame with twelve dotted lines showing examples of
annotated walking routes in PWPD dataset (source [26] ).

####  2.2.15 PWPD dataset

The Pedestrian walking route dataset (PWPD) dataset [26] was published in the
year 2015. It is a 1-hour crowd surveillance video recorded in a terminal
station. It provides 12,684 complete trajectories of people from the time they
enter the scene to exit time. The video has 100,000 frames with a resolution
of 1920×1080 recorded at 25 FPS. Authors uniformly sample each $20^{th}$ frame
from the video, and thus the frame rate of annotation is 1.25. The
trajectories from these frames are manually annotated. Some trajectories are
shown in Fig. 25. The average and the maximum number of pedestrians observed
in each frame are 123 and 332, respectively. This dataset is a larger dataset
with a dense crowd and complex scene. It is useful for tracking people end to
end in the scene and walking pattern-based anomalous behavior analysis.

Fig. 26: Example frames in RE-DID dataset (source [27]).

####  2.2.16 RE-DID dataset

The RE-DID (Real-Life Events-Dyadic Interactions Dataset) dataset [27] is a
fight detection dataset published in 2015. It has 30 real footage from car-
mounted Dash-Cams, and mobile phones scraped from YouTube (see example frames
in Fig. 26). The length of clips varies from 20 to 242 seconds. Also, the
clips have variable lighting conditions, weather conditions, camera views, and
resolutions. Clips were upsampled to a fixed resolution of 1280x720 for
homogeneity. The dataset recorded a total of 73 instances of fight scenes in
its 30 clips. The start and end of fight scenes are annotated along with
bounding box information.

Fig. 27: Representative frames for MED dataset. First row shows normal frames.
Row 2 to 5 shows abnormal frames from the four categories: panic, fight,
obstacle, and congestion.

####  2.2.17 MED dataset

The Motion-Emotion dataset (MED) [28] was published in 2016. It has 31 video
sequences (554 × 235, 30 FPS) recorded at two different indoor scenarios and
variable lighting conditions. There is a total of 29713 normal frames and
13913 abnormal frames. The video mainly records pedestrians, which are
regarded as normal events. A group of people acted out a predefined script
which led to anomalies in the dataset. Anomalies in this dataset are
categorized into 4 different classes, which are panic, fight, congestion, and
presence of obstacle or unattended object. Some of the example frames for
normal and abnormal classes are shown in Fig. 27. The dataset is also
annotated with six categories of emotions and hence can be used for emotion
detection as well.

Fig. 28: Example frames in CCTV-Fight dataset (source [29] ).

####  2.2.18 CCTV-Fights dataset

The CCTV-Fights dataset [29], released in 2019, is a database of real fight
scenes. It has 1000 clips collected from YouTube by using search keywords like
CCTV Fight, Mugging, Violence, Surveillance, Physical violence, etc. The clips
were generally CCTV footage or recorded from mobile cameras. Some example
frames are pasted in Fig. 28. A total of 280 videos of various types of fight
scenes were acquired, with each ranging from 5 to 12 minutes. Further, 720
additional clips (3-7 seconds each) of real fights from other sources (mobile
cameras, dash-cams, drones, helicopters) were added to the database. Thus it
has a total of 100 videos. Frame-level annotation is available for this
dataset.

###  2.3 Specific Anomaly datasets: Traffic Surveillance based

The datasets discussed here are mainly developed for traffic monitoring. Some
examples of anomalies in these datasets are wrong direction, illegal parking,
illegal turn, accidents, etc. Now, we will be discussing the public datasets
having traffic-related specific anomalies. Although, some of these datasets
also have non-traffic-related datasets. Since they are released together so we
have discussed them together.

Fig. 29: Frames for i-Lids bag (first row) and vehicle detection (second row)
challenge.

####  2.3.1 i-Lids dataset

The i-lids dataset [30] was released as a bag and vehicle detection challenge
in AVSS 2007. It mainly focused on two specified anomalous situations. The
first is abandon baggage detection, and the other is parked vehicles in a no
parking zone (representative frames in Fig. 29). Abandoned baggage and illegal
parking of vehicles are clearly defined for the specified scene in the
challenge. The dataset has 7 video sequences, 3 for the first task and 4 for
the other. The data collection site are various locations in the UK. For
baggage detection, a train station is selected, and for vehicle detection, a
suburban street is chosen. The primary event in street videos includes the
left-lane vehicle going up, the right-lane vehicle going down, vehicle
turning, pedestrian crossing the road. The street clips have slight camera
motion and light variations. The database has a total of 35000 frames with
resolution 720×576 sampled at 25 Hz.

Fig. 30: Representative frames with anomaly category and clip number in QMUL
dataset (image source [31]).

####  2.3.2 QMUL dataset

The QMUL dataset [31], published in the year 2008, is intended for traffic
surveillance applications. The authors [31] recorded a one-hour video with a
25 frame rate is recorded at a busy traffic junction. There are four roads,
two walkways, and numerous buildings in the background. It also has a
significant number of cars, bicycles, and pedestrians, as well as lighting
changes making the scene complex. The events include vehicles straight
driving, turning, waiting for the red lights, and pedestrians crossing the
road. The authors identified 4 vehicle flow types (trajectories) in the video.
Any event interrupting the normal traffic flow, vehicles not following the
allowed set of trajectories, rare behavior of vehicles, etc., are regarded
anomalies here. Authors also categorized observed anomalies into 3 categories:
A (Abnormal behaviors that are visually obvious), B (Rare and ambiguous
behaviors), and C (Abnormal behaviors supported by weak evidence). Some
example of anomalies with their category and clip number is given in Fig. 30.
The database has 112 non-overlapping clips, manually extracted from the one-
hour footage resulting in a total of 22 minutes of duration. All the clips are
resized to a resolution of 360×288 pixels. 73 clips (22000 frames) were used
for training, and 39 clips (12000 frames) were used for testing. For
annotations, authors manually examined each frame exhaustively. 3 clips from
the test set and 5 clips from the train set are identified as anomalous.

Fig. 31: Representative frames for the U-turn dataset. First row shows normal
frames and second row shows abnormal frames.

####  2.3.3 U-turn dataset

The U-turn traffic dataset [32], released in 2009, consists of eight short
clips. Six clips are recorded in a university, and the rest two are acquired
by a static camera positioned at a road intersection and viewing the traffic
(Fig. 31). Anomalies in the first six clips are rare objects/events such as
abandoned baggage, a car, a person running, etc., and the rest two clips have
anomalies as illegal u-turns. Each of the clips has mainly one type of
anomaly. The clips have the same resolution (360 × 240) but variable FPS
(15-30). Normal and abnormal representative frames from some of the clips are
shown in Fig. 31. The authors consider some very specific objects/events as
anomalies; on the contrary presence of cycle in the last column of the figure
is not regarded anomaly, just the abandoned baggage was of interest. Further,
the very short duration of clips (18-407 seconds), an environment with
stationary background, and less crowd make it less challenging.

Fig. 32: Representative frames for the ldiap dataset (image source [33]).

####  2.3.4 ldiap dataset

The ldiap dataset [33] released in 2009 is another video dataset for traffic
surveillance. The dataset is recorded at a traffic-controlled road junction.
It has approximately 45 minutes of video recorded at 25 Hz with a resolution
of 288×360. Some of the observed anomalies in this dataset are people crossing
the road at the wrong place (far away from zebra crossing), vehicles parked at
the pedestrian path, or vehicles stopping ahead of the stop line while the
stop sign is red, etc. The dataset has fewer anomalies, and the video has
zoomed view of traffic; thus, only a small portion of road traffic is visible
at any point in time (see Fig. 32).

Fig. 33: Representative frames with variable illuminations in the IITH
Accident dataset (image source [34]).

####  2.3.5 IITH Accidents dataset

The IITH Accidents dataset [34] published in 2018 is focused on road accident
detection tasks. It offers real footage collected from multiple CCTV cameras
deployed in Hyderabad city in India. Few scenes are shown via Fig. 33. Videos
are recorded at 30 FPS. Each clip starts with some normal frames, some
accident events followed by normal frames. There are a total of 127138 normal
and 863 abnormal frames. The authors used 94720 normal frames for training.
For testing, 32417 normal and 863 abnormal (accident) frames were used. The
dataset has only one type of anomaly, limiting its applicability for generic
surveillance.

####  2.3.6 HTA dataset

The HTA dataset [35] was released in 2020. It is curated from the Berkeley
DeepDrive [36] collection, which contains 100k high-resolution dashcam
recordings (1280 × 720, 30FPS) from vehicles in New York and the Bay Area.
Authors select only highway clips that are of good visual quality. The dataset
is intended to be utilized by self-driving cars for anomaly detection or
monitoring other vehicles. This dataset is different from other existing
anomaly detection datasets in terms of frequent camera motion and dynamic
scene changing scenarios. Five types of anomalies, namely speeding vehicle,
speeding motorcycle, vehicle accident, close merging vehicle, and halted
vehicle, were observed in this dataset. Some of the abnormal samples and
normal samples are pasted via Fig. 34. The training set contains 286 videos of
normal traffic clips, whereas the test set has 78 normal and 25 abnormal
traffic condition footage. For this dataset, too, only temporal annotations
are available.

Fig. 34: Representative frames for the HTA dataset. First row shows normal
frames and second row shows abnormal frames.

####  2.3.7 NVDIA AI CITY dataset

These are traffic surveillance datasets [37] offered under AI City Challenge
organized by NVIDIA. Till now, a total of 5 challenges from 2017-2021 have
been organized. The challenge provides large datasets from real traffic
footage collected in the wild as well as some synthetic footage. The main
tasks in these challenges are turn counting, multi-camera-based vehicle re-
identification/tracking, anomaly detection (e.g., crashes, stalled vehicles,
etc.), natural language-based vehicle track retrieval, etc. The dataset can be
acquired by filling a dataset request form provided on the dataset website.

###  2.4 Additional datasets

The datasets discussed here are not primarily meant or suitable for anomaly
detection but other tasks such as human activity classification, group
analysis in crowded scenarios, specific object detection, human behavior
classification, people tracking, counting, etc. However, they have the
potential to be used for anomaly detection after re-annotating them. Multiple
short datasets can be merged for this purpose too. These pubic datasets are
discussed below.

Fig. 35: Representative frames for the CAVIAR dataset (image source [38]).

####  2.4.1 CAVIAR dataset

The CAVIAR dataset [38] released in 2004 is primarily meant for activity
detection. It has video clips from two indoor scenarios. The dataset was
recorded from webcams deployed in two different indoor scenarios; the first is
the entrance lobby of the INRIA Labs at Grenoble, France, and the second
scenario is a shopping center in Lisbon (Fig. 35). Most of the activities of
interest were acted out by the CAVIAR project team members. Some of the
activities include people walking alone, meeting with others, window shopping,
entering and exiting shops, fighting, leaving a package in a public place,
etc. Videos in both scenarios are recorded at 25 FPS and 384 × 288 resolution.
Annotations of people in all frames are provided as bounding boxes.

Fig. 36: Sample frames from the MIT Traffic dataset. Vehicle motions are
marked by red color and pedestrian motions are marked by green color (image
source [39]).

####  2.4.2 MIT Traffic dataset

The MIT Traffic dataset [39] is a 90-minute long traffic surveillance video
sequence published in the year 2008. A traffic scenario with walking
pedestrians and moving vehicles in a far-field at a street junction is
captured in this video. It has 168822 frames with resolution 720×480 recorded
at 30 FPS. The video footage is divided into 20 clips where 10 clips were used
for training, and the other 10 clips were used for testing by the authors. For
each clip, the ground truth of pedestrians of some sampled frames is manually
labeled. The dataset is mainly used for car/pedestrian detection/tracking.
Fig. 36 shows some samples frames with marking on pedestrians (green) and
vehicles (red).

Fig. 37: Representative frames from the two outdoor scene in the BEHAVE
dataset. These frames also show some potential anomalies.

####  2.4.3 BEHAVE dataset

The BEHAVE dataset [40] was published in 2010, mainly for the purpose of
detecting different human activities in outdoor environments. The dataset has
four video clips (640×480, 25 FPS) from two different outdoor scenes
(representative frames in Fig. 37). The four videos have lengths 52 minutes 11
seconds, 19 minutes 26 seconds, 56 minutes 59 seconds, and 21 minutes 22
seconds, respectively. Out of these four videos, the only first video is
chunked into 8 sub-videos and annotated yet. Authors annotate start and end
frames of activities taking place in the video. The bounding boxes around
humans are also provided to localize the actions. Actions are mainly human-
centric and categorized into 10 classes such as people walking in a group, two
groups of people approaching each other, one group chasing another, groups
fighting, groups splitting from one another, etc. Action such as fighting,
running can be of interest to be detected for the purpose of crime-oriented
abnormal behavior detection.

Fig. 38: Few example scene from the VIRAT dataset (image source [41]).

####  2.4.4 VIRAT dataset

The first release of the VIRAT dataset [41] in 2011 is mainly meant for
activity recognition in an outdoor environment. It records various daily-life
activities from 16 different outdoor scenes. Frames from some of the scenes
are shown in Fig. 38. The dataset is important in terms of realistic scenes,
diverse types of human actions and human–vehicle interactions, and
environmental noise such as clutter, camera shake. Videos are collected at
multiple sites distributed throughout the USA. Two releases of the dataset are
available, being 2.0 the current one. Annotations for objects (person,
vehicle) and events are available in the form of bounding boxes. The dataset
can be used to detect context-based anomalies like wrong parking, standalone
vehicle on the road, etc.

Fig. 39: Representative scene in the CUHK Square dataset (image source [42]).

####  2.4.5 CUHK Square dataset

The CUHK Square dataset [42] was recorded in a similar fashion to the MIT
Traffic dataset in the year 2012. It offers one continuous footage of 60
minutes captured by a stationary camera overlooking traffic. Compared to MIT
Traffic, it has more challenging perspective distortion. A total of 352 frames
uniformly sampled from the first 30 minutes and 100 frames uniformly sampled
from the last 30 minutes are annotated and used for train and test,
respectively. This dataset is also mainly used for person/vehicle
detection/tracking. Since it offers untrimmed footage of traffic, it can also
be used for traffic analysis or surveillance.

Fig. 40: Example frames in the Mall dataset. Person with baby-Carrier,
cleaning material, walking stick in above frames can be regarded anomalies.

####  2.4.6 Mall dataset

The Mall dataset [43], published in 2012, was collected from a public
surveillance camera deployed in a shopping mall. It has 2000 video frames
taken at ¡2 FPS with resolution 640×480. It provides annotations of 60,000
pedestrians in this dataset. The head position of pedestrians is provided.
This dataset covers a more diverse range of crowd density, activity patterns,
illumination conditions compared to the UCSD dataset. It also has more
perspective distortion, which results in larger differences in the size and
appearance of objects at different depths of the scene, as well as a higher
occurrence of occlusion. This dataset can be used for person counting,
tracking, crowd behavior modeling, etc. People running, presence of baby-cart
can be considered as anomalies in this dataset (see Fig. 40). However, the
annotation is mainly done for tracking and counting.

Fig. 41: Example frames from the Collective Motion dataset (image source
[44]).

####  2.4.7 Collective Motion dataset

The Collective Motion dataset [44] was published in the year 2013, mainly for
the purpose of estimating collectiveness in crowd videos. It has clips from 62
different crowded scenes (see representative frames from a different scene in
Fig. 41). The database contains 413 video clips, where 297 videos were
collected by authors and 116 videos were taken from Gitty Images. Each clip
has roughly 100 frames and a variable amount of collective motions. Clips are
annotated with three levels of motion as low, medium, and high with the help
of 10 annotators.

Fig. 42: Example frames from the CUHK Crowd dataset (image source [45]).

####  2.4.8 CUHK Crowd dataset

As the name suggests, the CUHK Crowd dataset [45] is mainly focused on group
profiling from crowd scenarios. It was published in 2014. It is a large
dataset with 474 video clips collected from different sources. 419 clips are
taken from Pond5 and Getty Images, and the rest 49 are collected by authors.
The video clips are from different environments such as shopping malls,
airports, parks, etc., (some example frames from different scenes are shown in
Fig. 42). They vary in terms of crowd densities and shapes. The number of
frames and resolution are also not static in these clips. Authors provide gKLT
tracker-based trajectories for each video clip. Also, locations and velocities
of tracked feature points and annotate the group label of each point is made
available.

Fig. 43: Example frame with groups marked in the Student003 datasets (image
source [46]). Fig. 44: Example frames with groups marked in various Group
Detection datasets. Fig. (a-e) are from MPT-20x100, Fig. (f) is from GVEII,
and Fig. (g), (h) are from the two sequence (Eth and hotel) from BIWI dataset
(image source [46]).

####  2.4.9 Group Detection datasets

Few datasets such as BIWI, GVEII, and Student003 are specially developed and
used for group detection in the crowd. Trajectories and groups information is
provided for all these datasets. See some sample frames for these datasets in
Fig. 43 and 44. The datasets can also be utilized for crowd analysis, people
counting, tracking, etc. They are described below.
BIWI: The BIWI dataset [46] was released in the year 2009, offers two scenes,
namely ETH and Hotel. These scenes (ETH and Hotel) are recorded outside a
university and at a bus stop, respectively. This dataset is mainly used for
group detection and crowd surveillance. 750 different pedestrians are
encountered in each scene. The motion patterns in this dataset are more varied
and chaotic.
GVEII: The GVEII dataset [46] released in the year 2015 is also meant for
group detection. It is a single sequence dataset (resolution 1280×720)
recorded from a camera mounted at a high elevation. The dataset was collected
by ImageLab at the University of Modena and Reggio Emilia and has 2400 frames.
Student003: The Student003 dataset [46] is used for group
formation/dispersion/detection task. It was released in the year 2015 by
ImageLab at the University of Modena and Reggio Emilia. It has a resolution of
720×576 and contains more than 5,000 frames. The dataset is challenging as the
density of the pedestrians is significantly high, and there are multiple entry
and exit points present in the scene. These datasets show densely populated
real-world environments with hundreds of non-linear trajectory variations.
Couples walking together, groups crossing each other, and groups forming and
dispersing are some of the challenging crowd scenarios here.
MPT-20x100: The MPT-20x100 [46] dataset released in the year 2015 is composed
of 20 clips of 100 frames each. It has a variety of videos captured from a
public camera placed at different crowded places such as malls, streets, etc.
Thus it offers a high number of pedestrians and heterogeneous scene
conditions.

Fig. 45: Some representative frames from the worldExpo’10 dataset (image
source [47]). Person heads are marked in red color.

####  2.4.10 WorldExpo’10 dataset

The WorldExpo’10 dataset [47], published in 2015, is one of the largest video
datasets for crowd counting acquired from Shanghai 2010 WorldExpo. The video
sequences were obtained from 108 CCTV cameras stationed there, each with a
different bird view, and thus cover a wide range of scenes (see some frames in
Fig. 45). There is a total of 1132 video clips, 1127 clips of 1-minute each
for training and 5 clips of 1 hour each for testing. Frames are uniformly
sampled from each of these video sequences and annotated. For training, set
frames are sampled at a width of 15 seconds, resulting in 3380 frames (from
1132 clips). For test frames, sampling is done at a width of 30 seconds,
resulting in a total of 600 frames (from 5 clips). The dataset provides 199923
annotated pedestrians (labeled at the centers of their heads) from train and
test frames. The pedestrian count in the scene ranges from 1 to 220.

Fig. 46: Sample frames from the MuseumVisitor dataset. First and second row
show indivisual and groups from the three camera installed in a museum (image
source [48]).

####  2.4.11 MuseumVisitors dataset

The MuseumVisitors dataset [48] was published in the year 2015. It is under
construction. The dataset is aimed for various crowd-based analysis tasks such
as group detection, occlusion handling, tracking, re-identification, gaze
estimation, and behavior analysis. The dataset was collected by deploying
three IP cameras inside the National Museum of Bargello in Florence. It
captures data from two scenarios: the first one consists of individuals
watching artwork, and the second scenario consists of groups of people
watching artwork (some frames are pasted in Fig. 46). The data is recorded at
5 frames per second and with a resolution of 1280×800 pixels. The dataset has
a total of 4808 frames with ground-truth annotation for each pedestrian in the
form of bounding boxes.

Fig. 47: Example of images collected for the spectators and the rink in the
S-Hock dataset. The example annotation file is also shown (image source [49]).

####  2.4.12 S-Hock dataset

The S-Hock dataset [49] was released in the year 2015. It is aimed at
analyzing the behavior of spectator crowds at stadiums. Particularly, 5
cameras from multiple angles were installed focusing on a different part of
the stadium. A full HD camera (1920×1080, 30 FPS, focal length 4mm) for the
ice rink, another one for a panoramic view of all the bleachers, and 3 high-
resolution cameras (1280×1024, 30 FPS, focal length 12mm) focusing on
different parts of the spectator crowd was installed. The dataset is unique
compared to other crowd datasets as the people are semi-static, centered most
of the time on a unique position/seat. This dataset can be used for multiple
spectator crowd analysis purposes such as spectator recognition and
segmentation, global head orientation, automatic highlight production, gesture
segmentation, and social signal processing. It has 31 seconds long 15
sequences for each camera; thus, in total, it has 75 clips. The annotation is
done with the help of an annotator and some surveys from the spectator crowd
itself. For each frame, multiple information that is interesting for crowd
analysis were annotated. They are as follows: people detection (full body
bounding boxes), head detection (bounding boxes of head), head pose (left,
right, down, etc.), body position (sitting, standing, etc.), posture (crossed
arms, elbows on legs, hands on hips, etc.), locomotion (walking, jumping),
action/interaction (waving arms, whistle), supported team (which one), best
action (most exciting action), social relation (if know the neighbor). An
example of the spectator and rink from one of the cameras is shown in Fig. 47.
Each person from the crowd is zoomed in, and their above-mentioned details are
created as shown in the annotation file there.

Fig. 48: Example frames and their annotation in the WWW dataset (image source
[50]). The videos in the first row have ambiguous attributes (marked with
question mark). The videos in the second row have multiple attributes of
where, who, and why.

####  2.4.13 WWW dataset

The WWW dataset [50] published in 2016 is the largest crowd video dataset
collected in the wild. It has 10000 clips from 8257 scenes. Authors built this
dataset from various online sources such as surveillance footage, Getty
Images, Pond5, YouTube, and movies. They used keywords such as street,
stadium, rink, marching, chorus, graduation, etc., for collecting crowd-
related videos from these sources. The diversity in the collected video makes
it complete and suitable for crowd surveillance. It has more than 8 million
frames with 640×360 resolution. The clips are annotated with the help of 19
annotators. Crowd attributes such as Who is in the crowd? Where is the crowd?
And why is the crowd here? is provided. A total of 94 meaningful attributes
covering common crowded places, subjects, actions, and events were specified
in the allowed annotation list. See an example of annotation with sample
frames in Fig. 48.

Fig. 49: Illustration of 10 different flow types of crowd in the Crowd-11
dataset. The flows of the first 5 categories are illustrated by a same color.
The individuals that do not belong to any group are shown by gray color (image
source [51]).

####  2.4.14 Crowd-11 dataset

The Crowd-11 dataset [51] is one of the largest crowd datasets consisting of
6272 video clips, released in 2017. It is collected using web scrapping. Some
of videos are also selected from existing public crowd datasets (WWW [50],
CUHK [45], Violent-Flows [20], Worldexpo’10 Crowd Counting dataset [47],
Agoraset [22], PETS [15], UMN [14], WWW [50], Hockey Fight and Movies [18]).
The procedure to prepare the dataset from these sources is elaborated in the
paper. Moreover, the authors provide the list of URLs on email request. Videos
have variability of resolutions, ranging from 220 × 400 to 700 × 1250. Authors
model the crowd type as different flow types studied in fluid dynamics are
label the crowd dataset into 10 different categories which are Gas Free, Gas
Jammed, Laminar Flow, Turbulent Flow, Crossing Flows, Merging Flow, Diverging
Flow, Static Calm, Static Agitated, and Interacting Crowd (see Fig. 49). And
the last class is of no crowd. Thus the dataset can be seen as 11 class
classification problem too.

Fig. 50: Representative frames from the BOSS dataset.

####  2.4.15 BOSS dataset

The BOSS dataset [52], published in 2017, is a collection of CCTV footage from
10 different cameras installed in various locations of a moving train. Each
footage (720 × 576, 25 FPS) is of duration 60-232 seconds. The groups of
people (single to a crowd of maximum 13 people) are asked to intentionally
create scenes for various types of human interaction (helping, kissing,
handshaking, etc.), pose (sitting, lying, standing), and actions (walking,
running in panic, etc.). Mainly these poses, actions, and interactions at the
frame-level are annotated and hence primarily useful for the pose, action, and
interaction recognition. However, some actions such as a panic attack,
fighting, falling, camera tampering, etc., can be of interest to detect for
anomaly detection model. Some frames with such events and different
illumination conditions are shown in Fig. 50. Additionally, most of the clips
have audio information as well; however, it will be challenging to utilize
them as they are mainly overlapped with noise created from moving trains.
Shadow, fluctuations in the video due to moving train make this video dataset
more challenging.

##  3 Discussion on video datasets

Initial datasets for video based anomaly detection consisted of simple events,
scenarios, and a very constrained amount of anomalies. The anomalies are
mainly performed by a group of people intentionally and therefore lack a
natural flow of events. They are of very short duration too. The presence of a
few rare objects or events in clean background events was regarded anomaly
there. Some examples of such datasets are Canoe (a boat occurs in scene one
time and this is regarded anomaly), UMN (few people acted for sudden
evacuation which is regarded anomaly here), Web (panic-escape, crowd fighting
is regarded anomaly), Subway entrance/exit (the wrong direction is regarded
anomaly), various sub-clips in abnormal behavior dataset (appearance of only
one type of rare object/event), etc. Later datasets were consists of more
types of anomalies and scene variability such as UCSD, AVENUE, ARENA,
ShanghaiTech. In contrast, some recently developed datasets are gigantic in
terms of duration and variability of scene and events such as UCF-Crime,
VIRAT, LV, ADOC, HTA, Rodriguez’s, etc. They leverage the online sources of
CCTV footage and public videos and curate large databases of realistic video
clips from our daily life. The developed dataset over time for anomaly
detection offers a rich and large collection of datasets. They are equipped
with a variety of events, actions, the scene complexity level of scene and
events and thus offers a large collection to select application-specific
dataset for anomaly modeling. Some datasets are suitable for generic scene
monitoring, such as UCSD, ADOC, AVENUE, ShanghaiTech, etc., while others are
suitable for specific classes of anomaly detection such as Subway
entrance/exit, UMN, Web, i-Lids, etc. Further, some datasets offering
detection of specific anomaly classes are important datasets to be used for
frameworks intended for life-threatening event detection, such as UCF-Crime,
LV, NVIDIA AI CITY, etc.

We present a systematic comparison and remark on video datasets discussed in
Section 2. For this, we describe these datasets on various attributes through
Tables 1 to 6. Table 1 shows a broader category of datasets according to the
type of anomalies present. In the first category, i.e., heterogeneous
anomalies, the included anomalies are of different categories, i.e., any
object, situation, the appearance of people which are rare in that dataset, is
treated anomaly. Object/scene representation requires considering both motion
and appearance for the detection of such anomalies. This dataset category
allows researchers to focus on generic anomaly detection framework
development. On the other hand, datasets with specific anomalies, which are
further categorized as crowd and traffic-based, poses a constraint on the
allowed type of anomalies. Crowd-specific ones have mainly focused on one type
crowd centric anomaly event. Datasets for traffic surveillance are mainly
focused on vehicle trajectory-related anomalies such as illegal turns, vehicle
passing from restricted areas, etc. Frameworks to detect such contextual
anomalies must consider the trajectories of vehicles in any form. And these
frameworks then do not focus on anomalies other than trajectory-related. These
datasets are specially designed for evaluating traffic surveillance systems.
The additional dataset category has video datasets that are not primarily
meant for anomaly due to a lack of anomaly-related annotations and dataset
formulation. However, they can be reformulated and be utilized for anomaly
detection. We compare datasets through Tables 2-6 following the sequence of
categories mentioned in Table 1 and the year of their release (increasing
order).

Table 1: Video anomaly dataset categorization

Dataset category | Datasets name
---|---
Heterogeneous Anomaly | UCSD Pedestrian 1/2, AVENUE, ARENA, ShanghaiTech, LV, UCF-Crime, Street Scene, ADOC
Specific Anomaly (Crowd) | Canoe, Subway Entrance/Exit, Web, PETS2009, Rodriguezs, UCF Crowd, Voilent-Flows, RE-DID,
| Grand Cantral Station, AGORASET, Meta-Tracking, PWPD, MED, Hockey, Movie,
CCTV-Fights
Specific Anomaly (Traffic) | i-Lids, QMUL, U-Turn, ldiap, IITH Accidents, HTA, NVIDIA AI CITY
Additional Dataset | CAVIAR, MIT Traffic, BEHAVE, VIRAT, Collective Motion, MALL, CUHK Square,
| CUHK Crowd, WorldExpo’10, MuseumVisitors, S-Hock, WWW, Crowd-11, BOSS

For each dataset, we list the year of publication, the approximate number of
times it is refereed by researchers, and the applications where it has been
used for bench-marking through Table 2-3. The number of times a dataset is
referred also signifies how many times it is used for bench-marking. However,
it is not exactly equal because there are many papers that just mention the
particular dataset paper for a survey paper, use the approach provided along
with the dataset, referred to compare with own dataset, etc. Also, the dataset
which can be applied for bench-marking in multiple applications gets more
references. Additionally, We can observe that a dataset published earlier has
been referenced more compared to a recently released one. Recently released
datasets are less cited as they are younger but normally provide better ground
truth information. For some datasets such as Canoe, PETS2009, QMUL, etc., the
number of references is less, in spite of being released a long time ago. This
may be due to various reasons such as the dataset website is not maintained,
annotations are not sufficient/poor, the dataset is recorded under too many
constraints to look unrealistic, has constrained amount of anomalies, etc.,
resulting in limited applicability of the dataset. The column showing the area
of application gives an idea of different applications where the data has been
used for bench-marking.

Table 2: Applications of various video datasets

Dataset | Year | # Ref. | Area of Application
---|---|---|---
USCD Ped1 | 2010 | 1196 | crowd profiling/counting [53, 54],
|  |  | anomaly detection [55, 56],
|  |  | crowd density estimation [57]
UCSD Ped2 | 2010 | 1196 | anomaly detection [58, 59],
|  |  | crowd profiling/counting [53],action classification [60]
Avenue | 2013 | 588 | aomaly detection [61, 62]
ARENA | 2014 | - | abnormal activity/behaviour detection [63, 64, 65],
|  |  | group walking event detection [66],
|  |  | Human Full-Body/Body-Parts Detection and Tracking [67]
ShanghaiTech | 2017 | 214 | anomaly detection [61, 68],
|  |  | action classification [60], crowd counting [69]
LV | 2017 | 21 | anomaly detection [70, 71, 72],
|  |  | panic detection [73]
UCF-Crime | 2018 | 468 | anomaly detection [74, 75],
|  |  | action classification [60]
Street Scene | 2020 | 27 | anomaly detection [76], road surveillance [77]
ADOC | 2020 | 2 | anomaly detection [76]
Canoe | 2008 | 43 | anomaly detection [16, 78]
Subway Entrance | 2008 | 810 | Anomaly detection [79, 80],
|  |  | abnormal behavior modeling [81, 82]
Subway Exit | 2008 | 810 | Anomaly detection [79, 80],
|  |  | abnormal behavior modeling [81, 82]
UMN | 2009 | 1691 | anomaly detection [4, 79],
|  |  | abnormal crowd behaviour detection [83, 84],
|  |  | Crowd aggregation detection [85],
|  |  | crowd escape behaviour detection [86]
Web | 2009 | – | anomaly detection [87]
PETS2009 | 2009 | 54 | tracking [88, 89],
|  |  | crowd profiling/counting [53, 90],
|  |  | crowd analysis [84],
|  |  | human detection [91, 92],
|  |  | person re-identification [93],
|  |  | crowd escape behaviour detection [86]
Train | 2010 | 106 | anomaly detection [94, 59, 78]
Belleview | 2010 | 106 | anomaly detection [94, 78]
Boat-Sea | 2010 | 106 | anomaly detection [95, 78]
Boat-River | 2010 | 106 | anomaly detection [78, 82]
Caouflage | 2010 | 106 | anomaly detection [82]
Airport-WrongDir | 2010 | 106 | anomaly detection [96, 82]
Rodriguez’s | 2011 | 255 | crowd profiling/counting [90],
|  |  | crowd saliency detection [97],
|  |  | tracking [89], crowd analysis [98],
|  |  | crowd segmentation [99]
Movie | 2011 | 316 | violence detection [100, 101, 102, 103]
Hockey | 2011 | 316 | violence detection [104, 102, 103]
UCF Crowd | 2012 | 265 | abnormal crowd detection [105],
|  |  | crowd profiling/counting [90],
|  |  | crowd saliency detection [97],
|  |  | crowd segmentation [106]
Voilent-Flows | 2012 | 337 | abnormal crowd behaviour detection [83],
|  |  | violence detection [100, 102]
Grand Central Station | 2012 | 326 | pedestrian trajectory prediction [107],
|  |  | pedestrian behavior modeling [108],
|  |  | crowd behavior analysis [109, 110],
|  |  | human re-identification [111], tracking [112],
|  |  | tracking [113], crowd counting [54, 114]

Table 3: Continued from Table 2: Applications of various video datasets

Dataset | Year | # Ref. | Area of Application
---|---|---|---
AGORASET | 2012 | 31 | panic behaviour detection [115],
|  |  | crowd flow tracking [116],
|  |  | crowd motion classification [117],
|  |  | crowd behaviour analysis [118]
Meta-tracking | 2013 | 62 | tracking [119, 120],
|  |  | measuring collectiveness [119],
|  |  | crowd segmentation [121],
|  |  | anomalous walking pattern detection [120]
PWPD | 2015 | 195 | trajectory prediction [122], anomaly detection [122],
|  |  | pedestrian speed detection [123],
|  |  | crowd behaviour analysis [124]
RE-DID | 2015 | 24 | fight detection [27]
MED | 2016 | 35 | anomaly detection [125], panic detection [126]
CCTV-Fights | 2019 | 20 | fight detection [127]
i-Lids | 2007 | – | abandon baggage detection [128],
|  |  | traffic surveillance [129]
QMUL | 2008 | 9 | vehicle tracking [112],
|  |  | trajectory classification [130],
|  |  | anomaly detection [131, 132, 133]
U-Turn | 2009 | 244 | traffic based anomaly detection [134, 135, 136]
ldiap | 2009 | 148 | anomaly detection [132, 133],
|  |  | recurrent activity mining [137]
IITH Accidents | 2018 | 56 | road accident detection [34]
HTA | 2020 | 1 | anomalous motion detection [35]
NVDIA AI CITY | 2021 | – | multi-camera based vehicle re-identification/tracking [138],
|  |  | traffic anomaly detection [139]
CAVIAR dataset | 2004 | 185 | tracking [89], violence detection [140],
|  |  | pedestrian detection [141],
|  |  | re-identification [142],
|  |  | face detection [143],
|  |  | anomalous behaviour detection [144],
|  |  | abandon baggage detection [128],
|  |  | body pose detection [145]
MIT Traffic | 2008 | 891 | crowd behavior analysis [109], vehicle tracking [112],
|  |  | recurrent activity mining [137]
BEHAVE dataset | 2010 | 174 | anomalous behaviour/activity detection [129],
|  |  | violence detection [146]
VIRAT | 2011 | 649 | Activity recognition/forecasting [147],
|  |  | Surveillance Video Summarization [148]
CUHK Square | 2012 | 137 | people/vehicle detection [149]
Mall | 2012 | 503 | crowd density estimation [57],
|  |  | crowd counting [54]
Collective Motion | 2013 | 230 | group collectiveness or crowd analysis [150]
CUHK Crowd | 2014 | 252 | pedestrian trajectory prediction [107],
|  |  | Crowd aggregation detection [85],
|  |  | crowd segmentation [121, 106],
|  |  | anomaly detection, tracking, person counting, crowd behaviour analysis [151]
WorldExpo’10 | 2015 | 904 | pedestrian behavior modeling [108],
|  |  | pedestrian speed detection [123],
|  |  | crowd counting [152]
MuseumVisitors | 2015 | 12 | user interest profiling [153], group identification/tracking [48]
S-Hock | 2015 | 44 | head counting [154, 155], head pose detection [156],
|  |  | crowd analysis [157]
WWW | 2016 | 34 | crowd analysis [158]
Crowd-11 | 2017 | 15 | crowd analysis [159, 160]
BOSS | 2017 | 5 | attack detection [161] people detection [52]

Table 4 shows the following attributes: if the video is recorded in
continuation, i.e., it is untrimmed (column 2), the dataset is recorded from
how many locations (column 3), where is the data recorded (column 4), the
recording location is indoor or outdoor (column 5), what is the density of
people in the scene (column 6). Continuity (yes or no) and location (how many)
helps to see the applicability of a dataset for the task at hand such as if
the aim is to examine the context adaptation of anomaly detection framework,
then the footage should be untrimmed as well as recorded at a single location
[162]. The adaptive models need sufficient data to learn the context on their
own and adapt over time; hence a dataset with too short clips from different
spatial and temporal aspects is not useful for evaluation. For datasets having
multiple recording locations, we mention the exact number of locations if
countable or provided by authors; otherwise, we write it as multiple.
Information about dataset collection scene, i.e., whether it is a mall or an
airport or a traffic junction, etc., may help to choose scene-specific dataset
for the evaluation of specific surveillance frameworks. Also, whether the
dataset is recorded indoor or outdoor may help to evaluate the robustness or
claim that the frameworks work in indoor and/or outdoor scenarios. Some works
are specifically designed to handle crowded scenes, whereas some work only for
low density of people; thus, the information of density in a dataset may help
to choose suitable datasets for evaluations.

Table 4: Comparison of video datasets based on data collection information

Dataset | Continuity | Location | Scene | Indoor/Outdoor | Density
---|---|---|---|---|---
USCD Ped1 | no | 1 | campus (walkway at UCSD) | outdoor | high/medium
UCSD Ped2 | no | 1 | campus (walkway at UCSD) | outdoor | high/medium
Avenue | no | 1 | campus (avenue) | outdoor | medium/low
ARENA | no | 4 | campus (University of Reading) | outdoor | low
ShanghaiTech | no | 13 | campus (Shanghai Tech) | outdoor | medium/low
LV | no | 30 | multiple | both | low/medium
UCF-Crime | no | multiple | multiple | both | high/medium/low
Street Scene | no | 1 | road (two-lane street) | outdoor | medium/low
ADOC | yes | 1 | campus (walkway) | outdoor | high/medium/low
Canoe | yes | 1 | river | outdoor | low
Subway Entrance | yes | 1 | subway entrance | indoor | medium/low
Subway Exit | yes | 1 | subway exit | indoor | medium/low
UMN | no | 3 | field, courtyard, hallway | both | medium
Web | no | multiple | multiple | both | dense/high
PETS2009 | no | 8 | university | outdoor | medium
Train | yes | 1 | inside train | outdoor | low
Belleview | no | 1 | road (intersection) | outdoor | medium
Boat-Sea | yes | 1 | sea | outdoor | low
Boat-River | yes | 1 | river | outdoor | low
Caouflage | yes | 1 | room | indoor | low
Airport-WrongDir | no | 1 | security check | indoor | medium
Rodriguez’s | no | multiple | multiple | outdoor | dense
Hockey | no | 1 | Ice hockey | indoor | medium/low
Movies | no | multiple | movies | both | medium/low
UCF Crowd | no | multiple | multiple | outdoor | dense
Violent-Flows | no | multiple | multiple | both | dense
Grand Central Station | yes | 1 | terminal station | indoor | medium/high
AGORASET | no | multiple | multiple | indoor | high/dense
Meta-tracking | no | multiple | multiple | outdoor | low/high/dense
PWPD | yes | 1 | terminal station | indoor | medium/high
RE-DID | no | multiple | multiple | both | low/medium
MED | no | 3 | campus (walkways) | outdoor | high/medium
CCTV-Fights | no | multiple | multiple | both | low/medium
i-Lids | no | 2 | railway station, road | outdoor | low/medium
QMUL | no | 1 | road (traffic junction) | outdoor | medium/high
U-Turn | no | 4 | road (intersection), university | outdoor | medium/low
ldiap | yes | 1 | road | outdoor | low/medium
IITH Accidents | no | multiple | road (intersection, junction) | outdoor | medium
HTA | no | multiple | multiple | outdoor | medium/low
NVDIA AI CITY | no | multiple | multiple | outdoor | high/medium/low
CAVIAR | no | 3 | entrance lobby, shopping centre | indoor | medium/low
MIT Traffic | yes | 1 | road (intersection) | outdoor | medium/high
BEHAVE | yes | 2 | campus (parking) | outdoor | medium/low
VIRAT | no | 16 | parking | outdoor | medium/low
CUHK Square | yes | 1 | road | outdoor | medium/high
Mall | yes | 1 | shopping mall | indoor | medium/high
Collective Motion | no | 62 | multiple | outdoor | high/dense
CUHK Crowd | no | multiple | multiple | outdoor | high/dense
WorldExpo’10 | no | 108 | Shanghai 2010 WorldExpo | outdoor | high/dense
MuseumVisitors | no | 3 | Museum | indoor | low/medium
S-Hock | no | 5 | stadium | indoor | medium/high
WWW | no | 8257 | multiple | both | high/dense
Crowd-11 | no | multiple | multiple | both | high/dense
BOSS | no | 10 | inside train | indoor | medium/low

In Table 5, we add details of anomalies and annotation. In the ground truth
column, the availability type of annotation is provided. Some dataset gives
only clip level annotation; if a clip is annotated anomalous, then it is
assumed that anomaly is present or spanned to all the frames in the clip. The
assumptions hold true in most cases, especially for small databases, but it is
not scalable to validate this assumption for larger datasets. Hence, some
frames may be wrongly annotated. Frame-level annotations are more precise and
accurate. For some datasets, pixel-level annotations are also provided, which
are helpful for models which aim to precisely localize the occurrence of an
anomaly in the spatial domain along with the temporal domain. Pixel-wise
annotations are too precise but difficult to manually do; therefore, some
datasets such as Train, Belleview, Boat-Sea, BoatRiver, Airport-WrongDir, etc.
first divide the frame into grids (a grid can have multiple pixels) and then
annotates the grids that can be served for spatial anomaly localization. Some
also provide the rectangle bounding boxes around the area of interest, i.e.,
anomalous object/person. Some datasets, especially for people tracking crowd
datasets, provide the trajectories and anomaly information on these. some
datasets in additional dataset category viz. S-Hock and WWW offer categorical
information such as head position, body position, details of scene (who is
there, what is in the scene, where is it located), etc. We list the nature of
anomalies, i.e., anomalies are acted or naturally occurred if acted, then how
many actors were there, how many distinct types of anomalies are there in the
dataset, and some examples of anomaly events through columns 3 to 6. Getting
anomalous natural events for human-centric anomaly datasets is difficult;
therefore, many of the datasets have acted as anomalies. The acted events add
unrealistic flavor, and hence less complex anomaly events are generated.
Generally, traffic surveillance-related datasets get natural anomaly samples
as accidents, illegal turns, or other traffic rule violations are a bit
common. Anomalies in the densely crowded scene are also easier to spot.
However, very few types of anomalies are found in specific anomaly categories.
On the contrary, heterogeneous datasets for public surveillance offer more
diverse types of anomalies. We also observe that the amount of anomalies in a
dataset has increased in recent years (such as LV, UCF-Crime, Street Scene,
ADOC).

Table 5: Comparison of video datasets based on anomaly information

Dataset | Ground truth |  | Nature of
---
Anomaly
# Actor |  | Types of
---
Anomaly
Example anomalies/event
USCD Ped1 | frame/pixel | natural | none | 5 | bikers, small carts, walking across walkways
UCSD Ped2 | frame/pixel | natural | none | 5 | bikers, small carts, walking across walkways
Avenue | frame/pixel | acted | 1-3 | 5 | loitering, running, throwing objects, new object
ARENA | frame | acted | 1-5 | 8-12 | abnormal behaviour, threats
ShanghaiTech | frame/pixel | both | 2-5 |  $<$10 | bicycle, small carts, fight, etc.
LV | frame/bounding box | natural | none | 14 | realistic security threats
UCF-Crime | clip/frame | natural | none | 13 | abuse, arrest, assault, accident, burglary, etc
Street Scene | frame/pixel | natural | none | 17 | jaywalking across road, pedestrians loitering, u-turns
ADOC | frame/pixel | natural | none | 27 | walking with balloons/umbrella/dog/suitcase,
|  |  |  |  | person on vehicle, crowd gathering, etc.
Canoe | grid | natural | none | 1 | canoe
Subway Entrance | frame | acted | 2-4 |  $<$3 | wrong direction
Subway Exit | frame | acted | 2-4 |  $<$3 | wrong direction
UMN | frame | acted |  $\approx$ 15 | 1 | run
Web | clip | natural | none |  $<$4 | panic-escape, clashing/fighting, etc.
PETS2009 | frame | acted |  $>$15 |  $<$5 | run, panic
Train | grid | natural | none | 1 | people movement
Belleview | grid | natural | none | 1 | illegal turns
Boat-Sea | grid | natural | none | 1 | boat
Boat-River | grid | natural | none | 1 | boat
Caouflage | grid | acted | 1 | 1 | person in Caouflage
Airport-WrongDir | grid | acted | 2-6 | 1 | movement in wrong direction
Rodriguez’s | clip | natural | none |  $<$5 | anomalous trajectory
Hockey | clip | natural | none | 1 | fight
Movie | clip | natural | none | 1 | fight
UCF Crowd | clip | natural | none |  $<$5 | anomalous trajectory
Violent-Flows | clip | natural | none |  $<$5 | crowd violence
Grand Central Station | bounding box | natural | none |  $<$3 | rare walking pattern
AGORASET | trajectories | synthetic | – |  $<$3 | evacuation, dispersion
Meta-tracking | trajectories | natural | none |  $<$3 | anomalous trajectory
PWPD | trajectories | natural | none |  $<$3 | anomalous trajectory, abnormal crowd
RE-DID | frame/bounding box | natural | none | 1 | fight
MED | frame | acted |  $\approx$5-15 | 5 | Panic, fight, congestion, obstacle, neutral
CCTV-Fights | frame | natural | none | 1 | fight
i-Lids | frame | natural | none | 2 | abandon bag, illegally parked vehicle
QMUL | clip | natural | none |  $<$5 | unusual traffic trajectory, rare behaviour of vehicles
U-Turn | - | both | 1-2 |  $<$4 | illegal u-turns, running, abandon baggage
ldiap | frame | natural | none |  $<$4 | wrong road crossing, wrong vehicle parking, etc.
IITH Accidents | bounding box | natural | none | 1 | accidents
HTA | frame | natural | none | 5 | accident, speeding vehicle, close merge,
NVIDIA AI CITY | - | - | - | - | accidents
CAVIAR | frame/pixel | both | 2-4 |  $<$5 | leaving object, fighting, run, chase, etc.
MIT Traffic | bounding box | natural | none |  $<$3 | road surveillance
BEHAVE | frame/pixel | both | 2-5 |  $<$4 | fighting, chasing, run
VIRAT | frame/pixel | natural | none |  $<$3 | wrong parking, standalone vehicle
CUHK Square | bounding box | natural | none |  $<$3 | road surveillance related
Mall | bounding box | natural | none |  $<$3 | running, baby carrier
Collective Motion | clip | natural | none |  $<$3 | crowd based behaviour modeling
CUHK Crowd | trajectories | natural | none |  $<$3 | crowd based behaviour modeling
WorldExpo’10 | bounding box | natural | none |  $<$2 | unusual walking pattern
MuseumVisitors | bounding box | natural | none |  $<$2 | unusual walking pattern
S-Hock | categorical/bounding box | natural | none |  $<$4 | shout, interesting events in game
WWW | categorical | natural | none |  $<$4 | anomalous behaviour modeling
Crowd-11 | frame | both | - |  $<$4 | anomalous behaviour modeling
BOSS | frame | acted | 2-13 |  $<$4 | Harass, panic

Table 6 presents a comparison of datasets on various attributes of recorded
videos such as duration, number of frames, resolution, frame rate, etc. The
approximate amount of duration is specified in $3^{rd}$ columns. Some of the
dataset papers do not specify the exact duration; rather, an approximate value
is specified, some provided just the number of frames. Hence in the lack of
FPS details, it is not possible to estimate the dataset duration; hence that
entry is vacant in the table. Some datasets specially specify the number of
normal/abnormal clips. Sometimes a clip has all normal or abnormal frames, and
sometimes a clip has both types of frames. Generally, authors have used clips
having only normal frames in training and clips with normal and abnormal
frames in the testing phase. If authors have specified the exact number of
normal (N), abnormal (A), normal-abnormal (AN) clips, then we have added this
information in $5^{th}$ column; the otherwise total number of clips is
specified there. Some datasets, mainly scraped from online sources (youTube,
Pond5, Getty Images), have multiple resolutions; therefore, they need to be
scaled on the same scale before processing. Also, for the same reason, these
datasets may have multiple FPS. Some datasets have camera motion present too.
A different amount of motion is also found among these datasets. For some of
the datasets, motion is intentionally added to make the dataset more
challenging and realistic (surveillance in moving trains such as Train, BOSS,
etc., surveillance with head-mounted cameras such as HTA, etc.).

Table 6: Specifications of video datasets

Dataset | Release | Total | Total No. of | Total No. of | Resolution | FPS | Camera
---|---|---|---|---|---|---|---
| Year | Duration | Frames | Videos |  |  | motion
USCD Ped1 | 2010 |  $\approx$ 5-7 min | 14000 | 34(N)+36(AN) | 238×158 | - | none
UCSD Ped2 | 2010 |  $\approx$ 2-3 min | 4560 | 16(N)+14(AN) | 360×240 | - | none
Avenue | 2013 | 20 min 26 sec | 30652 | 16(N)+21(AN) | 640×360 | 25 | slight in few
ARENA | 2014 | – | – | 22 | 1280 x 960 | 30 | none
ShanghaiTech | 2017 |  $\approx$ 3-3.5 hrs | 317398 | 330(N)+107(AN) | 856×480 | 24 | none
LV | 2017 | 3.93 hrs | - | 30 | multiple | multiple | in some
UCF-Crime | 2018 |  $\approx$ 128 hrs |  $\approx$ 13 million | 950(N)+950(A) | 320×240 | 30 | slight in few
Street Scene | 2020 |  $\approx$ 3-4 h | 203257 | 46(N)+35(AN) | 1280×720 | 15 | none
ADOC | 2020 | 24 hrs | 259127 | 4 | 2048 × 1536 | 3 | none
Canoe | 2008 | 34 s | 1050 | 1 | 320 × 240 | 30 | none
Subway Entrance | 2008 | 96 m 9 s | 144225 | 1 | 512×384 | 25 | none
Subway Exit | 2008 | 43 m 16 s | 64900 | 1 | 512×384 | 25 | none
UMN | 2009 | 4 min 17 sec | 7710 | 11 (available as 1) | 320×240 | 30 | none
Web | 2009 | 7 min 35 sec | 11,962 | 12(N)+8(A) | multiple | multiple | slight jerks
PETS2009 | 2009 |  $\approx$ 1-2 hrs | 42182 | 59 | 768x576 | 7 | none
|  |  |  |  | 720x576 |  |
Train | 2010 | 12 min | 19218 | 1 | 288×386 | 25 | moving
Belleview | 2010 | 4 min 51 sec | 2918 | 1 | 320 × 240 | 10 | jitter
Boat-Sea | 2010 | 1 min 56 sec | 450 | 1 | 720×576 | 19 | none
Boat-River | 2010 | 1 min 8 sec | 250 | 1 | 704×576 | 5 | none
Caouflage | 2010 | 54 sec | 1629 | 1 | 320×240 | 29.97 | none
Rodriguez’s | 2011 | 10 hrs 24 min | - | 520 | 720x480 | - | none
Hockey | 2011 | 27 min | 50000 | 1000 | 720×576 | - | none
Movie | 2011 | 6 min | - | 200 | multiple | multiple | in some
UCF Crowd | 2012 |  $\approx$11 min |  $\approx$16320 | 38 | multiple | multiple | none
Voilent-Flows | 2012 | 14 min 6 sec | 22,156 | 246 | 320×240 | multiple | significant
Grand Central Station | 2012 | 33 min 20 sec | 50010 | 1 | 480×720 | 25 | none
AGORASET | 2012 |  $\approx$20 min |  $>$33641 | 23 | 640×480 | 30 | none
Meta-tracking | 2013 | – |  $>$4000 | 12 | multiple | multiple | none
PWPD | 2015 | 1 hour | 5000 | 1 | 1920×1080 | 1.25 | none
RE-DID | 2015 |  $<$2 hrs | - | 30 | 1280X720 | multiple | in some
MED | 2016 |  $\approx$24-25 min | 43,626 | 31 | 554×235 | 31 | none
CCTV-Fights | 2019 |  $\approx$ 10 hrs | - | 1000 | multiple | multiple | in some
i-Lids | 2007 |  $\approx$24 min | 35000 | 7 | 720×576 | 25 | none
QMUL | 2008 | 22 min | 34000 | 104(N)+8(A) | 360×288 | 25 | none
U-Turn | 2009 |  $\approx$20 min |  $\approx$25182 | 8 | 360 × 240 | multiple | none
ldiap | 2009 | 44.13 min | 66324 | 1 | 288×360 | 25 | none
IITH Accidents | 2018 | – | 128001 |  | 30 | none |
HTA | 2020 |  $\approx$ 4 hrs |  $\approx$ 0.4 million | 286(N)+107(AN) | 1280×720 | 30 | moving
CAVIAR dataset | 2004 |  $\approx$17-18 min | 26419 | 28 | 384 x 288 | 25 | none
MIT Traffic | 2008 | 90 min | 168822 | 20 | 720×480 | 30 | none
Airport-WrongDir | 2010 | 1 min 39 sec | 2200 | 1 | 300×300 | 25 | none
BEHAVE dataset | 2010 |  $\approx$2.5 hrs | 225024 | 4 | 640×480 | 25 | none
VIRAT | 2011 | 29 hrs |  $\approx$ 3 million | 354 | multiple | multiple | varying
CUHK Square | 2012 | 60 min | - | 1 | 720 × 576 | - | none
Mall | 2012 | – | 2000 | 1 | 640×480 |  $<$2 | none
Collective Motion | 2013 | – |  $\approx$41300 | 413 | – | – | none
CUHK Crowd | 2014 | – | – | 474 | multiple | multiple | none
WorldExpo’10 | 2015 | 1427 min | 3980 | 1132 | 576×720 | 1/30, 1/15 | none
MuseumVisitors | 2015 |  $\approx$17 min | 4808 | 3 | 1280×800 | 5 | none
S-Hock | 2015 |  $\approx$39 min | 69750 | 75 | 1920×1080 | 30 | none
|  |  |  |  | 1280×1024 |  |
WWW | 2016 | – |  $>$8 million | 10,000 | 640×360 | 30 | none
Crowd-11 | 2017 |  $\approx$ 4-6 hrs | 621196 | 6,272 | multiple | multiple | none
BOSS | 2017 | 27 min |  $\approx$ 40500 | 129 | 720×576 | 25 | moving

##  4 Audio Anomaly datasets

Audio anomaly detection is gaining a lot of attention in many applications in
various fields, such as traffic surveillance, industries, music, medicines,
etc. Unlike for video, not many datasets for audio are available for study. A
few datasets have been created based on applications and further provided
publicly. In this section, we describe and discuss the features of publicly
available audio datasets.

###  4.1 MIVIA road audio events dataset

The MIVIA road audio events dataset [163] was released in the year 2014. It
was mainly intended for road surveillance applications [164, 163]. Originally,
59 samples of crashes and 45 of tire skidding, together with the sound of 23
different road locations, were collected. Thus, in total, a collection of 400
events, namely tire skidding with the duration of 522.5 seconds and car
crashes for a duration of 326.38 seconds, were created. The duration for
background audio amounted to 2732 seconds. The audios are available as wav
files and have been recorded and sampled at 32 kHz, quantized at 16 bits per
PCM sample. The signal-to-noise ratio for the data is 15 dB. In order to
account for cross-validation experiments, these events are divided into 4
folds of 100 events each. Each fold comprises a number of audio files of a
duration of about 1 minute, wherein a series of dangerous events are
superimposed to a regular road background sound. Each audio file consists of a
different background sound in order to simulate a number of different real
situations.

###  4.2 AudioSet soundscape

AudioSet Soundscape dataset was collected for the experiment carried out in
[165] from different ecosystems. A total of over 2750 hours of audio
recordings were collected. In order to detect anomalies, each recording was
embedded with 0.96 seconds of eco-acoustic data in 128-dimensional feature
space from Google’s AudioSet dataset. AudioSet is a collection of over 2
million human-labeled, short audio clips from a wide range of sources
available on YouTube [166]. The majority of the sound clips in AudioSet are
unrelated to natural soundscapes, with the largest sections comprising of
music, human speech, and machine noise, such as gunshots and chainsaws. The
dataset collected from different ecosystems has been mentioned below. A
summary of the datasets collected has been described in Table 7.

Table 7: AudioSet_Sounscape Dataset Location | Time of recording | Total hrs | Sampling rate
---|---|---|---
Sabah, Malaysia | February 2018 - June 2019 | Tascam: 274 hrs 40 min | 44.1 kHz
|  | Audiomoth: 748 hrs | 16 kHz
Ithaca, New York | January 2016 - December 2019 | 797 hrs | 48 kHz
| From May 2017 | 638 hrs | 48 kHz
New Zealand | 8-20 December, 2016 | 240 hrs | 32 kHz
Sulawesi | August 2018 | 64 hrs | 48 Hz
Republic of Congo | December 2017 - July 2018 | 238 hrs 20 min | 8 kHz

####  4.2.1 Sabah, Malaysia

Two datasets were collected from lowland rainforests across a gradient of
habitat degradation in Sabah, Malaysia. It was collected between February 2018
and June 2019 across an ecological gradient encompassing primary forest,
logged forest, cleared forest, and oil palm sites.

Tascam dataset: In this, 20 minutes sound files were recorded at a sampling
rate of 44.1 kHz. At each site, one 20 minutes file was recorded per hour,
resulting in a total of 27 hours 40 minutes recording.

Audiomoth dataset: In the Audiomoth dataset, continuous recording of sound was
carried out in consecutive 5 minutes sound files, sampled at 16 kHz, resulting
in a total of 748 hours of audio. Audiomoths were secured to trees.

####  4.2.2 Ithaca, New York

From the protected temperate broadleaf forests in Sapsucker Woods, Ithaca, New
York, two datasets were collected.

The first dataset was collected continuously over 3 years, between January
2016 and December 2019 (inclusive) from only one location. Data was audio-
digitized at a sampling rate 48 kHz and comprised of a total of 797 hours of
audio.

The second dataset comprises 24 hours of audio dated from 13 May 2017 across
an area of 220 acres. The audio was recorded continuously in consecutive
1-hour files, sampled at 48 kHz, with recorders attached to trees at eye
height. A total of 638 hours of audio was collected.

####  4.2.3 New Zealand

Audio data was collected from the protected temperate broadleaf forests in
Abel Tasman National Park, New Zealand, by deploying 10 units of semi-
autonomous recorders from the New Zealand Department of Conservation from 8 to
20 December 2016. Of these, 5 units were installed on the mainland and the
other 5 on Adele Island. Continuous recording of audio was performed in
consecutive 15 minutes files, audio-digitized at 32 kHz, with recorders
attached to trees at eye height. A total of 240 hours of audio was collected.

####  4.2.4 Sulawesi

In Sulawesi, audio was recorded from protected lowland tropical rain forests,
Tangkoko National Park. Continuous recording of audio was carried out from
four locations during August 2018, in consecutive 40 minutes files, sampled at
k48 Hz. A total of 64 hours of audio was collected.

####  4.2.5 Republic of Congo

Audio data was collected from 10 sites of protected and logged lowland
rainforest in and surrounding Nouabalé-Ndoki National Park between the period
from December 2017 to July 2018. Continuous recording of audio was carried out
in consecutive 24 hours files, sampled at 8 kHz. Habitat types spanned mixed
forest and Gilbertiodendron spp. from within a protected area, areas within a
6-year-old logging concession, and within active logging concessions.
Recorders were installed at the height of 7 to 10 meters from ground level and
suspended below tree limbs. Overall, 238 hours 20 minutes of audio were taped.

###  4.3 DCASE 2017 dataset

The 2017 DCASE Challenge Task 2 [167] was released in the year 2017. It
comprises an audio dataset for the detection of infrequent sound events from a
set of artificially created mixtures. The background recordings in the audio
dataset have been collected from the TUT Acoustic Scenes 2016 development
dataset [168], which consists of 30 seconds audio recordings from 15 distinct
acoustic indoor as well as outdoor environments, as mentioned in Table 8. Each
acoustic scene comprises of 78 segments, totaling 1170 samples, resulting in
the final audio of 39 minutes.

Three rare sound events are included in the database for anomaly detection.
These are baby cry (106 training, 42 test instances, mean duration 2.25
seconds), glass break (96 training, 43 test instances, mean duration 1.16
seconds), and gunshot (134 training, 53 test, mean duration 1.32 seconds). The
sound events are recorded at levels of +6Db, 0Db, and -6Db when compared to
the background audio (Event-to-Background Ratio). The events have been
recorded in the 24-bit format and at the sampling rate of 44.1 kHz.

Table 8: Acoustic environments in the DCASE 2017 dataset

Condition | Environment
---|---
Bus: traveling by bus in the city | outdoor: vehicle
Cafe/ Restaurant: small cafe/restaurant | indoor
Car: driving/ traveling as a passenger | outdoor: vehicle
City center | outdoor
Forest path | outdoor
Grocery store: medium size grocery store | indoor
Home | indoor
Lakeside beach | outdoor
Library | indoor
Metro station | indoor
Office: multiple persons, typical work day | indoor
Residential area | outdoor
Train: traveling | outdoor: vehicle
Tram: traveling | outdoor: vehicle
Urban park | outdoor

###  4.4 ICBHI 2017 dataset

The ICBHI 2017 dataset [169] was collected and made available in the year 2017
for medical purposes, such as for respiratory anomaly classification. For the
same, data were acquired from the International Conference on Biomedical and
Health Informatics (ICBHI) Challenge 2017, task 1 [170]. The respiratory
sounds for this challenge were collected from 128 patients, resulting in an
audio recording of over 5.5 hours. A total of 920 audio clips were recorded,
where each recording consists of one to four distinct types of cycles, namely
Crackle, Wheeze, Crackle and Wheeze, and Normal accompanied by onset and
offset time labels. The recording lengths of the cycles varied from 0.2 to
16.2 seconds, with the unbalanced number of cycles,i.e., 1864 cycles for
Crackle, 886 for Wheeze, 506 for Crackle and Wheeze, and 3642 Normal cycles.
The entire set of audio recordings comprise different lengths from 10 to 90
seconds, with a wide range of sampling frequencies varying from 4 to 44.1 kHz.

###  4.5 SMD anomaly detection dataset

The SMD Anomaly Detection Dataset [171] was released in year 2018. The audio
recordings for this dataset were acquired from a Surface-Mount Device (SMD)
assembly machine. Two sets are provided in this data: C-line and B-line. In
this dataset, one production C-line is considered normal, and the rest
everything is abnormal. Three anomalous categories have been described in this
dataset, viz., change in assembly part (from C-Line to B-line), the addition
of artificial noise (in order to represent broken internal components), and
removal of grease in the same production C-line.

For anomaly detection using the SMD dataset, 2048 samples transmuted into
frequency domain with an overlap length of 512 samples, and a sampling rate of
16300 Hz. The spectrogram so obtained was split into 32 columns and
represented about 1.5 seconds. A total of 2100 segments for C-line were
designed, out of which 2000 segments were used for the training dataset and
the rest 100 as testing data. The anomalies were introduced only in the
testing dataset. The audio records used in the experiment have been listed in
Table 9.

Table 9: Audio records (in minutes) for SMD Dataset Category | Length
---|---
C-line (Normal) | 24:49
Non-greased line (Anomaly) | 15:53
C-line with intermittent noise (Anomaly) | 06:08
B-line (Anomaly) | 11:00

###  4.6 MIMII DUE dataset

MIMII DUE audio dataset [172] was developed in the year 2019. It comprises of
Malfunctioning Industrial Machine Investigation and Inspection with domain
shifts due to variations in operational and environmental conditions (MIMII
DUE). It is a subset dataset used in the creation of DCASE 2021 Challenge Task
2. Hence, the dataset is entirely similar to the data that is included in the
development as well as the additional training dataset. It includes normal and
anomalous audios from five distinct types of industrial machines: fans,
gearboxes, pumps, slide rails, and valves. Each category comprises data from
two domains, i.e., the source domain and target domain, collected in different
conditions like the speed of operation or noise from the environment. The
anomalous conditions for each machine type are mentioned in Table 10. The
entire dataset comprises audio recordings for more than 420000 seconds. Each
single-channel 16-bit audio sample is a recording of 10 seconds, sampled at 16
kHz. For the source domain in each section, the training dataset comprises
1000 audio clips, whereas the test data consists of around 100 audio clips of
normal and 100 for anomalous conditions. For the target domain, each section
contains only three audio clips for the training data, whereas the test data
comprises around 100 sound recordings of normal and 100 for anomalous
conditions.

Table 10: Anomalous conditions for each machine type in the MIMII DUE dataset Machine | Anomalous condition
---|---
type |
Fan | Wing damage, unbalanced, over voltage, etc.
Gearbox | Gear damage, overload, over voltage, etc.
Pump | Contamination, clog, leakage, dry run, etc.
Slide rail | Rail damage, loose belt, no grease, etc.
Valve | Contamination

###  4.7 ToyADMOS dataset

The ToyADMOS dataset [173] was released in the year 2019. It refers to a
machine operating sounds dataset consisting of approximately 540 hours of
normal machine operating sounds and over 12000 samples of anomalous audio
collected using four microphones, sampled at the rate of 48 kHz. The ToyADMOS
dataset consists of three sub-datasets for three types of anomaly detection in
machine operating sounds (ADMOS) tasks. A different toy is considered in each
task: product inspection (toy car), fault diagnosis for a fixed machine (toy
conveyor), and fault diagnosis for a moving machine (toy train). In order to
simulate different noise levels, individual recordings of machine-operating
sounds and environmental have been carried out. The dataset for each task
comprises over 180 hours of normal machine-operating sounds and over 4000
samples of anomalous sounds collected at a sampling rate of 48 kHz. All audio
events were stored as different wav files classified into two types:
individual (IND) and continuous (CNT). Each of the three sub-datasets consists
of three types of acoustic data: normal, anomalous, and environmental. Normal
sound refers to the operating sound when the target machine normally performs
in accordance with its specified characteristics. Anomalous sound is defined
as the operating sound when the target machine is forced to perform
anomalously by intentionally altering its elements or by adding foreign parts.
The anomalous conditions for each miniature machine have been mentioned in
Table 11. Environmental noise refers to the noise used for the purpose of
simulation of a factory environment. For this purpose, noise samples such as
collision, drilling, pumping, and airbrushing were assembled from various
locations in an actual factory. The sub-datasets used in the ToyADMOS dataset
are described below. The total audio duration and number of samples for each
sub-dataset are summarized in Table 12.

####  4.7.1 Toy car

The toy car sub-dataset contains normal and anomalous audio clips of a toy car
called “mini 4WD”. It is mainly designed for the purpose of product
inspection. Each IND normal and anomalous wav file was of a duration of 11
seconds, recording 1350 IND samples in each case and channel, resulting in 66
hours of IND normal sounds. In the case of CNT recordings, approximately 150
CNT samples were recorded for each case and channel. A total number of 135
hours of CNT normal sounds was recorded, which made half the total length of
CNT files. In order to add anomalous sounds, parts of the machine, namely
shaft, gears, tires, and voltage, were intentionally altered. Using their
combinations (53 patterns), a total of 250 anomalous sound samples were
generated for each case and channel. In addition, environmental noise was
recorded only once for 12 hours.

####  4.7.2 Toy conveyor

The toy conveyor sub-dataset contains normal and anomalous audio clips of a
toy conveyor fixed on a desk. It is mainly designed for the purpose of fault
diagnosis of a fixed machine. Three different sizes of the conveyor were
considered as ‘cases’ in this sub-dataset. Each normal and anomalous wav-file
of IND sounds was of a duration of 10 seconds, recording a total of 1800 IND
samples for each case and channel. In total, 60 hours of IND normal sounds
were recorded. For CNT data, at least 124 audio samples were taped in each
case, resulting in a total of 120 hours of CNT normal sounds. In order to add
anomalous sounds, parts of the machine, namely tension pulley, trail pulley,
and belt, were intentionally altered or voltage was excessively
lowered/raised. A total of 355 anomalous sound samples were recorded with the
help of these combinations (60 patterns). In addition, environmental noise was
recorded only once for 12 hours.

####  4.7.3 Toy train

The toy train sub-dataset contains normal and anomalous audio clips of HO-
scale (large) and N-scale (small) model railways. It is mainly designed for
the purpose of fault diagnosis of a moving machine. Four cases were designed
for this sub-dataset. Each ‘case’ is configured by combining two kinds of
trains, i.e., commuter and a bullet, and scales, viz., HO-scale and N-scale.
Each normal and anomalous wav-file of IND sounds was of a duration of 11
seconds, recording a total of 1350 IND samples for each case and channel. In
total, 66 hours of IND normal sounds were recorded. For CNT data, 74 audio
samples were recorded for every case and channel, resulting in a total of 197
hours of CNT normal sounds. Anomalous sounds were taped by intentionally
altering the first/last carriage and straight/curved railway track. A total of
270 anomalous sound samples were recorded with the help of these combinations
(54 patterns). In addition, environmental noise was recorded only once for 12
hours.

Table 11: Anomalous conditions for each miniature machine in the ToyADMOS
dataset

Toy car | Toy conveyor | Toy train
---|---|---
Parts | Condition | Parts | Condition | Parts | Condition
Shaft | -Bent | Tension Pulley | -Excessive tension | First Carriage | -Chipped wheel axle
Gears | -Deformed | Tail Pulley | -Excessive tension | Last Carriage | -Chipped with axle
| -Melted |  | -Removed |  |
Tires | -Coiled (plastic ribbon) | Belt | -Attached metallic object 1 | Straight Railway Track | -Broken
| -Coiled (steel ribbon) |  | -Attached metallic object 2 |  | -Obstructing stone
|  |  | -Attached metallic object 3 |  | -Disjointed
Voltage | -Over voltage | Voltage | -Over voltage | Curved Railway Track | -Broken
| -Under voltage |  | -Under voltage |  | -Obstructing stone
|  |  |  |  | -Disjointed

Table 12: Audio collection in ToyADMOS dataset | Toy car | Toy conveyor | Toy train
---|---|---|---
| # of IND normal sounds per case and channel
---
Total hours
| 1350 samples
---
66 hours
| 1800 samples
---
60 hours
| 1350 samples
---
66 hours
| # of CNT normal sounds per case and channel
---
Total hours
|  $\approx$150 samples
---
135 hours
| at least 124 samples
---
120 hours
| 74 samples
---
197 hours
# of IND anomalous sounds per case and channel |  $\approx$ 250 samples | 355 samples | 270 samples
| # of CNT environmental noise samples per case and channel
---
Total hours
| 72 samples
---
48 hours
| 72 samples
---
144 hours
| 72 samples
---
96 hours

###  4.8 LIFE DYNAMAP dataset

Detection of anomalous noise events in the LIFE DYNAMAP project makes use of
audio events of real road traffic noise (RTN) of the ring road surrounding the
city of Barcelona [174]. These audio events are synthetically combined with
anomalous noise events (ANE) samples. The synthetic noise events have been
gathered from free online repositories and consist of up to 15 noise types:
horns, ambulance sirens, car collisions, church bells, birds, crickets, rain,
thunders, etc. The noise-free field recordings of road traffic were obtained
at a sampling rate of 48 kHz and a broadband linear frequency range of 4.2 Hz
- 22.4 kHz. An overall length of 250 seconds of acoustic samples of RTN and
300 seconds of ANE were used for training. Each noise type in the mixtures was
adjusted to obtain RTN-ANE ratios of -6 and -12 dB.

###  4.9 ToyADMOS2 datase

The ToyADMOS2 dataset [175] is also a subset of the dataset for DCASE 2021
Challenge Task 2. This dataset consists of operating sounds for miniature
machines for detecting anomalous sounds under domain shift conditions. It
comprises of two audio categories, viz., toy car and toy train.

####  4.9.1 Toy car:

This sub-dataset consists of normal and anomalous sound clips of a toy car. It
is mainly designed for the purpose of product inspection. Each recorded sample
is of a duration of 12 seconds. Thus, there are a total of 35k samples times
five channels, resulting in a total of over 177k audio samples. The anomalies
were produced by an intentionally damaging shaft, gears, and tires of the
machine to the depth three levels. This resulted in producing over 8k samples
of anomalous audios, including 300 patterns of their combinations.

####  4.9.2 Toy train:

The toy train sub-dataset is designed for the detection of faults in a moving
machine. Here, similar to the toy car sub-dataset, each recorded sample is of
a duration of 12 seconds. Thus, there are a total of 35k samples times two mic
types, resulting in a total of over 71k audio samples. The anomalies were
produced by an intentionally damaging shaft, gears, and tires of the machine
to the depth three levels. This resulted in producing over 8k samples of
anomalous audios, including 300 patterns of their combinations.

The anomalous conditions for each sub-dataset are described in Table 13.

Table 13: Anomalous conditions for each miniature machine in the ToyADMOS2 dataset Toy car | Toy train
---|---
Parts | Anomaly | Parts | Anomaly
Shaft | -Bent | Carriage | -Flat tire
Wheels | -Damaged |  | -Broken shaft
Gears | -Deformed | Railway track | -Disjointed
| -Melted |  | -Obstructing stone

###  4.10 DCASE 2021 dataset

The 2021 DCASE Challenge Task 2 [176] was released in the year 2021. It
comprises normal/ anomalous operating sounds from seven different kinds of
machines, viz., Fan, Gearbox, Pump, Slide rail (also called slider), ToyCar,
ToyTrain, and Valve. Each recording in the dataset is 10 seconds, single-
channel audio consisting of sounds of a machine as well as its related
equipment and environmental sounds. The dataset is categorized into three
subsets: development dataset, additional training dataset, and evaluation
dataset.

####  4.10.1 Development dataset

The development dataset includes three sections for each type of machine,
where each section comprises training as well as test data. Each section
comprises of (i) around 1000 clips of normal audio in a source domain for the
purpose of training, (ii) three clips of normal audio in a target domain for
training, (iii) around 100 clips of both normal as well as anomalous audio in
the source domain for testing, and (iv) around 100 clips of both normal as
well as anomalous audio in the target domain for testing.

####  4.10.2 Additional training dataset

Additional training dataset comprises another three sections for each type of
machine. Each section includes (i) around 1000 normal audio clips in the
source domain for training and (ii) three clips of normal audio clips in the
target domain for training.

####  4.10.3 Evaluation dataset

The evaluation dataset provides test clips for three sections for each machine
type, similar to the additional training dataset. Here, each section comprises
(i) test audio clips in the source domain and (ii) test audio clips in the
target domain, where none of the clips has a condition label (i.e., normal or
anomaly).

##  5 Discussion on Audio datasets

In this section, a discussion on various audio datasets described above has
been carried out. Table 14 gives a description of the datasets based on
applications. As it can be seen from the table, so far, most of the anomaly
detection applications have used DCASE 2017 database, followed by the MIVIA
road audio event dataset. The datasets have been used for applications in
different fields, such as in industries for machine monitoring and inspection,
road surveillance, etc. A dataset for anomaly detection in respiratory
patterns in human bodies has also been created. Most of the datasets have been
artificially generated by deliberately adding anomalies and made available to
the public recently. The year of release for every dataset has also been
mentioned in the table.

Table 14: Applications of various audio datasets Dataset | Year of | # of | Areas of application
---|---|---|---
| release | references |
MIVIA | 2014 | 184 | applications in road surveillance [177, 178, 179]
AudioSet Soundscape | 2015 | 25 | anomaly detection from eco-acoustic data [165]
DCASE 2017 | 2017 | 407 | detection of rare sounds [180, 181]
ICBHI 2017 | 2017 | 94 | anomalous respiratory pattern detection in humans [182, 183, 184]
SMD | 2018 | 70 | machine monitoring in industries [171]
MIMII DUE | 2019 | 24 | investigation and inspection of industrial machine
ToyADMOS | 2019 | 81 | anomalous sound detection in miniature machines [185, 186]
LIFE DYNAMAP | 2020 | 31 | detection and removal of anomalous events for road traffic mapping [187, 188]
ToyADMOS2 | 2021 | 24 | anomalous sound detection in miniature machines
DCASE 2021 | 2021 | 25 | detection of anomalous sounds during machine monitoring [189, 190, 191]

Table 15 describes expressively the type of anomalies that are present in
various datasets, along with some examples. As it can be seen from the table,
most of the audio datasets have been created artificially, except for ICBHI
2017 dataset, which records the natural respiratory sounds from a human body.
The datasets have been created at various indoor and outdoor locations, such
as industry, roads, forests, etc. Since it is difficult to generate natural
audio datasets consisting of anomalies, hence anomalies have been generated
deliberately to simulate the system for its detection and for future anomaly
detection applications. Some of the anomalies that have been generated are
mentioned for every dataset in the table.

Table 15: Comparison of audio datasets based on anomaly information

Dataset | Scene |  | Indoor/
---
Outdoor
| Nature of
---
anomaly
| # of
---
anomalies
Example anomalies
MIVIA | road | outdoor | artificial | 2 | crashes and tire skidding
AudioSet Soundscape | nature | outdoor | both | random | music, human speech, machine noise, etc.
DCASE 2017 | city | both | artificial | 3 | baby cry, glass break, gun shot
ICBHI 2017 | human body | indoor | natural | 3 | crackle, wheeze, crackle and wheeze
SMD | industry | indoor | both | 3 | non-greased line, B-line, C-line to B-line
MIMII DUE | industry | indoor | artificial |  $>$15 | wing damage, clogging, gear, contamination etc.
ToyADMOS | industry | indoor | artificial | 23 | deformed gears, over/under voltage, pulley,
|  |  |  |  | chipped wheel, axle, excessive tension, etc.
LIFE DYNAMAP dataset | road | outdoor | both | 15 | horns, church bells, birds, thunder etc.
ToyADMOS2 | industry | indoor | artificial | 8 | bent shaft, melted gears, flat tire etc.
DCASE 2021 | industry | indoor | artificial |  $>$30 | wing damage, clogging, chipped wheel, axle etc.

Table 16 gives the specifications of the audio samples collected for different
datasets. The specifications include the time duration of each dataset, the
number of samples collected, and the sampling rate for audio-digitization. It
can be from the table that the duration of audio for different datasets varies
from a few seconds to a number of hours. AudioSet soundscape dataset has the
longest duration of audios, followed by ToyADMOS2 and ToyADMOS datasets. A
very small recording of 550 seconds has been collected for the LIFE DYNAMAP
project.

Further, from the table, it can be seen that the sampling rate for different
audio datasets varies between the range of 4kHz to 48 kHz. The sampling rate
of 48 kHz has been mostly considered for audio digitization in various
datasets. Also, for ICBHI 2017, there is no fixed value of sampling rate for
the data; rather, it varies between the range of 4 to 44.1 kHz. Thus,
different sampling rates have been used as per the requirement of the tasks.

Table 16: Specifications of audio datasets Dataset | Total duration | No. of samples | Sampling rate
---|---|---|---
MIVIA | 3580 seconds | - | 32 kHz
AudioSet Soundscape: Ithaca, New York | #1: 797 hrs | - | 48 kHz
| #2: 638 hrs | - | 48 kHz
AudioSet Soundscape: Sabah, Malaysia | Tuscam: 27 hrs 40 min | - | 44.1 kHz
| Audiomoth: 784 hrs | - | 16 kHz
AudioSet Soundscape: New Zealand | 240 hrs | - | 32 kHz
AudioSet Soundscape: Sulawesi | 64 hrs | - | 48 kHz
AudioSet Soundscape: Republic of Congo | 238 hrs 20 min | - | 8 kHz
DCASE 2017 | 39 min | 1170 | 44.1 kHz
ICBHI 2017 |  $>$ 5.5 hrs | - | 4 to 44.1 kHz
SMD |  $\approx$ 1 hour | 2048 | 16.3 kHz
MIMII DUE |  $>$420000 sec | 32157 | 16 kHz
ToyADMOS |  $\approx$ 540 hrs |  $>$ 12000 | 48 kHz
LIFE DYNAMAP dataset | 550 sec | - | 48 kHz
ToyADMOS2 | 604 hrs |  $\approx$264k | 48 kHz
DCASE 2021 |  $\approx$82 hours | 29463 | 16 kHz

##  6 Audio-visual Datasets

So far, we have discussed a number of publicly available datasets for video
and audio anomaly detection applications individually. Since a course of
actions consist of both audio and video components dependant on each other,
hence, audio-visual analysis can result in more accurate results compared to
audio and video recordings being used independently. In this present section,
we describe the available audio-visual datasets for anomaly detection in
various applications. All the public audio-visual datasets contain specific
kind of anomalies, therefore no further categorization is done.

###  6.1 Human–human Interactions dataset777Available upon request

The Human-human Interaction dataset [192] developed in 2014 aimed at stressful
situation detection in human-human interactions at help-desks. Generally, this
stressful situation which is also regarded anomaly, is generated due to
various reasons such as there is a case of emergency, communication is not
running smoothly, or problems are not solved as expected, etc. These
situations are reflected in hand gestures and speech, which also reflect the
emotional state of a person. Thus analyzing gestures and speech can also help
in detecting stressful situations, i.e., anomalies. Authors create such a
dataset with the help of a multicultural group of nine professional actors (4
male, 5 female). Four scripted scenarios were designed and asked to be
performed by the actors that create a stressful situation in human-human
interaction. The actors were divided into two disjoint groups and played each
script twice; thus, a total of eight sessions were recorded, resulting in
32.24 minutes. Each scene has an employee and a customer. In few cases, there
were three visitors in front of the service desk. Two cameras, one facing to
the customer’s face and the other to the employee and the complete scene, were
installed. Each person has a microphone clipped to their shirt. Representative
frames for both the camera are shown in Fig. 51. Video of the customer, i.e.,
the image on the left of Fig. 51, is taken for analysis in the paper [192].
The audio has Dutch and English speech depending upon one’s nationality.
Moreover, they were not provided any detailed script of conversation but asked
to create a source of conflict in between. The recordings were annotated on a
scale of 5, where label 1 was attributed to normal situations, 2 and 3 to
moderate stress levels, and 4 and 5 to extremely negative situations.

Fig. 51: First camera showing the details of the visitor’s behavior (left) and
an overview shot by the second camera (right) in Human–human Interactions
dataset.

###  6.2 VSD dataset

The VSD dataset [193] released in 2015, is an audio-visual dataset for
violence detection. The dataset is developed as a part of the MediaEval Affect
task for the detection of violent scenes in movies. The clips are curated from
18 movies from the different genres such as extremely violent ones, war
movies, action movies, movies with no violent content, etc. The total duration
of the training set is 28 hours 34 minutes, and that of the test set is 6
hours 44 minutes. Seven visual violence categories and three audio violence
categories were identified. For video, they are as follows: the presence of
blood, fights, presence of fire, presence of guns, presence of cold weapons,
car chases, and gory scenes; for audio, they are: gunshots, explosions, and
screams. The start and end frames of these violent events are annotated.

Fig. 52: Representative frame for the VSD dataset (image source [193])

###  6.3 EMOLY dataset

The EMOLY dataset [194] was published in 2018. The dataset is made of speech
and facial video records of subjects which also contains controlled anomalies.
This is the first dataset of its kind. The dataset constitutes 123 audio-
visual recordings of face and voice from 41 student participants (11 females
and 30 males) from the age group of 22±2 years. Each participant had to read
11 slides made from the French version of the tale ”the handless maiden” (”La
jeune fille sans main”) from Brothers Grimm. 37 participants were native
French speakers, and 4 were non-native French speakers. Anomalous facial
expression and speech were triggered by inducing anomalous content in the
slide show. Participants were unaware of this aim. They were just asked to
read the tale to the camera as if they were telling the tale to a child. The
purpose was to get expressive speech and facial expressions from them. The
dataset was collected in three rounds, i.e., each subject went through all
slides three times. In the first round, the session was without anomaly. In
the second session, different backgrounds in the slides and rotated text were
incorporated to induce anomalies in participants. In the third session (done
after a week delay), they were asked to perform anomalous acting of their
choice. Annotation is done with the help of eleven members of the EXPRESSION
research team at IRISA. Manual annotations of the sample on which an anomaly
inductor has been triggered are made available by the authors. Additional
information about each subject, such as age, gender, french native, self-
measurement of oral abilities, etc., are also available.

Fig. 53: Representative frame for the EMOLY dataset. Face reaction at the
beginning of the slide containing the anomaly induction for the normal session
(left) and the induced session (right).

###  6.4 XD-Violence dataset

The XD-Violence dataset [195], released in the year 2020, is the largest
audio-visual dataset for violence detection. Authors consider six classes of
violence, namely Abuse, Car Accident, Explosion, Fighting, Riot, and Shooting.
The choice is such that they are frequently occurring and important from a
safety point. The clips are collected from movies and YouTube (in-the-wild
scenes), resulting in violence from various sources such as movies, cartoons,
sports, games, music, fitness, news, live scenes, captured by CCTV cameras,
captured by hand-held cameras, captured by car driving recorders, etc. A total
of 91 movies (violent and non-violent) were used to collect violent and non-
violent events. Further, text search queries were used to collect both types
of events from YouTube. The dataset has 2405 violent videos and 2349 non-
violent videos, resulting in a total duration of 217 hours. The author also
specified a train-test split as 3954 videos (1905 violent and 2049 non-
violent) in the training set and 800 videos (500 violent and 300 non-violent)
in the test set. For each violent video, labeling is done with the help of
multiple human annotators. They marked the start and end frame of the violent
event. There can be multiple labels to the same frame, i.e., multiple types of
violence can co-occur; hence multiple labels are available. Some of the sample
frames representing violence and non-violence are shown in Fig. 54.

Fig. 54: Representative frame for the XD-Violence dataset (image source [195])

###  6.5 BAREM dataset999Available upon request

BAREM stands for Behaviour Analysis for Reverse Efficient Modeling. Released
in the year 2021, BAREM is an audio-visual dataset consisting of humans
interacting with the e-service platform. BAREM dataset was designed for
automatic detection of changes in the behavior of a user while interacting
with e-service [196]. Thus, the level of frustration was considered an anomaly
for the task. A total of 18 subjects, consisting of 11 men and 7 women, were
considered for collecting the database. The dataset has been created in a
geographical area that is unfamiliar to the subject. The main target was to
create a frustrating environment for the subjects by proposing frustrating and
challenging tasks within a time frame of 20 minutes. Four tasks have been
created, as mentioned in Table 17. Thus, with each subject being given the
mentioned tasks, a total of 72 videos were generated. Two annotations have