task [3.8em] 2.3em

#  hyppo: A Multivariate Hypothesis Testing Python Package

Sambit Panda1 Satish Palaniappan1 Junhao Xiong1 Eric W. Bridgeford2 Ronak
Mehta1 Cencheng Shen3 and Joshua T. Vogelstein1,4 Corresponding author:
<EMAIL_ADDRESS>Johns Hopkins University, Baltimore, MD 2 Johns Hopkins School
of Public Health, Baltimore, MD 3 University of Delaware, Newark, DE 4
Progressive Learning

###### Abstract

We introduce hyppo, a unified library for performing multivariate hypothesis
testing, including independence, two-sample, and k-sample testing. While many
multivariate independence tests have R packages available, the interfaces are
inconsistent and most are not available in Python. hyppo includes many state
of the art multivariate testing procedures. The package is easy-to-use and is
flexible enough to enable future extensions. The documentation and all
releases are available at https://hyppo.neurodata.io.

###### keywords:

Python, multivariate, independence, k-sample, hypothesis

##  1 Introduction

Examining and identifying relationships between sets of high-dimensional
variables is critical to advance understanding and planning of future
numerical and physical experiments. Hypothesis testing enables formally
testing models to identify such discrepancies.

Many correlation measures have been proposed the problem of independence
testing, such as Pearson’s correlation [1], but many are unsuited to detect
nonlinear and high-dimensional dependence structures within data. Recently,
several statistics have been proposed that operate well on high-dimensional
(potentially non-Euclidean) data, such as distance correlation [2, 3, 4, 5]
and Hilbert-Schmidt independence criterion [6, 7, 8], which are actually
exactly equivalent in Sejdinovic et al. [9], Shen and Vogelstein [10]. Heller,
Heller and Gofrine proposed another nonparametric independence test with
particularly high power in certain nonlinear relationships [11]. Multiscale
Graph Correlation is a test that has demonstrated higher statistical power on
many multivariate, nonlinear, and structured data when compared to other
independence tests [12, 13, 14], which combines and extends the nearest
neighbors and energy statistics to detect underlying relationships. For each
of these tests, p-values can be calculated using a random permutation test
[15, 16, 17]. These tests can be modified and extended to such applications as
time-series testing [18].

To approach the problem of two-sample testing, Student’s t-test [19] is
traditionally used, while a few nonparametric alternatives have been proposed
that operate well on multivariate, nonlinear data such as Energy [20], and
maxmimal mean discrepency [21], and Heller Heller and Gorfine’s test [11]. The
two-sample testing problem can be generalized to the k-sample testing problem
and here analysis of variance (ANOVA) [22] or its multivariate analogue
multivariate ANOVA (MANOVA) [23] can be used, but these statistics either fail
to, or operate poorly upon, multivariate and nonlinear data. In addition,
ANOVA and MANOVA in particular suffer from fundamental assumptions that are
not generally present in real data [24, 25]. There are a few nonparametric
alternatives to ANOVA and MANOVA, such as multivariate k-sample Heller Heller
Gorfine [26], and distance components (DISCO) [27]. Recently, Shen et al. [28]
has shown that nonparametric distance and kernel k-sample tests can be
formulated by reducing the k-sample testing problem to the independence
testing problem.

This manuscript introduces hyppo, a hypothesis package that provides various
tests with high statistical power on multidimensional and nonlinear data.
hyppo is a well-tested, multi-platform, Python 3 compatible library that
allows users to conduct hypothesis tests on their data, and is also flexible
enough to allow developers to easily add in their own tests. hyppo also
provides benchmarks for each of these tests by comparing statistical power
over many statistical models. The contribution of this manuscript is therefore
to provide: (1) an overview of the library and examples of how to use some of
the tests in the package, and (2) comparisons of the test statistics and wall
times with similar R packages.

##  2 Library Overview

Inspired by the desire to allow for convenient use of these independence
tests, hyppo has been developed as a hypothesis testing package. The package
structured is modeled on the scikit-learn and energy R packages’ API. Links to
source code, documentation, and tutorials can be found here:
https://hyppo.neurodata.io.

### Included Tests

We have included a host of notable and novel hypothesis tests that we
determined to be useful for the end user. Shen and Vogelstein [10] has shown
that distance and kernel methods are equivalent and thus, we have 1
implementation that is able to perform both with a proper bijective
transformation. We have implemented k-sample tests as specified in Panda et
al. [29] and every algorithm in the following list can be used as a k-sample
test this way. The included algorithms are:

  * •

Multivariate generalizations of Pearson’s product moment correlation: RV [30,
31] and Cannonical correlation analysis (Cca) [32].

  * •

Heller-Heller-Gorfine (Hhg) [11]: Multivariate distance-based test.

  * •

Distance correlation (Dcorr): Both biased [4] and unbiased [33] and a fast
$\mathcal{O}(n\log n)$ variant that runs on euclidean one-dimensional data
[34]. A chi-square fast statistic is also implemented [35].

  * •

Hilbert-Schmidt independence criterion (Hsic): Both biased and unbiased [36]
kernel-based statistics. A chi-square fast statistic is also implemented [35].

  * •

Multiscale graph correlation (Mgc) [14]: An independence tests that combines
k-nearest neighbors and energy statistics. Recently, Mgc has been accepted
into scipy.stats and this implementation wraps the scipy implementation.

  * •

Multivariate analysis of variance (Manova) [37, 24] and Hotelling $T^{2}$
(Hotelling) [38].

  * •

Maximum mean discrepancy (Mmd) [21]: A kernel two-sample test.

  * •

Energy [20]: A distance two-sample test.

  * •

Distance components (Disco) [27]: A distance-based k-sample test.

  * •

Discriminability test [39]: A highly accurate and powerful discriminability
test.

A number of algorithms have been implemented that lack an open source
implementation elsewhere. These include:

  * •

Kernel k-sample tests: k-sample Hsic and k-sample Mgc formulated using the
k-sample formulation as above [29].

  * •

Kernel mean embedding random forest (Kmerf) [40]: A kernel test that leverages
random forest kernel induced similarity matrix to generate a test statistic.

  * •

Fast Implementations of Dcorr (Fast Dcorr) [41]: An approximation to Dcorr
when calculating the p-value.

  * •

Time-series Mgc and Dcorr: Applying Mgc and Dcorr to time-series data.

  * •

Maximal Margin Correlation (MaxMargin) [42]: A highly accurate formulation of
independence tests in high dimensions with minimal added computational
complexity.

### Structure of hyppo

The modules of hyppo are as follows: independence, ksample, discrim,
time_series, and tools. Each test within hyppo contains a .test method which
the user runs that returns at least a statistic and p-value in all cases, and
parallelization of the permutation tests are performed using joblib. Test
statistic code is compiled and cached with numba, which is a just-in-time
(JIT) compiler. tools contains common helper functions, simulations, and
finite-sample statistical power functions to test differences between each
test in hyppo.

##  3 Benchmarks

|
---|---

Figure 1: Benchmarks of hyppo implementations against corresponding R
implementations. Average wall times (over 3 repetitions) (left) are shown for
Dcorr in energy and kernlab as compared against hyppo implementations of Mgc,
Dcorr, Fast Mgc, and Fast Dcorr. Test statistic comparisons (right) between
Dcorr, Mmd, and Hhg in hyppo are compared against their respective reference R
implementations. Test statistics are nearly identical for each implementation.

### Wall Time Comparisons

Figure 1a shows the computational efficiency of hyppo’s implementations
against existing implementations in commonly used R packages—specifically
energy [43], kernlab [44], and HHG [45]. When comparing performance, wall
times are averages of p-value computations (1000 replications when permutation
tests are used) 3 trials calculated on a univariate noisy linear simulation
with number of samples increasing from 50 to 10,000. All computations were
performed on an Ubuntu 18.04.3 LTS system with access to 96 cores. When sample
sizes are above a few hundred, all algorithms achieve approximately quadratic
times, with different slopes. Hhg was the slowest as expected, though had
comparable speeds to the other algorithms at low sample sizes. Mgc and Dcorr
are next, and still only requires tens of minutes to run when sample sizes are
around 10,000. At low sample sizes, the energy package’s Dcorr is faster than
kernlab’s implementation of Mmd (Dcorr is equivalent to Mmd for all finite
sample sizes [10]) even at a sample size of 10,000. hyppo’s Fast Dcorr, which
uses a fast statistic [34] and p-value approximation [35] is the fastest, even
though both energy and kernlab both use highly optimized C++ versions.

### Implementation Validation

Next, we verify that hyppo’s test statistics are equivalent to existing R
implementations of the tests. Specifically, hyppo’s implementations were
compared to: Dcorr from the energy package [43]. Mmd from the kernlab package
[44], and Hhg from the HHG package [45]. The evaluation uses a spiral
simulation with 1000 samples and 2 dimensions for each test and compares test
statistics over 20 repetitions. Figure 1b shows the difference between the
hyppo implementation of the independence test and the respective R package
implementation of the independence test. Test statistics are nearly equivalent
for each implementation.

##  4 Conclusion

hyppo is an extensive and extensible open-source Python package for
multivariate hypothesis testing. As hyppo continues to grow and add
functionality, it will enhance tools scientists use when determining
relationships within their investigations.

### Acknowledgements

This work is graciously supported by the Defense Advanced Research Projects
Agency (DARPA) Lifelong Learning Machines program through contract
FA8650-18-2-7834, the National Institute of Health awards RO1MH120482 amd
T32GM119998, and the National Science Foundation award DMS-1921310. The
authors would like to acknowledge the NeuroData Design class and the NeuroData
lab at Johns Hopkins University for helpful feedback.

## References

  * Pearson [1895] Karl Pearson.  Note on regression and inheritance in the case of two parents.  _Proceedings of the Royal Society of London_ , 58(347-352):240–242, 1895.
  * Székely and Rizzo [2009] Gábor J Székely and Maria L Rizzo.  Brownian distance covariance.  _The Annals of Applied Statistics_ , 3(4):1236–1265, 2009.
  * Székely and Rizzo [2013a] Gábor J Székely and Maria L Rizzo.  The distance correlation t-test of independence in high dimension.  _Journal of Multivariate Analysis_ , 117:193–213, 2013a.
  * Székely et al. [2007] Gábor J Székely, Maria L Rizzo, and Nail K Bakirov.  Measuring and testing dependence by correlation of distances.  _The Annals of Statistics_ , 35(6):2769–2794, 2007.
  * Lyons [2013] Russell Lyons.  Distance covariance in metric spaces.  _The Annals of Probability_ , 41(5):3284–3305, 2013.
  * Gretton and László [2010] Arthur Gretton and Györfi László.  Consistent nonparametric tests of independence.  _Journal of Machine Learning Research_ , 11(Apr):1391–1423, 2010.
  * Gretton et al. [2005] Arthur Gretton, Ralf Herbrich, Alexander Smola, Olivier Bousquet, and Bernhard Schölkopf.  Kernel methods for measuring independence.  _Journal of Machine Learning Research_ , 6(Dec):2075–2129, 2005.
  * Muandet et al. [2017] Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schölkopf.  Kernel mean embedding of distributions: A review and beyond.  _Foundations and Trends® in Machine Learning_ , 10(1-2):1–141, 2017.
  * Sejdinovic et al. [2013] Dino Sejdinovic, Bharath Sriperumbudur, Arthur Gretton, and Kenji Fukumizu.  Equivalence of distance-based and rkhs-based statistics in hypothesis testing.  _The Annals of Statistics_ , 41(5):2263–2291, 2013.
  * Shen and Vogelstein [2018] Cencheng Shen and Joshua T Vogelstein.  The exact equivalence of distance and kernel methods for hypothesis testing.  _arXiv preprint arXiv:1806.05514_ , 2018.
  * Heller et al. [2012] Ruth Heller, Yair Heller, and Malka Gorfine.  A consistent multivariate test of association based on ranks of distances.  _Biometrika_ , 100(2):503–510, 2012.
  * Shen et al. [2020a] Cencheng Shen, Carey E Priebe, and Joshua T Vogelstein.  From distance correlation to multiscale graph correlation.  _Journal of the American Statistical Association_ , 115(529):280–291, 2020a.
  * Lee et al. [2019] Youjin Lee, Cencheng Shen, Carey E Priebe, and Joshua T Vogelstein.  Network dependence testing via diffusion maps and distance-based correlations.  _Biometrika_ , 106(4):857–873, 2019.
  * Vogelstein et al. [2019] Joshua T Vogelstein, Eric W Bridgeford, Qing Wang, Carey E Priebe, Mauro Maggioni, and Cencheng Shen.  Discovering and deciphering relationships across disparate data modalities.  _eLife_ , 8:e41690, 2019.
  * Collingridge [2013] Dave S Collingridge.  A primer on quantitized data analysis and permutation testing.  _Journal of Mixed Methods Research_ , 7(1):81–97, 2013.
  * Dwass [1957] Meyer Dwass.  Modified randomization tests for nonparametric hypotheses.  _The Annals of Mathematical Statistics_ , pages 181–187, 1957.
  * Good [2006] Phillip I Good.  _Permutation, parametric, and bootstrap tests of hypotheses_.  Springer Science & Business Media, 2006.
  * Mehta et al. [2019] Ronak Mehta, Cencheng Shen, Ting Xu, and Joghua T Vogelstein.  A consistent independence test for multivariate time-series.  _arXiv preprint arXiv:1908.06486_ , 2019.
  * Student [1908] Student.  The probable error of a mean.  _Biometrika_ , pages 1–25, 1908.
  * Székely and Rizzo [2013b] Gábor J Székely and Maria L Rizzo.  Energy statistics: A class of statistics based on distances.  _Journal of statistical planning and inference_ , 143(8):1249–1272, 2013b.
  * Gretton et al. [2012] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  _Journal of Machine Learning Research_ , 13(Mar):723–773, 2012.
  * Fisher [1919] Ronald A Fisher.  Xv.—the correlation between relatives on the supposition of mendelian inheritance.  _Earth and Environmental Science Transactions of the Royal Society of Edinburgh_ , 52(2):399–433, 1919.
  * Bartlett [1947] Maurice S Bartlett.  Multivariate analysis.  _Supplement to the journal of the royal statistical society_ , 9(2):176–197, 1947.
  * Warne [2014] Russell Warne.  A primer on multivariate analysis of variance (manova) for behavioral scientists.  _Practical Assessment, Research, and Evaluation_ , 19(1):17, 2014.
  * Stevens [2002] JP Stevens.  Applied multivariate statistics for the social sciences. lawrence erlbaum.  _Mahwah, NJ_ , pages 510–1, 2002.
  * Heller et al. [2016] Ruth Heller, Yair Heller, Shachar Kaufman, Barak Brill, and Malka Gorfine.  Consistent distribution-free k-sample and independence tests for univariate random variables.  _The Journal of Machine Learning Research_ , 17(1):978–1031, 2016.
  * Rizzo et al. [2010] Maria L Rizzo, Gábor J Székely, et al.  Disco analysis: A nonparametric extension of analysis of variance.  _The Annals of Applied Statistics_ , 4(2):1034–1055, 2010.
  * Shen et al. [2019] Cencheng Shen, Carey E Priebe, and Joshua T Vogelstein.  The exact equivalence of independence testing and two-sample testing.  _arXiv preprint arXiv:1910.08883_ , 2019.
  * Panda et al. [2020] Sambit Panda, Cencheng Shen, Ronan Perry, Jelle Zorn, Antoine Lutz, Carey E. Priebe, and Joshua T. Vogelstein.  Nonparametric manova via independence testing, 2020.
  * Escoufier [1973] Yves Escoufier.  Le traitement des variables vectorielles.  _Biometrics_ , pages 751–760, 1973.
  * Robert and Escoufier [1976] Paul Robert and Yves Escoufier.  A unifying tool for linear multivariate statistical methods: the rv-coefficient.  _Journal of the Royal Statistical Society: Series C (Applied Statistics)_ , 25(3):257–265, 1976.
  * Hardoon et al. [2004] David R Hardoon, Sandor Szedmak, and John Shawe-Taylor.  Canonical correlation analysis: An overview with application to learning methods.  _Neural computation_ , 16(12):2639–2664, 2004\.
  * Székely et al. [2014] Gábor J Székely, Maria L Rizzo, et al.  Partial distance correlation with methods for dissimilarities.  _The Annals of Statistics_ , 42(6):2382–2412, 2014.
  * Chaudhuri and Hu [2019] Arin Chaudhuri and Wenhao Hu.  A fast algorithm for computing distance correlation.  _Computational Statistics & Data Analysis_, 135:15 – 24, 2019.  ISSN 0167-9473.  https://doi.org/10.1016/j.csda.2019.01.016.  URL http://www.sciencedirect.com/science/article/pii/S0167947319300313.
  * Shen and Vogelstein [2020] Cencheng Shen and Joshua T. Vogelstein.  The chi-square test of distance correlation, 2020.
  * Gretton et al. [2008] Arthur Gretton, Kenji Fukumizu, Choon H Teo, Le Song, Bernhard Schölkopf, and Alex J Smola.  A kernel statistical test of independence.  In _Advances in neural information processing systems_ , pages 585–592, 2008.
  * Carey [1998] Gregory Carey.  Multivariate analysis of variance (manova): I. theory.  _Retrieved May_ , 14:2011, 1998.
  * Hotelling [1992] Harold Hotelling.  The generalization of student’s ratio.  In _Breakthroughs in statistics_ , pages 54–65. Springer, 1992.
  * Wang et al. [2020] Zeyi Wang, Eric Bridgeford, Shangsi Wang, Joshua T. Vogelstein, and Brian Caffo.  Statistical analysis of data repeatability measures, 2020.
  * Shen et al. [2020b] Cencheng Shen, Sambit Panda, and Joshua T. Vogelstein.  Learning interpretable characteristic kernels via decision forests, 2020b.
  * Shen [2020a] Cencheng Shen.  High-dimensional independence testing and maximum marginal correlation.  _arXiv preprint arXiv:2001.01095_ , 2020a.
  * Shen [2020b] Cencheng Shen.  High-dimensional independence testing and maximum marginal correlation, 2020b.
  * Rizzo and Szekely [2018] Maria Rizzo and Gabor Szekely.  _energy: E-Statistics: Multivariate Inference via the Energy of Data_ , 2018.  URL https://CRAN.R-project.org/package=energy.  R package version 1.7-5.
  * Karatzoglou et al. [2004] Alexandros Karatzoglou, Alex Smola, Kurt Hornik, and Achim Zeileis.  kernlab – an S4 package for kernel methods in R.  _Journal of Statistical Software_ , 11(9):1–20, 2004.  URL http://www.jstatsoft.org/v11/i09/.
  * Brill and Kaufman [2019] Barak Brill and Shachar Kaufman.  _HHG: Heller-Heller-Gorfine Tests of Independence and Equality of Distributions_ , 2019.  URL https://CRAN.R-project.org/package=HHG.  R package version 2.3.2.


# Crime scene classification from skeletal trajectory analysis in surveillance
settings

Alina-Daniela Matei<EMAIL_ADDRESS>Estefanía Talavera
<EMAIL_ADDRESS>Maya Aghaei<EMAIL_ADDRESS>Bernoulli Institute, University of Groningen, Nijenborgh 9, 9747 AG,
Groningen, The Netherlands  Faculty of Electrical Engineering, Mathematics and
Computer Science of the University of Twente, Drienerlolaan 5, 7522 NB,
Enschede, The Netherlands  Computer Vision and Data Science Professorship, NHL
Stenden University, Rengerslaan 10, 8917 DD Leeuwarden, The Netherlands

###### Abstract

Video anomaly analysis is a core task actively pursued in the field of
computer vision, with applications extending to real-world crime detection in
surveillance footage. In this work, we address the task of human-related crime
classification. In our proposed approach, the human body in video frames,
represented as skeletal joints trajectories, is used as the main source of
exploration. First, we introduce the significance of extending the ground
truth labels for HR-Crime dataset and hence, propose a supervised and
unsupervised methodology to generate trajectory-level ground truth labels.
Next, given the availability of the trajectory-level ground truth, we
introduce a trajectory-based crime classification framework. Ablation studies
are conducted with various architectures and feature fusion strategies for the
representation of the human trajectories. The conducted experiments
demonstrate the feasibility of the task and pave the path for further research
in the field.

###### keywords:

Forensics, Human-related crime classification, Human behaviour analysis,
Surveillance videos

##  1 Introduction

From the pattern recognition perspective, anomaly detection involves detecting
events that do not conform to an expected behaviour (Chandola et al., 2009).
Nonetheless, a universal definition of what the term ‘anomaly’ entails is hard
to achieve due to the general nature of abnormal events. Given the wide range
of content that the ‘anomaly’ term can refer to, the task of anomaly detection
is relevant to a multitude of areas of research: violence detection (Febin et
al., 2020), car and traffic accident detection (Yun et al., 2014), crime
detection (Loganathan et al., 2019) or security screening (Akçay et al.,
2019), among others.

Despite the intricacies involved with anomaly detection, many efforts have
been targeted towards automatizing this task due to its significant
applicability in the field of forensics, especially within the video
surveillance setting (Ramachandra et al., 2020)(Ahmed et al., 2018)(Lee et
al., 2018). Surveillance cameras capturing continuous footage of public spaces
are increasingly being used to ensure public safety (Lin and Purnell, 2019).
Nonetheless, the human monitoring capabilities of the security agencies are
overwhelmed by the big volume of video surveillance data to be supervised
(Collins et al., 2000). This leads to surveillance inefficiencies and,
eventually, to undesirable situations that could have been prevented given the
human monitors-cameras ratio would have been kept within manageable bounds
(Collins et al., 2000). This human monitoring deficiency can be compensated by
automating the monitoring process: ultimately, real-time anomaly detection and
prediction could be employed (Arroyo et al., 2015) (Rabie, 1990).

Focusing on the relevance of surveillance video data in the context of anomaly
detection, one significant aspect is that this type of data captures a wide
range of unrestricted and unscripted human behaviour in a variety of scenes
and situations. In this ample and accessible body of data, we focus on human-
related anomaly detection. This task is closely related to human activity
recognition and targets the identification of abnormal activities through the
analysis of human behavior (Emonet et al., 2011).

To date, human-related anomaly detection received limited attention from the
research community, which perhaps can be motivated by the lack of human-
related anomaly datasets. Recently, the novel HR-Crime dataset was introduced
in (Boekhoudt et al., 2021). HR-Crime, as the human-related fraction of the
original UCF-Crime (Sultani et al., 2018a), focuses on human-related criminal
activities in surveillance videos. The availability of the novel HR-Crime
dataset enables creating a benchmark for human-centered anomaly detection
(Boekhoudt et al., 2021). Another notable step towards the detection of human-
related anomalies is the work of (Morais et al., 2019), which is centered on
human body movements within the surveillance videos.

In this work, we aim at human-related anomaly classification into -unevenly
distributed- crime categories in HR-Crime. To this end, we build on top of
(Morais et al., 2019) feature representation aiming to discover human body
behavioural patterns that could uniquely identify a depicted crime. Our
contributions are as follows:

  1. 1.

We propose a supervised as well as an unsupervised approach for expanding the
HR-Crime ground truth with trajectory-level annotations.

  2. 2.

We propose various data augmentation techniques to mitigate the dataset
imbalance of HR-Crime.

  3. 3.

We build on top of the MPED-RNN architecture (Morais et al., 2019),
introducing a crime classification framework based on the skeletal trajectory
representation of human bodies in surveillance videos.

  4. 4.

We conduct experiments with varying architectures, trajectory representations
and feature fusions for the task of human-related crime classification and
their distinction from human normal (non-crime) behaviour.

  5. 5.

We exhaustively discuss the obtained results, also from the social science
perspective, and provide insights within the context of crime scenes.

The rest of the paper is structured as follows: Section 2 describes the
related works. The dataset and our proposed pipeline for trajectory-level
annotation and dataset augmentation are described in Section 3. Section 4
introduces the proposed methodology together with the corresponding
experimental framework. In Section 5, we discuss the obtained results.
Finally, Section 6 draws our final conclusions.

##  2 Related work

A common methodology widely applied for anomaly detection is novelty
detection. This involves training the detection system with what is considered
as ‘normal’ events, which sets the reference point for the later detection of
particularities in the environment. The anomaly detection system is
essentially trained to accurately reproduce the ‘normal’ video data, hence
yields in high errors when faced with anomalous videos (Markou and Singh,
2003). This approach generally entails the use of encoder-decoder
architectures (Malhotra et al., 2016).

A high-level taxonomy of anomaly detection methodologies broadly distinguishes
between traditional and deep learning methods. Traditional methods entail the
extraction and analysis of features such as optical flow (Zhou and Kwan, 2018)
or histograms of oriented gradients (Dalal and Triggs, 2005) in order to
detect anomalies in video settings. More recently, deep learning techniques
have been vastly exploited for the task of anomaly detection (Hasan et al.,
2016; Luo et al., 2017). One popular approach entails using Generative
Adversarial Networks (GANs) within the novelty detection approach (Doshi and
Yilmaz, 2021). Works such as (Akcay et al., 2018) are used to generate
internal frame representations based on a given frame and its optical flow.
Apart from GANs, Convolutional Neural Networks (CNNs) and Long-Short Term
Memory (LSTM) networks are used for features extraction (e.g. motion and
visual features) and anomaly detection in (Ullah et al., 2021). Due to the
visual nature of surveillance data, anomaly detection is usually approached
from a visual standpoint (Sultani et al., 2018b) with an emphasis on object
detection (Doshi and Yilmaz, 2020) or visual attention regions in the frames
(Nasaruddin et al., 2020). These approaches have been tested and benchmarked
on anomaly datasets that usually capture a limited and simple range of
anomalies such as people running or throwing objects, among others.

A recent approach to anomaly detection is presented in (Morais et al., 2019),
where authors investigate how human body motion patterns relate to the
concepts of (ab)normality. Unlike the other works presented before, this
method uses skeleton trajectories extracted for each person in a video.
Skeleton coordinates capture local body movements, as well as global body
motion in the scene throughout the video. The anomaly detection technique
follows the novelty detection approach and the architecture used is a single-
encoder-dual-decoder (i.e. one decoder reconstructs, the other is used for
predictions) referred to as MPED-RNN. MPED-RNN has two structural branches:
one local and one global branch which process the corresponding trajectory
features and exchange information. The anomaly score of the video is computed
per frame, by selecting the maximum of the reconstruction and prediction
errors corresponding to the skeleton trajectories in the frame.

Anomaly detection extends to a classification task given prior knowledge of
the type of abnormalities. In this regard, the authors of (Sultani et al.,
2018a) introduced a video surveillance dataset targeted towards the task of
anomaly detection and classification, referred to as UCF-Crime. UCF-Crime
consists of normal videos and videos depicting 13 anomaly categories that
correspond to crime categories. UCF-Crime poses an additional challenge for
the task of anomaly detection since both the ‘abnormal’ and ‘normal’ videos
show a high degree of diversity, both in terms of environment scenes and in
the complexity of events. The complexity of events is highly influenced by the
number of people present in the video and the wide range of activities, which
translate into high intra-class variance. The proposed methodology for anomaly
detection on the UCF-Crime dataset given in (Sultani et al., 2018a) introduces
a multiple instance learning (MIL) (Maron and Lozano-Pérez, 1998) approach
with weakly labelled training videos. Since manually labelling each frame in a
video is expensive, the authors leveraged one label of the entire video: if a
video is labelled as part of an anomaly class it can be assumed that the
anomaly event occurs at some point in the video. Each video, whether ‘normal’
or ‘abnormal’, is divided into video segments; a video is then represented by
a ‘bag’ of its segments from which C3D features (Tran et al., 2015) are
extracted. A fully connected CNN is trained on the extracted features from
both ‘normal’ and ‘abnormal’ videos. A ranking loss function is used which
computes the ranking loss between the highest scored segments in the ‘normal’
and ‘abnormal’ bags. The aim is to train the model and adjust the loss such
that higher anomaly scores are appointed to the video segments in the
‘abnormal’ bags.

Our proposed methodology addresses the lack of tools for anomaly
classification, most specifically for human-related crime classification. We
approach this by extending the work presented in (Morais et al., 2019), with
the focus on the human-related fraction of UCF-Crime, HR-Crime (Boekhoudt et
al., 2021).

##  3 HR-Crime trajectory-level annotation

The HR-Crime dataset (Boekhoudt et al., 2021), as a subset of the UCF-Crime
dataset (Sultani et al., 2018a) only consisting of human-related crime videos,
is a significantly complex dataset due to several factors, such as low video
quality and lack of content clarity even for the human eye. Sample frames
extracted from the dataset are shown in Figure 1, where the taxonomy of HR-
Crime is also introduced. HR-Crime is a balanced dataset in terms of ‘normal’
vs. ‘abnormal’ video samples. However, as shown in Table 1, within the
‘abnormal’ category, the classes are largely imbalanced.

Figure 1: Example frames from HR-Crime following the class structure of the
dataset.

From the ground truth perspective, each video is defined by the human
skeletons and trajectories of the people appearing in it. A human skeleton is
defined by 17 $(x,y)$ coordinates corresponding to 17 body joints tracked over
the video. Each trajectory is split into segments of $12$ frames to ensure the
requirement of models demanding fixed length inputs (Boekhoudt et al., 2021).

The HR-Crime dataset solely provides ground truth labels at video-level. Each
video is annotated with the corresponding crime class or with the ‘normal’
label for videos that do not contain an anomaly. However, it is worth noting
that not all people in a crime scene are necessarily participants to the
criminal activity. The video might capture bystanders or people exhibiting
normal behaviour out of the abnormal event boundary. It is to be highlighted
that in our approach, we aim to investigate the role of human skeletons
displacement in the classification of crimes. This faces us with the lack of
ground truth annotations in the HR-Crime dataset at trajectory level.

Our proposed train:test split is based on the total number of trajectories per
class, rather than the number of videos. This is enabled by the novel
trajectory-level ground truth introduced in Section 3.1. Table 1 shows the
trajectory distribution over the 13 crime classes of HR-Crime with a 0.8:0.2
train:test ratio. The validation set is randomly set during the training phase
as 0.2 of the training set.

Set |  Abuse |  Arrest |  Arson |  Assault |  Burglary |  Explosion |  Fighting |  | Road
---
Accidents

Robbery |  Shooting |  Shoplifting |  Stealing |  Vandalism |  Abnormal |  Normal
Trajectory based distribution
Train | 221 | 186 | 101 | 290 | 227 | 50 | 309 | 59 | 579 | 107 | 588 | 178 | 45 | 2,940 | 28,622
Test | 55 | 47 | 25 | 72 | 57 | 12 | 78 | 15 | 145 | 27 | 147 | 45 | 11 | 736 | 7,156
Video based distribution
Train | 30 | 34 | 39 | 38 | 77 | 21 | 31 | 54 | 116 | 37 | 40 | 78 | 37 | 632 | 625
Test | 8 | 8 | 9 | 9 | 19 | 5 | 8 | 14 | 29 | 9 | 10 | 20 | 9 | 157 | 157

Table 1:  Trajectory and video based dataset distributions; the trajectory
based distribution follows the trajectory-level ground truth introduced in
Section 3.1

###  3.1 Trajectory-level ground truth generation

In order to compensate for the lack of ground truth annotations at trajectory
level, we propose a supervised as well as an unsupervised methodology for
differentiating between the ‘normal’ vs. ‘abnormal’ trajectories over the
trajectories appearing in the HR-Crime dataset.

We employ the MPED-RNN model trained de novo on HR-Crime, as proposed in
(Boekhoudt et al., 2021), in order to assign to each trajectory an anomaly
score. The anomaly score, $\alpha$, per trajectory, $s_{i}$, is computed based
on the reconstruction of the trajectory, following Equation 16 in (Morais et
al., 2019), described below in Equation 1:

$\alpha_{s_{i}}=\frac{\sum_{u\in S_{i}}L_{p}(u)}{|S_{i}|}$ (1)

where $S_{i}$ represents the set of reconstructed segments that compose
$s_{i}$, for which perceptual loss $L_{p}$ is computed. Perceptual loss
$L_{p}$ is defined as the mean squared error between the original trajectory
segment and its reconstruction.

We propose identifying the two clusters corresponding to the ‘normal’ and
‘abnormal’ trajectories given the anomaly scores $\alpha$. Based on the
obtained cluster labels, we establish a new ground truth for the anomaly class
classification following the steps:

  1. 1.

Removing ‘normal’ trajectories corresponding to videos labelled with one of
the 13 anomaly classes based on the video-level ground truth, adding them to
‘normal’ category.

  2. 2.

Removing from the trajectories within the ‘normal’ video-level ground truth,
the falsely generated trajectories which may resemble anomalies.

Step 1 addresses the fact that not necessarily all the people captured in an
abnormal video are actively participating in the ‘abnormal’ activity, while
Step 2 mainly captures outlier trajectories in the ‘normal’ cluster. By
removing these mismatched trajectories we ensure that the HR-Crime dataset is
consistent for tasks that are trajectory-related, reducing the amount of noise
in the data***The trajectory-level ground truth will be publicly available
upon the publication of the manuscript..

As mentioned earlier, we employ a supervised and an unsupervised method to
distinguish the wrongly labelled trajectories due to the video label
propagation into trajectories. On one side, Gaussian mixture model (Reynolds,
2009) clustering is used. GMM is used with two mixture components
corresponding to the ‘normal’ and ‘abnormal’ categories with their own general
covariance matrices. GMM is implemented using an Expectation-maximization
probabilistic algorithm (EM) (Reynolds, 2009) with 100 iterations and
convergence threshold of $1e-3$.

As per the supervised approach, we plot next to each other the anomaly score
of all the ‘normal’ and ‘abnormal’ trajectories in HR-Crime as determined by
the original ground truth at video-level. We select a threshold where visually
a difference in value can be spotted. We also define few more thresholds in
that proximity. For each value of the threshold, we compute the Silhouette
score of the resulting ‘normal’ and ‘abnormal’ trajectory clusters. We select
the threshold which leads to the highest Silhouette score as the threshold
value to separate between ‘normal’ and ‘abnormal’ trajectories. The final
threshold is equal to the anomaly score of $0.0102$. The entire process is
depicted in Figure 2.

Figure 2: Depiction of the identification process for the proposed threshold
in the supervised paradigm. The anomaly scores of the trajectories are plotted
before applying the threshold (bottom) based on the video-level ground truth
and after applying the threshold (top) based on the trajectory-level ground
truth.

We evaluate the quality of the obtained clusters of ‘normal’ and ‘abnormal’
trajectories based on the Silhouette score. The obtained results of using
either supervised or unsupervised technique are 0.796 and 0.752, respectively.
This relatively high score indicates the decent quality of the clusters, i.e.
the coherence among the grouped samples. We also evaluate the number of
trajectories that received a different trajectory-level label than the video-
level label provided by HR-Crime: 11,987 and 11,748 trajectories originally
belonged to the ‘abnormal’ class are relabelled as ‘normal’, using supervised
and unsupervised approach, respectively. Furthermore, 24,533 and 24,549
trajectories originally labelled as ‘normal’ are found to be outliers that
resemble the ‘abnormal’ class using supervised and unsupervised approach,
respectively.

Figure 3: Examples of video-level ‘normal’ trajectories clustered as
‘abnormal’ within the normal category, detected by the unsupervised approach.
a) skeleton merged from different skeleton joints, b) skeleton is too distant
and joints overlap, c) skeleton joints not connected to an actual person are
detected incorrectly.

Qualitatively, the results of the GMM categorization of ‘normal’ vs.
‘abnormal’ trajectories can be seen in Figure 3. This shows some examples of
trajectories detected in ‘normal’ video-level labelled videos that are
categorized as ‘anomaly’. It can be seen that these trajectories do not
necessarily capture anomalous crime activities, but are outliers within the
HR-Crime dataset (Boekhoudt et al., 2021).

###  3.2 Dataset augmentation

In order to alleviate the issue of class imbalance of the HR-Crime dataset, we
opt for the following data augmentation techniques.

Skeletal joints coordinate shift: A human skeleton trajectory is defined by a
collection of coordinates which identifies the movement of the body over the
video frames in which the body is visible. One approach to generate new
trajectories is to shift the original trajectory within the bounds of the
coordinate system that describe the skeletons. The shift is implemented by
aggregating a small factor, $\delta$, to the skeletal coordinates in the
original trajectory, resembling the skeleton being moved within the frame by a
small distance. Figure 4(a) shows a trajectory segment in its original format
as well as the shifted one. The ‘shifted’ trajectory can be seen as a
translation of the original trajectory.

(a) The original and ‘shifted’ skeleton trajectories.

(b) The noisy trajectories propagated by the skeletal ‘shifting’ technique.

Figure 4: Illustration of the skeletal joints coordinate shift; ‘shifted’
skeleton is shown in red.

The value of the $\delta$ is fully dataset dependent; an inappropriate
$\delta$ might negatively impact the movement behaviour captured by the
original trajectory and possibly leads to transforming a normal trajectory
into an abnormal one or vice versa. Hence, $\delta$ needs to accommodate two
important aspects: first, represent the movement behaviour of each skeleton
individually, as different people might show different particularities in
terms of their body movements. Second, uniquely represent each of the
$17(x,y)$ key points that define the skeleton movement behaviour. This is
important since, for instance, hand movements do not follow the same behaviour
as torso movement. In order to satisfy these two constraints, $\delta$ is
computed per trajectory and per body joint based on the movement coordinates
of the original trajectory. This is achieved as follows: each $\delta$ is
defined as the average movement of the joint; joint movement is identified as
the difference in joint coordinates between two consecutive frames in the
original trajectory. The original trajectory is then used to generate any
number of new trajectories by appending $\delta$ (i.e. shift to right) or
$-\delta$ (i.e. shift to left) to the respective joint coordinates. Note that
any number of new trajectories can be generated by altering $\delta$ with some
small degree of randomness. This augmentation approach is suitable for the
encoded-based classification described in Section 4.1, which entails an
initial feature decomposition step applied to the trajectory. A disadvantage
of applying this augmentation technique is that the resulted trajectories
might introduce some degree of noise, especially for the classes which have a
limited number of samples in the original dataset. This occurs due to the fact
that $\delta$ is adjusted by a random value for a larger number of iterations
in order to achieve the target number of trajectories. This effect is more
undesirable considering the propagation of noisy trajectories appearing in the
original dataset into the augmented dataset, as illustrated in Fig. 4(b).

Trajectory data point oversampling: Overcoming the dataset imbalance is
possible by augmenting the under-represented classes with synthetic samples.
Here, this is achieved using the Synthetic Minority Oversampling Technique
(SMOTE) (Chawla et al., 2002). SMOTE is a suitable technique for data points
with consistent size, therefore it is applicable to the trajectory segments
that describe HR-Crime. Considering this aspect, SMOTE is used in relation
with the decoded-based classification proposed in Section 4.2. This technique
proposes the oversampling of the minority classes by creating synthetic
example data until all classes have a size equal to the size of the initial
majority class. The majority class in HR-Crime is Shoplifting with almost
$600$ trajectories leading to $243,272$ trajectory segments of $12$ time-
steps. The minority classes are oversampled to reach this same amount of
trajectory segments. The SMOTE technique generates new data while minimizing
the added noise due to the technique of creating new samples based on existing
neighbouring data points. This approach can add more range to the data,
however, depending on the nature of the initial dataset, it is prone to result
in additional noise.

##  4 HR-Crime trajectory-based classification

Our proposed approach for classification of crime categories builds upon
(Morais et al., 2019), as we aim to study whether certain anomalous human body
movements correspond to a particular crime category. The proposed models
explore two types of data representation of human motion: human trajectory
encodings and decodings obtained by the MPED-RNN model (Morais et al., 2019).

###  4.1 Encoded-based classification

With this experiment, our aim is to investigate to what extent the normality
encoding ability of MPED-RNN enables capturing the distinctive characteristics
of each crime category. To this end, we first train MPED-RNN as per
instructions given by the authors in (Morais et al., 2019) and then, isolate
the trained encoder part of MPED-RNN from the rest of the architecture. Next,
we freeze the isolated encoder layers in order to preserve the gained
knowledge. As shown in Figure 5, the encoder becomes the base element of the
encoded-based crime classification. The local and global latent spaces
resulted from the encoding process denote the feature representation
corresponding to the input trajectory segment. These representations are
further processed in order to reach the final classification. An overview of
the architecture can be seen in Figure 5.

Figure 5: Illustration of the encoded-based classification models.
Architecture $A_{x}$ refers to architectures presented in Figure 7; $z_{l}$,
$z_{g}$ represent the local and global feature representations which are
merged into $z_{f}$ during the early fusion step.

####  4.1.1 MPED-C: crime activities classification

MPED-RNN encoder architecture is a two-branch RNN structure, processing the
local and global representation of skeletal trajectories. In our proposed
model, MPED-C, we opt for merging the local and global latent representations
obtained by the encoder in an early fusion step to create a unified
representation of the trajectory segment, as per Equation 2:

$z_{f}=w_{l}*z_{l}+w_{g}*z_{g},$ (2)

where $z_{l}$ and $z_{g}$ are the local and global representations and $w_{l}$
and $w_{g}$ their corresponding aggregation weights. The multiplication is
element-wise, which results in a fused trajectory segment representation,
$z_{f}$, of the same size as $z_{l}$ and $z_{g}$, meaning that all information
captured by $z_{l}$ and $z_{g}$ is preserved. The weights $w_{l}$ and $w_{g}$
are learned to specifically fine-tune the fusion process to the dataset. We
also experiment with concatenating the obtained features $z_{l}$ and $z_{g}$,
such that $z_{f}=[z_{l},z_{g}]$. The local and global latent representations
are individually processed prior to the fusion step. The fused representation
is then followed by a 13 class classifier.

Figure 6: Illustration of late fusion of the local and global encodings;
$Architecture\ A_{x}$ is a choice of the $A_{1}$, $A_{2}$, $A_{3}$
architectures visualized in Figure 7.

Furthermore, we conduct an ablation study focused on feature fusion
strategies. To this end, we experiment with late fusion as follows: applying
the same architecture for each of the encoding branches results into two
probability vectors which yield the classification of the trajectory segments,
based exclusively on their local or global features, respectively. The late
fusion step involves the element-wise aggregation of the two probability
vectors. The resulting vector is processed by a normalization step which gives
the final classification probabilities. The key to this approach is appending
the same model architecture in parallel to the local and global encoding
branches. The architecture is chosen from Architectures A1, A2 and A3 shown in
Figure 7. An overview of this experiment is shown in Figure 6.

We experiment with the early fusion strategy only over the best performing
architecture resulting from the late fusion experiments. As we will see in
Section 5, the best performing architecture with late fusion is A3 and to
apply early fusion with it, we discard its last two Dense layers (i.e. ReLu
and softmax). Moreover, to choose the best fitting layer configuration, we
conduct an ablation study for the winning architecture (i.e. A3) and the
number of filters of the convolution. We run experiments with 8, 16, 32, 64,
128, 256 filters.

Figure 7: Visual representation of the network architectures A1, A2 and A3.
The latent space refers to the MPED-RNN encoder representations.

####  4.1.2 MPED-NC: normal and crime activities classification

The MPED-C model presented above is extended to recognize and classify between
$14$ classes by adding the ‘normal’ activity class to the $13$ crimes
classification task. This new model is entitled MPED-NC. The adjustments
required for the $14$ categories are minimal; Figures 5 and 6 showcase the
architectures of both MPED-C and MPED-NC, the only difference being the size
of the last softmax activated layer with the output probability vector of $14$
instead of $13$ elements. We extend the architectural and fusion experiments
presented for MPED-C in Section 4.1.1 to the MPED-NC model.

###  4.2 Decoded-based classification

The reconstruction error is the basis of the anomaly score proposed by (Morais
et al., 2019), which essentially represents how much a trajectory deviates
from ‘normality’. Hence, by using reconstructed trajectories as input for
classifying the crimes, we implicitly capture a higher-level representation
that inherently carries information about the abnormality of the trajectories
in the form of the deviation of the reconstructed trajectory from the original
one. With this approach, our focus is merely on the crime classification to
investigate how the degree of trajectory abnormality translates to the
comprehension and distinction of the 13 crime categories, hence experiments
with addition of ‘normal’ class has been avoided.

The reconstructed trajectories by the MPED-RNN architecture are structured in
a similar fashion as the original trajectories. MPED-RNN is able to
reconstruct global and local features describing the skeleton, in a similar
fashion to the local and global decomposition of the unprocessed skeleton; for
the trajectories reconstructions, we opt for the same feature space as the
original trajectories: $17(x,y)$ pair coordinates correlated to $17$ joints of
the human skeleton embedding the local and global features. The input to the
crime decoded-based classifier is the reconstructed trajectories corresponding
to the crime videos, structured in a $12$ time-step segment fashion.

The reconstructed skeleton trajectories have a preeminent temporal
characteristic, which we assume to play a key role in solving the crime
classification task. They capture a unified representation of the
reconstructed body joints tracked over time, where the joints are defined by
local and global features. In order to leverage the particularities of this
data, we propose a classifier centered on a Long Short Term Memory (LSTM)
layer (Greff et al., 2016). An LSTM network also acts as an encoder: it
reduces the sequential input data size by reducing its dimensions to a pre-
defined size, therefore creating a condensed encoding of the input data. The
encodings can thereafter be manipulated further for other tasks. We simply
place a fully connected layer (i.e. dense) after the LSTM for classification.
The output of the proposed model is a $13$ dimension probability vector which
indicates the probability of the input trajectory segment belonging to the
$13$ crime classes. An overview of the proposed classifier is shown in Figure
8.

Figure 8: The proposed LSTM model for the 13 crimes classification task in
Section 4.2

###  4.3 Implementation details

The experiments are conducted with models trained using the Adam optimizer
(Kingma and Ba, 2014), with a learning rate of $0.001$, and the categorical
cross-entropy loss (Zhang and Sabuncu, 2018). In order to avoid overfitting,
early stopping is implemented to monitor both training and validation losses
with a patience of 3 epochs.

For the extended MPED models, the training process is done conventionally
using the trajectory-based split described in Section 3, since the encoder
based is separately trained only on normal examples and later frozen. This
implies that MPED-C, is presented with crime trajectories exclusively during
training and evaluation. MPED-NC, on the other hand, is trained and evaluated
both on normal and crime trajectories in order to solve the 14 class task.

For evaluation purposes, we conduct k-fold cross-validation for all the
experiments discussed above. We opted for k=3 and trained the models on each
fold for 25 epochs. The folds were shuffled for the training set, leaving the
testing set unchanged.

###  4.4 Evaluation metrics

For classification evaluation, we use multi-class categorical evaluation
metrics such as Accuracy, Precision, Recall and F-score. We also report the
weighted average (W) over the metric performance per class based on the number
of samples in the respective class, in order to account for the imbalanced
nature of the original dataset. To compensate for the complexity of the
classification task, we also evaluate Top-3 and Top-5 Accuracy.

Furthermore, we conduct the statistical analysis on the weighted accuracy
results obtained by the models during k-fold cross-validation. The models are
compared pairwise using first a Shapiro-Wilk normality test (Shapiro and Wilk,
1965). If the P-value obtained is greater than $\alpha=0.05$, a paired two-
sided t-test is performed (Hsu and Lachenbruch, 2005); this implies that the
data comes from a normal distribution. If the P-value obtained is less than
$\alpha=0.05$, a Wilcoxon test (Gehan, 1965) is used; this is a non-parametric
version of the t-test, which is more suitable for data that does not come from
a normal distribution.

We also present the relations between the quantitative performance of the
crime classes in the form of a confusion matrix. These relations are further
explored by investing the common top performance results for the class
predictions and by investigating common misclassification patterns.

##  5 Results and discussion

###  5.1 Encoded-based classification results

This Section presents and discusses the results of the experiments conducted
with the encoded-based classifiers introduced in Section 4.1.

####  5.1.1 Convolution filter size ablation study results

#filters | Accuracy (M) | Accuracy (W) | Precision (W) | Recall (W) | F-score (W) | IBA
---|---|---|---|---|---|---
8 | 0.364 $\pm$ 0.013 | 0.132 $\pm$ 0.013 | 0.328 $\pm$ 0.003 | 0.364 $\pm$ 0.013 | 0.297 $\pm$ 0.015 | 0.274 $\pm$ 0.012
16 | 0.394 $\pm$ 0.006 | 0.189 $\pm$ 0.003 | 0.364 $\pm$ 0.007 | 0.394 $\pm$ 0.006 | 0.361 $\pm$ 0.003 | 0.313 $\pm$ 0.004
32 | 0.419 $\pm$ 0.002 | 0.219 $\pm$ 0.002 | 0.397 $\pm$ 0.003 | 0.419 $\pm$ 0.002 | 0.395 $\pm$ 0.001 | 0.340 $\pm$ 0.002
64 | 0.422 $\pm$ 0.010 | 0.239 $\pm$ 0.015 | 0.404 $\pm$ 0.015 | 0.422 $\pm$ 0.010 | 0.405 $\pm$ 0.014 | 0.343 $\pm$ 0.012
128 | 0.407 $\pm$ 0.012 | 0.234 $\pm$ 0.013 | 0.393 $\pm$ 0.015 | 0.407 $\pm$ 0.012 | 0.393 $\pm$ 0.013 | 0.333 $\pm$ 0.011
256 | 0.410 $\pm$ 0.017 | 0.254 $\pm$ 0.023 | 0.399 $\pm$ 0.028 | 0.410 $\pm$ 0.017 | 0.401 $\pm$ 0.023 | 0.339 $\pm$ 0.020

Table 2: Ablation study on the number of filters for the 1D Convolution layers
included in the MPED-C model; training was conducted on the non-augmented
version of HR-Crime.

As mentioned earlier, we conduct an ablation study focused on the number of
filters of the convolution layers in Architecture A3 (see Figure 7). The
results presented in Table 2 show that the model designed with 64 filters
overall achieves the best performance with scores of $0.42$ and $0.24$
respectively in macro and weighted accuracy. For this ablation study, MPED-C
was trained on the non-augmented version of HR-Crime. The number of filters
used corresponds to the number of features that the convolution layer learns
from the data; the quantitative results in Table 2 show that using a limited
amount of filters (e.g. 8 filters) tends to under-fit the data achieving
$0.13$ weighted accuracy score, while adding excessive filters does not
improve the comprehension of the data. Following these results, convolution-
based architecture A3 is modelled with one-dimensional convolution layers with
a filter size of 64.

| Model
---
ID
| Archi. &
---
fusion
| Data
---
aug.
Accuracy (M) | Accuracy (W) | Precision (W) | Recall (W) | F-score (W) |  | Top-3
---
accuracy
| Top-5
---
accuracy
Unsupervised trajectory-level ground truth
MPED-C1.1 | A1-Late | None | 0.367 $\pm$ 0.004 | 0.140 $\pm$ 0.008 | 0.310 $\pm$ 0.008 | 0.367 $\pm$ 0.004 | 0.310 $\pm$ 0.007 | 0.677 $\pm$ 0.005 | 0.824 $\pm$ 0.005
MPED-C1.2 | A1-Late | Shift | 0.286 $\pm$ 0.021 | 0.193 $\pm$ 0.004 | 0.322 $\pm$ 0.014 | 0.280 $\pm$ 0.021 | 0.285 $\pm$ 0.019 | 0.571 $\pm$ 0.005 | 0.749 $\pm$ 0.002
MPED-C2.1 | A3-Late | None | 0.403 $\pm$ 0.009 | 0.219 $\pm$ 0.018 | 0.378 $\pm$ 0.013 | 0.403 $\pm$ 0.008 | 0.380 $\pm$ 0.014 | 0.706 $\pm$ 0.007 | 0.839 $\pm$ 0.003
MPED-C2.2 | A3-Late | Shift | 0.330 $\pm$ 0.020 | 0.270 $\pm$ 0.010 | 0.395 $\pm$ 0.012 | 0.330 $\pm$ 0.020 | 0.344 $\pm$ 0.018 | 0.621 $\pm$ 0.017 | 0.797 $\pm$ 0.010
MPED-C3.1 | A2-Late | None | 0.406 $\pm$ 0.002 | 0.244 $\pm$ 0.007 | 0.393 $\pm$ 0.001 | 0.406 $\pm$ 0.002 | 0.395 $\pm$ 0.002 | 0.700 $\pm$ 0.005 | 0.837 $\pm$ 0.003
MPED-C3.2 | A2-Late | Shift | 0.345 $\pm$ 0.017 | 0.271 $\pm$ 0.017 | 0.407 $\pm$ 0.017 | 0.345 $\pm$ 0.017 | 0.361 $\pm$ 0.017 | 0.633 $\pm$ 0.013 | 0.802 $\pm$ 0.001
MPED-C4.1 | A3-Early(c) | None | 0.423 $\pm$ 0.010 | 0.243 $\pm$ 0.012 | 0.408 $\pm$ 0.011 | 0.423 $\pm$ 0.010 | 0.408 $\pm$ 0.011 | 0.714 $\pm$ 0.007 | 0.850 $\pm$ 0.003
MPED-C4.2 | A3-Early(c) | Shift | 0.363 $\pm$ 0.009 | 0.302 $\pm$ 0.009 | 0.420 $\pm$ 0.013 | 0.363 $\pm$ 0.009 | 0.376 $\pm$ 0.010 | 0.660 $\pm$ 0.003 | 0.820 $\pm$ 0.001
MPED-C5.1 | A3-Early(a) | None | 0.432 $\pm$ 0.002 | 0.255 $\pm$ 0.004 | 0.418 $\pm$ 0.005 | 0.432 $\pm$ 0.002 | 0.419 $\pm$ 0.002 | 0.718 $\pm$ 0.002 | 0.850 $\pm$ 0.002
MPED-C5.2 | A3-Early(a) | Shift | 0.364 $\pm$ 0.010 | 0.304 $\pm$ 0.008 | 0.419 $\pm$ 0.014 | 0.364 $\pm$ 0.010 | 0.373 $\pm$ 0.011 | 0.659 $\pm$ 0.003 | 0.816 $\pm$ 0.003
Supervised trajectory-level ground truth
MPED-C6.1 | A1-Late | None | 0.356 $\pm$ 0.005 | 0.140 $\pm$ 0.003 | 0.323 $\pm$ 0.006 | 0.356 $\pm$ 0.005 | 0.297 $\pm$ 0.006 | 0.655 $\pm$ 0.005 | 0.815 $\pm$ 0.005
MPED-C6.2 | A1-Late | Shift | 0.305 $\pm$ 0.004 | 0.184 $\pm$ 0.006 | 0.324 $\pm$ 0.007 | 0.305 $\pm$ 0.004 | 0.300 $\pm$ 0.004 | 0.577 $\pm$ 0.008 | 0.759 $\pm$ 0.007
MPED-C7.1 | A3-Late | None | 0.413 $\pm$ 0.010 | 0.223 $\pm$ 0.012 | 0.389 $\pm$ 0.011 | 0.413 $\pm$ 0.010 | 0.388 $\pm$ 0.014 | 0.690 $\pm$ 0.012 | 0.834 $\pm$ 0.008
MPED-C7.2 | A3-Late | Shift | 0.371 $\pm$ 0.029 | 0.269 $\pm$ 0.025 | 0.408 $\pm$ 0.033 | 0.371 $\pm$ 0.029 | 0.379 $\pm$ 0.031 | 0.634 $\pm$ 0.016 | 0.798 $\pm$ 0.009
MPED-C8.1 | A2-Late | None | 0.418 $\pm$ 0.008 | 0.236 $\pm$ 0.005 | 0.397 $\pm$ 0.007 | 0.418 $\pm$ 0.008 | 0.398 $\pm$ 0.009 | 0.688 $\pm$ 0.007 | 0.830 $\pm$ 0.005
MPED-C8.2 | A2-Late | Shift | 0.397 $\pm$ 0.015 | 0.290 $\pm$ 0.011 | 0.434 $\pm$ 0.016 | 0.397 $\pm$ 0.015 | 0.406 $\pm$ 0.016 | 0.657 $\pm$ 0.006 | 0.812 $\pm$ 0.002
MPED-C9.1 | A3-Early(c) | None | 0.450 $\pm$ 0.012 | 0.263 $\pm$ 0.012 | 0.434 $\pm$ 0.015 | 0.450 $\pm$ 0.012 | 0.433 $\pm$ 0.015 | 0.711 $\pm$ 0.006 | 0.840 $\pm$ 0.003
MPED-C9.2 | A3-Early(c) | Shift | 0.385 $\pm$ 0.010 | 0.283 $\pm$ 0.017 | 0.425 $\pm$ 0.020 | 0.385 $\pm$ 0.019 | 0.394 $\pm$ 0.019 | 0.667 $\pm$ 0.012 | 0.818 $\pm$ 0.008
MPED-C10.1 | A3-Early(a) | None | 0.444 $\pm$ 0.015 | 0.251 $\pm$ 0.013 | 0.424 $\pm$ 0.017 | 0.444 $\pm$ 0.015 | 0.421 $\pm$ 0.015 | 0.711 $\pm$ 0.004 | 0.841 $\pm$ 0.004
MPED-C10.2 | A3-Early(a) | Shift | 0.403 $\pm$ 0.012 | 0.291 $\pm$ 0.016 | 0.439 $\pm$ 0.015 | 0.4031 $\pm$ 0.012 | 0.349 $\pm$ 0.012 | 0.650 $\pm$ 0.015 | 0.804 $\pm$ 0.005

Table 3: Results of the MPED-C experiments for the classification of 13
crimes; the Architecture column refers to the different network architectures
in Figure 7 appended to the MPED-RNN encoder, fusion is either late or early
(being (a) aggregation and (c) concatenation). Data augmentation is achieved
by Shifting the original trajectories.

####  5.1.2 MPED-C: crime activities classification results

The results of the experiments conducted for the MPED-C model are presented in
Table 3. The best performing model is MPED-C5.2, with a performance in
weighted accuracy of $0.30$. For the remaining quantitative evaluation
metrics, the best results are obtained by early fusion models, where the
fusion is implemented, for most models, as aggregation. This indicates that
the local and global encodings of the trajectories capture highly
complementary features. Results indicate that applying the classification
individually, through late fusion, does not account for the local and global
inter-relation, which leads to a decrease of $0.06$ in performance. This is
observed for architecture A1 in combination with late fusion. Comparing the
performance achieved by architectures A2 and A3 in the context of late fusion,
we observe that similar results are obtained; MPED-C3.2 and MPED-C2.2 both
achieve aprox. $0.27$ in weighted accuracy, despite the more evolved LSTM
based architecture of MPED-C3.2.

In terms of dataset labelling, the two different resulted trajectory-level
ground truths, described in Section 3.1, influence the results achieved by the
MPED-C experiments. It can be observed that the top weighted accuracy is
obtained by the model MPED-C5.2, trained on the trajectory-level ground truth
obtained by GMM unsupervised methodology. For the remaining metrics however,
top performance is achieved for models trained on the trajectory ground truth
following the supervised operation. This is consistent with the results in
Table 5. Manually assuming a threshold for separating the ‘normal’ vs.
‘abnormal’ trajectory is customized specifically to the dataset. However,
unlike the unsupervised method, it does not account for hidden relations in
the data. The results also confirm the correlation between the dataset
imbalance and the complexity of the 13 crimes classification task. Models
trained on the augmented variant of HR-Crime, with new data samples being
generated by ‘shifting’ the original trajectories, obtain, on average, an
increase of $0.04$ in weighted accuracy as compared to the same models trained
on the original HR-Crime. Considering the methodology for the dataset
augmentation, this circumstance might occur due to the potential added noise
per class resulting from ‘shifting’ the original trajectories, especially for
those crime categories originally poorly represented.

We report a significant increase in accuracy for the Top-3 and Top-5 accuracy
metrics. Our proposed MPED-C model (with ID MPED-C5.2) obtains up to $0.82$ in
Top-5 accuracy, which is significantly higher than the Top-1 accuracy of
$0.36$. In general, we observe that the scores achieved in Top-3 accuracy are
approximately double the macro accuracy, while Top-5 performances show even
more significant increase. This indicates that the approach of representing
the data through MPED-RNN encodings does not entirely overcome the challenges
of low inter-class variation.

Figure 9: Confusion matrix depicting the average results using MPED-C5.2
setting.

We apply the statistical test comparing the top performing model MPED-C5.2
with the next Top-5 best performing networks in terms of weighted accuracy:
MPED-C4.2, MPED-C10.2, MPED-C8.2, MPED-C9.2, and MPED-C3.2 in Table 3. The
obtained P-values are of 0.778, 0.083, 0.415, 0.153, 0.151, respectively. With
a confidence threshold of $\alpha=0.05$, we accept the null hypothesis for all
the 5 comparisons, based on the reported P-values. This similarity in achieved
results can be accounted for by the fact that all compared models were trained
on the augmented variant of HR-Crime.

The confusion matrix representing the results obtained by the best performing
MPED-C model is presented in Figure 9. As can be seen, class Road Accidents
shows the highest class accuracy of $0.60$. The achieved top result can be
correlated with the distinctive characteristics of this class in comparison to
the remaining 12 crimes in the dataset. Road Accidents samples present a
limited number of human participants, usually their skeletons being to some
extent obstructed by their location: for example, people in cars or other
vehicles. The anomaly window is also quite brief, corresponding to the moment
of impact. Classes Shoplifting and Abuse obtain, on average, results near the
$0.50$ score, with $0.51$ and $0.48$ accuracy for Shoplifting and Abuse,
respectively. The rest of the crime classes achieve accuracy scores on the
lower end with Vandalism obtaining the lowest average performance of $0.07$
accuracy.

Analysing the misclassification pattern of the results, we observe that the
model faces some difficulties in demarcating the 13 crime categories. The
misclassifications are spread across the entire confusion matrix, with classes
like Abuse, Arrest, Assault and Shoplifting being most of the time wrongly
predicted. The highest misclassification rates are reported for Vandalism
versus Arrest and Robbery versus Shoplifting with an occurrence of
aproximmately $19\%$. Class Vandalism is also misclassified with a high
frequency for classes Abuse and Burglary. This indicates that the model is not
able to properly learn the Vandalism category. This can be explained by the
low representation of the class in the dataset, which even the augmentation
procedure does not help in identifying the features of the crime category
sufficiently.

The misclassification between Arrest and Abuse categories can be explained by
the highly aggressive nature of the actions within these classes and the large
overlap they share: the nature of the representative videos include police
officials showcasing abusive behaviour. Class Fighting frequently overlaps
with classes such as Arrest, Assault, Shoplifting which are characterized by
increased levels of body aggression.

Figure 10: Trajectories relevant for crime classes Shoplifting, Burglary and
Robbery depicting the body movement and stance similarities shared by the
three classes; the trajectories present people with a inner drawn posture
inspiring a dishonest behaviour.

One crime that is commonly misattributed is class Shoplifting. This crime
category has the most original trajectory samples, with 588 trajectories in
the training set, therefore almost no data augmentation was required. This
class also achieves the second best accuracy score of $0.54$, which can be
justified by the high number of original trajectories. The most striking
misclassification occurs with class Robbery, with an average frequency of
$19\%$. The similarities between the two crimes lie in the main activity in
which the human participants are involved. This also explains the $15\%$
overlap with class Burglary. The three classes are focused on the act of
stealing, which implies semantic similarities. Given the common activity in
which the participants are involved, human movement behaviour is shared over
these categories, as showcased in Figure 10 through the exposed trajectories.
Shoplifting is focused on indoor shop environments in which one or two people
are attempting to steal some object(s) without being noticed; these people
usually leave the shop without any altercations. The Robbery category
showcases crimes occurring in shops where the cashier or other employees are
threatened, in some cases at gunpoint, by the perpetrator in order to obtain
money or some other valuable objects. The Robbery category involves more
violence than Shoplifting which can also be deduced from its increased
misclassification rates with classes Abuse and Assault, whereas Shoplifting is
misclassified as Assault only $11\%$ of the time. However Abuse, Arrest,
Assault are, on average, wrongly predicted as Shoplifting at a $0.13$ rate.
This indicates that more than one label per video, and thus, per trajectory,
might be necessary in order to more appropriately describe the essence of a
criminal activity.

Other categories that involve the act of stealing are Burglary and Stealing.
Burglary is misclassified as Shoplifting with an average rate of $0.14$. This
occurrence can be explained by the semantic nature of the class: Burglary
captures breaking and enterings in private homes or stores, with mostly one
human subject involved (i.e. the perpetrator); the majority of Burglary
anomalies are captured at night-time. The overlap between Shoplifting and
Stealing is minimal. The Stealing category focuses on events occurring
outdoors that usually involve a vehicle, either as a means of escape or, more
predominantly, as the object of interest for the perpetrators. The movement
behaviour is more varied and usually involves rapid, hastened local body
movements describing rather short trajectories.

####  5.1.3 MPED-NC: normal and crime activities classification results

The MPED-NC results are presented in Table 4. The highest achieving setting is
the MPED-NC5, achieving a $0.24$ weighted accuracy score. This implies that
for the 14 class classification task, early fusion and trajectory-level ground
truth obtained using the unsupervised method achieves top classification
results. Comparing the macro and weighted accuracy over all the experiments,
we observe that the performance gap is relatively small; macro accuracy is on
average with only $0.02$ higher than weighted accuracy, which indicates that
the used dataset setting is suitable for the experiments involving 14 classes.

Model ID | Arch. | Fusion | Accuracy (M) | Accuracy (W) | Precision (W) | Recall (W) | F-score (W) |  | Top-3
---
accuracy
| Top-5
---
accuracy
Unsupervised trajectory-level ground truth
MPED-NC1 | A1 | Late | 0.229 $\pm$ 0.011 | 0.178 $\pm$ 0.006 | 0.459 $\pm$ 0.008 | 0.229 $\pm$ 0.011 | 0.274 $\pm$ 0.014 | 0.418 $\pm$ 0.015 | 0.533 $\pm$ 0.015
MPED-NC2 | A3 | Late | 0.180 $\pm$ 0.003 | 0.214 $\pm$ 0.010 | 0.473 $\pm$ 0.024 | 0.180 $\pm$ 0.003 | 0.188 $\pm$ 0.007 | 0.337 $\pm$ 0.006 | 0.442 $\pm$ 0.008
MPED-NC3 | A2 | Late | 0.240 $\pm$ 0.011 | 0.239 $\pm$ 0.004 | 0.513 $\pm$ 0.003 | 0.240 $\pm$ 0.011 | 0.275 $\pm$ 0.014 | 0.411 $\pm$ 0.016 | 0.521 $\pm$ 0.012
MPED-NC4 | A3 | Early(c) | 0.201 $\pm$ 0.013 | 0.224 $\pm$ 0.006 | 0.517 $\pm$ 0.021 | 0.201 $\pm$ 0.013 | 0.225 $\pm$ 0.020 | 0.381 $\pm$ 0.013 | 0.500 $\pm$ 0.014
MPED-NC5 | A3 | Early(a) | 0.243 $\pm$ 0.013 | 0.244 $\pm$ 0.010 | 0.529 $\pm$ 0.010 | 0.243 $\pm$ 0.013 | 0.277 $\pm$ 0.015 | 0.450 $\pm$ 0.018 | 0.579 $\pm$ 0.021
Supervised trajectory-level ground truth
MPED-NC6 | A1 | Late | 0.213 $\pm$ 0.014 | 0.172 $\pm$ 0.015 | 0.450 $\pm$ 0.011 | 0.213 $\pm$ 0.014 | 0.248 $\pm$ 0.015 | 0.340 $\pm$ 0.019 | 0.517 $\pm$ 0.021
MPED-NC7 | A3 | Late | 0.205 $\pm$ 0.001 | 0.218 $\pm$ 0.006 | 0.494 $\pm$ 0.012 | 0.205 $\pm$ 0.001 | 0.224 $\pm$ 0.005 | 0.370 $\pm$ 0.005 | 0.481 $\pm$ 0.009
MPED-NC8 | A2 | Late | 0.230 $\pm$ 0.022 | 0.221 $\pm$ 0.019 | 0.491 $\pm$ 0.026 | 0.237 $\pm$ 0.022 | 0.271 $\pm$ 0.027 | 0.381 $\pm$ 0.016 | 0.481 $\pm$ 0.012
MPED-NC9 | A3 | Early(c) | 0.204 $\pm$ 0.008 | 0.231 $\pm$ 0.012 | 0.540 $\pm$ 0.010 | 0.204 $\pm$ 0.008 | 0.221 $\pm$ 0.013 | 0.359 $\pm$ 0.008 | 0.468 $\pm$ 0.009
MPED-NC10 | A3 | Early(a) | 0.218 $\pm$ 0.016 | 0.237 $\pm$ 0.011 | 0.549 $\pm$ 0.005 | 0.218 $\pm$ 0.016 | 0.243 $\pm$ 0.022 | 0.392 $\pm$ 0.028 | 0.507 $\pm$ 0.030

Table 4: Results for the MPED-NC experiments for the classification of the 14
classes. The ‘Arch.’ column refers to the architectures in Figure 7 appended
to the MPED-RNN encoder. (a) refers to aggregation and (c) to concatenation of
features.

We can observe an overall performance drop that results from the addition of a
14th ‘normal’ class ($0.24$ vs. $0.30$ for MPED-C in weighted accuracy). This
might be due to the fact that the ‘normal’ category shows the most intra-class
variation in terms of descriptive features and trajectories. The complexity of
the classification problem is therefore enhanced by the addition of the class
since the model is faced with a 2-fold challenge: learn the inner variation of
the ‘normal’ trajectories while differentiating it from the rest of the 13
crime categories.

We apply the statistical test comparing the top performing model MPED-NC5 with
the next Top-5 best performing networks in terms of weighted accuracy: MPED-
NC3, MPED-NC10, MPED-NC9, MPED-NC4, and MPED-NC8 in Table 4. The obtained
P-value is of 0.451, 0.683, 0.027, 0.198, and 0.154, respectively. Based on
the resulted P-values, we accept the null hypothesis with a confidence
threshold of $\alpha=0.05$ for all 5 comparisons. This entails that the models
analysed produce consistent results, which can be accounted by the uniform
data settings on which all models are trained.

Figure 11: Confusion matrix depicting the average results using MPED-NC5
setting.

The confusion matrix shown in Figure 11 displays the classification results
obtained by the MPED-NC as the average over the 3-folds of the cross-
validation. The top results per class are obtained for Abuse with a score of
$0.47$ and Shoplifting with $0.42$. The remaining classes, including the
‘normal’ class, obtain, on average, an individual accuracy of $0.24$, with no
class over-passing the $0.4$ score. We exclude from this account classes
Explosion and Vandalism, which achieve low score of about $0.07$. The Road
Accidents category achieves an accuracy score of $0.33$, while the MPED-C (see
Figure 9) obtained significantly higher results around the $0.6$ accuracy
score. The decrease in performance follows from the notable misclassification
rate of $0.14$ reported for classes Road Accidents and the ‘normal’ class.
This can be attributed to the shared environment, the ‘normal’ class
containing videos captured by traffic cameras. Moreover, as Figure 12 shows,
trajectories specific to the two classes share some commonalities. Since
surveillance footage has not been pre-processed to only capture the anomaly
window, situations such as the one presented in Figure 12 are frequent: the
trajectory extracted from the Road Accident example is ‘normal’ (marked by a
green box) previous to the moment of impact (i.e. the accident); afterwards,
the movement behaviour anomaly (marked by a red box) becomes evident.
Comparing the ‘normal’ segment of the Road Accidents trajectory, we observe
that it shares similarities with the trajectory captured in the ‘normal’
video: both trajectories depict pedestrians walking on the sidewalk or over a
cross-walk. This observation indicates that a ground truth labelling of the
trajectory segments might be beneficial for the classification since the
anomaly window would be isolated, resulting in a more accurate definition of
the events.

Figure 12: Depiction of two human skeleton trajectories from categories
‘normal’ and Road Accidents; displayed frames are not consecutive. Green boxes
identify ‘normal’ behaviour, while red boxes identify ‘abnormal’.

The ‘normal’ class achieves an average of $0.18$ individual accuracy, with
some misattributions occurring. This indicates that MPED-NC faces difficulties
in generalizing the trained knowledge about the ‘normal’ set. Considering the
high intra-class variance as well as the random under-sampling that is applied
to the ‘normal’ class prior to training, it might be the case that the
examples shown to the model during the training phase are not entirely
relevant to the overall contents of the ‘normal’ category. Revising the under-
sampling step would potentially improve the per class performance for the
‘normal’ category, given that the model would be trained on possibly more
relevant examples. One way to approach this would be by providing an inner-
class taxonomy and equally sampling from the ‘normal’ sub-classes.

The frequency of misclassification observed in Figure 11 is consistent with
the quantitative results presented in Table 4. We can observe that
misclassifications occur with a higher frequency; classes such as Abuse,
Arrest, Assault, Burglary and Shoplifting are often incorrectly predicted. The
classification is especially diffused between the first five classes. The most
striking misclassification result is observed for Vandalism and Arrest, with
Arrest being predicted $17\%$ of the time for samples belonging to class
Vandalism. This might indicate that the trajectories capture two anomalous
activities which can be accounted for by multi-label ground truth. The
increased amount of varied misclassification is an indicator of the added
complexity from including the 14th class.

###  5.2 Decoded-based classification results

The results obtained by the model proposed in Section 4.2 for the 13 crimes
classification task, are presented in Table 5. Model M1.2 obtains the highest
performance of $0.382$ in weighted accuracy. A similar performance is obtained
by the model trained on the trajectory-level ground truth after supervised
thresholding; this model achieves $0.364$ in weighted accuracy on the SMOTE
augmented dataset. There is an increase in weighted accuracy for the model
applied on the augmented as opposed to the original dataset. Therefore, we can
state that the dataset imbalance has a significant influence on the complexity
of the classification task.

| Model
---
ID
| Ground
---
truth
| Data
---
aug.
Accuracy (M) | Accuracy (W) | Precision (W) | Recall (W) | F-score (W) |  | Top-3
---
accuracy
| Top-5
---
accuracy
M1.1 | unsuper. | None | 0.478 $\pm$ 0.005 | 0.324 $\pm$ 0.003 | 0.477 $\pm$ 0.002 | 0.478 $\pm$ 0.005 | 0.476 $\pm$ 0.004 | 0.629 $\pm$ 0.007 | 0.861 $\pm$ 0.004
M1.2 | unsuper. | SMOTE | 0.408 $\pm$ 0.007 | 0.382 $\pm$ 0.005 | 0.497 $\pm$ 0.001 | 0.408 $\pm$ 0.007 | 0.428 $\pm$ 0.006 | 0.577 $\pm$ 0.010 | 0.846 $\pm$ 0.003
M2.1 | super. | None | 0.503 $\pm$ 0.005 | 0.318 $\pm$ 0.002 | 0.494 $\pm$ 0.005 | 0.503 $\pm$ 0.005 | 0.495 $\pm$ 0.004 | 0.642 $\pm$ 0.006 | 0.855 $\pm$ 0.005
M2.2 | super. | SMOTE | 0.444 $\pm$ 0.004 | 0.364 $\pm$ 0.003 | 0.521 $\pm$ 0.004 | 0.444 $\pm$ 0.004 | 0.466 $\pm$ 0.004 | 0.596 $\pm$ 0.004 | 0.843 $\pm$ 0.004

Table 5: Results for the decoded-based classification model in combination
with different trajectory-level ground truth generation techniques.

By looking at the variations in weighted accuracy, the models trained on the
augmented dataset achieve significantly larger results as compared to the
original and imbalanced version of HR-Crime, while the macro accuracy tends to
show some decrease. This indicates that the models are focusing on
consistently learning an account of all the crime classes when being presented
with the same number of trajectory segment samples per class. Comparing M1.2
and M2.2 based on macro accuracy, the model trained on the supervisedly
thresholded data (i.e. model M2.2) obtains slightly higher results than the
unsupervised based one. In terms of weighted F-score, the results are slightly
over $0.4$ score, which indicates that the models are balanced as of precision
vs. robustness.

Figure 13: Confusion matrix depicting the average results using M1.2 setting.

We present the Top-3 and Top-5 macro accuracy for the same experiments
presented in Table 5. The Top-3 accuracy results for all the conducted
experiments reach a performance around the $0.6$ score, while for the Top-5
accuracy, the average performance is around the $0.8$ score. The accuracy
almost doubles as compared to the Top-1 accuracy scores presented in Table 5.
The model M1.2 achieves $0.577$ in Top-3 and $0.846$ in Top-5 accuracy. This
again, indicate low inter-class variability in HR-Crime.

We compare the best performing model M1.2 with the rest of decoded-based
classification experiments in Table 5: M2.2, M1.1, and M2.1. The statistical
analysis shows that the obtained results of the models are drawn from a normal
distribution, except for the comparison of M1.2 and M2.1. With a confidence
threshold $\alpha=0.05$, we apply the null hypothesis stating that the results
of the compared models come from the similar distributions. The obtained
P-values are 0.056, 0.001, and 0.109, respectively. These values indicate that
the null hypothesis can be accepted for 3 of the comparisons. Comparison of
our model M1.2 with models M2.2 and M1.1 obtain low P-values which implies
that the models do not produce similar results.

For a further quantitative interpretation of the results presented in Table 5,
we analyse the confusion matrix in Figure 13 for M1.2 model. The per crime
results are consistent with the decrease in overall accuracy noted for MPED-C,
as compared to the decoded-based model. Figure 13 shows that the most accurate
classification is achieved for class Road Accidents, with an average accuracy
of $0.64$. This is explained by the very distinctive nature of actions
captured by the videos in this crime class: the number of skeletons
participating in this type of crime is limited since the person driving the
vehicle is usually occluded, while the person being affected by the crime
(i.e. the accident) is more clearly visible, especially in the first time-
steps of the accident. Other classes for which an accuracy performance close
to the $0.5$ score is obtained are Abuse, Arson, Shoplifting, and Stealing.
Surprisingly, there is almost no confusion between classes Stealing and
Shoplifting given the similar nature of the actions involved with these two
crimes. This indicates a good demarcation between these two classes. As
indicated by the confusion matrix in Figure 13, crimes such as Explosion,
Shooting, and Vandalism are often misclassified as class Fighting. This
situation occurs due to the violent and aggressive nature of actions belonging
to this class, which can be quite common to crimes in general. What appears
more striking from this confusion matrix is the misinterpretation between
class Explosion and Shooting which occurs 21% of the time. This might be the
case since people have similar fleeing reactions triggered by the loud noise
factor that is common for the two events.

The confusion matrix in Figure 13 also shows that two crime classes obtain
accuracy scores around the $0.1$ score: Explosion and Vandalism.

We deduce that there exist two influencing factors: low-class representation
and feature involvement. The two classes are the least represented in the non-
augmented HR-Crime dataset, with 50 trajectories for class Explosion and 45
for Vandalism (see Table 1).

Quantitatively comparing the proposed classification model based on decoded
trajectory reconstructions and the MPED-C model (see Section 4.1.1) shows that
the decoded-based model (M1.2 model ID) achieves a marginally better
performance (MPED-C5.2 model ID). Both models are trained on the proposed
unsupervised trajectory-level generated ground truth with an augmented version
of 13 crime categories of HR-Crime dataset. The decoded-based model M1.2 is
quantitatively superior with an average increase of $0.04$ over all the
evaluation metrics. The most significant increase is reported for the weighted
accuracy: model M1.2 achieves a $0.38$ score, while the MPED-C5.2 obtains a
result of $0.30$. A plausible explanations for the superior performance of the
decoded-based model might be regarding the data augmentation techniques
applied to each setting. SMOTE, used for model M1.2, is an automatized
methodology that infers new data points from neighbouring original samples.
The ‘shifting’ technique used for MPED-C5.2 is more laborious and prone to
error since the ‘shifting’ factor must be determined empirically. Moreover, it
is difficult to generalize the same factor for all samples. Due to some
limitations detailed in Section 3.2, the SMOTE technique could not be applied
for the case of model MPED-C5.2. Therefore, these incongruencies in data
augmentation between the models might explain the performance gap.

Furthermore, the input data of the decoded-based M1.2 model is the
trajectories reconstructions produced by MPED-RNN. Following from the
reconstruction process used by MPED-RNN, these representations carry an
implicit notion of the anomaly score proposed by MPED-RNN. The MPED-C5.2
model, however, receives the trajectory encodings of MPED-RNN; the encodings
are produced by the first component (i.e. encoder) of the MPED-RNN network, at
which point the anomaly score is not yet known. This indicates that the MPED-C
model is required to infer independently the anomaly characteristics of the
trajectories. In this sense, the reconstruction based model M1.2 has an
advantage since it implicitly includes that information.

##  6 Conclusion and Future work

In this work, we explored the task of human-related crime classification from
surveillance footage using the HR-Crime dataset. Given the extracted skeletal
trajectories, we proposed a supervised as well as an unsupervised methodology
to generate trajectory-level abnormality ground truth and introduced two
methods for trajectory-level augmentation of HR-Crime. Later, we present our
crime classification framework that is based on human skeleton trajectories
and builds on top of the MPED-RNN encoder-decoder architecture for human-
related anomaly classification. We present an extensive body of experiments to
assess the proposed framework and obtained top-weighted accuracy performances
of $0.30$ (encoded-based) and $0.38$ (decoded-based) for the 13 crimes
classification. The obtained increase in top-5 accuracy performance reinforced
the complexity of the dataset, following from inter-class similarities. For
the normal and crime activities classification, the obtained top-weighted
accuracy performance of $0.24$ indicated the added challenge posed by the high
intra-class variance of the ‘normal’ category. With our experiments we
confirmed the negative impact of the dataset imbalance and the suitability of
the proposed data augmentation techniques for the task at hand leading to
significant boost in classification performance.

Through our research, we demonstrated that human skeletal trajectory analysis
is a feasible approach to crime-related anomaly classification. Possibilities
for future work include extending the novel HR-Crime dataset with labels that
capture more accurately the nature and boundaries of anomalous events in the
surveillance videos. As confirmed by the significantly high top-5 performances
of our proposed approach, we expect the multi-label definition of crime
categories as well as more accurate definition of non-crime activities to be a
sensible direction for future endeavours in the field.

## References

  * Ahmed et al. (2018) Ahmed, S.A., Dogra, D.P., Kar, S., Roy, P.P., 2018.  Surveillance scene representation and trajectory abnormality detection using aggregation of multiple concepts.  Expert Systems with Applications 101, 43–55.
  * Akcay et al. (2018) Akcay, S., Atapour-Abarghouei, A., Breckon, T.P., 2018.  Ganomaly: Semi-supervised anomaly detection via adversarial training.  Asian Conference on Computer Vision .
  * Akçay et al. (2019) Akçay, S., Atapour-Abarghouei, A., Breckon, T.P., 2019.  Skip-ganomaly: Skip connected and adversarially trained encoder-decoder anomaly detection.  Int. Joint Conf. on Neural Networks .
  * Arroyo et al. (2015) Arroyo, R., Yebes, J.J., Bergasa, L.M., Daza, I.G., Almazán, J., 2015.  Expert video-surveillance system for real-time detection of suspicious behaviors in shopping malls.  Expert systems with Applications 42, 7991–8005.
  * Boekhoudt et al. (2021) Boekhoudt, K., Matei, A., Aghaei, M., Talavera, E., 2021\.  Hr-crime: Human-related anomaly detection in surveillance videos.  Conf. on Computer Analysis of Images and Patterns .
  * Chandola et al. (2009) Chandola, V., Banerjee, A., Kumar, V., 2009.  Anomaly detection: A survey.  ACM Computing Surveys 41, 1–58.
  * Chawla et al. (2002) Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P., 2002\.  Smote: synthetic minority over-sampling technique.  Artificial Intelligence Research 16, 321–357.
  * Collins et al. (2000) Collins, R.T., Lipton, A.J., Kanade, T., 2000.  Introduction to the special section on video surveillance.  Transactions on Pattern Analysis and Machine Intelligence 22, 745–746.
  * Dalal and Triggs (2005) Dalal, N., Triggs, B., 2005\.  Histograms of oriented gradients for human detection.  Conf. on Computer Vision and Pattern Recognition .
  * Doshi and Yilmaz (2020) Doshi, K., Yilmaz, Y., 2020\.  Any-shot sequential anomaly detection in surveillance videos.  Conf. on Computer Vision and Pattern Recognition .
  * Doshi and Yilmaz (2021) Doshi, K., Yilmaz, Y., 2021\.  Online anomaly detection in surveillance videos with asymptotic bound on false alarm rate.  Pattern Recognition , 107865.
  * Emonet et al. (2011) Emonet, R., Varadarajan, J., Odobez, J.M., 2011.  Multi-camera open space human activity discovery for anomaly detection.  Conf. on Advanced Video and Signal Based Surveillance .
  * Febin et al. (2020) Febin, I., Jayasree, K., Joy, P.T., 2020.  Violence detection in videos for an intelligent surveillance system using mobsift and movement filtering algorithm.  Pattern Analysis and Applications 23, 611–623.
  * Gehan (1965) Gehan, E.A., 1965.  A generalized wilcoxon test for comparing arbitrarily singly-censored samples.  Biometrika 52, 203–223.
  * Greff et al. (2016) Greff, K., Srivastava, R.K., Koutník, J., Steunebrink, B.R., Schmidhuber, J., 2016.  Lstm: A search space odyssey.  Transactions on Neural Networks and Learning Systems 28, 2222–2232.
  * Hasan et al. (2016) Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A.K., Davis, L.S., 2016.  Learning temporal regularity in video sequences.  Conf. on Computer Vision and Pattern Recognition .
  * Hsu and Lachenbruch (2005) Hsu, H., Lachenbruch, P.A., 2005\.  Paired t test.  Encyclopedia of Biostatistics 6.
  * Kingma and Ba (2014) Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization.  Conf. on Learning Representations .
  * Lee et al. (2018) Lee, W.K., Leong, C.F., Lai, W.K., Leow, L.K., Yap, T.H., 2018.  Archcam: Real time expert system for suspicious behaviour detection in atm site.  Expert Systems with Applications 109, 12–24.
  * Lin and Purnell (2019) Lin, L., Purnell, N., 2019.  A world with a billion cameras watching you is just around the corner.  The Wall Street Journal .
  * Loganathan et al. (2019) Loganathan, S., Kariyawasam, G., Sumathipala, P., 2019.  Suspicious activity detection in surveillance footage.  Conf. on Electrical and Computing Technologies and Applications .
  * Luo et al. (2017) Luo, W., Liu, W., Gao, S., 2017.  A revisit of sparse coding based anomaly detection in stacked rnn framework.  Conf. on Computer Vision .
  * Malhotra et al. (2016) Malhotra, P., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., Shroff, G., 2016\.  Lstm-based encoder-decoder for multi-sensor anomaly detection.  Conf. on Machine Learning .
  * Markou and Singh (2003) Markou, M., Singh, S., 2003\.  Novelty detection: A review—part 1: Statistical approaches.  Signal Processing 83, 2481–2497.
  * Maron and Lozano-Pérez (1998) Maron, O., Lozano-Pérez, T., 1998\.  A framework for multiple-instance learning.  Advances in Neural Information Processing Systems .
  * Morais et al. (2019) Morais, R., Le, V., Tran, T., Saha, B., Mansour, M., Venkatesh, S., 2019.  Learning regularity in skeleton trajectories for anomaly detection in videos.  Conf. on Computer Vision and Pattern Recognition .
  * Nasaruddin et al. (2020) Nasaruddin, N., Muchtar, K., Afdhal, A., Dwiyantoro, A.P.J., 2020\.  Deep anomaly detection through visual attention in surveillance videos.  Journal of Big Data 7, 1–17.
  * Rabie (1990) Rabie, S., 1990.  Applications of expert systems to network surveillance, in: Network Management and Control. Springer, pp. 249–262.
  * Ramachandra et al. (2020) Ramachandra, B., Jones, M., Vatsavai, R.R., 2020.  A survey of single-scene video anomaly detection.  Transactions on Pattern Analysis and Machine Intelligence .
  * Reynolds (2009) Reynolds, D.A., 2009.  Gaussian mixture models.  Encyclopedia of Biometrics 741, 659–663.
  * Shapiro and Wilk (1965) Shapiro, S.S., Wilk, M.B., 1965\.  An analysis of variance test for normality (complete samples).  Biometrika 52, 591–611.
  * Sultani et al. (2018a) Sultani, W., Chen, C., Shah, M., 2018a.  Real-world anomaly detection in surveillance videos.  Conf. on Computer Vision and Pattern Recognition .
  * Sultani et al. (2018b) Sultani, W., Chen, C., Shah, M., 2018b.  Real-world anomaly detection in surveillance videos.  Conf. on Computer Vision and Pattern Recognition .
  * Tran et al. (2015) Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M., 2015.  Learning spatiotemporal features with 3d convolutional networks.  Conf. on Computer Vision .
  * Ullah et al. (2021) Ullah, W., Ullah, A., Haq, I., Muhammad, K., Sajjad, M., Baik, S.W., 2021.  Cnn features with bi-directional lstm for real-time anomaly detection in surveillance networks.  Multimedia Tools and Applications 80, 16979–16995.
  * Yun et al. (2014) Yun, K., Jeong, H., Yi, K.M., Kim, S.W., Choi, J.Y., 2014\.  Motion interaction field for accident detection in traffic surveillance video.  Conf. on Pattern Recognition .
  * Zhang and Sabuncu (2018) Zhang, Z., Sabuncu, M.R., 2018\.  Generalized cross entropy loss for training deep neural networks with noisy labels.  Conf. on Neural Information Processing Systems .
  * Zhou and Kwan (2018) Zhou, J., Kwan, C., 2018.  Anomaly detection in low quality traffic monitoring videos using optical flow.  Pattern Recognition and Tracking XXIX 10649, 122–132.