[table]capposition=top

# A Novel Skeleton-Based Human Activity Discovery Using Particle Swarm
Optimization with Gaussian Mutation

Parham Hadikhani*, Daphne Teck Ching Lai and Wee-Hong Ong P. Hadikhani, D.T.C.
Lai, and W.H. Ong are with the School of Digital Science, Universiti Brunei
Darussalam, Brunei. E-mail: {20h8561, daphne.lai<EMAIL_ADDRESS>
###### Abstract

Human activity discovery aims to cluster the activities performed by humans
without any prior information on what defines each activity. Most methods
presented in human activity recognition are supervised, where there are
labeled inputs to train the system. In reality, it is difficult to label
activities data because of its huge volume and the variety of human
activities. This paper proposes an unsupervised framework to perform human
activity discovery in 3D skeleton sequences. First, an approach for data pre-
processing is presented. In this stage, important frames are selected based on
kinetic energy. Next, the displacement of joints, statistical displacements,
angles, and orientation features are extracted to represent the activities
information. Since not all extracted features have useful information, the
dimension of features is reduced using PCA. Most methods proposed for human
activity discovery are not fully unsupervised. They use pre-segmented videos
before categorizing activities. To deal with this, we have used a sliding time
window to segment the time series of activities with some overlapping. Then,
activities are discovered by our proposed Hybrid Particle swarm optimization
(PSO) with Gaussian Mutation and K-means (HPGMK) algorithm to provide diverse
solutions. PSO is used due to its straightforward idea and powerful global
search capability which can identify the ideal solution in a few iterations.
Finally, k-means is applied to the outcome centroids from each iteration of
the PSO to overcome the slow convergence rate of PSO. The experiment results
on five datasets show that the proposed framework has superior performance in
discovering activities compared to the other state-of-the-art methods and has
increased accuracy of at least 4 % on average.

###### Index Terms:

Human activity discovery, Unsupervised learning, Clustering, Feature
extraction, Dimension reduction, Skeleton sequence, Particle swarm
optimization

##  I Introduction

Human Activity Recognition (HAR) has attracted much attention due to its
applications in fields such as human-computer interaction, intelligent
transportation systems, and monitoring applications [1]. Activity recognition
aims to identify actions and activities that humans perform in different
environments automatically. The input to a vision-based HAR system is a
sequence of frames of a person performing different movements. The output is a
set of labels representing the actions taken or activities in those movements.
Many existing works use visual data as input. But such data have considerable
complexity detrimental to the performance of HAR systems. These complexities
include cluttered background, changes in brightness and points of view. 3D
skeleton data partially overcomes these complexities and protects people’s
privacy when RGB data is not captured. Each frame represented by 3D
coordinates of the main body joints is appropriate for representing human
actions [2] and can be easily obtained in real-time with low-cost depth
sensors [3].
As shown in Fig. 1, there are at least seven steps in vision-based HAR
systems. Vision sensors capture activities performed by a person. The skeletal
information comprising joints coordinate are extracted from captured videos,
containing image sequences called frames. Meaningful features are then
extracted for more accurate activity discovery. The system without using
manual annotations and having any guidance for activities discovers them by
clustering the most similar activities from a set of different activities. In
other words, the system tries to differentiate observed activities based on
the likeness of extracted features. The discovered activity clusters are used
in the learning process to model each cluster of activity and recognize future
activities.
Significant progress has been made in the supervised learning of activity
models [4], illustrated in blocks (f) and (g) of Fig. 1. The learning and
recognition steps rely on human-labeled training data to categorize activities
if activity discovery (block e) was not performed. Human activity discovery is
a part of the HAR process where activities are categorized based on their
similarities without any knowledge of activity labels or any information that
characterizes an activity, making this step particularly challenging. In other
words, activity discovery is like a child’s learning. There is no prior
information to define a specific sequence of movements to mean a particular
activity such as crawling or waving and so forth to the child learner. Using
the ability to differentiate, they learn from unlabeled data and form a model
that can post-label new data based on that training. In human activity
discovery, there is no known information or knowledge about a particular
movement, including its start to end, for example when someone is picking up
something. This means the input is a series of movements without knowing the
start and endpoints to indicate each activity. Some existing work has
segmented the input data by activity [2]. Thus, the start and endpoints of the
activities are already known, although the method of grouping activities may
be unsupervised.
In this paper, we focus on the less developed activity discovery comprising
the block (d) to (e) of Fig. 1 by developing an effective methodology to
extract good features and cluster activities without any label. Keyframe
selection [5] and PCA are used to remove redundant frames and features to
reduce time complexity and increase accuracy.A pre-processing and feature
extraction methodology are proposed to prepare information and extract
features from the most informative joints and bones, including joint
displacement, joint orientation, and statistical time domain. As our first
study, a hybrid PSO with Gaussian Mutation and k-means (HPGMK) clustering that
requires a known prior of the cluster number is proposed to find activities.
Sometimes particles converge to a specific point between the best global and
personal positions and get trapped in local optima. This difficulty arises
when the swarm’s variety reduces and the swarm cannot escape from a local
optimum [6]. To address this, a hybrid PSO with Gaussian mutation is proposed
to promote diversity to avoid early convergence. Then, K-means is applied to
the centroids obtained by PSO to refine their location and get the best
possible solution. Our methodology performs activity discovery using
unsegmented input data and the proposed techniques used are unsupervised with
no prior knowledge of the labels of the different activities.

Figure 1: Overview of HAR system: (a) performed activities are (b) captured by
a Kinect sensor. (c) After that, pose of humans are estimated by extracting
joints. (d) To make raw data more usable, their salient and defining features
are identified. (e) Based on the similarities and differences, activities are
discovered. (f) Afterwards, the system begins to learn from the discovered
activities and (g) finally human activities are recognized.

The main contributions of this paper are:

  * •

A methodology consists of keyframe selection, feature extraction, and PCA to
represent human activities. The features based on displacement, statistical
and orientation are extracted simultaneously to represent all the movement
aspects of human activities. This makes the discovery part perform better
because it has comprehensive information about the activities.

  * •

A hybrid clustering algorithm called HPGMK to discover and group unlabelled
human activities observations into individual activity classes. PSO is
customized by applying the Gaussian mutation, based on the advantages of two
methods [7] and [8], on the global best’s centroids to increase the diversity
of selected clusters in the global best. Also, due to the increase in the
variety of solutions, the proposed algorithm reaches the desired solution in a
smaller number of iterations. That is why the clustering time is reduced.

  * •

Integrating K-means to refine the obtained cluster centers from the PSO to
improve the exploitation of the algorithm. In PSO, when approaching the final
solution, the speed of the particles decreases greatly and it becomes
difficult to reach the best optimum solution. For this reason, the problem of
PSO is solved by using the advantage of k-means in local search and applying
it to the obtained solution from PSO.

  * •

An evaluation of the performance of the proposed methodology compared with the
latest fully unsupervised methodology for HAD and the state-of-the-art HAD and
deep clustering methods. The proposed method demonstrates superior performance
on five challenging 3D skeleton-based activity analysis datasets.

In this paper, the background and related methods are discussed in section II.
The methodology is described in section III. We present the evaluation of the
proposed approach, comparing with state-of-the-art (SOTA) techniques in
Section IV, and finally, the conclusion is stated in Section V.

##  II Related works

Feature extraction from 3D skeletal human activities data. Skeletal data
includes the number of joints, and each joint contains three-dimensional
coordinates. Since the motion of the joints has essential information for any
activity, feature extraction is vital. There are various methods for
representation of the motion of skeletal joints such as calculating the
difference between the joints in the same frame and the same joints in
different frames [9], using Histogram Oriented of Joints [10], Dynamic Time
Warping algorithm [11], Covariance of 3D Joints [12], generating joint
rotation matrix concerning the person’s torso [13], and extracting the angles
and orientations of the most informative body joints [14]. However, most of
these methods extract one aspect of the skeletal data features, leading to
other important aspects of activities being overlooked. As a result, there is
a decrease in accuracy in the final result because of the insufficient
discriminating ability of the extracted features. Moreover, due to the
complexity of feature calculations, some of these methods cause computational
latency. The difference between our paper with previous works for extracting
features is that We have combined the feature extraction techniques from [9,
15, 14] to extract three skeletal features from informative joints and
keyframes. Previously, these features have been applied separately and most of
them have used all of the joints and frames, which increases additional
information. This increases the time complexity and reduces identifying the
activities’ performance. Some methods like [5] and [16] have tried to select
some frames that are more distinguishing compared to other frames and remove
redundant information with the assistance of kinetic energy. However, these
methods are applied to each activity sample separately. In other words, in [5]
and [16], keyframes were selected in a supervised manner. In contrast, we
apply the above methods in our proposed method to select keyframes without
knowing the beginning and end of the activities.
Particle Swarm Optimization Clustering and dimension reduction. One of the
important methods to do discovery is to use clustering. Clustering is a method
that categorizes data points based on similarities and dissimilarities. One of
the common methods of clustering is K-means. But it has problems such as poor
convergence rate and local optimum. One of the ways to overcome these problems
is to use evolutionary algorithms like PSO. PSO is a population-based
algorithm where each individual represents a potential solution, which makes a
better problem space search. PSO greatly reduces the possibility of getting
stuck in local optimum due to using local search and global search
simultaneously. To create a novel clustering algorithm, Malarvizhi et al. [17]
combined PSO and feature linkage-based weight reduction. The method determined
the weight of each feature using the Mahalanobis distance to choose the
feature and do the clustering automatically. Sharma et al. [18] developed a
hybrid PSO clustering for network-based sustainable computing. They applied
the mutation operator to ensure diversity among the solutions to keep the
algorithm’s balance between exploration and exploitation. Rengasamy et al.
[19] Introduced a new memory dimension termed family memory and added to the
two already existing ones of cognitive memory and social memory. This memory
was used to collect the information from the particles that favor a certain
cluster. Additionally, they utilized the K-means to initialize centroids for
PSO clustering to enhance the traditional PSO. Cai et al. [20] proposed a new
clustering method based on combining density peaks clustering with PSO. They
employed a technique to compute density peaks to avoid falling into a local.
They also presented a new fitness criteria function to optimally explore K
cluster centers to obtain the optimal global solutions. However, these methods
do not address all the weak points of PSO. Some of them either deal with the
issue of reducing the speed of PSO when approaching the optimal solution or
the issue of reducing the diversity among the solutions during the search.
Different from the above-mentioned methods, we overcome the weakness noted for
PSO simultaneously by using Gaussian mutation and K-means. By employing
Gaussian mutation, the variety of the solutions the PSO’s capacity to exploit
around potential solutions gets effectively enhanced. We also utilize K-means
to solve the PSO problem: its convergence speed slows down when it approaches
the global optimum. We also use K-means to solve the problem of reducing the
convergence speed of PSO when approaching the global optimum by using the fast
speed of K-means in local search. Zhang et al. [21] introduced a feature
selection method based on PSO that combined fuzzy clustering and feature
importance (PSOFS-FC). They presented a new objective function based on
F-measure and filling risk for PSO with fuzzy clustering to assess the impact
of missing data in class imbalance. To overcome the dimensionality curse, Song
et al. [22] presented a three-phase feature selection technique based on
correlation-guided clustering and particle swarm optimization. First, they
combined a filter approach and a feature clustering-based method to reduce the
search space. Then, an enhanced integer PSO was used to select the best
feature subset. Unlike PSOFS-FC and other mentioned methods, which combine
clustering and feature selection, we reduce extracted features’ dimensions
before clustering. Although good results have been obtained in these methods,
due to the simultaneity of feature selection and clustering, clustering
becomes problematic in high-dimensional data and the clustering time increases
greatly. In HAD, spending time to discover activities is important because of
its applications, such as use in security areas to identify suspicious
behavior or in hospitals to check patients’ status. For this reason, our
proposed method reduces the feature dimensions before clustering by using PCA
to make the data more clusterable. PCA speeds up the clustering algorithm by
removing correlated features that do not influence decision-making. As a
result, the algorithm’s clustering time decreases dramatically with fewer
features. Thus, not only the speed of clustering increases but also the
accuracy of clustering increases because the clustering process is performed
on highly important features. Regarding detecting outlier and noisy data,
Hubert et al. [23] proposed a combination method to make PCA robust to
outliers. They combined projection pursuit [24] with h robust covariance
estimation in lower dimensions [25]. Moreover, they applied a diagnostic plot
to detect the outliers. Candes et al. [26] proposed a technique to improve the
performance of PCA. They used a low-rank and sparse component for PCA to avoid
outliers and achieved good performance in the application of Alzheimer’s
Disease Recognition [27]. Rahmani et al. [28] presented a provable algorithm
to identify the outliers based on PCA. For this reason, they employed a convex
optimization problem to evaluate the data points based on the innovation
search method. Despite the very good performance of the presented methods to
improve PCA, they have more execution time than the original PCA. On the other
hand, the focus of our work is on the improvement of feature extraction and
discovery for human activities. As mentioned before, time is very important in
HAD. That is why we use PCA to prevent the increase in the computation of time
for the presented framework. It is worth mentioning that the obtained results
show that PCA has reduced the dimensions of the features and improved HAD
performance significantly. Recognition and discovery of 3D skeletal human
activity. Many studies in HAR used supervised approaches [29, 30, 31]. Yadav
et al. [32] combined long-short term memory networks and convolutional neural
networks for recognizing human activity and fall detection. They used some
handcrafted features, including geometrical and kinematic features to guide
their proposed model. Zhang et al. [33] proposed an end-to-end semantics-
guided neural networks framework. They provided two semantic forms based on
joints and frames and used GCN and CNN layers to find the dependence of joints
and frames, respectively. Si et al. [34] proposed a novel model based on a
recurrent network. They applied a graph convolutional layer into the LSTM
network to improve the performance of traditional LSTM. They also introduced
an attention gate inside the LSTM to capture discriminative features. Xia et
al. [35] provided a graph convention network based on spatial and temporal.
They applied an attention layer to the model to generate discriminative
features and modified feature maps. Then, a softmax classifier was used to
categorize the activities. Cai et al. [36] introduced a scheme to capture
visual information surrounding each skeleton joint and achieve local motion
cues. They extracted features from skeleton and RGB data using two graph
convolutional networks. Then, both types of features were concatenated and
activities were classified by calculating a score based on linear blending.
The problem with these approaches is that they require activity labels in the
training data. Humans annotated the labels during data preparation. It makes
these methods impractical with real-life data that are mainly unlabeled. Our
work does not use labels for training in our algorithm. The algorithm
discovers activities by looking for similar features between them. In
addition, the methods mentioned above use deep learning techniques. In
contrast, as a first study, we do not use them in our method and the focus is
on developing a comprehensive model for HAD as a baseline.
Several approaches try to address the HAR in an unsupervised way. Wang et al.
[37] presented a deep clustering method based on a dual-stack auto-encoder to
map raw data to spatio-temporal features. After extracting features, the
radial basis function neural network was used to classify the activities. Su
et al. [4] provided an unsupervised model by employing a bi-directional
recurrent neural network and used K-NN to classify the activities. Liu et al.
[38] designed a spatial-temporal asynchronous normalization method to reduce
redundant information related to time and normalize the spatial features.
Next, they used a gated recurrent unit auto-encoder to feature vectors. First,
all of these methods received the activities already segmented which has
enabled them to be aware of the differences between the activities before
performing the recognition. Second, in most of these methods, only feature
extraction was performed without supervision. The supervised classification
method was used for the rest of the operations to learn activity models using
activity labels.
On the other hand, human activity discovery can automatically categorize human
activity in a fully unsupervised way and the challenge is learning from
unlabeled data. The majority of existing methods were developed for sensor-
based [39, 40] and RGB video data [41, 42]. The challenges of the sensor-based
approach are difficult to implement in the environment and take a long time to
install [43]. Furthermore, it is impractical for people to wear sensors
everywhere. With RGB videos, the problems faced are millions of pixel values,
illumination variations, viewpoint changes, and cluttered backgrounds [3]. In
this work, we concentrate on 3D skeleton-based data as it does not have the
problems of the other two data types. One of the first works in HAD was
performed by Ong et al. [43]. They proposed an autonomous learning technique
based on the mixture of the Gaussian hidden Markov model. They introduced an
incremental clustering approach based on k-means to discover the activities to
deal with the undefined number of clusters. An issue with their approach is
that they have used k-means to discover the activities that get stuck into the
local optimum easily and they have not examined all aspects of the skeleton
data features. Moreover, they extracted all the features from all joints,
resulting in more redundant data and increased discovery errors. Recently
several approaches have been proposed by [2] to solve HAR without labels. In
their proposed methods, several clustering methods, including spectral
clustering (SC), elastic net subspace clustering (ENSC), and sparse subspace
clustering (SSC) were used, which used covariance descriptors. They used an
affinity matrix to find similarities and then applied spectral clustering. In
addition, a time stamp pruning approach was used to remove redundant data to
normalize temporal features. Although they have achieved impressive results,
the data used were already segmented by activity before performing discovery.
It means that the activities are already categorized. Because each sample
contains an activity that performs completely. In other words, the beginning
and end of each activity are clear.
In a nutshell, many methods have been proposed to recognize human activities
in a supervised and unsupervised manner and have obtained very acceptable
results. But the problem with these methods is that they ignore the discovery
step. These methods are useless because labeling activities do not occur in
real-time. They also need a lot of computation time for training. In reality,
training data are not available. If we have a dynamic big and growing video
related to human activity, we are not sure of the labels to predefine the
rules. This can be a real challenge. On the other hand, due to the variety of
human activities, these methods need to be retrained for new activities,
making them not scalable. In the case of HAD, in addition to the fact that
there are very limited methods, these methods have problems including using
shallow methods which are not very accurate or not fully performing the
discovery process in an unsupervised manner. In this paper, we propose an
approach to discovering activities from untrimmed videos without knowing the
label of activities. It makes this method suitable for use in real-world
scenarios. In addition, we use a feature extraction approach to examine most
aspects of skeleton data along with a keyframe selector to reduce redundant
information and discovery accuracy.

##  III Proposed human activity discovery

The proposed framework consists of two main stages. In the first phase, we
propose an approach that can extract high-quality features in an unsupervised
manner. Three crucial factors should be taken into account in this matter.
First, not all the captured frames are important. Due to the similarity of
frames and noises, HAD performance reduces sharply. Therefore, we need frames
that show the salient features of the activities. Also, not all the Joints
have an effective role in discovering activities, such as the torso, which is
constant in most activities. Extracting features from these joints increase
time complexity. Second, the extracting features should accurately reflect all
aspects of human activity by considering all factors such as both spatial and
temporal. Third, the feature dimension needs to be minimal to make the
clustering operation easier (a larger dimension confuses the clustering
process). To cover the first phase, we present a pre-processing method as
shown in Fig. 2(Stage 1). We employ an innovative approach based on kinetic
energy to select representative frames of video sequences as keyframes. In
other words, we seek to select the frames that show the most prominent
characteristics of the activities as the keyframe. However, the selection of
particular Keyframes without losing the required information is a challenging
task. When only the local maximum kinetic energies are considered keyframes,
the sequence of activities may break up and no longer represent activities.
For example, in the walking activity, considering keyframes with high local
energy, only positions where both legs move away from each other are
considered keyframes. In contrast, positions where both legs are placed
together show a part of the walking process that is lost in this state. For
this reason, we employ keyframe selection based on local maximum and minimum
kinetic energy that can find representative frames and reduce complexity.
Moreover, it can maintain the order of the activity. To avoid increasing
computational time and overlapping among activities, we select joints
(informative) that have a vital role in displaying activities based on
experimental tests. To increase the discovery performance, it is necessary to
extract features to represent all aspects of each activity. for this purpose,
we design a method to represent the activities based on spatial and temporal
displacement, statistical, and orientation features. The displacement-based
representations provide the view-invariant Spatio-temporal human
representations due to invariant to positions and orientations of humans with
respect to the camera. To obtain features that are invariant to human scale
changes, orientation-based representations are extracted to find relative
information between human joints. Statistical features describe how activity
evolves over time, especially when separating actions involving the arms and
legs. Therefore, statistical time-domain characteristics represent changes in
a set of postures for a time-domain activity. We employ a PCA to address the
third factor to reduce dimensionality and make the extracted feature more
clusterable while high-importance features are kept. We adopt a sliding window
over the untrimmed skeleton streaming sequences to perform activity discovery.
To increase the number of samples and avoid pruning important events like a
transition between activities we employ overlapping sliding windows increases
performance. In the second phase, we propose a novel clustering algorithm
based on hybrid PSO called HPGMK to discover human activities (Fig. 2(Stage
2)). The key benefit of PSO is that there are fewer parameters to set.
Contrary to genetic algorithms, PSO does not use complex evolutionary
operators like the crossover, making it less complicated. However, the issue
is that its convergence speed slows down when it approaches the global
optimum. For this reason, we combine PSO with k-means to use the fast speed of
k-means to reach the local optimum to improve PSO’s performance. The Gaussian
mutation is also employed on the global best particle to search for areas
around it that have a high potential to be selected and generate diverse
solutions and strike a balance between exploitation and exploration. In this
section, the proposed method is presented, as shown in Fig. 2. Keyframes are
first extracted from the input video based on the kinetic energy of all
frames. Three types of features including spatial and temporal displacement,
mean and standard deviation differences, and orientation and angle features
are extracted. Next, feature vectors are reduced by PCA and frames are sampled
in specific periods to segment the activity steam. Finally, the proposed
clustering technique assigns each sample to an appropriate category of
activities. The details of each part of the methodology are described below.

Figure 2: Methodology of the proposed approach has two stage pre-processing
and discovery. In pre-processing, keyframes are selected from the video
sequence by computing kinetic energy. Then, features based on different
aspects of skeleton including displacement, orientation, and statistical are
extracted from informative joints and bones. Principal components are then
chosen by applying PCA on the features. Next, overlapping time windows is used
to segment a series of keyframes as activity samples. In discovery stage
(HPGMK), Hybrid PSO clustering with Gaussian mutation operator is used to
discover the groups of activities. Eventually, K-means clustering is applied
to the resultant cluster centers to refine the centroids.

###  III-A Keyframe selection

Keyframe selection is a process of selecting frames reflecting the main
activities in the video. Some methods like [5] and [16] have tried to select
some frames that are more distinguishing compared to other frames. However,
these methods were applied to each activity sample separately. In other words,
in methods [5] and [16], keyframes were selected in a supervised manner. In
our proposed method, keyframes are selected without knowing the beginning and
end of the activities. To find the keyframes, the kinetic energy E($f_{i}$) of
each frame $f_{i}$ is calculated [5] using Eq.(1), based on the displacement
of joints over time. In this way, the movement of a joint j between frame i
and i-1 is calculated for all joints (J). The sum of the movements for all
joints is the energy of the current frame. Frames with local maxima and minima
amount of kinetic energy compared to neighboring frames are considered
keyframes (see Fig. 3) Because these are the energy’s extreme points, which
are meant to resemble crucial posture data.

$E(f_{i})=\sum_{j=1}^{J}E(f^{j}_{i})=1/2\sum_{j=1}^{J}(f^{j}_{i}-f^{j}_{i-1})^{2}$
(1)

Figure 3: Illustration of the keyframe selection.

###  III-B Feature extraction

To represent the activities, a set of statistical displacements, angles and
orientation features for encoding key aspects of activities are extracted.
These important features are extracted from selected (informative) joints in
the data to describe the shape and movement of human. Selected joints have
been obtained based on experimental tests that included left and right hand,
foot, hip, shoulder, elbow and knee. These joints have more movement and
contribution than other joints such as torso in activities. We use information
related to the position and movement of joints, the orientation and angle
between a pair of bones and activity variation over time. The normalization
procedure [9] is performed on all features.

####  III-B1 Displacement features

Joint displacement-based features encode information on the position and
motion of body joints, particularly displacement between joints of a pose and
position differences of skeleton joints across time [9].

  * •

Spatial joint displacement is computed using pairwise Euclidean distances
between joints $P_{i}$ and $P_{j}$ (i $\neq$ j) in 3D space in the same frame,
Eq.(2). The joint pairs used are both hands, hands and head, and hip and feet
at both sides, giving 5 features per frame, (see Fig. 4(a)).

$PairwiseDistances=\sqrt{\sum_{x,y,z}(P_{i}-P_{j})^{2}}$ (2)

  * •

Temporal joint displacement is calculated based on two modes. $T_{cp}$ is the
difference between each selected joint $P_{i}$ in current frame $P^{c}_{i}$
and previous frame $P^{c-1}_{i}$ (see Fig. 4(b)) to determine the small
changes in joint movement over time (Eq.(3)). $T_{cn}$ is the difference
between each selected joint of current frame and the frame of neutral pose (We
randomly select a standing position as a neutral position) $P^{n}_{i}$,
illustrated in Fig. 4(c), to find general changes in joint movements as given
in Eq.(4).

$T_{cp}=P^{c}_{i}-P^{c-1}_{i}$ (3) $T_{cn}=P^{c}_{i}-P^{n}_{i}$ (4)

Figure 4: (a) Spatial displacement of pairwise joints in the same frame.
Temporal displacement of current frame from (b) previous frame and (c) the
neutral frame.

####  III-B2 Statistical features

The mean and standard deviation of time-domain features express how activity
changes over time, particularly in distinguishing between activities related
to the arms and legs. Thus, statistical time-domain features encode variations
across a collection of poses of an activity in time-domain. These features are
calculated by the difference of selected joint $P_{i}^{c}$ in current frame
from mean $P_{(i,mean)}$ and standard deviation $P_{(i,std)}$ of the selected
joint coordinates within an activity sequence as given by Eq.(5) and (6) [9].

  * •

Joint coordinate-mean difference

$P_{i_{(mean)}}^{c}=P_{i}^{c}-P_{(i,mean)},P_{(i,mean)}=\frac{1}{N}\sum_{c=1}^{N}P_{i}^{c}$
(5)

N is the number of frames.

  * •

Joint coordinate-standard deviation difference

$P_{(i,std)}^{c}=P_{i}^{c}-\sqrt{\frac{\sum_{i=c}^{N}(P_{i}^{c}-P_{(i,mean)})^{2}}{N}}$
(6)

####  III-B3 Orientation features

The three-dimensional coordinate system {x,y,z $\in$ $R^{3}$} represents
points as joints. x, y, and z denote the 3D coordinates of joints. Joints and
bones can be described by the orthonormal vectors [15] as follow:

$P^{f}_{i}=x^{f}_{i}e_{1}+y^{f}_{i}e_{2}+z^{f}_{i}e_{3}$ (7)
$B^{f}_{ij}=(x^{f}_{i}-x^{f}_{j})e_{1}+(y^{f}_{i}-y^{f}_{j})e_{2}+(z^{f}_{i}-z^{f}_{j})e_{3}$
(8)

where $P_{i}^{f}$ is the ith skeleton joint in the fth frame and
$e_{1},e_{2},e_{3}$ are orthonormal vectors. $B_{ij}^{f}$ is the bone between
two adjacent joints $P_{i}^{f}$ and $P_{j}^{f}$. Moreover, magnitude and
direction of two bones a and b are represented by geometric product where this
product is the sum of internal ($a.b$) and external ($a\wedge b$) product.
where the inner product is used to compute the length and angle between two
bones a and b. The outer product of two bones can be regarded as an oriented
plane containing a and b. The orientation and angles between bones features
are obtained in the process described as follows.

  * •

The rotation matrix is a transformation matrix that describes the rotation
from a bone to another. Three angles are required to define the rotation
matrix between two bones. The rotation angles are considered as orientation
features. The elements of these features are the rotation of bones relative to
the x, y, and z axes (see Fig. 5(a)).

  * •

The angle features consist of the angles between the bones of elbow-wrist and
shoulder-elbow at both sides and the angles between the bones of hip-knee and
knee-ankle at both sides. These angles are highlighted in Fig. 5(b) and
calculated as below:

$\theta=180\times\frac{arctan^{2}(\frac{\lVert bone_{i}\wedge
bone_{j}\rVert}{\lVert bone_{i}.bone_{j}\rVert})}{\pi}+180$ (9)

where $bone_{i}$ and $bone_{j}$ are determined by Eq.(8)

Figure 5: (a) Illustration of the rotation between two bones A and B.
$\alpha$, $\beta$, and $\gamma$ are the orientation of angles. (b) The angles
of the selected body bones. The angles of elbow-wrist and shoulder-elbow at
both sides and angles between the bones of hip-knee and knee-ankle at both
sides are used to calculate angle features.

###  III-C Feature selection and sampling

For fast clustering and complexity reduction, key features are extracted by
PCA. Then, sliding windows are used to segment frames into time windows. Each
window comprises of 15 frames. The overlap of sliding windows increases
performance. Because it increases the number of samples and avoid pruning
important events like transition between activities [44]. The first 15 frames
do not overlap while in other samples, their first frame starts from the last
frame of the previous sample (see Fig. 6).

###  III-D Proposed clustering

PSO is a population-based optimization algorithm [45]. A population is made up
of a number of particles and each particle represents a solution and moves
according to its speed. The changes in velocity and position of the particles
are calculated based on the following formula:

$x_{i}(t+1)=x_{i}(t)+v_{i}(t)$ (10) $\begin{split}v_{i}(t+1)=w\times
v_{i}(t)+c_{1}\times rand_{1}\times(pbest_{i}(t)-x_{i}(t))\\\ +c_{2}\times
rand_{2}\times(gbest(t)-x_{i}(t))\end{split}$ (11)
$w=\frac{w_{max}+t\times(w_{max}-w_{min})}{t_{max}}$ (12)
$c_{1}(t+1)=(c_{1_{max}}-c_{1_{min}})\times\frac{t}{t_{max}}+c_{1_{max}}$ (13)
$c_{2}(t+1)=(c_{2_{max}}-c_{2_{min}})\times\frac{t}{t_{max}}+c_{2_{max}}$ (14)

In Eq.(10) and (11) $x_{i}$(t) and $v_{i}$(t) are the position and velocity of
the particle i at time t respectively. $pbest_{i}$ is the best position found
by the particle i. $gbest$ is the best position found in the population. $w$
is the inertial weight defined by Eq.(12) and starts to decrease from
$w_{max}$. $c_{1}$ and $c_{2}$ are acceleration coefficients expressed by
Eq.(13) and (14). The $c_{1_{max}}$, $c_{2_{max}}$ and $c_{1_{min}}$,
$c_{2_{min}}$ are initial and final values, respectively, t is the number of
iterations and $t_{max}$ is the maximum number of iterations [20]. $rand_{1}$
and $rand_{2}$ are random variables between 0 and 1. Each solution is
evaluated by Eq.(15) which should be minimized to achieve proper clustering.

$SSE=\sum_{k=1}^{K}\sum_{\forall x\in c_{k}}\|x_{i}-\mu_{k}\|^{2}$ (15)

$x_{i}$ is a data point belonging to the cluster $C_{k}$ and $\mu_{k}$ is the
mean of the cluster $C_{k}$. $k$ is the number of clusters specified. To avoid
in local optimum, a Gaussian mutation operator based on [7] and [8] is applied
to the global particle as follows:

$\begin{split}v^{{}^{\prime}}_{gbest}(d)=v_{gbest}(d)\times G(0,h)\\\
\times(x_{max}(d)-x_{min}(d))\end{split}$ (16)

$x^{{}^{\prime}}_{gbest}(d)=x_{gbest}(d)+G(0,h)\times
v^{{}^{\prime}}_{gbest}(d)$ (17)

where $x_{gbest}$ and $v_{gbest}$ represent the position and velocity of
global best particle. $x_{max}$ and $x_{min}$ are the maximum and minimum
value in $d^{th}$ dimension. Gaussian (0,h) is Gaussian distribution with the
mean 0 and the variance h. the value of h is start with a high value to
increase the exploration ability of the algorithm to find interested region at
the beginning of the search. Then h decreases linearly during each iteration
according to Eq.(18), to increase the power of exploitation at the end of
search to reach optimum solution.

$h(t+1)=h(t)-(1/t_{max})$ (18)

where $t_{max}$ is the maximum number of iterations. Fig. 7 is an illustration
of the Gaussian mutation.
In general, the core of HPGMK is based on PSO which is a population-based
algorithm. The position of each particle in the population represents a
solution. In other words, each particle contains the position of the cluster
centers. Each particle updates its position using its velocity to reach the
optimum solution [46]. Furthermore, we have one objective function which is
SSE (sum square error). We evaluate each individual based on SSE (Eq.(15)). In
this process, an individual is chosen as the global best in each iteration
with the lowest SSE value among the rest of the individuals. To increase the
diversity of solutions, a Gaussian mutation is applied to the position and
velocity of the global best particle.

The velocity of the particles reduces quickly as PSO approaches the global
optimum, and in most circumstances, the ideal solution is not achieved. For
this reason, K-means is applied to the obtained centroids from PSO to refine
them. After the completion of the PSO process, the global best solution is
selected based on the SSE value. Then, the selected solution is modified by
averaging the position of data points in each cluster to select the best
position for the cluster centers. This continues until the position of the
clusters does not change. The routine of the proposed clustering algorithm is
shown in Algorithm 1.

1

Input: D={$d_{1}$,$d_{2}$,$\ldots$,$d_{n}$} //Set of data points

$k$ //Number of desired activities (clusters)

Output: Set of $k$ clusters

2 Initialize a population of particles with random positions and velocities in
the search space

3 for  _t=1 to the maximum number of iteration_ do

4 for  _each particle i_ do

5 Update position and velocity of particle i according to Eq.(10) and Eq.(11)

6 Evaluate fitness value of particle i according to the fitness function in
Eq.(15)

7 Update $pbest_{i}$(t) and gbest(t) if necessary

8 for  _T times (T is the number of iteration for mutation and set 10)_ do

9 Mutate gbest(t) according to Eq.(16) and (17)

10 Compare mutated gbest(t) with previous and choose the best as new gbest(t)

11

12

13

14Use gbest(t) as the initial centroids

15

16while  _until no change_ do

// Refining the centroids

17 Calculate distances of data points to centroids

18 Assign data points to the closest cluster

19 Centroids are updated using the following equation
$centroid_{i}=\frac{1}{n_{i}}\sum_{\forall d_{i}\in C_{i}}d_{i}$,

20 where $n_{i}$ is the number of data points in the cluster i

21

Algorithm 1 Hybrid PSO with Gaussian Mutation and K-means (HPGMK) Figure 6:
Illustration of sampling based on overlapping sliding windows. Each sample
($S_{1}$, $S_{2}$,$\ldots$, $S_{n}$), except the first sample, starts with the
last frame of the previous sample. Figure 7: Visualization of the Gaussian
Mutation Operator. In each iteration of hybrid PSO, one centroid ($C_{q}$) is
chosen from $x_{gbest}$ randomly. Then, Gaussian distribution is applied on
position and velocity of the selected centroid based on Eq.(16) and (17) to
create a new offspring $C_{q}^{{}^{\prime}}$. The new global best
($x^{{}^{\prime}}_{gbest}$) is then compared to $x_{gbest}$. If
$x^{{}^{\prime}}_{gbest}$ has better fitness value than $x_{gbest}$,
$x^{{}^{\prime}}_{gbest}$ is replaced with new global best.

##  IV Experiments

###  IV-A Datasets

Five datasets were used to evaluate the effectiveness of proposed method:
Cornell Activity Dataset (CAD-60) [13], UTKinect-Action3D (UTK) [10],
Florence3D (F3D) [47], Kinect Activity Recognition Dataset (KARD) [31], and
MSR DailyActivity3D (MSR) [48]. These datasets have different dimensions,
features, and activities. Table I shows the statistical information of these
datasets. They are discussed as follows.

TABLE I: Number of activities, subjects and videos in the five datasets used Dataset | CAD-60 | UTK | F3D | KARD | MSR
---|---|---|---|---|---
Activities | 14 | 10 | 9 | 18 | 16
Subjects | 4 | 10 | 10 | 10 | 10
Videos | 60 | 200 | 215 | 2160 | 320

CAD-60: This dataset includes 14 activities: rinsing mouth, brushing teeth,
wearing contact lens, talking on the phone, drinking water, opening pill
container, cooking (chopping), cooking (stirring), talking on couch, relaxing
on couch, writing on whiteboard, still (standing), working on computer and
random. Each activity was performed by 4 subjects including one left-handed
person. They were performed in 5 different environments: bathroom, bedroom,
kitchen, living room, and office. It contains activities of cyclic nature such
as brushing teeth and activities with similar postures such as drinking water
and talking on the phone.

UTK: There are 10 activities in this dataset: walk, sit down, stand up, pick
up, carry, throw, push, pull, wave hands, and clap hands. These activities
were performed by 10 subjects and repeated twice by each subject. The
significant intra-class and viewpoint variations are the main challenges of
this dataset.

F3D: This dataset includes 9 activities: wave, drink from a bottle, answer
phone, clap, tight lace, sit down, stand up, read watch, and bow. These
activities were repeated twice or thrice by 10 subjects. The main challenge
with this dataset is that the activities were performed at high speed. This
provides a small number of frames for the algorithm to sample and learn from.

KARD: This dataset contains 2160 videos and consists of 18 activities. These
activities were performed by 10 different people. The 18 activities are
horizontal arm wave, high arm wave, two hand wave, catch cap, high throw, draw
X, draw tick, toss paper, forward kick, side kick, take umbrella, bend, hand
clap, walk, phone call, drink, sit down, and stand up. This dataset is
challenging due to the large number of activities with high intra-class
variation such as different individuals performing the same activity such as
catch cap but in different ways, this makes learning of the same activity with
large differences in movements difficult.

MSR: In this dataset there are 16 activities: drink, eat, read book, call
cellphone, write on a paper, use laptop, use vacuum cleaner, cheer up, sit
still, toss paper, play game, lie down on sofa, walk, play guitar, stand up,
and sit down. There are 10 subjects, and each subject performed all activities
in both standing and sitting positions. This makes the dataset challenging
because the extracted features for both sitting and standing positions in each
activity are different. Another challenge is data corruption. In some frames,
the skeletal gesture structure suddenly collapses completely and lose their
coherence and become meaningless.

###  IV-B Method

Through preliminary experiments, the best values for swarm size and the number
of iterations were 20 and 50, respectively. The experiment was repeated 30
times and the average was obtained. The parameter settings of the HPGMK is
summarized in Table II. The performance of our proposed method (HPGMK) was
with three state-of-the-art (SOTA) methods for HAD incuding ENSC (elastic net
subspace clustering), SSC (Sparse Subspace Clustering) and SC (Subspace
Clustering) [2] and three recent deep clustering methods Deep Clustering
Network (DCN) [49], Structural Deep Clustering Network (SDCN) [50] and
incomplete multi-view clustering via contrastive prediction (Completer) [51].
In addition, we compared our method with conventional and well-known
clustering methods including K-means Clustering (KM) and PSO. All parameters
of each compared method, such as dimensions and numbers of layers, have been
adjusted as described in their papers. K-means Clustering (KM) and PSO have
been chosen for comparison as our proposed HPGMK is based on them. ENSC was
found to be most similar to our work as an unsupervised algorithm requiring
known cluster number while SSC and SC were the original algorithms that ENSC
was based on. The three deep clustering methods were chosen to compare our
method with the latest methods that use deep learning tools for clustering. To
compare the performance of the methods, the accuracy metric (calculated based
on [52]) was used. Moreover, F-score was used to show the performance of each
method in categorizing each activity and the confusion between them was shown
in the confusion matrix. The convergence test and clustering time of HPGMK
were measured to evaluate the benefits of each component used in the HPGMK on
its performance.

TABLE II: Parameters setting used in the experiment. Parameter | Description | Value
---|---|---
$c_{1_{max}},c_{2_{max}}$ | acceleration coefficients | 2.5
$c_{1_{min}},c_{2_{min}}$ | acceleration coefficients | 0
$w_{max}$ | inertial weight | 0.9
$t$ | maximum number of iteration | 50
$T$ | number of iteration for mutation | 10
$np$ | swarm size | 20

###  IV-C Computation complexity analysis

The Computation complexity of HPGMK in the initial stage (step 1 in Algorithm
1) is equal to $O(D.k.dim.np)$, where $D$ is the number of data points, $k$ is
the number of clusters, $dim$ is the dimension of data points and $np$ is the
population size of particles. The time complexity when particles are updated
(step 2 to 9) is equal to $O(t.dim.T.np)$, where $t$ is maximum number of
iteration and $T$ is the number of iteration for mutating gbest(t). In
Refining obtained centroids (step 11 to 15), the time complexity is equal to
$O(log(D.k.dim))$. Therefore, the overall computation complexity of proposed
algorithm is equal to $O(t.T.np.D.k.dim+log(D.k.dim))$. Table III shows the
comparison of the computation complexity of the well-known clustering
algorithms including KM and BIRCH along with the state-of-the-art algorithms
developed for clustering with the proposed algorithm. Based on the results,
PSO-based clustering algorithms take substantially longer to execute than non-
PSO-based clustering methods. But despite the low time complexity of KM and
BIRCH clustering, their accuracy is very low and they need to be executed many
times to reach the desired solution if they do not get stuck in the local
optimum. HPSOK-means, MinMaxK-means, PSC-RCE, PSOLFK and PSOSCALFK have around
the same time complexity. However, we developed an algorithm, which has
powerful global search capabilities due to increasing the variety of solutions
during the execution of the algorithm by applying Guassina mutation and
modifying the cluster using KM to increase the ability of PSO in local search.

Distribution Method and Lévy Flight

TABLE III: The computation complexity of HPGMK and eight related algorithms. Algorithm | Computation Complexity
---|---
KM | $O(N.K.D.t)$
BIRCH | $O(N)$
HPSOK-means | $O(N.K.D.t)$
MinMaxK-means | $O(N.K.D.t)$
PSC-RCE | $O(N.K.D.n_{m}.t)$
PSOLFK | $O(N.K.D.t)$
PSOSCALFK | $O(N.K.D.t)$
KMM | $O((N.dim+m^{2}+m.k).t+m.dim)$
GLPSOK | $O(K.dim.(K.dim+N).t)$
HPGMK | $O(t.T.np.D.k.dim+log(D.k.dim))$

###  IV-D Results and discussion

Fig. 8 shows the accuracy of HPGMK with the SOTA techniques for all subjects
of each dataset based on the maximum, minimum and average accuracies. The
average overall accuracy of the HPGMK was 77.53 % for CAD-60, 56.54 % for UTK,
66.84 % for F3D 46.02 % for KARD and, 40.12 % for MSR. As seen in Fig. 8,
HPGMK has the best performance in terms of maximum and average accuracy in all
datasets. This shows the effectiveness of the HPGMK for human activity
discovery. By utilizing the Gaussian mutation and KM along with PSO, our
approach brings performance improvement compared to the other methods. ENSC
and SSC, which are subspace clustering algorithms, do not use an efficient
search strategy [53]. In these methods, there is no strategy for maintaining
the balance between exploitation and exploration in their search. Moreover,
Parameters are required to be set and finding the right values for them is
tricky and complex such as size of subspace [54]. In contrast, HPGMK are not
dependent to parameters like SSC and ENSC and has several strategies for
searching. First, it used the PSO to search in large space area by using
several particles as potential solutions. To promote diversity, Gaussian
mutation is used. KM is also used to search in a small area of the global best
solution to refine the obtained centroids from PSO. These search strategies,
enable HPGMK has a relatively good performance compared to the SSC and ENSC.
Moreover, HPGMK has performed better than the methods that use deep learning.
In HPGMK, both spatial features from each 3D skeleton frame and temporal
features from sequences along with Orientation and statistical information are
extracted. However, in the deep clustering this information is ignored. On the
other hand, unlike deep clustering methods that have used shallow clustering,
HPGMK has different search strategies for exploration and exploitation to
determine better clusters.

Figure 8: The average accuracy for all subjects in (a) CAD-60, (b) UTK, (c)
F3D, (d) KARD and (e) MSR

Fig. 9(a) to (c) show the effect of each component of each type of the
proposed hybrid feature extraction method based on the discovery accuracy of
the activities performed by subject one in the CAD-60 dataset. Percentages
represent the discovery accuracy using the different combination of features
and each piece of graphs shows the ratio of the impact of one component in
discovery improvement to the rest of the other components in each type of
feature. This ratio is obtained based on dividing discovery accuracy obtained
by one of the components from a feature type by summation of discovery
accuracy obtained by all components of that feature type. Overall, in three
feature extraction methods comprising displacement, statistical, and
orientation features, when all their components are combined, the discovery
accuracy significantly increased and obtained 65.45 %, 62.54 %, and 57.09 %
respectively. By contrast, if each component of the feature extraction methods
is used alone without considering other components in the feature extraction,
the accuracy of discovery decreases. Fig. 9(d) shows the effect of different
combinations of each of the feature extraction methods. The size of each
circle indicates the effectiveness of the features. Based on the obtained
results, it is shown that the highest detection accuracy of 85.45 % is
obtained by combining all the three methods. It indicates that in order to
better differentiate between activities, it is necessary to extract features
from different aspects of activities.

Figure 9: The effect of the each components of the each type of feature
approach including (a) Displacement (D), (b) Statistical (S), (c) Orientation
(O) features and (d) their various combinations together. Capital letters
stand for different methods and putting these letters together means combining
relevant methods. Figure 10: Illustration of the effect of selecting the
keyframe based on kinetic on ”walking” in MSR where distinct frames compared
to adjacent frames are selected from similar frames. A window of frames is
shown with a few selected frames based on the local maximum (blue) and local
minimum (red) energy.

Fig. 10 shows the selected keyframes from walking activity in MSR. As shown,
there are a lot of frames with high similarity that by extracting their
features, not only do not help to improve discovery but also increase the
computational complexity and increase the overlap between other activities
because these gestures occur in other activities. However, using local maximum
and minimum kinetic energy can find representative frames and reduce
complexity. Looking at a window of frames, we can see the selected frames
based on the maximum and minimum local energy value. The selected frames show
the most differentiation to display the activity sequence. It is worth
mentioning that selecting keyframes maintains the order of the activity.
To show activity discovery performance, confusion matrix in on subject 10 in
KARD was shown in Fig. 11 for different methods. This figure demonstrates that
HPGMK produced distinctively meaningful clusters. In this figure, cluster
overlapping appears relatively high in the other methods especially KM and SC
in all datasets. There were a lot of overlapping due to the large number of
activities that are very similar including similar body gestures and similar
hand movement. For instance, activity overlapping were seen between High arm
wave and High throw with Drink.

Figure 11: Comparison of confusion matrix of the result on subject 10 in KARD.
Activity list: (1) Horizontal arm wave; (2) High arm wave; (3) Two hand wave;
(4) Catch cap; (5) High throw; (6) Draw X; (7) Draw tick; (8) Toss paper; (9)
Forward kick; (10) side kick; (11) Take umbrella; (12) Bend; (13) Hand clap;
(14) Walk; (15) Phone call; (16) Drink; (17) Sit down; and (18) Stand up.
Figure 12: The average F-score for all subjects in CAD-60. Activity list: (1)
Brushing teeth; (2) Cooking (chopping); (3) Rinsing mouth with water; (4)
Still(standing); (5) Taking on the couch; (6) Talking on the phone; (7)
Wearing contact lenses; (8) Working on computer; (9) Writing on whiteboard;
(10) Drinking water; (11) Cooking (stirring); (12) Opening pill container;
(13) Random; and (14) Relaxing on couch. AVG is the average F-score for all
activities. Figure 13: The comparison of clustering time of different
components of proposed algorithm on all subjects in CAD-60. Figure 14: The
comparison of convergence in subject 1 of CAD-60.

Fig. 12 shows the average F-score for each activity of all subjects in CAD-60.
By examining the average F-scores in most activities, it shows HPGMK
outperforming other methods and achieved slightly under 75 % on CAD-60 in
average for all activities. Although F-score was high in most of the
activities for HPGMK compared to other methods, cooking (chopping) and talking
on the phone were discovered with low F-scores. This was due to the high
similarity between cooking (chopping) with cooking (stirring) and talking on
the phone with wearing contact lenses.

Fig. 13 shows the average clustering time of the different combinations of
components used in the proposed algorithm in milliseconds on subject 1 in
CAD-60. This experiment evaluates the impact of the different components in
the proposed HPGMK algorithm. As can be seen, the clustering time of the
proposed algorithm (red line) was relatively low. The reason is that our
approach has benefited from dimension reduction methods, including PCA and
keyframe selector. Fig. 14 indicates the effect of each component of HPGMK on
convergence rate. As it is indicated, each combination has a different
convergence rate. However, using all components enables the proposed method to
achieve the best convergence compared to the other combinations. It is also
confirmed from Fig. 14 that by combining both KM and PSO algorithms, the
convergence speed has increased. Table IV indicates a significant difference
between the HPGMK and KM, HPGMK and PSO, HPGMK and SSC, HPGMK and ENSC and
HPGMK and SDCN through 30 independent runs using the Kruskal–Wallis test
(p-value). Since the p-value of almost all of the datasets is less than 0.05
(significance level) with the 95 % confidence intervals for each median, we
reject the null hypothesis, and conclude there is a significant difference
between the proposed method with KM, PSO, SSC, ENSC, and SDCN. In cases where
the p-value is higher than the significant level (these cases are specified
with⋆ in Table IV), there is not enough evidence to reject the null
hypothesis.

TABLE IV: Comparison of Kruskal-Wallis test (p-value) between HPGMK and KM,
HPGMK and PSO, HPGMK and SSC, HPGMK and ENSC and HPGMK and SDCN through 30
independent runs

Datasets | HPGMK vs KM | HPGMK vs PSO | HPGMK vs SDCN | HPGMK vs ENSC | HPGMK vs SSC
---|---|---|---|---|---
CAD-60 |  $\approx$0 | 0.00048 |  $\approx$0 |  $\approx$0 |
UTK |  $\approx$0 | 0.00014 | 0.02369⋆ |  $\approx$0 | 0.03325⋆
F3D |  $\approx$0 |  $\approx$0 |  $\approx$0 |  $\approx$0 |  $\approx$0
KARD |  $\approx$0 | 0.00544 | 0.023696⋆ |  $\approx$0 | 0.39117⋆
MSR |  $\approx$0 | 0.88246⋆ |  $\approx$0 |  $\approx$0 | 0.018736⋆

  * •

⋆ p-value $>$ 0.05: The differences between the medians are not
statistically significant.

##  V Conclusion

Most of the proposed HAR frameworks are supervised or semi-supervised, making
them unusable in real-world situations due to a lack of access to the ground
truth. In this paper, a Hybrid Particle Swarm Optimization with Gaussian
Mutation and k-means (HPGMK) approach was proposed to solve human activity
discovery on skeleton-based data with no prior knowledge of the label of the
activities in the data. Five different datasets were used to assess the
performance of the method. The results obtained have shown that HPGMK achieved
an average overall accuracy of 77.53 %, 56.54 %, 66.84 %, 46.02 %, and 40.12 %
in datasets CAD-60, UTK, F3D, KARD, and MSR, respectively and validate the
superiority of HPGMK over other methods compared. In activities with high
intra-class variation, corrupted data and the same activity performed in
sitting and standing positions, HPGMK has performed better in activity
discovery than other SOTA methods.
We have examined the impact of each feature used. It was found that the
simultaneous combination of features together further improves the results.
The impact of the different components in the proposed algorithm has shown
that Gaussian mutation has evolved particles to improve search algorithm and
k-means has increased discovery efficiency and improved the convergence rate.
This work paves the way toward implementing fully unsupervised human activity
discovery in practical applications using skeleton-based data. There are
various factors in the proposed methods that need to be addressed to develop
an effective activity discovery algorithm. One factor is the number of
clusters that were pre-configured in the proposed algorithm. The HPGMK needs
to be further extended to automatically address human activity discovery by
estimating the number of activities by itself. Another factor is detecting
outlier or noisy data. Outliers shift the cluster centers towards themselves,
thus affecting optimal cluster formation. Using outlier detection methods in
HPGMK to reject outliers will be beneficial. The manual procedure used to
build the PSO structure in the suggested technique was based on the
information and expertise that was gained. It takes a long time to manually
introduce changes because of trial-and-error, which makes it challenging to
thoroughly explore all potential algorithm setups. A potential future study to
address these concerns is automating the suggested method’s setup to make it
more effective in handling various circumstances and datasets.

## Acknowledgment

This work was supported by Grant UBD/RSCH/1.11/ FICBF(b)/2019/001 from
Universiti Brunei Darussalam.

## References

  * [1] H. V. Chandrashekhar _et al._ , “Human activity representation, analysis, and recognition,” 2006.
  * [2] G. Paoletti, J. Cavazza, C. Beyan, and A. Del Bue, “Subspace clustering for action recognition with covariance representations and temporal pruning,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 6035–6042.
  * [3] J. Han, L. Shao, D. Xu, and J. Shotton, “Enhanced computer vision with microsoft kinect sensor: A review,” _IEEE transactions on cybernetics_ , vol. 43, no. 5, pp. 1318–1334, 2013.
  * [4] K. Su, X. Liu, and E. Shlizerman, “Predict & cluster: Unsupervised skeleton based action recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9631–9640.
  * [5] J. Shan and S. Akella, “3d human action segmentation and recognition using pose kinetic energy,” in _2014 IEEE international workshop on advanced robotics and its social impacts_. IEEE, 2014, pp. 69–75.
  * [6] O. M. Nezami, A. Bahrampour, and P. Jamshidlou, “Dynamic diversity enhancement in particle swarm optimization (ddepso) algorithm for preventing from premature convergence,” _Procedia Computer Science_ , vol. 24, pp. 54–65, 2013.
  * [7] B. Jana, S. Mitra, and S. Acharyya, “Repository and mutation based particle swarm optimization (rmpso): A new pso variant applied to reconstruction of gene regulatory network,” _Applied Soft Computing_ , vol. 74, pp. 330–355, 2019.
  * [8] C. Li, S. Yang, and I. Korejo, “An adaptive mutation operator for particle swarm optimization,” 2008.
  * [9] D. A. Adama, A. Lotfi, C. Langensiepen, K. Lee, and P. Trindade, “Human activity learning for assistive robotics using a classifier ensemble,” _Soft Computing_ , vol. 22, no. 21, pp. 7027–7039, 2018.
  * [10] L. Xia, C.-C. Chen, and J. K. Aggarwal, “View invariant human action recognition using histograms of 3d joints,” in _2012 IEEE computer society conference on computer vision and pattern recognition workshops_. IEEE, 2012, pp. 20–27.
  * [11] M. Tabejamaat and H. Mohammadzade, “Embedded feature representation in dynamic time warping space for 3d action recognition using kinect depth sensor,” _Journal of Machine Vision and Image Processing_ , vol. 9, no. 3, pp. 19–34, 2022.
  * [12] M. E. Hussein, M. Torki, M. A. Gowayyed, and M. El-Saban, “Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations,” in _Twenty-third international joint conference on artificial intelligence_ , 2013.
  * [13] J. Sung, C. Ponce, B. Selman, and A. Saxena, “Unstructured human activity detection from rgbd images,” in _2012 IEEE international conference on robotics and automation_. IEEE, 2012, pp. 842–849.
  * [14] W. Cao, Y. Lu, and Z. He, “Geometric algebra representation and ensemble action classification method for 3d skeleton orientation data,” _IEEE Access_ , vol. 7, pp. 132 049–132 056, 2019.
  * [15] X. Liu, Y. Li, and R. Xia, “Rotation-based spatial–temporal feature learning from skeleton sequences for action recognition,” _Signal, Image and Video Processing_ , vol. 14, no. 6, pp. 1227–1234, 2020.
  * [16] M. M. Arzani, M. Fathy, A. A. Azirani, and E. Adeli, “Switching structured prediction for simple and complex human activity recognition,” _IEEE transactions on cybernetics_ , 2020.
  * [17] K. Malarvizhi and K. Amshakala, “Data clustering using hybrid of feature linkage weight based feature reduction and particle swarm optimization,” _Materials Today: Proceedings_ , 2021.
  * [18] M. Sharma and J. K. Chhabra, “Sustainable automatic data clustering using hybrid pso algorithm with mutation,” _Sustainable Computing: Informatics and Systems_ , vol. 23, pp. 144–157, 2019.
  * [19] S. Rengasamy and P. Murugesan, “Pso based data clustering with a different perception,” _Swarm and Evolutionary Computation_ , vol. 64, p. 100895, 2021\.
  * [20] J. Cai, H. Wei, H. Yang, and X. Zhao, “A novel clustering algorithm based on dpc and pso,” _IEEE Access_ , vol. 8, pp. 88 200–88 214, 2020.
  * [21] Y. Zhang, Y.-h. Wang, D.-w. Gong, and X.-y. Sun, “Clustering-guided particle swarm feature selection algorithm for high-dimensional imbalanced data with missing values,” _IEEE Transactions on Evolutionary Computation_ , 2021.
  * [22] X.-F. Song, Y. Zhang, D.-W. Gong, and X.-Z. Gao, “A fast hybrid feature selection based on correlation-guided clustering and particle swarm optimization for high-dimensional data,” _IEEE Transactions on Cybernetics_ , 2021.
  * [23] M. Hubert and S. Engelen, “Robust pca and classification in biosciences,” _Bioinformatics_ , vol. 20, no. 11, pp. 1728–1736, 2004.
  * [24] M. Hubert, P. J. Rousseeuw, and S. Verboven, “A fast method for robust principal components with applications to chemometrics,” _Chemometrics and intelligent laboratory systems_ , vol. 60, no. 1-2, pp. 101–111, 2002.
  * [25] M. Hubert, P. J. Rousseeuw, and K. Vanden Branden, “Robpca: a new approach to robust principal component analysis,” _Technometrics_ , vol. 47, no. 1, pp. 64–79, 2005.
  * [26] E. J. Candès, X. Li, Y. Ma, and J. Wright, “Robust principal component analysis?” _Journal of the ACM (JACM)_ , vol. 58, no. 3, pp. 1–37, 2011\.
  * [27] M. Alessandrini, G. Biagetti, P. Crippa, L. Falaschetti, S. Luzzi, and C. Turchetti, “Eeg-based alzheimer’s disease recognition using robust-pca and lstm recurrent neural network,” _Sensors_ , vol. 22, no. 10, p. 3696, 2022.
  * [28] M. Rahmani and P. Li, “Outlier detection and robust pca using a convex measure of innovation,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [29] L. Shi, Y. Zhang, J. Cheng, and H. Lu, “Two-stream adaptive graph convolutional networks for skeleton-based action recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 12 026–12 035.
  * [30] C. Li, C. Xie, B. Zhang, J. Han, X. Zhen, and J. Chen, “Memory attention networks for skeleton-based action recognition,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2021.
  * [31] S. Gaglio, G. L. Re, and M. Morana, “Human activity recognition process using 3-d posture data,” _IEEE Transactions on Human-Machine Systems_ , vol. 45, no. 5, pp. 586–597, 2014.
  * [32] S. K. Yadav, K. Tiwari, H. M. Pandey, and S. A. Akbar, “Skeleton-based human activity recognition using convlstm and guided feature learning,” _Soft Computing_ , vol. 26, no. 2, pp. 877–890, 2022.
  * [33] P. Zhang, C. Lan, W. Zeng, J. Xing, J. Xue, and N. Zheng, “Semantics-guided neural networks for efficient skeleton-based human action recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1112–1121.
  * [34] C. Si, W. Chen, W. Wang, L. Wang, and T. Tan, “An attention enhanced graph convolutional lstm network for skeleton-based action recognition,” in _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 1227–1236.
  * [35] H. Xia and X. Gao, “Multi-scale mixed dense graph convolution network for skeleton-based action recognition,” _Ieee Access_ , vol. 9, pp. 36 475–36 484, 2021.
  * [36] J. Cai, N. Jiang, X. Han, K. Jia, and J. Lu, “Jolo-gcn: mining joint-centered light-weight information for skeleton-based action recognition,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 2735–2744.
  * [37] T. Wang, W. W. Ng, J. Li, Q. Wu, S. Zhang, C. Nugent, and C. Shewell, “A deep clustering via automatic feature embedded learning for human activity recognition,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2021.
  * [38] M. Liu, Y. Bao, Y. Liang, and F. Meng, “Spatial-temporal asynchronous normalization for unsupervised 3d action representation learning,” _IEEE Signal Processing Letters_ , 2022.
  * [39] P. Gupta, R. McClatchey, and P. Caleb-Solly, “Tracking changes in user activity from unlabelled smart home sensor data using unsupervised learning methods,” _Neural Computing and Applications_ , vol. 32, no. 16, pp. 12 351–12 362, 2020.
  * [40] W. Qi, H. Su, and A. Aliverti, “A smartphone-based adaptive recognition and real-time monitoring system for human activities,” _IEEE Transactions on Human-Machine Systems_ , vol. 50, no. 5, pp. 414–423, 2020.
  * [41] X. Yan, Y. Ye, X. Qiu, and H. Yu, “Synergetic information bottleneck for joint multi-view and ensemble clustering,” _Information Fusion_ , vol. 56, pp. 15–27, 2020.
  * [42] H. Zhang, W. Zhou, and L. E. Parker, “Fuzzy temporal segmentation and probabilistic recognition of continuous human daily activities,” _IEEE Transactions on Human-Machine Systems_ , vol. 45, no. 5, pp. 598–611, 2015.
  * [43] W.-H. Ong, L. Palafox, and T. Koseki, “Autonomous learning and recognition of human action based on an incremental approach of clustering,” _IEEJ Transactions on Electronics, Information and Systems_ , vol. 135, no. 9, pp. 1136–1141, 2015.
  * [44] L. L. Presti and M. La Cascia, “3d skeleton-based human action classification: A survey,” _Pattern Recognition_ , vol. 53, pp. 130–147, 2016.
  * [45] P. Hadikhani and P. Hadikhani, “An adaptive hybrid algorithm for social networks to choose groups with independent members,” _Evolutionary Intelligence_ , vol. 13, no. 4, pp. 695–703, 2020.
  * [46] J. Kennedy and R. Eberhart, “Particle swarm optimization,” in _Proceedings of ICNN’95-international conference on neural networks_ , vol. 4. IEEE, 1995, pp. 1942–1948.
  * [47] L. Seidenari, V. Varano, S. Berretti, A. Bimbo, and P. Pala, “Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2013, pp. 479–485.
  * [48] J. Wang, Z. Liu, Y. Wu, and J. Yuan, “Mining actionlet ensemble for action recognition with depth cameras,” in _2012 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2012, pp. 1290–1297.
  * [49] B. Yang, X. Fu, N. D. Sidiropoulos, and M. Hong, “Towards k-means-friendly spaces: Simultaneous deep learning and clustering,” in _international conference on machine learning_. PMLR, 2017, pp. 3861–3870.
  * [50] W. Van Gansbeke, S. Vandenhende, S. Georgoulis, M. Proesmans, and L. Van Gool, “Scan: Learning to classify images without labels,” in _European Conference on Computer Vision_. Springer, 2020, pp. 268–285.
  * [51] Y. Lin, Y. Gou, Z. Liu, B. Li, J. Lv, and X. Peng, “Completer: Incomplete multi-view clustering via contrastive prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 11 174–11 183.
  * [52] B. Peng, J. Lei, H. Fu, L. Shao, and Q. Huang, “A recursive constrained framework for unsupervised video action clustering,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 1, pp. 555–565, 2019.
  * [53] Y. Lu, S. Wang, S. Li, and C. Zhou, “Particle swarm optimizer for variable weighting in clustering high-dimensional data,” _Machine learning_ , vol. 82, no. 1, pp. 43–70, 2011.
  * [54] P. Agarwal, S. Mehta, and A. Abraham, “A meta-heuristic density-based subspace clustering algorithm for high-dimensional data,” _Soft Computing_ , pp. 1–20, 2021.