# Skeleton-based Human Action Recognition via Convolutional Neural Networks
(CNN)

Ayman Ali, Ekkasit Pinyoanuntapong, Pu Wang, Mohsen Dorodchi  College of
Computing and Informatics
University of North Carolina at Charlotte
Charlotte, United States
aali26, epinyoan, pwang13<EMAIL_ADDRESS>
###### Abstract

Recently, there has been a remarkable increase in the interest towards
skeleton-based action recognition within the research community, owing to its
various advantageous features, including computational efficiency,
representative features, and illumination invariance. Despite this,
researchers continue to explore and investigate the most optimal way to
represent human actions through skeleton representation and the extracted
features. As a result, the growth and availability of human action recognition
datasets have risen substantially. In addition, deep learning-based algorithms
have gained widespread popularity due to the remarkable advancements in
various computer vision tasks. Most state-of-the-art contributions in
skeleton-based action recognition incorporate a Graph Neural Network (GCN)
architecture for representing the human body and extracting features. Our
research demonstrates that Convolutional Neural Networks (CNNs) can attain
comparable results to GCN, provided that the proper training techniques,
augmentations, and optimizers are applied. Our approach has been rigorously
validated, and we have achieved a score of $95\%$ on the NTU-60 dataset.

##  I Introduction

Recently, the research community has been witnessing a surging interest in
skeleton-based action recognition, owing to its many advantageous attributes
such as computational efficiency, informative features, and immunity to
changes in lighting conditions. The field continues to explore the optimal way
of representing human actions through skeleton representation and feature
extraction. This has resulted in a marked increase in the number of human
action recognition datasets. On the other hand, deep learning-based algorithms
have gained immense popularity as a result of their effectiveness in various
computer vision tasks. Most cutting-edge contributions in the realm of
skeleton-based action recognition adopt Graph Neural Network (GCN)
architecture for representing the human body’s articulated structure and
extracting features. However, our findings demonstrate that Convolutional
Neural Networks (CNNs) can deliver comparable results to GCN if proper
training methods, data augmentation techniques, and the appropriate optimizer
are utilized.

The prospect of equipping machines with human-like visual capabilities has
garnered significant attention among researchers, inspiring the development of
various technologies, algorithms, and techniques to facilitate this task. To
date, numerous computer vision tasks have been successfully addressed,
including image classification [1] and object detection [2]. Among the active
research areas in computer vision is human activity analysis in videos,
including human action detection, recognition, and prediction.

The human gesture, which is characterized by a shorter duration and less
complex movements performed by a limited number of body parts, represents the
first category of human action. Examples of human gestures include hand waving
and head nodding. On the other hand, human action encompasses longer-duration
movements involving more body parts. A sequence comprising multiple actions is
referred to as human activity. Finally, human interaction contains human
interactions with the surrounding environment, including both human-to-human
and human-to-object interactions. The presence of multiple humans or
interactions with various objects increases the complexity of motion analysis,
which can further be complicated by online or offline human behavior analysis.

In the past, capturing human actions through human-performance systems often
necessitated the application of markers on the subjects’ bodies and the use of
distinctive attire. Despite overcoming these limitations, the high cost of
cameras remained a hindrance to widespread adoption [3]. However, recent
advancements have led to the development of cost-effective contactless sensing
devices, such as Microsoft Kinect, Intel Realsense, and Doppler radar.
Historically, the RGB modality in human action capture has been challenged by
various factors, including illumination, occlusions, background clutter, frame
rate, viewpoint, and biometric variations. By contrast, RGB-D sensors have
mitigated some of these difficulties, particularly with regard to
illumination, and provide a crucial advantage by enabling the generation of 3D
structural information of the scene. As a result, estimating 3D human skeleton
joints has become a relatively straightforward process in the RGB-D modality.

For many years, machine learning algorithms have been the primary technologies
incorporated to address various computer vision problems, including action
recognition. The conventional approach to human motion analysis involves
capturing and representing spatiotemporal information, which enhances the
accuracy and robustness of the analysis. Typically, features are extracted
manually and incorporated into classical machine learning algorithms to
perform different tasks. The research community has explored various features
representations, including joint coordinates [4], the center of gravity [5],
the angle between skeleton joints [6], motion velocity [7], and co-occurrence
features [8]. The selection of the appropriate algorithm also recreates a
vital role in the machine learning era. Many algorithms have been utilized,
including Support Vector Machine (SVM) [9], Linear Discriminant Analysis (LDA)
[10], Naive Bayes Nearest Neighbor [11], Logistic Regression [12], and KNN
[13]. However, the generalization of machine learning algorithms is
challenging and requires significant effort in feature engineering.

Recent advancements in deep learning-based techniques have demonstrated
superior performance in various computer vision problems, such as image
classification [1], object detection [2], action recognition [14], and action
detection [15]. The increasing interest in skeleton-based representation has
been driven by the abundant and discriminative features that can be derived
from skeleton joints. For instance, features such as Skeleton Map [16], Joint
Trajectory Map [17], Joint Distance Map [18], and many others can be extracted
from the skeleton joint data alone. Consequently, the utilization of deep-
learning algorithms has emerged as a prevalent approach for skeleton-based
human action recognition. Our contributions are summarized as follows:

  * •

Construct an easy-to-integrate and modular Convolutional Neural Network (CNN)
for the action recognition task, which attains results comparable to the
State-of-the-Art (SOTA) methods.

  * •

Despite the prevalence of graph neural network-based methods in the SOTA
contributions to the action recognition task, we demonstrate that CNNs can
attain comparable results by implementing various training techniques.

  * •

Our results indicate that incorporating a diverse set of augmentation
techniques leads to an improvement in the generalization and robustness of the
model.

  * •

Our findings reveal that utilizing a margin-based cosine loss function instead
of the conventional cross-entropy loss leads to a significant enhancement in
performance.

##  II Related Work

Deep-learning approaches have been demonstrated superiority over conventional
machine learning algorithms in various computer vision tasks, including image
classification as demonstrated in [1] in the ImageNet dataset, object
detection as introduced by Ren et al. [2] with their Faster R-CNN framework,
action recognition as recently revisited by Duan et al [14], and action
detection as proposed by Wang et al. [15]. In recent years, skeleton-based
representations have garnered increasing attention due to the wealth of rich
and discriminative features that can be obtained from the skeleton joints.
Examples of such representations include the Skeleton Map[16], Joint
Trajectory Map[17], Joint Distance Map [18], among others, all of which are
based solely on features extracted from the skeleton joints data.

####  II-1 Convolution Neural Network - CNN Approaches

Wang et al. [17] proposed the Joint Trajectory Map (JTM), a representation
that captures the spatiotemporal information of a sequence in the form of 2D
images. This work emphasizes human motion magnitude and speed as the core
features, represented by the saturation value in the HSV (Hue-Saturation-
Value) color space, with higher motion magnitude and speed yielding a higher
saturation value. Du et al. [16] introduced a method for encoding the
spatiotemporal features of an action sequence into an image matrix. This
representation is constructed by vertically encoding the skeleton’s joint
coordinates into the RGB channels and horizontally encoding the sequence
frames. The encoded information is then quantified, normalized, and
transformed into an image for use with a CNN classification network. In
addition to the skeleton map representation proposed by Du et al., Li et al.
[19] proposed a two-stream CNN-based network that leverages the skeleton map.
Li et al. [18] extracted discriminative features from the pair-wise distances
between skeleton joints in a human action sequence to construct a Joint
Distance Map (JDM), which maps the skeleton sequence into images. Bilinear
interpolation was applied to address the issue of variable sequence duration.
Li et al. [8] embarked on the problem of joint co-occurrence by proposing an
end-to-end hierarchical framework that facilitates better feature learning,
gradually aggregating point-level features into global co-occurrence features.
Ke et al. [20] proposed a translation, scale, and rotation invariant body part
vector-based representation, using geometrical features of the skeleton to
generate two sets of features: cosine distance and normalized magnitude. The
human skeleton joints are grouped into five parts: trunk, right arm, left arm,
right leg, and left leg. Both cosine distance and normalized magnitude
features are computed within the body and skeleton, yielding ten feature
vectors fed into a CNN for further feature extraction and training.

####  II-2 Recurrent Neural Network - RNN Approaches

The recognition of human actions is a problem of spatiotemporal
representation. To capture the discriminative features, researchers have
proposed utilizing conventional RNN networks. Yet, the vanishing gradient
problem has necessitated adapting memory gating techniques, such as long-short
term memory (LSTM) and gated recurrent unit (GRU). To address this issue, Du
et al. [21] proposed a hierarchical bidirectional recurrent neural network
(BRNN) that models the long-term temporal sequence through an end-to-end
approach. To facilitate better feature representation, the human body is
divided into five parts and processed through five different BRNN subnets. The
outputs are fused and classified in higher layers before being fed into the
final BRNN. To further improve the system’s robustness, Du et al. [22]
implemented random rotation transformation during preprocessing and scale
transformation to account for varying human sizes. Salient motion patterns
have been leveraged as essential features in various human action-related
tasks. However, LSTM struggles to capture the spatiotemporal dynamics. To
resolve this, Veeriah et al. [23] proposed a differential gating for LSTM-RNN
networks that quantifies the salient motion between frames through the
derivative of state (DoS). Skeleton-based action recognition presents several
challenges, including the intra-frame joint spatiotemporal information of all
frames in a sequence. To address these challenges, Sogn et al. [24] proposed
an end-to-end spatiotemporal attention model that adaptively learns intra-
frame and inter-frame dependencies through recurrent RNN and LSTM. Human
skeleton data contains discriminative features such as acceleration, angles,
spatial coordinates, orientation, and velocity. Zhang et al. [25] evaluated
eight geometric features on a 3-layer LSTM network and found that computing
the distance between joints and selected lines outperformed the other
features.

####  II-3 Graph Convolution Network - GCN Approaches

Yan et al. [26] introduced the initial spatiotemporal graph convolutional
network (ST-GCN), which established edges between joints in both the intra-
frame and inter-frame dimensions. Typically, in graph-based techniques, the
convolution operation is performed over the neighboring joints within the
receptive field. However, conducting a convolution over non-Euclidean data
necessitates a judicious partition strategy to attain a label map. Although
ST-GCN [26] achieved substantial results in the human action recognition task,
it faced several limitations. The graph representation in ST-GCN was created
based on the human body’s physical structure, which may not be ideal for
characterizing human action. For example, in an act like clapping, the
relationship between the hands is crucial in classifying the action. In this
case, ST-GCN fails to capture such a relationship since the two hands are not
connected and are distant from each other in the kinematic tree. Additionally,
certain geometric features, such as bone orientation and length, are
challenging to represent in graph-based algorithms. To overcome these
limitations, Shi et al. [27] proposed an end-to-end data-driven multi-stream
attention-enhanced adaptive graph convolutional network (MS-AAGCN), where the
graph topology is learned and generated adaptively from the input data. In
practical scenarios, occlusion is a prevalent challenge that is often
unavoidable. Song et al. [28] proposed a GCN-based model that is trained on
incomplete spatiotemporal skeleton data, with intra-frame joints and inter-
frame frames masked to imitate occlusion effects. Moreover, Yoon et al. [29]
incorporated noise into the skeleton data to enhance the model’s robustness.

##  III Data Preprocessing

Typically, skeleton data is represented in a camera coordinate system, leading
to a diverse representation of captured sequences from different viewpoints,
as illustrated in Figures 3 (A), (B), and (C). To mitigate the impact of view
variation, a common approach is to transform the skeleton data into a unified
coordinate system [30, 31, 32, 33]. This is typically accomplished through a
sequence of geometric translation, rotation, and normalization operations, as
depicted in Figures 3 (D), (E), and (F). There are various transformation
strategies presented in the literature. In frame-based approaches, the
transformation operation is applied to each frame in the sequence, which,
however, results in the loss of some relative motions. For example, when
applied to the walking action, this technique produces an effect as if walking
on a treadmill. On the other hand, in sequence-based strategies, the first
frame in the sequence is designated as the reference frame, and all
transformations applied to the subsequent frames are relative to this
reference frame. This approach provides a more realistic representation of
human skeleton motions.

Figure 1: Action representation from NTU-D 60 dataset A) -45°skeleton
visualization, B) 0 °skeleton visualization, C) 45°skeleton visualization. (D,
E, F) are the transformed skeleton for the same skeletons in (A, B, C)

###  III-A Encoding Skeleton to Image

Given a sequence of skeletons $S$, where $s\in(1,...,S)$, and the $j^{th}$
joint in the $t^{th}$ frame of skeleton $s$ is represented as
$s_{t,j}=[x_{t,j},y_{t,j},z_{t,j}]$, with $j\in(1,...,J)$ denoting the number
of joints in a frame and $t\in(1,...,T)$ denoting the total number of frames
in the sequence $S$.

Inspired by the work of Du et al. [16], we transform the raw skeleton data
into a skeleton map image that preserves spatiotemporal information, as
depicted in Fig. 2. Given an RGB image of dimensions $[H,W,C]$, where $H$
represents height, $W$ represents width, and $C$ represents the conventional
RGB image channels, we map the action sequence $S$, represented in the
dimension of $T\times N\times 3$, to an image of dimensions $[H,W,C]$. Here,
$T$ represents the number of frames and maps to the height of the image, $N$
represents the number of joints and maps to the width of the image, and the
last dimension represents the 3D joint coordinates of the joints in a frame,
which are mapped to the three channels of the image.

Since raw skeleton data may have different value ranges than the image, pixel
normalization is necessary to ensure that the mapped values are in the range
of $0-255$. This is accomplished by:

$P_{t,j}=floor(255\times\frac{s_{t,j}-C_{min}}{C_{max}-C_{min}})$ (1)

Figure 2: The pipeline of generating the skeleton map image

##  IV Data Augmentation

Data augmentation is a well-established technique utilized to enhance the
performance of machine learning algorithms by diversifying the training data.
This approach involves synthesizing new samples from the existing training
data by employing transformations such as scaling, rotation, translation, and
other deformations. Adding these transformed samples to the training set can
improve the generalization performance of the machine learning algorithm,
making it more resilient to variations in the input data. Data augmentation is
widely applied across several deep-learning tasks, including image
classification, object detection, and natural language processing.

On the other hand, Skeleton-based data augmentation is used explicitly for
tasks involving 3D pose information, such as 3D human pose estimation. This
approach involves transforming the 3D joint positions of a skeleton to
generate new pose samples. Image-based data augmentation is utilized for tasks
where the input data consists of images, including image classification and
object detection.

In this research, we investigated various image-based and skeleton-based
techniques for data augmentation, as outlined in TABLE. Our approach is
inspired by RandAugmentation [34], which randomly injects the training
pipeline with predefined augmentations based on the number and magnitude of
augmentations to be applied.

The Flipping Sequence augmentation technique, as described in [35], involves
the generation of synthetic pose sequences by horizontally flipping the input
pose sequences. This is achieved through the reflection of the 3D joint
positions across a horizontal plane when applied to skeleton data and by
flipping in both the horizontal and vertical planes when applied to image
data. As a result, a new pose sequence with actions performed in the opposite
direction is produced.

The Geometric Rotation augmentation, as discussed in [34], generates new
synthetic samples by applying random rotations to the input data. This is
achieved by rotating the input images or 3D objects around a fixed point,
utilizing a specified angle and center of rotation.

The Cutout augmentation, as presented in [36], is a regularization technique
utilized in deep learning. This technique involves randomly masking a portion
of an image during the training process, which necessitates the model to learn
to ignore these regions and focus on the remaining parts. This can enhance the
model’s generalization to new data, thus improving its performance. Cutout
augmentation is frequently adopted in image classification tasks, where it can
aid the model in recognizing objects in images, despite variations in their
appearance. It can be easily implemented in most deep learning frameworks as a
simple and effective regularization method for deep learning models.

The Zoom augmentation, as described in[34], involves rescaling the input
images or 3D objects using a specified zoom factor, which determines the
extent of the zooming. This augmentation method is appropriate for both image
and skeleton data. Additionally, the Shear augmentation, as discussed in Cubuk
et al. (2020)[34], is achieved through the utilization of a linear
transformation that maps the x-axis or y-axis coordinates of the input data to
new positions. The Translate augmentation, also presented in Cubuk et al.
(2020)[34], involves the shifting of the x-axis, y-axis, or both coordinates
of the input data by a specified amount.

The introduction of various types of noise to image data during training has
been demonstrated to enhance the generalizability and robustness of machine
learning models [37]. Salt-and-Pepper noise [38] is a form of noise that
mimics the effect of corrupted or missing pixels in the input data. This is
achieved by randomly setting a specified proportion of pixels to either the
minimum or maximum intensity value, creating the appearance of salt and pepper
grains on the image. The remaining pixels remain unaltered.

Another noise-based augmentation strategy is Localvars noise [37]. This method
generates random noise samples from a specified distribution. Speckle noise
[37] generates random noise samples from a Poisson distribution with a
specified mean. In contrast, Gaussian noise [37] generates random noise
samples from a Gaussian distribution with a specified mean and standard
deviation.

In the context of skeleton data, Bone Shuffling augmentation is applied. This
is achieved by randomly permuting the 3D joint positions of the skeleton,
resulting in a new pose sequence in which a different body configuration
performs the same actions. Additionally, Bone Masking augmentation is
implemented by setting the 3D joint positions of the skeleton to zero for a
randomly selected subset of bones. A similar technique can also be applied on
the frame level rather than the bone level.

Figure 3: Various augmentation implementation

##  V Loss function

In the realm of Deep Learning, a loss function serves as an evaluation metric
of a model’s ability to predict the desired outcome. The objective of training
a model is to find the optimal set of parameters that minimize the loss
function, leading to the model’s improved accuracy in making predictions on
unseen data. The use of classification loss is prevalent in classifying
outputs into discrete class labels. The Cross-Entropy loss is a popular choice
for action recognition classification problems [39]. Although the Cross-
Entropy loss often proves effective in training models, it has certain
limitations, including sensitivity to the relative magnitude of predicted and
true values, rendering it challenging to use in specific scenarios.

One critical observation when utilizing the Cross-Entropy loss function is
that the distance between samples of different classes is minimal. As a
result, the classifier trained with Cross-Entropy is susceptible to making
incorrect predictions. An intuitive alternative to the Cross-Entropy loss is
the implementation of metric learning techniques, which involve learning a
distance metric from the data. Unlike conventional Machine Learning, where the
aim is to learn a mapping function between inputs and outputs, the goal of
metric learning is to learn a metric that can be utilized to measure the
distance between data points [39]. This distance metric can then be leveraged
for making predictions or other tasks, maximizing the inter-class distance and
minimizing the intra-class distance. The Additive Angular Margin Loss (AAML)
[40] is a widely used loss function in Deep Learning for face recognition. It
is designed to learn a discriminative feature representation for face images
by maximizing the angular margin between the features of the same identity
while minimizing the angular margin between the features of different
identities. The AAML loss function is based on learning a weight vector and a
feature vector for each identity. The angle between the weight and feature
vectors is maximized for the same identity and minimized for different
identities. The loss function encourages the weight and feature vectors to lie
on the surface of a hypersphere. The large margin between the vectors of the
same and different identities enhances the learned feature representation’s
discriminative power. The AAML loss function has been demonstrated to
outperform other loss functions for face recognition on several benchmark
datasets, resulting in a 1.5% improvement in classifier accuracy.

##  VI Deep-Learnig Optimizer and Schedulers

###  VI-A Training Optimizers

The purpose of an optimizer in deep learning is to determine the set of
parameters that minimize the loss function, thus yielding the model
configuration most representative of the data. This is achieved through
iterative refinement of the model parameters, guided by various algorithms and
techniques, with the optimizer making progressive adjustments to the
parameters until an optimal solution is reached. Commonly used optimizers
include Stochastic Gradient Descent, Adam [41], and RMSprop [42], each of
which can significantly impact the performance of a deep learning model. Thus,
the selection of the appropriate optimizer is a critical consideration.

The MadGrad [43] optimizer, a variation of the commonly used Adam [41], aims
to improve the generalization performance of deep learning models. This is
achieved by incorporating multiplicative noise into the gradients during the
training process. Specifically, the gradients are multiplied by a random noise
tensor at each iteration, derived from a Gaussian distribution with a mean of
1 and a slight variance, and scaled by a factor that decreases over time.
Results have demonstrated that MadGrad outperforms Adam on various tasks,
including image classification and natural language processing. This study
validated its effectiveness, resulting in a 1.1% improvement in accuracy.

###  VI-B Learning rate schedulers

The objective of a learning rate scheduler in deep learning is to modulate the
learning rate. This crucial hyperparameter determines the magnitude of updates
made to the model’s parameters by the optimizer. Ineffective regulation of the
learning rate can result in either significant, erratic updates that lead to
suboptimal convergence or slow, incremental updates that impede the pace of
the training process. A learning rate scheduler resolves these issues by
dynamically adjusting the learning rate during training, using various
techniques such as fixed scheduling, scheduling based on the training
progress, or scheduling based on model performance. This contribution
demonstrates the efficacy of the combination of the Cosine Annealing scheduler
[44] and the ReducedLR scheduler [45] in stabilizing the learning process.

The ReducedLR scheduler [45] is a commonly utilized approach in deep learning
for regulating the learning rate during the training process. This scheduler
operates by reducing the learning rate by a predetermined factor when the
training loss fails to improve over a specified number of epochs, thereby
mitigating the risk of getting stuck in a suboptimal local minimum and
enhancing the generalization performance of the model. Implementable as a
callback function within deep learning frameworks, the ReducedLR scheduler
offers the flexibility to specify the reduction factor and the number of
epochs for which the learning rate reduction should be triggered.

The CosineAnnealing scheduler [44] is a learning rate scheduler that adjusts
the learning rate following a cosine curve, which commences at a high value
and gradually decreases to a lower value as training progresses. This
technique helps to improve the convergence of the training process and address
the challenge of premature saturation, where the learning rate becomes
excessively low and the training slows down.

Architecture | Technique | NTU
---|---|---
CS | CV
RNN-based | S-trans+RNN | 76.0 | 82.3
S-trans+RNN (aug.) | 77.0 | 85.0
VA-RNN | 79.4 | 87.6
VA-RNN (aug.) | 79.8 | 88.9
CNN-Based | S-trans+CNN | 87.5 | 92.2
S-trans+CNN (aug.) | 87.9 |  93.5
VA-CNN | 88.2 | 93.8
VA-CNN (aug.) | 88.7 | 94.3
Our + ArcFace | - | 95.0
TABLE I: Effectiveness (in accuracy ($\%$)) of applying (regularlization, Madgrad Optimizer, multiple learning schedulers)  Augmentation | Image-based | Weak | Strong
---|---|---|---
Flipping | Both | 93.70 | 93.62
Rotation | Both | 94.21 | 94.1
Zoom | Image | 93.60 | 93.40
Shear | Image | 94.03 | 94.20
Translate-x | Image | 93.71 | 94.03
Translate-y | Image | 94.24 | 94.1
Cutout | Image | 94.03 | 94.1
Salt and pepper noise | Image | 93.52 | -
Bone Shuffling | Pose | 94.13 | 93.96
Bone Masking | Pose | 93.80 | 93.8
Frame Masking | Both | 93.8 | 93.75
Gaussian Noise | Image | 93.5 | -
Speckle Noise | Image | 93.45 | -
Localvars Noise | Image | 93.5 | -
Salt noise | Image | 93.5 | -
Pepper Noise | Image | 93.5 | -
TABLE II: Various augmentations applied on the encoded skeleton image map

##  VII Regularization

Regularization is a widely employed technique in machine learning for
mitigating overfitting, which refers to the scenario in which a model
demonstrates superior performance on the training data yet subpar performance
on unseen data. Overfitting transpires when the model has learned specific
patterns in the training data that do not generalize to other data, resulting
in poor performance on previously unseen data. Regularization helps to address
overfitting by incorporating a penalty term in the loss function during the
training process. This promotes the model to learn more generalizable
parameters, thereby yielding enhanced performance on new data. There exist
several types of regularization methods, such as label smoothing
regularization, dropout, batch normalization, and early stopping, which can be
utilized in a complementary manner to improve the generalizability of machine
learning models.

Label Smoothing Regularization [46] is a popular regularization technique
utilized in deep learning to enhance the generalization performance of
classifiers. The objective of label smoothing is to reduce the confidence of
the classifier’s predictions, thereby improving its generalization
capabilities on new data. This technique is commonly utilized in image
classification tasks, where instead of using one-hot encoded training data
labels with a label vector of [0, 0, 1, 0] to indicate that an image belongs
to class 3, label smoothing would modify the label vector to [0.1, 0.1, 0.8,
0.1], suggesting that the image has a high probability of belonging to class 3
but also a tiny likelihood of belonging to the other classes. This regularizes
the classifier and reduces the risk of overfitting the training data.
Essentially, one-hot encoded training data labels should not contain zero
values for the non-class index.

Early stopping, as described in [47], is a prevalent regularization strategy
utilized in deep learning to mitigate overfitting. The technique entails
prematurely halting the training process prior to the model’s convergence to
its optimal performance. This is achieved by continuously monitoring the
model’s performance on a validation set and interrupting the training when the
performance either ceases to improve or begins to decline. The objective of
early stopping is to restrict overfitting, which is a situation where a model
performs well on the training data but poorly on unseen data. By curtailing
the training before the model attains full convergence, early stopping can
enhance its ability to generalize to novel data and improve its overall
performance. Early stopping is a straightforward and practical approach that
can be effortlessly integrated into most deep-learning frameworks.

Dropout, as a regularization technique in deep learning, aims to enhance the
generalization performance of neural networks [srivastava2014dropout]. The
approach involves randomly setting a specified fraction of the activations of
the neurons in the network to zero during training. This reduction in the co-
adaptation of neurons leads to a network that is less sensitive to the
specific weights of individual neurons. As a result, training a network with
dropouts can lead to learning more robust and diverse feature representations,
which are less prone to overfitting the training data. Dropout is commonly
applied to fully-connected layers in neural networks but can also be
implemented in other layer types, such as convolutional and recurrent layers.
It is usually deployed with a high dropout rate (e.g., 0.5) during training
and a low dropout rate (e.g., 0.1) or no dropout during inference.

Batch normalization, as discussed in [48], is a technique employed in deep
learning to improve the performance and stability of neural networks. The
strategy entails normalizing the inputs to each layer in the network, which
can expedite the training process and enhance the generalization performance
of the network. A batch normalization layer is inserted between the input and
output of a neural network. It is designed to normalize the activations of the
preceding layer using the mean and variance of the current mini-batch of data.
This reduction of internal covariate shift, which refers to the phenomenon of
the distribution of inputs to each layer evolving over time during training,
can enhance the network’s convergence speed and stability and make it more
resilient to initialization parameter choices.

##  VIII Dataset and Performance Measures

###  VIII-A Dataset

The design of a data collection environment for capturing actions is dictated
by the target task. For instance, the setup required for a gesture recognition
task will differ from that required for a human interaction task. In most
cases, the captured action occurs at the beginning of the sequence, obviating
the need for action recognition without action detection. Additionally, many
datasets use a global capturing length, which simplifies the data
preprocessing by eliminating the requirement for consistent length. The
presence of non-action-related subjects in the scene is another challenge
commonly addressed by capturing data in a controlled environment to ensure
that only sequences relevant to the task are captured.

The NTU RGB+D dataset, proposed by Shahroudy et al. [49], is a substantial
multi-view action dataset with 56,000 samples from 40 individuals and 60
action classes, which are classified into three categories: 40 daily activity
classes, nine health-related classes, and 11 interaction classes. The
Microsoft Kinect device was used for data collection, providing RGB, depth,
skeleton, and infrared modalities. Three fixed camera setups were utilized,
with capturing angles ranging from -45 to 45 degrees, and the cameras’
distance and height were also varied to increase view variations. Liu et al.
[50] further extended the NTU-60 dataset using the same capturing system and
modalities, with 106 subjects performing 120 action classes and 114,500 video
samples.

###  VIII-B Performance Measures

Standardizing evaluation metrics is crucial for fairly comparing various
approaches. The choice of metric depends on the task, and some metrics may be
more intuitive than others. The choice of evaluation protocol may also play a
role in demonstrating the difficulty and complexity of the scenario. The
Cross-views evaluation protocol, proposed by Shahroudy et al. [49] and Liu et
al. [51], assumes that samples from the same view cannot be used for both
training and testing, for instance, camera 1 and 3 for training and camera 2
for testing only.

Cross-subject evaluation protocol, also proposed by the creators of the NTU-D
60 dataset [49, 51], dictates that the subjects selected for training cannot
be used for testing, with 20 of the 40 subjects selected for training and the
remaining 20 for testing. Lastly, Liu et al. [51] also proposed the cross-
setup evaluation protocol, which uses different vertical heights, distances,
and backgrounds during the capturing to include natural variations, while
keeping the horizontal three-camera fixed in terms of capturing angle.

##  IX Results

We replicated the results of the End-to-End Two Streams Network for Action
Recognition as reported by et al. [33]. We established it as a baseline for
our experiments. In their work, Zhang et al. et al. [33] built upon their
previous contribution in [52] by proposing an End-to-End Two Streams Network
for Action Recognition. The network comprises two streams: one is similar to
the one presented in [52], while the other is a CNN-based stream that includes
a view adaptation subnetwork with a similar architecture. This CNN stream
leverages the skeleton representation proposed by Du et al. [16]. To enhance
the robustness of the network, random rotation augmentation was incorporated
on-the-fly. Finally, late score fusion was applied with different stream
weights, favoring the CNN stream. Our results outperform those of the Two-
Stream View Adaptive Module, demonstrating the potential of our approach with
simple training techniques. We incorporated all mentioned strategies and
techniques to achieve the results shown in the tables.

##  X Conclusion

The proliferation of sensing devices has increased the availability of diverse
datasets in multiple modalities. Among these, skeleton-based modality holds
promise due to its computation efficiency and the wealth of information the
human skeleton provides. Accurately estimating human pose from the video is a
prerequisite for extracting 3D skeleton data from various modalities. In this
work, we have shown that Convolutional Neural Networks (CNNs) utilizing
diverse training techniques can achieve state-of-the-art (SOTA) results
comparable to those obtained using Graph Neural Networks (GNNs) for action
recognition tasks. Furthermore, our findings indicate that using various data
augmentation techniques can improve the generalization and robustness of the
model. This results in improved performance on unseen data and reduced
sensitivity to variations or distortions in the input. Additionally, we have
demonstrated that using MadGrad as the optimizer and implementing a learning
rate scheduler can improve the model’s accuracy. Furthermore, we have shown
that using a margin-based cosine loss function instead of the traditional
cross-entropy loss can enhance the model’s performance, leading to more
accurate predictions and improved overall results. Regularization techniques
can further prevent overfitting, improving the model’s performance on unseen
data.

## References

  * [1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25:1097–1105, 2012.
  * [2] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28:91–99, 2015\.
  * [3] Susan Barker, Rebecca Craik, William Freedman, Nira Herrmann, and Howard Hillstrom.  Accuracy, reliability, and validity of a spatiotemporal gait analysis system.  Medical engineering & physics, 28(5):460–467, 2006.
  * [4] Junqin Huang, Xiang Xiang, Xuan Gong, Baochang Zhang, et al.  Long-short graph memory network for skeleton-based action recognition.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 645–652, 2020.
  * [5] Bharath N Raj, Anand Subramanian, Kashyap Ravichandran, and Dr N Venkateswaran.  Exploring techniques to improve activity recognition using human pose skeletons.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, pages 165–172, 2020.
  * [6] Du Q Huynh.  Metrics for 3d rotations: Comparison and analysis.  Journal of Mathematical Imaging and Vision, 35(2):155–164, 2009\.
  * [7] Sijie Yan, Zhizhong Li, Yuanjun Xiong, Huahan Yan, and Dahua Lin.  Convolutional sequence generation for skeleton-based action synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4394–4402, 2019.
  * [8] Chao Li, Qiaoyong Zhong, Di Xie, and Shiliang Pu.  Co-occurrence feature learning from skeleton data for action recognition and detection with hierarchical aggregation.  arXiv preprint arXiv:1804.06055, 2018.
  * [9] Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik.  A training algorithm for optimal margin classifiers.  In Proceedings of the fifth annual workshop on Computational learning theory, pages 144–152, 1992.
  * [10] Mengyuan Liu, Qinqin He, and Hong Liu.  Fusing shape and motion matrices for view invariant action recognition using 3d skeletons.  In 2017 IEEE International Conference on Image Processing (ICIP), pages 3670–3674. IEEE, 2017.
  * [11] Junwu Weng, Chaoqun Weng, and Junsong Yuan.  Spatio-temporal naive-bayes nearest-neighbor (st-nbnn) for skeleton-based action recognition.  In Proceedings of the IEEE Conference on computer vision and pattern recognition, pages 4171–4180, 2017.
  * [12] Nick C Tang, Yen-Yu Lin, Ju-Hsuan Hua, Ming-Fang Weng, and Hong-Yuan Mark Liao.  Human action recognition using associated depth and skeleton information.  In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4608–4612. IEEE, 2014.
  * [13] Sebastián Ubalde, Francisco Gómez-Fernández, Norberto A Goussies, and Marta Mejail.  Skeleton-based action recognition using citation-knn on bags of time-stamped pose descriptors.  In 2016 IEEE International Conference on Image Processing (ICIP), pages 3051–3055. IEEE, 2016.
  * [14] Haodong Duan, Yue Zhao, Kai Chen, Dian Shao, Dahua Lin, and Bo Dai.  Revisiting skeleton-based action recognition.  arXiv preprint arXiv:2104.13586, 2021.
  * [15] Xiang Wang, Zhiwu Qing, Ziyuan Huang, Yutong Feng, Shiwei Zhang, Jianwen Jiang, Mingqian Tang, Changxin Gao, and Nong Sang.  Proposal relation network for temporal action detection.  arXiv preprint arXiv:2106.11812, 2021.
  * [16] Yong Du, Yun Fu, and Liang Wang.  Skeleton based action recognition with convolutional neural network.  In 2015 3rd IAPR Asian conference on pattern recognition (ACPR), pages 579–583. IEEE, 2015.
  * [17] Pichao Wang, Zhaoyang Li, Yonghong Hou, and Wanqing Li.  Action recognition based on joint trajectory maps using convolutional neural networks.  In Proceedings of the 24th ACM international conference on Multimedia, pages 102–106, 2016.
  * [18] Chuankun Li, Yonghong Hou, Pichao Wang, and Wanqing Li.  Joint distance maps based action recognition with convolutional neural networks.  IEEE Signal Processing Letters, 24(5):624–628, 2017.
  * [19] Chao Li, Qiaoyong Zhong, Di Xie, and Shiliang Pu.  Skeleton-based action recognition with convolutional neural networks.  In 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 597–600. IEEE, 2017.
  * [20] Qiuhong Ke, Senjian An, Mohammed Bennamoun, Ferdous Sohel, and Farid Boussaid.  Skeletonnet: Mining deep part features for 3-d action recognition.  IEEE signal processing letters, 24(6):731–735, 2017.
  * [21] Yong Du, Wei Wang, and Liang Wang.  Hierarchical recurrent neural network for skeleton based action recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1110–1118, 2015.
  * [22] Yong Du, Yun Fu, and Liang Wang.  Representation learning of temporal dynamics for skeleton-based action recognition.  IEEE Transactions on Image Processing, 25(7):3010–3022, 2016.
  * [23] Vivek Veeriah, Naifan Zhuang, and Guo-Jun Qi.  Differential recurrent neural networks for action recognition.  In Proceedings of the IEEE international conference on computer vision, pages 4041–4049, 2015.
  * [24] Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, and Jiaying Liu.  An end-to-end spatio-temporal attention model for human action recognition from skeleton data.  In Proceedings of the AAAI conference on artificial intelligence, 2017.
  * [25] Songyang Zhang, Xiaoming Liu, and Jun Xiao.  On geometric features for skeleton-based action recognition using multilayer lstm networks.  In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 148–157. IEEE, 2017.
  * [26] Sijie Yan, Yuanjun Xiong, and Dahua Lin.  Spatial temporal graph convolutional networks for skeleton-based action recognition.  In Thirty-second AAAI conference on artificial intelligence, 2018\.
  * [27] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu.  Skeleton-based action recognition with multi-stream adaptive graph convolutional networks.  IEEE Transactions on Image Processing, 29:9532–9545, 2020.
  * [28] Yi-Fan Song, Zhang Zhang, and Liang Wang.  Richly activated graph convolutional network for action recognition with incomplete skeletons.  In 2019 IEEE International Conference on Image Processing (ICIP), pages 1–5. IEEE, 2019.
  * [29] Yongsang Yoon, Jongmin Yu, and Moongu Jeon.  Predictively encoded graph convolutional network for noise-robust skeleton-based action recognition.  Applied Intelligence, pages 1–15, 2021.
  * [30] Alban Main De Boissiere and Rita Noumeir.  Infrared and 3d skeleton feature fusion for rgb-d action recognition.  IEEE Access, 8:168297–168308, 2020.
  * [31] Inwoong Lee, Doyoung Kim, Seoungyoon Kang, and Sanghoon Lee.  Ensemble deep learning for skeleton-based action recognition using temporal sliding lstm networks.  In Proceedings of the IEEE international conference on computer vision, pages 1012–1020, 2017.
  * [32] Hossein Rahmani and Mohammed Bennamoun.  Learning action recognition model from depth and skeleton videos.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5832–5841, 2017.
  * [33] Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, and Nanning Zheng.  View adaptive neural networks for high performance skeleton-based human action recognition.  IEEE transactions on pattern analysis and machine intelligence, 41(8):1963–1978, 2019.
  * [34] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020.
  * [35] Haocong Rao, Shihao Xu, Xiping Hu, Jun Cheng, and Bin Hu.  Augmented skeleton based contrastive action learning with momentum lstm for unsupervised action recognition.  Information Sciences, 569:90–109, 2021.
  * [36] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * [37] Connor Shorten and Taghi M Khoshgoftaar.  A survey on image data augmentation for deep learning.  Journal of big data, 6(1):1–48, 2019.
  * [38] Xiaogeng Liu, Haoyu Wang, Yechao Zhang, Fangzhou Wu, and Shengshan Hu.  Towards efficient data-centric robust machine learning with noise-based augmentation.  arXiv preprint arXiv:2203.03810, 2022.
  * [39] Juan Luis Suárez, Salvador García, and Francisco Herrera.  A tutorial on distance metric learning: Mathematical foundations, algorithms, experimental analysis, prospects and challenges.  Neurocomputing, 425:300–322, 2021.
  * [40] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019.
  * [41] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [42] Mahesh Chandra Mukkamala and Matthias Hein.  Variants of rmsprop and adagrad with logarithmic regret bounds.  In International conference on machine learning, pages 2545–2553. PMLR, 2017.
  * [43] Aaron Defazio and Samy Jelassi.  Adaptivity without compromise: a momentumized, adaptive, dual averaged gradient method for stochastic optimization.  Journal of Machine Learning Research, 23:1–34, 2022.
  * [44] Tristan Cazenave, Julien Sentuc, and Mathurin Videau.  Cosine annealing, mixnet and swish activation for computer go.  In Advances in Computer Games, pages 53–60. Springer, 2022.
  * [45] Ayman Al-Kababji, Faycal Bensaali, and Sarada Prasad Dakua.  Scheduling techniques for liver segmentation: Reducelronplateau vs onecyclelr.  arXiv preprint arXiv:2202.06373, 2022.
  * [46] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  Advances in neural information processing systems, 32, 2019.
  * [47] Lutz Prechelt.  Early stopping-but when?  In Neural Networks: Tricks of the trade, pages 55–69. Springer, 1998.
  * [48] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [49] Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang.  Ntu rgb+ d: A large scale dataset for 3d human activity analysis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010–1019, 2016.
  * [50] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot.  Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding.  IEEE transactions on pattern analysis and machine intelligence, 42(10):2684–2701, 2019.
  * [51] Chunhui Liu, Yueyu Hu, Yanghao Li, Sijie Song, and Jiaying Liu.  Pku-mmd: A large scale benchmark for continuous multi-modal human action understanding.  arXiv preprint arXiv:1703.07475, 2017.
  * [52] Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, and Nanning Zheng.  View adaptive recurrent neural networks for high performance human action recognition from skeleton data.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2117–2126, 2017.