[table]capposition=top normalsize[table]font=normalsize

# CNN based Multistage Gated Average Fusion (MGAF) for Human Action
Recognition Using Depth and Inertial Sensors

Zeeshan Ahmad, Student Member, IEEE, Naimul Khan, Senior Member, IEEE

###### Abstract

Convolutional Neural Network (CNN) provides leverage to extract and fuse
features from all layers of its architecture. However, extracting and fusing
intermediate features from different layers of CNN structure is still
uninvestigated for Human Action Recognition (HAR) using depth and inertial
sensors. To get maximum benefit of accessing all the CNN’s layers, in this
paper, we propose novel Multistage Gated Average Fusion (MGAF) network which
extracts and fuses features from all layers of CNN using our novel and
computationally efficient Gated Average Fusion (GAF) network, a decisive
integral element of MGAF. At the input of the proposed MGAF, we transform the
depth and inertial sensor data into depth images called sequential front view
images (SFI) and signal images (SI) respectively. These SFI are formed from
the front view information generated by depth data. CNN is employed to extract
feature maps from both input modalities. GAF network fuses the extracted
features effectively while preserving the dimensionality of fused feature as
well. The proposed MGAF network has structural extensibility and can be
unfolded to more than two modalities. Experiments on three publicly available
multimodal HAR datasets demonstrate that the proposed MGAF outperforms the
previous state-of-the-art fusion methods for depth-inertial HAR in terms of
recognition accuracy while being computationally much more efficient. We
increase the accuracy by an average of 1.5% while reducing the computational
cost by approximately 50% over the previous state-of-art.

###### Index Terms:

Convolutional neural network, deep learning, human action recognition, gated
average fusion, sensor fusion.

##  I Introduction

HUMAN action recognition has progressed by leaps and bounds over the last two
decades due to its many direct and indirect applications in numerous circles
of our daily life such as media industry [1], robotics [2], sports [3],
healthcare [4] and smart surveillance [5]. These applications are indeed the
true motivation behind the extensive research that has been carried out on HAR
over the past years. ††© 2020 IEEE. Personal use of this material is
permitted. Permission from IEEE must be obtained for all other uses, in any
current or future media, including reprinting/republishing this material for
advertising or promotional purposes, creating new collective works, for resale
or redistribution to servers or lists, or reuse of any copyrighted component
of this work in other works.

Earlier, the research on HAR was on single modality sensor, either the vision
sensor or inertial sensor. Vision sensors include RGB cameras and Kinect
sensors while the inertial sensors for HAR are accelerometer and gyroscope.
Using single modalities, several methods for HAR have been proposed over the
years and these approaches set some promising directions for human action
analysis [6] [7]. However, each modality has limitations and deficiencies. For
instance, conventional RGB cameras acquire 2D images and they are sensitive to
lighting conditions, background clutter, partial occlusion and thus making
action recognition task more challenging [8].

The revolution in imagery technology emerges a new opportunity for HAR in the
shape of cost-effective depth sensors such as Kinect sensor which is capable
of providing 3D action data, require less resources to operate and is less
sensitive to lighting changes and illumination [9]. Yet, there are limitations
associated with depth images such as view point variation, sensitiveness to
noise during image acquisition and effect on the privacy of patients while
using for healthcare applications. To address these limitations, researchers
started to use wearable inertial sensors for HAR such as accelerometer and
gyroscope [10]. Wearable sensors provide 3D action data in terms of
multivariate time series. These sensors provide data at a high sampling rate
and can work in gloomy and confounded conditions. Like vision based sensors,
inertial sensors also have limitations such as sensor drift, inadequate
onboard power and awkwardness of wearing them all the time [11].

From above discussion it is clear that no single modality can encounter the
challenging and unexpected situations that may arise in practice. The way out
is to integrate the data from two modalities that can complement each other.
Since depth images capture global features while inertial signals capture
local attributes, the fusion of these two modalities leads to robust and
improved recognition [12]. Furthermore, these two sensors are cost-effective,
provide 3D action data and their multimodal fusion requires less computational
complexities for HAR. We use depth images rather than the video camera images
because the images obtained from video cameras are sensitive to lightening and
illuminations, have limited viewing angle, needs aperture adjustments and
require computationally complex processing algorithms than depth images [11].

Earlier fusion methods for HAR using depth and inertial sensors were based on
hand crafted methods where features are extracted using statistical methods
like mean and variance in time domain [13] and the coefficients of Fast
Fourier transform in frequency domain [14]. The disadvantages with classical
methods are the requirement of domain knowledge about the data and the
separation of feature extraction part from the classification part [15].
Furthermore, these hand crafted feature methods capture only a subset of the
features, resulting in difficulty to generalize for unseen data. Outstanding
performance of deep learning models, especially the convolutional neural
network in the field of computer vision and image classification [16] [17],
trigger researchers to design deep learning based multimodal fusion frameworks
for HAR using depth and inertial sensors [18].

A major challenge in multimodal fusion is to decide when and where to
integrate the modalities for optimal result [19]. Mulimodal fusion of sensors
can be performed at feature level and decision. The major disadvantage with
decision level fusion is that it requires $N$ classifiers to train and test
for $N$ modalities. Furthermore, with decision level fusion the correlated
data necessary to improve recognition accuracy cannot be combined at earlier
stages. In conradiction, the feature-level fusion ensures the collection and
integration of correlated and concurrent information from the modalities
necessary for the classifier to make a rigorous decision. Thus, feature level
fusion assures the semantic information to flow from data to classifier [20].

To address the aforementioned deficiencies and to exploit the fact that CNN
allows to extract features from all the layers of its structure, in this
paper, we present a novel multistage gated average fusion (MGAF) network
capable of extracting and fusing features from all the layers of CNN using our
proposed GAF network. GAF network effectively fuses the features from CNN
layers using high boost kernel and gating mechanism. GAF network can be
smoothly unrolled to more than two modalities without increasing the
dimensionality of the fused features.

The key novelty of the proposed MGAF resides in an intelligent approach of its
gated average fusion (GAF) network that extracts, selects and fuses features
in an efficient manner so that the accuracy of the classification task could
be increased without increasing the computation cost as compared to the
existing works that depend upon concatenation and the creation of the
additional modalities.

The overview of the proposed MGAF network is shown in Fig. 1 and is explained
in detail in section III.

The novelty in the proposed framework can be summarized through the following
three key contributions :

  1. 1.

We propose a unique fusion network called multistage gated average fusion
(MGAF) network which takes advantage of intermediate features of CNN and
extracts discriminative, complementary and multiscale features from these
layers. The extracted features are then fused using a novel gated average
fusion (GAF) network which is an important componenet of MGAF. The proposed
MGAF has architectural flexibility and can be stretched out to more than two
modalities.

  2. 2.

Gated average fusion (GAF) network, which is the backbone of proposed MGAF,
serves two purposes : (1) Fusing incoming feature maps from modalities by
commissioning proper gated values to each pixel of feature maps. These values
are calculated through gating mechanism, similar to the input gate function of
long short-term memory (LSTM) network. (2) keeping the size of the fused
feature map equal to the size of the feature map of single input modality by
performing gated average fusion. Thus, the proposed GAF not only fuses the
features but also performs dimensionality reduction.

  3. 3.

We experiment on three publicly available datasets with an efficient CNN
consisting of only three convolutional layers. Achieving state of art outcomes
using small CNN infers the excellence of proposed MGAF network over existing
proposed fusion frameworks for depth-inertial HAR. We match or outperform all
the state-of-the-art methods, including the frameworks recently proposed by us
in [18], while significantly reducing both training and inference
computational cost.

The rest of the paper is organized as follows. Section II describes the
related works on HAR using depth and inertial sensors individually and their
conventional and deep learning based fusion. Section III provides detailed
description of the proposed method. In section IV, we provide detailed
experimental analysis, where the aforementioned contributions are analyzed in
detail through large number of experiments and comparison with state-of-the-
art models. Section V concludes the paper.

##  II Related Work

###  II-A Depth data based approaches

Large number of methods for HAR using depth data have been proposed in the
literature. A real time human action recognition system is developed in [21]
using depth motion maps generated from depth sequences and $l_{2}$ regularized
collaborative representation classifier. In [22], action graphs are generated
from depth map sequences to model the dynamics of the actions and a bag of 3D
points to characterize a set of salient postures that correspond to the nodes
in the action graph. Authors in [23] generate Histograms of Oriented Gradients
(HOG) from depth motion maps to build DMM-HOG descriptors for human action
recognition.

In [24], human action descriptor based on local spatio-temporal information
from depth video sequences is proposed. This descriptor works in three stages
and is capable of distinguishing similar actions performed with different
speeds. A fast and accurate real time human pose recognition system is
developed in [25] that uses an intermediate body parts representation to map
the difficult pose estimation problem into a simpler per-pixel classification
problem. A comprehensive survey on human action recognition using depth data
is provided in [26] where the recent work and challenges related to depth
modality are described in detail.

###  II-B Inertial and other sensor based approaches

HAR using inertial sensors became popular due to their certain advantages over
depth sensors such as insensitiveness against any lighting and illumination
conditions. In [27], CNN is used to classify eight different human action from
the data collected by tri-axial accelerometer. In [28], extreme learning
machine is used to classify ten actions using time series data obtained from
WiFi sensor. In [29], wearable sensor data is converted into activity images
and then CNN is employed for action classification. Experiments on three
datasets show that proposed method is robust and highly accurate. In [30],
limited data from accelerometer and gyroscope is used for HAR using CNN, LSTM
and classical machine learning algorithms. Accuracy of action recognition was
further increased by applying data augmentation and data balancing techniques.
A survey article [31] provides details about the performance of current deep
learning models and future challenges on sensor based activity recognition.

Other sensor modalities than inertial and depth for existing HAR are radar,
sonar and video cameras. In [32], a survey on different applications of
activity recognition such as healthcare, well being and sports using inertial
sensors is presented. All the related challenges from data collection to
classification and application directions are discussed in detail. In [33], a
wearable strain sensor technology is introduced for capturing and interpreting
the dynamics of human fingertip. The wearable device transmits raw deformation
data to an off-finger device for interpretation. Simple motions, gestures,
finger-writing, grip strength, and activation time, as well as more complex
idioms consisting of multiple grips, are identified and quantified.
Applications of proposed technology in different fields are also discussed. In
[34], the limitations of Kinect sensor is addressed by incorporating a set of
features to create a unique “Kinect signature” for identification of different
subjects during assessment of body kinematics of elderly people and patients
with neurological disorders while performing daily life activities at home. In
[35], wideband sonar is used to detect and classify human activities in indoor
environment. Wideband sonar is capable of tracking body parts precisely and
enhanced correlation properties of sonar distinguish between human and non-
human objects. The proposed technology can also be applied for monitoring
patients at home and for detecting intruders. In [36], experimental based
feasibility of applying deep learning methods for human activity recognition
using radar data is surveyed. Current research challenges and future
opportunities in various fields such as human–computer interaction, smart
surveillance and health assessment is discussed at length. In [37], a novel
RGB activity image-based DCNN classifier for the unobtrusive recognition of
the multi-resident activities is proposed. The proposed method is
experimentally proven to be helpful for increasing the recognition rate as
compared to the previous studies.

###  II-C Fusion of depth and inertial data

Some recent methods on HAR are based on fusing different modalities to get
better recognition results. Human action recognition by fusing depth and
inertial sensor data has gained significant attention due to their advantages
over other modalities. In [38], the accuracy of human action recognition is
improved by fusing features extracted from depth and inertial sensor data and
using collaborative representation classifier. Improved accuracy results were
achieved due to complementary aspect of data from both modalities. A decision
level fusion is performed between depth cameras and wearable sensors to
increase the capabilities of robots to recognize human actions [39]. An
efficient real-time human action recognition system is developed in [40] using
decision level fusion of depth and inertial sensor data. Depth and inertial
data is effectively merged in [41] to train hidden Markov model for improving
accuracy and robustness of hand gesture recognition.

In [42], computationally efficient real-time detection and recognition
approach is presented to identify actions in the smart TV application from
continuous action streams using continuous integration of information obtained
from depth and inertial sensor data. A novel, user friendly and safe method of
bilateral gait segmentation is proposed in [43] by multimodal fusion of
features obtained from thigh mounted inertial sensor and depth sensor with the
contralateral leg in its field of view. The proposed method can be used to
make lower limb assistive devices for patients with walking impairments. A
comprehensive survey on fusion of depth and inertial sensors is provided in
[12] where the recent success of the fusion and future challenges and trends
are disscussed in detail.

###  II-D Deep Learning based Fusion

After the knock out performance of deep learning models on numerous machine
learning and computer vision applications, several deep learning based fusion
frameworks are presented. In [44], we proposed a CNN based fusion framework
where depth and inertial sensor data is tranformed into images and then CNNs
are employed to extract features from the transformed images. Finally
concatenation fusion is performed between the features obtained from depth and
signal images. These fused features served as input to train multiclass SVM
classifier. Deep learning based decision level multimodal fusion framework is
proposed in [45]. CNN is used to extract features from depth images while
Recurrent Neural Netwrok (RNN) is used to capture features from inertial
sensor data. Data augmentation is also carried out to cop up with limited size
data. In [46], a supervised deep multimodal fusion framework for process
monitoring and verification in the medical and healthcare fields is presented
that depends on simultaneous processing of motion data acquired with wearable
sensors and video data acquired with body-mounted camera. A CNN based sensor
fusion system is developed in [47] to detect and monitor transition movements
between body states as well as falls in healthcare applications. A score level
sensor fusion is performed between the features extracted from depth and
inertial sensor data by two CNNs.

In [48], simultaneously captured video and inertial data are converted into 3D
video images and 2D inertial images respectively and are then fed as inputs
into a 3D convolutional neural network and a 2D convolutional neural network,
respectively, for recognizing actions. Then decision level and feature level
fusion are performed to improve the classification accuracy. In [49], deep
learning based fusion system based on fusing depth and inertial data is
presented which is capable of detecting and recognizing actions of interest
from continuous action streams. CNN is used to extract features from depth
images, and a combination of CNN and LSTM network is utilized for inertial
signals. First, segmentation is performed on each sensing modality and then
actions of interest were detected. Finally, decision level fusion is performed
for recognition. In [50], video and inertial data, introduced in [51] is
converted into 2D and 3D images which are used as input to 2D and 3D CNN.
Finally, decision level fusion is used to increase the recognition accuracy.

Figure 1:  Complete Overview of the Proposed Multistage Gated Average Fusion
(MGAF) network. Figure 2: Architecture of the proposed Gated Average Fusion
(GAF) network.

A common pitfall in the existing works is that the fusion strategies used are
either feature level or decision level. Furthermore, these fusion tactics are
exercised at a single level, thus failing to capture midlevel features which
could result in better classification performance.

Figure 3: Architecture of CNN used in proposed MGAF.

In [18], we proposed three novel fusion frameworks that outperformed existing
fusion frameworks for HAR using depth and inertial sensors. In all three
fusion frameworks, fusion was performed at two stages to overcome the
weaknesses of single stage fusion. Nevertheless, there are imperfections in
these proposed frameworks. Firstly, the two input modalities are converted
into multiple modalities, thus making computation more complex. In addition to
this, features are extracted only from fully connected layers and then two
stage consecutive fusion is performed, thus neglecting the complementary and
unique features present in the convolutional layers of CNN to take part in
fusion.

Some gated fusion networks are introduced in [52], [53] and [54]. The first
weakness in these fusion networks is that they apply concatenation at the
early stage of fusion network, limiting the scalability and extensibility of
networks for more modalities and ability to extract features from each layer
of CNN. Furthermore, concatenation creates the problem of the curse of
dimensionality, high computational cost, limited applicability and
deterioration of accuracy [55].

To address the shortcomings of existing works, in this paper, we propose a
multistage gated average fusion (MGAF) network which has the strength and
skill of extracting and fusing the features from all the layers of CNN while
reducing of dimensionality as well. A significant part of the proposed MGAF
network is gated average fusion (GAF) network as shown in Fig. 2. The proposed
MGAF beats or matches our previous work [18] with the current state-of-the-art
performance on benchmark datasets, while significantly reducing the training
and inference computational costs by utilizing a much smaller CNN, as
described in section IV-G.

##  III Proposed Multistage Gated Average Fusion Framework

In this section, we will explain in detail the proposed multistage gated
average fusion (MGAF) network shown in Fig. 1.

At the input of the proposed multistage gated average fusion (MGAF) network
shown in Fig. 1, both modalities are converted into images. Following our
recent work in [18], we convert the depth data into sequential front view
images (SFI) and inertial data into signal images (SI). Two CNNs of same
architecture, shown in Fig. 3, are trained on these images. Learned features
extracted from all convolutional layers and first fully connected layer are
fused using gated average fusion (GAF) network shown in Fig. 2 and explained
in section III-D.

###  III-A Formation of Sequential Front View Images

At the input of the proposed MGAF, we convert depth data into images called
sequential front view images (SFI) as shown in Fig. 4. The depth modality
provides 3D action information for generating front, top and side view of
depth motion maps. It is experimentally proved that only front view
information can be effectly utilized to recognize the actions as the fusion of
front view with other views doesnot significantly increase the recognition
accuracy [18]. For making sequential front view images (SFI), each depth
sequence has the following dimension.

$Depth=rows\times columns\times numberofframes$.

The number of SFI formed from each depth sequence is equal to the number of
frames with size $rows\times columns$. We resize the image to 64 x 64 to train
on CNN shown in Fig. 3. During fusion, supplementary information is provided
by the inertial dataset. Generating SFI using only one view reduces the
computational cost. SFI are like the motion energy images and motion history
images introduced in [56]. These SFIs provide aggregate knowledge about the
dynamics of action.

Figure 4: Five samples of Sequential Front View Images of Hand Catch

###  III-B Formation of Signal Images

The combination of accelerometer and gyroscope provide multivariate time
series data consists of six sequences of signals : three acceleration and
three angular velocity sequences.

We converted six sequences into 2D images called signal images based on the
algorithm in [29]. The conversion of time series data to signal image is shown
in Fig. 5. Signal image is obtained through row-by-row stacking of given six
signal sequences in such a way that each sequence appears alongside to every
other sequence. The signal images are formed by taking advantage of the
temporal correlation among the signals.

Row wise stacking of six sequences has the following order.

123456135246142536152616

Where the numbers 1 to 6 represent the sequence numbers in a raw signal. We
made sure that every sequence is a neighbor of every other sequence only once.
Using full sequence permutation order will cause redundant information in the
image and that may lead to the overfitting of the CNN during training. Thus
the final width of signal image becomes 24.

Length of signal image is decided by making use of sampling rate of datasets
which is 50Hz for our two datasets. Therefore, to capture granular motion
accurately and to facilitate the design of CNN, the length of the signal image
is finalized as 52, resulting in a final image size of 24 x 52. These signal
images are shown in Fig. 6 and it is observed that each signal image shows
unique pattern and thus enhance the performance of classification. Finally, we
resize the images to 64 x 64.

Figure 5: Formation of Signal Image from Inertial sensor data Figure 6: Signal
Images of three different actions

###  III-C Architecture of CNN

Architecture of CNN used in proposed MGAF is shown in Fig. 3. CNN consists of
three convolutional layers, two pooling layers, and a fully connected layer.
The first convolutional layer has 16 kernels of size 5x5, followed by pooling
layer of size 2x2 and stride 2. Second and third convolutional layers have 32
kernels of size 5x5 followed by 2x2 pooling layer with stride 2.

###  III-D Gated Average Fusion (GAF) Network

The structure of the gated average fusion (GAF) network is shown in Fig. 2.
The input to GAF network are the feature maps extracted from the layers of CNN
of each modality. The size of feature maps extracted from convolutional layer
is 4-dimensional (4D) and is given by

$Size\>offeature\>map=a\times b\times n\times N$

where,

$a$ = Length of the feature map

$b$ = Width of the feature map

$n$ = Number of feature maps

$N$ = Number of training samples per batch

To make the computation meaningful, uncomplicated and to make 2-dimensional
(2D) convolution possible with high boost kernel, we transform the 4D features
into 2D by following method.

$Size\>of2D\>feature\>map=N\times M$

where,

$N$ = Number of training samples per batch

$M=a\times b\times n$

Let $F_{1}$ and $F_{2}$ are the feature maps of size $N\times M$ obtained from
the CNNs of each modalitity respectively. These feature maps are now convolved
with high boost kernel $K$. During convolution, the size of feature maps
remain the same.

The kernel used for convolution with feature maps is a high boost filter that
highlights the fine details of feature maps and assigns proper boosted value
to each pixel of feature map according to its significance. High boost filter
has been used in [57] for watermark recovery in spatial domain by enhancing
the dissimilarity between watermarked and unwatermarked parts of the image.
High boost kernel can be of size 1D and 2D. However, when used as a separable
filter i.e when used to enhance the important specifics of the image, it works
better in 2D form. Thus, to have a meaningful and effective convolution with
2D kernel, the size of image or feature map should also be 2D. High boost
filter is a second order derivative filter obtained by subtracting the low
pass filtered version of the image from the scaled input image.

$f_{hb}(x,y)=Af(x,y)-f_{lp}(x,y)$ (1)

where $Af(x,y)$ and $f_{lp}(x,y)$ are the scaled and low pass versions of
original image $f(x,y)$

The general form of high boost filter is given by

$K=\begin{bmatrix}\par-1&-1&-1\\\ -1&A+8&-1\\\ -1&-1&-1\par\end{bmatrix}$ (2)

where $A$ is called amplification factor that controls the amount of weight
given to the feature maps during convolution.

For $A$ = 1, we observe optimal filter performance. Higher values of $A$
causes less boosting.

Thus, we empirically select the following high boost kernel that emphasizes
the important specifics.

$K=\begin{bmatrix}-1&-1&-1\\\ -1&9&-1\\\ -1&-1&-1\par\end{bmatrix}$ (3)

High boost filter enhances high frequency components while still preserving
the low frequency components.

After convolution with kernel, sigmoid function is applied to generate the
gated values $G_{1}$ and $G_{2}$ respectively as shown in Fig. 2. Sigmoid
function squishes the values between 0 and 1. Finally, the gated values
$G_{1}$ and $G_{2}$ are element-wise multiplied with feature maps $F_{1}$ and
$F_{2}$ respectively to perform gated avearge fusion and to generate fused
feature map. This gated fusion is performed with all feature maps obtained
from three convolutional layers and first fully connected layer of the CNNs of
both modalities as shown in Fig. 1. The operation of GAF network is summarized
by the following mathematical equations.

$G_{1}=\sigma(F_{1}\circledast\ K)$ (4)

$G_{2}=\sigma(F_{2}\circledast\ K)$ (5)

$F_{f}(j)=G_{1}\odot F_{1}(j)+G_{2}\odot F_{2}(j)$ (6)

where,

$\sigma(z)\triangleq\frac{1}{1+e^{-z}}$ : sigmoid function.

$y\circledast z$ : convolution operation

$y\odot z$ : element-wise multiplication

$F_{i}(j)$ : $j$th feature map of $i$th modality

$F_{f}(j)$ : $jth$ fused feature map

The working of equations 4 and 5 can be thought of as an input gate of LSTM
that decides which values will be updated by transforming the values to be
between 0 and 1. 0 means not important, and 1 means most important.

The GAF network scales very well for multimodal fusion, since the gated
average fusion output size remains the same regardless of the number of
modalities. If $F_{1},F_{2},F_{3},...,F_{N}$ are feature maps of $N$ input
modalities each of size $N\times M$, then the fused feature map $F_{f}$ will
also be of size $N\times M$. This can also be understood by transforming
equation 6 for $N$ modalities.

$F_{f}=G_{1}\odot F_{1}+G_{2}\odot F_{2}+G_{3}\odot F_{3}+....+G_{N}\odot
F_{N}$ (7)

###  III-E MGAF-Architecture

The fused features generated by the gated average fusion (GAF) network from
different CNN layers are collected at multimodal layer and concatenated and
finally sent to the SVM classifier for classification task. The multimodal
layer serves as input to the SVM classifier. The SVM classifier is trained
separately from the CNNs. CNNs are used as feature extractors, SVM is employed
for the final classification. We used support vector machine (SVM) as a
classifier since we experimentally proved in our previous work [44] that SVM
performs better than softmax, which is typically built into any CNN framework.
Softmax classifier reduces the cross entropy function while SVM employs a
margin based function. The more rigorous nature of classification is the
reason of better performance of SVM over softmax.

Although we explain MGAF with two modalities, the proposed MGAF can be
conveniently out-stretched to more than two modalities without increasing the
size of the fused feature map. This is due to the fact that GAF network of
MGAF architecture generates fused features by performing gated average fusion
which keeps the size of fused feature map equal to the size of single input
feature map and thus avoid the curse of dimensionality which is commonly
observed in case of concatenation fusion. To show extensibility, we also
perform experiments using three modalities and display results in section IV.

TABLE I: Dataset Information

Dataset Sampling Rate Modality Training Samples Test Samples UTD MHAD 50 Depth
46639 11660 Inertial 11031 2745 Berkeley MHAD 30 Depth 26400 6500 Inertial
2612 653 UTD Kinect V2 50 Depth 14098 3524 Inertial 3532 884

##  IV Experiments and Results

We experiment on three publicly available multimodal human action datasets,
namely, UTD-MHAD [58], Berkeley MHAD [59] and UTD Kinect-V2 dataset [60]. We
used subject specific setting for experiments on all datasets. In subject
specific setting, training and testing sets are split randomly across all
subjects.

For all experiments, we split the datasets into training and testing samples
by randomly splitting 80% data into training and 20% data into testing
samples. We ran the random split 20 times and report the average accuracy. The
number of training and testing samples after splitting are shown in Table I.

###  IV-A Ablation Study

In order to show the significance of the proposed MGAF, we carry out ablation
study on all datasets using concatenation, average fusion and gated fusion
without kernel. Average fusion is performed by assigning the value one to both
gated outputs i.e $G_{1}=1$ and $G_{2}=1$ in the GAF network. In gated fusion
without kernel, the sigmoid function is applied directly to input feature maps
$F_{1}$ and $F_{2}$ and the gated values are calculated by
$G_{1}=\sigma(F_{1})$ and $G_{2}=\sigma(F_{2})$. The results of ablation study
are reported in Tables II, III and IV along with proposed MGAF. The results of
ablation study are explained in detail in section IV-E.

Two Convolutional neural networks of same architecture shown in Fig. 3, are
trained on SFI and SI as shown in Fig. 1. In addition to this, momentum of
0.9, initial learn rate of 0.005, $L_{2}$ regularization of 0.004 and mini-
batch size of 64 is used to control overfitting. We reached these value
through using the grid search method. We conduct our experiments on Matlab
R2018b on a desktop computer with NVIDIA GTX-1070 GPU. We train CNNs on images
till the validation loss stops decreasing further.

TABLE II: Experimental results on UTD MHAD

Modalities Concatenation Average Fusion Gated Fusion without Kernel Proposed
Gated Average Fusion SFI + SI 97.8 99.05 99.13 99.3 SFI + SI + SFI (Prewitt)
94.7 99.1 99.22 99.4 SI(Spatial) + SI(DFT) + SI (GWT) 87.2 95 96.4 96.8

TABLE III: Experimental results on Berkeley MHAD

Modalities Concatenation Average Fusion Gated Fusion without Kernel Proposed
Gated Average Fusion SFI + SI 99.2 99.4 99.5 99.85 SFI + SI + SFI (Prewitt) 99
99.6 99.7 99.9 SI(Spatial) + SI(DFT) + SI (GWT) 91.2 98.2 98.6 98.9

TABLE IV: Experimental results on Kinect-V2 dataset

Modalities Concatenation Average Fusion Gated Fusion without Kernel Proposed
Gated Average Fusion SFI + SI 97 99.3 99.44 99.8 SFI + SI + SFI (Prewitt) 96.2
99.7 99.74 99.9 SI(Spatial) + SI(DFT) + SI (GWT) 91 98.8 99.1 99.4

Fusion Frameworks Training Speed ($minutes$) Inference Speed ($\mu$s) Training
Parameters Deep Multistage Feature Fusion Framework [18] 268 158 49232121 Deep
Hybrid Fusion Framework [18] 268 235 49232121 Computationally Efficient Fusion
Framework [18] 145 147 25180266 Proposed MGAF (SFI + SI) 58 132 1229814

TABLE V: Comparison of Computational Cost of Proposed MGAF with previous state
of art on UTD-MHAD Dataset

###  IV-B UTD-MHAD Dataset

The UTD-MHAD dataset consists of both depth and inertial components and is
collected in an indoor environment by 12 subjects with each subject repeating
the actions 4 times. The dataset contains 27 actions.

The inertial component of UTD-MHAD dataset is very challenging for deep
network training due to few flaws. The first drawback is that inertial sensor
was worn either on volunteer’s right wrist or right thigh depending upon the
nature of action. Hence the sensor is worn only on two positions for
collecting data of 27 actions which is insufficient to capture all the
dependencies and characteristics of data. Furthermore, the number of data
samples after converting to SI are only 2206, which is not enough to train a
deep network properly.

To combat these challenges, we perform data augmentation on signal images to
increase the number of samples by applying the data augmentation techniques
discussed in [44]. The number of samples shown in Table I are obtained after
augmentation of signal images.

We perform experiments on our proposed MGAF using UTD-MHAD datsets with same
number of samples shown in Table I and with same parameters of CNN described
in section IV-A. The results of all experiments in terms of recognition
accuracies are presented in Table II. These results are discussed in detail in
sections IV-E and IV-F.

Since proposed MGAF is extendable to more than two modalities as explained in
section III, we also perform experiments by creating more modalities to
validate the usability of the proposed MGAF. We create another modality called
SFI (Prewitt) from depth data by convolving SFI with Prewitt filter as
described in our previous work in [18]. The results obtained by experimenting
with three modalities i.e SFI, SI and SFI (Prewitt) are also reported in Table
II.

We also made inertial data multimodal by creating two more modalities using
Discrete Fourier transform (DFT) and Gabor Wavelet transform (GWT) as
illustrated in [61]. We obtained recognition accuracy of 96.8% as shown in the
last column of Table II beating previous state of art 95.8% as reported in
[61].

The comparison of recognition accuracies by using only two modalities i.e
depth and inertial with previous state-of-the-art on UTD-MHAD is shown in
Table VI.

Most of the work in Table VI used subject specific setting except in [62],
[45] and [49]. We used subject specific setting since this is a most commonly
used [60]. We compare the results on the basis of modalities i.e depth and
inertial.

Previous Methods | Accuracy%
---|---
C.chen et al. [58] | 97.1
Bulbul et al. [62] | 88.4
N.Dawar et al. [45] | 89.2
N.Dawar et al. [49] | 92.8
Chen et al. [40] | 97.2
Ehatisham et al. [11] | 98.3
Z.Ahmad et al. [44] | 98.4
Mahjoub et al. [63] | 98.5
Z.Ahmad et al. [18] | 99.2
Proposed MGAF (SFI + SI) | 99.3
TABLE VI:  Comparison of Accuracies of the proposed MGAF network with previous
methods on UTD-MHAD dataset using depth and inertial modalities.

###  IV-C Berkeley Multimodal Human Action Dataset

The Berkeley MHAD is another multimodal dataset containing both depth and
inertial modalities. The dataset contains 11 actions performed five times by
seven male and five female subjects.

Inertial part of the dataset contains six accelerometers and each generates
three sequences. For generating signal images explained in section III-B, we
need six sequences in a row. Thus we used two accelerometer ($A_{1}$ and
$A_{4}$) and stacked them row wise to make six sequences. The reason for
selecting $A_{1}$ and $A_{4}$ is that they are worn on the left wrist and
right hip, respectively and are able to generate more useful information than
those worn on both ankles [38]. We performed experiments on Berkeley MHAD
dataset using proposed MGAF network with same setting shown in Table I, with
the same parameters mentioned in section IV-A and using the same modalities as
that for UTD-MHAD in section IV-B. The results of experiments on the dataset
and their comparison with previous state-of-the-art are shown in Tables III
and VII respectively. Detailed discussion on experimental results is presented
in sections IV-E and IV-F.

Previous Methods | Accuracy%
---|---
F. Ofli et al. [59] | 97.81
Alireza Shafaei et al. [64] | 98.1
Earnest Paul Ijjina et al. [65] | 98.38
Chen Chen et al. [38] | 99.54
Z.ahmad et al. [18] | 99.8
Proposed MGAF (SFI + SI) | 99.85
TABLE VII: Comparison of Accuracies of proposed MGAF network with previous
methods on Berkeley MHAD dataset using depth and inertial modalities. Figure
7: Features Visualization of Berkeley MHAD. (a) Features without fusion. (b)
Features grouping after concatenation. (c) Features grouping after average
fusion. (d) Features grouping after gated average fusion.

###  IV-D UTD kinect-V2 Dataset

Kinect-V2 action dataset is another publicly available dataset that contains
both depth and inertial data. It is a new dataset using the second generation
of kinect. It contains 10 actions performed by six subjects with each subject
repeating the action 5 times.

We performed experiments on Kinect V-2 dataset with same settings shown in
Table I, with the same parameters of CNN described in section IV-A and using
the same modalities as that for UTD-MHAD in section IV-B and Berkeley MHAD in
section IV-C. The results in terms of recognition accuracies for the dataset
and their comparison with previous state-of-the-art are shown in Tables IV and
VIII respectively. Detailed analysis on experimental results is provided in
section IV-E.

Previous Methods | Accuracy%
---|---
Chen et al. [60] | 99.5
Z.Ahmad et al. [44] | 99.8
Z.Ahmad et al. [18] | 99.8
Proposed MGAF (SFI + SI) | 99.8
TABLE VIII: Comparison of Accuracies of proposed MGAF network with previous
methods on Kinect V2 dataset using depth and inertial modalities. ††Gated
average fusion code can be found at https://github.com/zaamad/Gated-Average-
Fusion

###  IV-E Discussion on Results

Experimental results show that the concatenation performance is poor compared
to other two methods, average fusion and gated average fusion, and it gets
worse with the increase in modalities as shown in Tables II, III, IV.
Concatenation of features from each convolutional layer with different visual
properties and dynamic ranges results in very high dimensionality of feature
space that contains inconsequential features and produces repeated and false
information that leads to the curse of dimensionality and in turns additional
computational complexity and degradation of classification accuracy [66].
Moreover, concatenation works better when features from different modalities
are strongly correlated [67].

Tables V, VI, VII and VIII justify the superiority of the proposed method over
the previous state-of-the-art. An important point to mention is that we
achieve higher accuracy using less computational resources as shown by the
inference speeds in Table V. Action recognition is likely to be a real-time
application, so we believe lower inference speed while surpassing the state-
of-the-art accuracy is a strong contribution. The theoretical framework behind
MGAF also has the potential to be applied in other application areas, since
MGAF can separate the feature space better, as shown by the feature
visualization in Fig. 7.

In the proposed method, we extract features from all convolutional layers and
first fully connected layer of the CNN. During multimodal learning, the
correlation among the features increases as we move from lower layers of CNN
to higher. We experimentally prove this fact by calculating the normalized
correlation coefficients for each dataset as shown in Table IX.

Datasets | conv1 | conv2 | conv3
---|---|---|---
UTD-MHAD | 0.00032 | 0.0026 | 0.033
Berkeley MHAD | 0.0037 | 0.012 | 0.015
Kinect V2 | 0.0003 | 0.0016 | 0.0089

TABLE IX: Normalized cross correlation coefficients between features extracted
from all three convolutional layers of CNN.

The normalized correlation coefficients are calculated using the following
equation.

$NCC(F_{1},F_{2})=\frac{1}{P}\frac{\sum_{x,y}(F_{1}-\bar{f_{1}})(F_{2}-\bar{f_{2}})}{\sigma_{1}\sigma_{2}}$
(8)

where,

$\bar{f_{1}}$ and $\bar{f_{2}}$ are the average pixel intensities of each
feature map. $\sigma_{1}$ and $\sigma_{2}$ are the standard deviation of pixel
intensities of each feature map. $P$ is the total number of pixels in the
feature map. The NCC value is between +1 and -1. If two feature maps are
highly correlated, NCC=1, and NCC = 0 if they are completely uncorrelated.

In Table IX, conv1, conv2 and conv3 denote the first, second and third
convolutional layers of CNN shown in Fig. 3.

As we can see since higher layers have higher NCC values, concatenation
performs better if the features are extracted from the higher layers of CNN
[44].

In order to alleviate these limitations, we presented in this paper a novel
fusion framework that selected the most consequential and meaningful features
while reducing dimensionality as well. As a result, the proposed method
outperforms the state-of-the-art methods on all three benchmark datasets, as
can be seen from Tables VI, VII and VIII. While the performance of the
proposed MGAF is very close to our recently proposed frameworks in [18], as we
show in Section IV-G, MGAF achieves state-of-the-art performance with a
significantly reduced training and inference time due to the gating mechanism
and dimensionality reduction.

###  IV-F Qualitative Analysis of Results

For qualitative analysis, we construct feature visualizations for Berkeley
MHAD by reducing high dimensional features to 2 dimensional using t-SNE [68]
as shown in Fig. 7. In Fig. 7(a), features are shown without fusion and we
observe that the classes seem to be inseparable. Fig. 7(b), (c) and (d) are
feature visualizations after concatenation, average fusion and gated average
fusion respectively. In concatenation, we inspect that between-class distance
is still small and classes make clusters at more than one places and hence
lead to misclassification.

For Average fusion, we recognize that between-classs distance has increased
and the clusters are more separated. However, there are actions like sit and
stand, sit down and stand up, creating complications due to resemblance
between them and thus their between-class distance is still insufficient for
segregation. This problem is solved by our proposed GAF as shown in Fig. 7(d).
We can now distinguish all classes clearly and clusters are more separable
than before.

In Fig. 7, we can see that features are distinctly separated and these
distinct features improve the performance of a classifier and thus highest
recognition accuracy is achieved as compared to the previous works. This shows
the supremacy of the proposed method. Since MGAF helps in separating the
feature space in a more distinctive manner, we believe that the application of
the proposed framework may help improving performance in other application
areas as well.

###  IV-G Comparison of Computational Cost of Proposed Method with Previous
State of Art

Table V shows that the proposed fusion network is computationally efficient
than previous state of art in terms of training speed, inference speed and
number of training parameters. In [18], computational cost, especially the
training speed and number of training parameters, became high due to the
creation of extra modalities and use of deeper CNN. Please note that the
Tables II, III and IV show that by utilizing more than two modalities, the
propsed MGAF can achieve an even better accuracy. However, for a balanced
comparison, we only report the results from two modalities (SFI + SI) in
Tables V, VI, VII and VIII.

Training speed is expressed in minutes and is defined as the total time to
train the whole framework. Inference speed is expressed in microseconds
($\mu$s) and is defined as a time taken by classifier to classify single test
sample.

Hence Tables V, VI, VII and VIII show that the proposed method beats the
previous state-of-the-art in terms of recognition accuracies and computational
cost.

##  V Conclusion

In this paper, we proposed a novel fusion network called multistage gated
average fusion (MGAF) which is capable of extracting features from all layers
of CNN and fuse them using our proposed gated average fusion (GAF) network. At
the input of the proposed MGAF, we transform both input modalities into
images. Features are extracted from both input modalities by utilizing two
CNNs of the same architecture. Extracted features are fused effectively by the
GAF network which assigns gated values to input feature maps. Experimental
results on three publicly available multimodal HAR datasets show the
superiority of the proposed MGAF over recently proposed multimodal fusion
methods for depth-inertial HAR in terms of recognition accuracy and
computational cost. In our furture work we are planning to implement the
proposed method on other modalities than inertial and depth. The proposed
method has some limitations. If the input image size is too large or if the
CNN is deeper, extracting and fusing features from all layers of the CNN will
not be computationally feasible. In future, we will investigate how
appropriate layers to fuse can be selected automatically in an intelligent
manner.

## References

  * [1] S. Samanta and B. Chanda, “Space-time facet model for human activity classification,” _IEEE Transactions on Multimedia_ , vol. 16, no. 6, pp. 1525–1535, 2014.
  * [2] W. Chi, J. Wang, and M. Q.-H. Meng, “A gait recognition method for human following in service robots,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 48, no. 9, pp. 1429–1440, 2017.
  * [3] G. Zhu, C. Xu, Q. Huang, Y. Rui, S. Jiang, W. Gao, and H. Yao, “Event tactic analysis based on broadcast sports video,” _IEEE Transactions on Multimedia_ , vol. 11, no. 1, pp. 49–67, 2008.
  * [4] P. Corbishley and E. Rodriguez-Villegas, “Breathing detection: towards a miniaturized, wearable, battery-operated monitoring system,” _IEEE Transactions on Biomedical Engineering_ , vol. 55, no. 1, pp. 196–204, 2007.
  * [5] C. Dhiman and D. K. Vishwakarma, “A robust framework for abnormal human action recognition using $R$-Transform and Zernike moments in depth videos,” _IEEE Sensors Journal_ , vol. 19, no. 13, pp. 5195–5203, 2019.
  * [6] M. Ramanathan, W.-Y. Yau, and E. K. Teoh, “Human action recognition with video data: research and evaluation challenges,” _IEEE Transactions on Human-Machine Systems_ , vol. 44, no. 5, pp. 650–663, 2014.
  * [7] L. Wang, D. Q. Huynh, and P. Koniusz, “A comparative review of recent kinect-based action recognition algorithms,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 15–28, 2019.
  * [8] P. Wang, W. Li, P. Ogunbona, J. Wan, and S. Escalera, “Rgb-d-based human motion recognition with deep learning: A survey,” _Computer Vision and Image Understanding_ , vol. 171, pp. 118–139, 2018.
  * [9] J. K. Aggarwal and L. Xia, “Human activity recognition from 3d data: A review,” _Pattern Recognition Letters_ , vol. 48, pp. 70–80, 2014.
  * [10] A. Y. Yang, R. Jafari, S. S. Sastry, and R. Bajcsy, “Distributed recognition of human actions using wearable motion sensor networks,” _Journal of Ambient Intelligence and Smart Environments_ , vol. 1, no. 2, pp. 103–115, 2009\.
  * [11] M. Ehatisham-Ul-Haq, A. Javed, M. A. Azam, H. M. Malik, A. Irtaza, I. H. Lee, and M. T. Mahmood, “Robust human activity recognition using multimodal feature-level fusion,” _IEEE Access_ , vol. 7, pp. 60 736–60 751, 2019\.
  * [12] C. Chen, R. Jafari, and N. Kehtarnavaz, “A survey of depth and inertial sensor fusion for human action recognition,” _Multimedia Tools and Applications_ , vol. 76, no. 3, pp. 4405–4425, 2017.
  * [13] L. Bao and S. S. Intille, “Activity recognition from user-annotated acceleration data,” in _International conference on pervasive computing_. Springer, 2004, pp. 1–17.
  * [14] A. Krause, D. P. Siewiorek, A. Smailagic, and J. Farringdon, “Unsupervised, dynamic identification of physiological and activity context in wearable computing.” in _ISWC_ , vol. 3, 2003, p. 88.
  * [15] T. Plötz, N. Y. Hammerla, and P. L. Olivier, “Feature learning for activity recognition in ubiquitous computing,” in _Twenty-second international joint conference on artificial intelligence_ , 2011.
  * [16] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [17] Z. Ahmad, K. Illanko, N. Khan, and D. Androutsos, “Human action recognition using convolutional neural network and depth sensor data,” in _Proceedings of the 2019 International Conference on Information Technology and Computer Communications_ , 2019, pp. 1–5.
  * [18] Z. Ahmad and N. Khan, “Human action recognition using deep multilevel multimodal (m2) fusion of depth and inertial sensors,” _IEEE Sensors Journal_ , vol. 20, no. 3, pp. 1445–1455, 2019.
  * [19] D. Ramachandram and G. W. Taylor, “Deep multimodal learning: A survey on recent advances and trends,” _IEEE Signal Processing Magazine_ , vol. 34, no. 6, pp. 96–108, 2017.
  * [20] P. K. Atrey, M. A. Hossain, A. El Saddik, and M. S. Kankanhalli, “Multimodal fusion for multimedia analysis: a survey,” _Multimedia systems_ , vol. 16, no. 6, pp. 345–379, 2010.
  * [21] C. Chen, K. Liu, and N. Kehtarnavaz, “Real-time human action recognition based on depth motion maps,” _Journal of real-time image processing_ , vol. 12, no. 1, pp. 155–163, 2016.
  * [22] W. Li, Z. Zhang, and Z. Liu, “Action recognition based on a bag of 3d points,” in _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops_. IEEE, 2010, pp. 9–14.
  * [23] X. Yang, C. Zhang, and Y. Tian, “Recognizing actions using depth motion maps-based histograms of oriented gradients,” in _Proceedings of the 20th ACM international conference on Multimedia_. ACM, 2012, pp. 1057–1060.
  * [24] C. Chen, M. Liu, H. Liu, B. Zhang, J. Han, and N. Kehtarnavaz, “Multi-temporal depth motion maps-based local binary patterns for 3-d human action recognition,” _IEEE Access_ , vol. 5, pp. 22 590–22 604, 2017.
  * [25] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, “Real-time human pose recognition in parts from single depth images,” 2011.
  * [26] B. Liang and L. Zheng, “A survey on human action recognition using depth sensors,” in _2015 International conference on digital image computing: techniques and applications (DICTA)_. IEEE, 2015, pp. 1–8.
  * [27] Y. Chen and Y. Xue, “A deep learning approach to human activity recognition based on single accelerometer,” in _2015 IEEE International Conference on Systems, Man, and Cybernetics_. IEEE, 2015, pp. 1488–1492.
  * [28] H. Yan, Y. Zhang, Y. Wang, and K. Xu, “Wiact: A passive wifi-based human activity recognition system,” _IEEE Sensors Journal_ , vol. 20, no. 1, pp. 296–305, 2019.
  * [29] W. Jiang and Z. Yin, “Human activity recognition using wearable sensors by deep convolutional neural networks,” in _Proceedings of the 23rd ACM international conference on Multimedia_. Acm, 2015, pp. 1307–1310.
  * [30] N. Tufek, M. Yalcin, M. Altintas, F. Kalaoglu, Y. Li, and S. K. Bahadir, “Human action recognition using deep learning methods on limited sensory data,” _IEEE Sensors Journal_ , 2019.
  * [31] J. Wang, Y. Chen, S. Hao, X. Peng, and L. Hu, “Deep learning for sensor-based activity recognition: A survey,” _Pattern Recognition Letters_ , 2018.
  * [32] A. Avci, S. Bosch, M. Marin-Perianu, R. Marin-Perianu, and P. Havinga, “Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: A survey,” in _23th International conference on architecture of computing systems 2010_. VDE, 2010, pp. 1–10.
  * [33] K. Sakuma, A. Abrami, G. Blumrosen, S. Lukashov, R. Narayanan, J. W. Ligman, V. Caggiano, and S. J. Heisig, “Wearable nail deformation sensing for behavioral and biomechanical monitoring and human-computer interaction,” _Scientific reports_ , vol. 8, no. 1, pp. 1–11, 2018.
  * [34] G. Blumrosen, Y. Miron, N. Intrator, and M. Plotnik, “A real-time kinect signature-based patient home monitoring system,” _Sensors_ , vol. 16, no. 11, p. 1965, 2016.
  * [35] G. Blumrosen, B. Fishman, and Y. Yovel, “Noncontact wideband sonar for human activity detection and classification,” _IEEE Sensors Journal_ , vol. 14, no. 11, pp. 4043–4054, 2014.
  * [36] X. Li, Y. He, and X. Jing, “A survey of deep learning-based human activity recognition in radar,” _Remote Sensing_ , vol. 11, no. 9, p. 1068, 2019.
  * [37] T.-H. Tan, M. Gochoo, S.-C. Huang, Y.-H. Liu, S.-H. Liu, and Y.-F. Huang, “Multi-resident activity recognition in a smart home using rgb activity image and dcnn,” _IEEE Sensors Journal_ , vol. 18, no. 23, pp. 9718–9727, 2018.
  * [38] C. Chen, R. Jafari, and N. Kehtarnavaz, “Improving human action recognition using fusion of depth camera and inertial sensors,” _IEEE Transactions on Human-Machine Systems_ , vol. 45, no. 1, pp. 51–61, 2015.
  * [39] A. Manzi, A. Moschetti, R. Limosani, L. Fiorini, and F. Cavallo, “Enhancing activity recognition of self-localized robot through depth camera and wearable sensors,” _IEEE Sensors Journal_ , vol. 18, no. 22, pp. 9324–9331, 2018.
  * [40] C. Chen, R. Jafari, and N. Kehtarnavaz, “A real-time human action recognition system using depth and inertial sensor fusion,” _IEEE Sensors Journal_ , vol. 16, no. 3, pp. 773–781, 2016.
  * [41] K. Liu, C. Chen, R. Jafari, and N. Kehtarnavaz, “Fusion of inertial and depth sensor data for robust hand gesture recognition,” _IEEE Sensors Journal_ , vol. 14, no. 6, pp. 1898–1903, 2014.
  * [42] N. Dawar and N. Kehtarnavaz, “Real-time continuous detection and recognition of subject-specific smart tv gestures via fusion of depth and inertial sensing,” _IEEE Access_ , vol. 6, pp. 7019–7028, 2018.
  * [43] B. H. Hu, N. E. Krausz, and L. J. Hargrove, “A novel method for bilateral gait segmentation using a single thigh-mounted depth sensor and imu,” in _2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob)_. IEEE, 2018, pp. 807–812.
  * [44] Z. Ahmad and N. Khan, “Towards improved human action recognition using convolutional neural networks and multimodal fusion of depth and inertial sensor data,” in _2018 IEEE International Symposium on Multimedia (ISM)_. IEEE, 2018, pp. 223–230.
  * [45] N. Dawar, S. Ostadabbas, and N. Kehtarnavaz, “Data augmentation in deep learning-based fusion of depth and inertial sensing for action recognition,” _IEEE Sensors Letters_ , vol. 3, no. 1, pp. 1–4, 2019.
  * [46] E. A. Bernal, X. Yang, Q. Li, J. Kumar, S. Madhvanath, P. Ramesh, and R. Bala, “Deep temporal multimodal fusion for medical procedure monitoring using wearable sensors,” _IEEE Transactions on Multimedia_ , vol. 20, no. 1, pp. 107–118, 2018.
  * [47] N. Dawar and N. Kehtarnavaz, “A convolutional neural network-based sensor fusion system for monitoring transition movements in healthcare applications,” in _2018 IEEE 14th International Conference on Control and Automation (ICCA)_. IEEE, 2018, pp. 482–485.
  * [48] H. Wei, R. Jafari, and N. Kehtarnavaz, “Fusion of video and inertial sensing for deep learning–based human action recognition,” _Sensors_ , vol. 19, no. 17, p. 3680, 2019.
  * [49] N. Dawar and N. Kehtarnavaz, “Action detection and recognition in continuous action streams by deep learning-based sensing fusion,” _IEEE Sensors Journal_ , vol. 18, no. 23, pp. 9660–9668, 2018.
  * [50] H. Wei and N. Kehtarnavaz, “Simultaneous utilization of inertial and video sensing for action detection and recognition in continuous action streams,” _IEEE Sensors Journal_ , 2020.
  * [51] H. Wei, P. Chopada, and N. Kehtarnavaz, “C-mhad: Continuous multimodal human action dataset of simultaneous video and inertial sensing,” _Sensors_ , vol. 20, no. 10, p. 2905, 2020.
  * [52] W. Ren, L. Ma, J. Zhang, J. Pan, X. Cao, W. Liu, and M.-H. Yang, “Gated fusion network for single image dehazing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3253–3261.
  * [53] H. Wu, H. Zhang, X. Zhang, W. Sun, B. Zheng, and Y. Jiang, “Deepdualmapper: A gated fusion network for automatic map extraction using aerial images and trajectories,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 01, 2020, pp. 1037–1045.
  * [54] J. Kim, J. Koh, Y. Kim, J. Choi, Y. Hwang, and J. W. Choi, “Robust deep multi-modal learning based on gated information fusion network,” in _Asian Conference on Computer Vision_. Springer, 2018, pp. 90–106.
  * [55] N. Manshor, A. A. Halin, M. Rajeswari, and D. Ramachandram, “Feature selection via dimensionality reduction for object class recognition,” in _2011 2nd International Conference on Instrumentation, Communications, Information Technology, and Biomedical Engineering_. IEEE, 2011, pp. 223–227.
  * [56] A. F. Bobick and J. W. Davis, “The recognition of human movement using temporal templates,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 23, no. 3, pp. 257–267, 2001.
  * [57] M. Alirezanejad, V. Saffari, S. Amirgholipour, and A. M. Sharifi, “Effect of locations of using high boost filtering on the watermark recovery in spatial domain watermarking,” _Indian Journal of Science and Technology_ , vol. 7, no. 4, p. 517, 2014.
  * [58] C. Chen, R. Jafari, and N. Kehtarnavaz, “Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor,” in _2015 IEEE International conference on image processing (ICIP)_. IEEE, 2015, pp. 168–172.
  * [59] F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy, “Berkeley mhad: A comprehensive multimodal human action database,” in _2013 IEEE Workshop on Applications of Computer Vision (WACV)_. IEEE, 2013, pp. 53–60.
  * [60] C. Chen, R. Jafari, and N. Kehtarnavaz, “Fusion of depth, skeleton, and inertial data for human action recognition,” in _2016 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2016, pp. 2712–2716.
  * [61] Z. Ahmad and N. M. Khan, “Multidomain multimodal fusion for human action recognition using inertial sensors,” in _2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM)_. IEEE, 2019, pp. 429–434.
  * [62] M. F. Bulbul, Y. Jiang, and J. Ma, “Dmms-based multiple features fusion for human action recognition,” _International Journal of Multimedia Data Engineering and Management (IJMDEM)_ , vol. 6, no. 4, pp. 23–39, 2015.
  * [63] A. B. Mahjoub and M. Atri, “An efficient end-to-end deep learning architecture for activity classification,” _Analog Integrated Circuits and Signal Processing_ , pp. 1–10, 2018.
  * [64] A. Shafaei and J. J. Little, “Real-time human motion capture with multiple depth cameras,” in _2016 13th Conference on Computer and Robot Vision (CRV)_. IEEE, 2016, pp. 24–31.
  * [65] E. P. Ijjina and C. K. Mohan, “Human action recognition based on mocap information using convolution neural networks,” in _2014 13th International Conference on Machine Learning and Applications_. IEEE, 2014, pp. 159–164.
  * [66] E. Akbas and F. T. Y. Vural, “Automatic image annotation by ensemble of visual descriptors,” in _2007 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2007, pp. 1–8.
  * [67] Z. Cai, L. Wang, X. Peng, and Y. Qiao, “Multi-view super vector for action recognition,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2014, pp. 596–603.
  * [68] L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” _Journal of machine learning research_ , vol. 9, no. Nov, pp. 2579–2605, 2008.

|  Zeeshan Ahmad received B.Eng. degree in Electrical Engineering from NED
University of Engineering and Technology Karachi, Pakistan in 2001, M.Sc.
degree in Electrical Engineering from National University of Sciences and
Technology Pakistan in 2005 and MEng. degree in Electrical and Computer
Engineering from Ryerson University, Toronto, Canada in 2017. He is currently
pursuing Ph.D. degree with the Department of Electrical and Computer
Engineering, Ryerson University, Toronto, Canada. His research interests
include Machine learning, Computer vision, Multimodal fusion, signal and image
processing.
---|---
|  Naimul Khan is an assistant professor of Electrical and Computer
Engineering at Ryerson University, where he co-directs the Ryerson Multimedia
Research Laboratory (RML). His research focuses on creating user-centric
intelligent systems through the combination of novel machine learning and
human-computer interaction mechanisms. He is a recipient of the best paper
award at the IEEE International Symposium on Multimedia, the OCE TalentEdge
Postdoctoral Fellowship, and the Ontario Graduate Scholarship. He is a senior
member of IEEE and a member of ACM.
---|---