# Unified Contrastive Fusion Transformer for Multimodal Human Action
Recognition

Kyoung Ok Yang, Junho Koh and Jun Won Choi∗ * Corresponding author: Jun Won
Choi.K. Yang is with Department of Artificial Intelligence, Hanyang
University, 222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea (email:
koyang@spa.hanyang.ac.kr)J. Koh is with Department of Electrical Engineering,
Hanyang University, 222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea
(email: jhkoh@spa.hanyang.ac.kr)J. W. Choi is with Department of Electrical
Engineering and Graduate School of Artificial Intelligence, Hanyang
University, 222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea (email:
<EMAIL_ADDRESS>
###### Abstract

Various types of sensors have been considered to develop human action
recognition (HAR) models. Robust HAR performance can be achieved by fusing
multimodal data acquired by different sensors. In this paper, we introduce a
new multimodal fusion architecture, referred to as Unified Contrastive Fusion
Transformer (UCFFormer) designed to integrate data with diverse distributions
to enhance HAR performance. Based on the embedding features extracted from
each modality, UCFFormer employs the Unified Transformer to capture the inter-
dependency among embeddings in both time and modality domains. We present the
Factorized Time-Modality Attention to perform self-attention efficiently for
the Unified Transformer. UCFFormer also incorporates contrastive learning to
reduce the discrepancy in feature distributions across various modalities,
thus generating semantically aligned features for information fusion.
Performance evaluation conducted on two popular datasets, UTD-MHAD and NTU
RGB+D, demonstrates that UCFFormer achieves state-of-the-art performance,
outperforming competing methods by considerable margins.

###### Index Terms:

Human Action Recognition, Sensor Fusion, Multimodal Fusion, Unified
Transformer, Factorized Attention, Contrastive Learning, UTD-MHAD, NTU RGB+D

##  I Introduction

Human Action Recognition (HAR) is a process that involves the automatic
identification and classification of human actions based on sensor data. HAR
has a wide range of applications, including healthcare monitoring [1], fitness
tracking [2], action analysis, gesture-based interfaces [3], and context-aware
systems [4]. The ability to automatically recognize and classify human
activities based on sensor measurements has the potential to enhance various
domains and improve user experiences.

Various sensors such as video camera, wearable devices [5], and environmental
sensors [6] can be utilized to acquire data for HAR. Based on the sensor
types, HAR methods can be broadly classified into two categories: those based
on visual data and those based on non-visual data. Visual data is acquired
from camera sensors, including video, depth, and infrared cameras. These
camera sensors capture high-resolution images that depict the movement and
posture of individuals. Deep learning models, especially convolutional neural
networks (CNNs), have been widely employed for HAR using visual data [7, 8, 9,
10]. On the other hand, non-visual data can be acquired from various sensors
including accelerometers, gyroscopes, magnetometers, and pressure sensors.
These sensors capture non-visual data related to the body’s physical
movements. Devices like smartwatches [11], fitness trackers [12], and
smartphones [13] integrate such sensors, allowing users to continually monitor
their activities. Various deep learning architectures for modeling sequential
data have been used to conduct HAR based on non-visual data [14, 15, 16].

The use of multimodal data is motivated by several factors. First, different
sensor modalities capture different aspects of human motion and provide
complementary information that enhances the accuracy of action recognition
[17]. Second, using multiple sensor modalities can offer redundancy, which can
improve the robustness of action recognition [18]. Third, multimodal data can
provide a more comprehensive representation of human action, improving
generalization and adaptability of HAR models [18, 19]. By capturing different
aspects of human motion across various sensor modalities, the models can be
trained to generalize across diverse scenarios, users, and environments.

However, utilizing multimodal data in HAR poses some challenges. One of the
main challenges is establishing a joint representation for multimodal data
that captures complex relational semantics. To maximize the benefits of sensor
fusion, HAR models should produce the complementary yet semantically well
aligned features from each modality. Another challenge is optimally combining
the data from different sensors and modalities. The varied forms of data are
typically noisy, intricate, and exhibit high variability. These challenges
necessitate an efficient sensor fusion algorithm capable of modeling their
relational structure and integrating data adaptively based on its quality.

To date, various multimodal fusion architectures have been proposed for HAR.
In [20], Meoncks et al. proposed adaptive feature processing for HAR using
RGBD and LiDAR, and in [21], Shahroudy et al. presented the shared specific
feature factorization network to combine multimodal signals. Hierarchical
Multimodal Self-Attention (HAMLET) [22] combined global and local features
extracted from RGB and IMU through Transformer. Multitask Learning-based
Guided Multimodal (MuMu) [23] integrated a multi-task learning strategy to a
multimodal fusion network. Vision-to-Sensor Knowledge Distillation (VSKD)
strategy [24] proposed the network for transferring the knowledge from video
modality to inertial modality. MAVEN [25] enhanced the performance of
multimodal fusion by using a memory-augmented recurrent network and aligning
representations using an attention mechanism.

In this study, we introduce a new unified multimodal fusion framework for HAR,
referred to as the Unified Contrastive Fusion Transformer (UCFFormer). This
framework effectively combines multimodal data of varying distributions,
including both visual and nonvisual data. The UCFFormer is different from the
existing fusion architectures in these two aspects. First, UCFFormer derives
joint representation of multimodal sensor data using a unified Transformer
architecture, which has been successfully used for vision-language modeling
lately [26]. It first extracts the embedding features of the same size from
each multimodal input and encodes them via Multimodal Self-attention capturing
pairwise interactions across both time and modality domains. We introduce an
efficient implementation of Multimodal Self-attention for the unified
Transformer. We devise a Factorized Time-Modality Self-attention to
independently encode the embeddings in both time and modality domains. Two
strategies are proposed for factorized attention: parallel and sequential
factorization. While parallel factorization conducts self-attention across
both time and modality domains simultaneously, sequential factorization
alternates between them in each Transformer layer.

Next, we propose a feature alignment strategy based on a contrastive learning
approach. By minimizing the cosine similarity metric defined between the
embedding vectors of different modalities, the proposed method can mitigate
the domain gap between the modalities and thereby boost the effectiveness of
multimodal feature fusion. As a result, the proposed Multimodal Contrastive
Alignment Network (MCANet) generates embedding features that are more coherent
and semantically aligned.

We evaluate the performance of UCFFormer on two widely used multimodal HAR
datasets, UTD-MHAD [27] and NTU RGB+D [28]. Our evaluation demonstrates that
UCFFormer outperforms existing HAR models by significant margins, recording a
new state-of-the-art performance. In particular, UCFFormer achieves remarkable
99.99% Top-1 accuracy on UTD-MHAD dataset.

The key contributions of this study are summarized as follows:

  * •

We present a novel multimodal fusion network called UCFFormer. Our approach
leverages a unified Transformer to enhance the embedding features extracted
from different sensor data modalities. This structure enables sensor fusion
whose design is agnostic to the type and number of multimodal inputs, without
the need for specific custom designs for different modality types.

  * •

We introduce the Factorized Time-Modality Self-attention for the efficient
encoding of embedding features. To this goal, we present two distinct
factorization strategies.

  * •

We further enhance the effects of sensor fusion by aligning the embedding
features semantically using contrastive learning. To the best of our
knowledge, we are the first to demonstrate the efficacy of contrastive
learning in multimodal HAR. Our contrastive learning-centric alignment
technique can be seamlessly integrated into any feature fusion module.

  * •

Our UCFFormer achieves the state-of-the-art performances on UTD-MHAD [27] and
NTU RGB+D [28] datasets.

##  II Related Works

###  II-A Multimodal Fusion Methods for HAR

Early studies in multimodal fusion presented a late fusion approach which
combined classification results obtained from each modality to obtain a final
prediction. In [29, 30], Dawar et al. encoded RGB images using CNN and
inertial sensor data using Long Short Term Memory (LSTM) and combined the
resulting classification outcomes using the weights derived from the decision
fusion scores. In [31], 1D CNN, 2D CNN, and Recurrent Neural Network (RNN)
were employed to predict action class based on Gyroscope data, RGB data, and
human joint pose data, respectively and the classification results were
combined. In [32], WiVi utilized a CNN backbone to represent WiFi signal and a
C3D backbone [33] to process RGB data. These were then integrated at the
decision level through an ensemble fusion model.

Several studies have explored achieving information fusion at the intermediate
feature level. In [34], a Keyless Attention method was presented to aggregate
the features extracted from multimodal data. HAMLET [22] employed Hierarchical
Multimodal Self-attention to obtain action-related spatio-temporal features.
MuMu [23] was trained to conduct multiple tasks, i.e., the target task of HAR
and an auxiliary task of human action grouping. The auxiliary task assists the
target task in extracting appropriate multimodal representations. VSKD [24]
utilized ResNet18 as a teacher network for video data and employed multi-scale
TRN [35] with BN-Inception as a student network for inertial data. Distance
and Angle-wise Semantic Knowledge (DASK) loss was proposed to account for the
modality differences between the vision and sensor domains. MAVEN [25]
employed the feature encoders to generate modality-specific spatial features
that were subsequently stored in memory banks. The memory banks were used to
capture long-term spatiotemporal feature relationships.

Our UCFFormer is different from aforementioned methods in that a joint
representation of multimodal data is found through the time-modality
factorized self-attention of the unified Transformer and using contrastive
learning framework.

###  II-B Contrastive Learning

Contrastive learning is a type of the self-supervised learning tasks that
provides a means of understanding the differences between representations.
This stems from the work of Bromley et al. who introduced the concept of a
Siamese Network, consisting of two identical networks that share weights for
metric learning [36]. Specifically, contrastive learning examines which pairs
of data points are similar and different to learn high-level data features
before performing classification or segmentation tasks.

Early studies used a contrastive learning framework to learn invariant
mappings for recognition using contrastive pair loss in discrimination models
[37, 38]. Inspired by triplet loss, recent studies applied feature extraction
methods that minimize the distance between representations of similar positive
pairs and maximize the distance between representations of different negative
pairs [39, 40, 41].

Recently, contrastive learning has widely been used for various image
classification tasks. Momentum Contrast (MoCo) [42, 43] used a momentum-
updated encoder to produce representations of negative samples, providing a
large and consistently maintained set of negatives for contrastive loss
calculations. SimCLR [44, 45] learned representations by maximizing the
similarity between different augmentations of the same data sample while
minimizing the similarity between different samples. Bootstrap Your Own Latent
(BYOL) [46] achieved self-supervised image representation learning without
using negative samples by creating two augmented views of the same image.

###  II-C Unified Transformer for Multimodal Task

Transformer architectures have achieved significant success in machine
learning tasks, including natural language processing [47, 48] and computer
vision [49]. However, they have mainly been limited to tasks within a single
domain or specific multimodal domains. To overcome this challenge, a Unified
Transformer model [50] has been proposed as a foundation model for multimodal
data. The Unified Transformer consists of transform encoders jointly encoding
multimodal inputs, and a transform decoder over the encoded input modalities,
followed by task-specific outputs applied to the decoder hidden states to make
final predictions for each task. The Unified Transformer handles multi-tasking
and multimodal in a single model with fewer parameters, moving toward general
intelligence.

The Unified Transformer has been used for a variety of tasks that involve
multimodal data. UFO [51] used a single transformer network for multimodal
data and implemented multi-task learning during vision-language pre-training.
The same transformer network was used as an image encoder, as a text encoder,
or as a fusion network in the different pre-training tasks. UniFormer [52]
unified 3D convolution and spatio-temporal self-attention to generate a
transformer embedding vector. UniFormer’s relation aggregator handles both
spatio-temporal redundancy and dependency by learning local and global token
correlations in shallow and deep layers, respectively. UniTranSeR [53]
embedded the multimodal features into a unified Transformer semantic space to
prompt inter-modal interactions, and then employed a Feature Alignment and
Intention Reasoning layer to perform cross-modal entity alignment.

##  III Unified Contrastive Fusion Transformer (UCF-Former)

Figure 1: Overall Architecture : The proposed UCFFormer first represents each
raw multimodal data in a shared feature space. FTMT encodes these embedding
features, capturing their dependencies within the time-modality domain using
the Unified Transformer. Subsequently, MCANet refines these embeddings by
aligning them across modalities through contrastive learning. These enhanced
embeddings are then aggregated, leading to the final classification results.

###  III-A Overview

Figure 1 depicts the overall structure of the proposed UCFFormer. The proposed
UCFFormer fuses the information from $N$ multimodal sensors. First, $N$
separate backbone networks are employed to extract sequential feature vectors
of length $T$ from each modality. The feature vectors are linearly projected
to produce the $NT$ embedding vectors of the same size. The projected
embedding vectors serve as basic semantic elements represented in time and
modality domain. Factorized Time-Modality Transformer (FTMT) encodes the
embedding vectors using a unified Transformer architecture. The unified
Transformer models both intra-modality and inter-modality interactions
simultaneously to produce the updated embedding vectors. To facilitate
effective interaction modeling, FTMT employs a factorized self-attention
mechanism that conducts the encoding process separately in temporal and
modality domains.

Next, the Multimodal Contrastive Alignment Network (MCANet) combines the
features generated by FTMT. Among $N$ multimodal sensors, we designate one as
main modality sensor and the others as $N-1$ sub-modality sensors. MCANet
boosts the effect of feature fusion by aligning the sub-modality features with
those of the main modality through contrastive learning. Finally, the combined
features are passed through a multi-layer perceptron (MLP) layer followed by a
softmax layer to generate the final classification result.

###  III-B Setup and notations

We make the assumption that $N$ sensor data are temporally synchronized. To
achieve this, we resample the data such that their sampling rates are all
identical. Our model takes $T$ consecutive samples of all modality data in a
sequential manner. We denote $T$ signal samples acquired from $N$ modality
sensors as $x_{1}[1:T],x_{2}[1:T],...,x_{N}[1:T]$, where
$x_{n}[1:T]=\\{x_{n}[1],...,x_{n}[T]\\}$. The dimension of each sample is
different for each modality.

###  III-C Multimodal Feature Extractor

Different backbone networks are employed to extract feature vectors from each
modality data. Suppose that the feature vectors $f_{n}[1:T]$ are obtained from
the input data $x_{n}[1:T]$. Then, we apply linear projection to map the
feature embedding vectors into the embedding vectors of common size $d$, i.e.,

$\begin{split}Z_{1}[1:T]&=W_{1}\cdot f_{1}[1:T]\\\ &\vdots\\\
Z_{N}[1:T]&=W_{N}\cdot f_{N}[1:T].\end{split}$ (1)

These $NT$ embedding vectors form the basis for finding the joint
representation of multimodal data.

###  III-D Factorized Time-Modality Transformer Encoder

FTMT encoder encodes the embedding vectors capturing their high-level inter-
dependencies. To this goal, we utilize the unified Transformer, which has been
utilized for vision-language multimodal data modeling [26]. The unified
Transformer leverages Transformer Self-attention to encode the embedding
vectors across different modalities and time steps.

However, given $NT$ embedding vectors, FTMT requires the computational
complexity of $\mathcal{O}(N^{2}T^{2})$ and high capacity network for learning
all possible pairwise relations among $NT$ elements. To cope with this issue,
we employ the Factorized Time-Modality Self-attention, which conducts self-
attention across time and modality domains independently. We present two
distinct versions of factorized self-attention, which are differentiated by
their respective arrangements of time-domain and modality-domain self-
attention operations.

####  III-D1 Module 1: Simultaneous Time-Modality Factorization

The Simultaneous Time-Modality Factorization (FTMT-Sim) approach conducts
self-attention operations concurrently in the time and modality domains and
merges the encoded features in the final attention layer. Figure 2 (a) depicts
the structure of FTMT-Sim. For each time step $t$, the embedding vectors
across different modalities are packed into a matrix
$Z[t]=[Z_{1}[t],...,Z_{N}[t]]$. Similarly, the embedding vectors across time
steps are packed into a matrix $Z_{n}=[Z_{n}[1],...,Z_{n}[T]]$. Then, self-
attention is applied independently to each axis in parallel. First, self
attention across modalities is performed as

$\begin{split}Z^{(l+1)}[t]&=\mathrm{ModalityAttention}(Z^{(l)}[t])\\\
&=\mathrm{Softmax}\left(\frac{Q^{(l)}[t]((K^{(l)}[t])^{T}V^{(l)}[t])}{\sqrt{d_{k}}}\right),\end{split}$
(2)

where

$\begin{split}Q^{(l)}[t]&=Z^{(l)}[t]\cdot W^{(l)}_{q}\\\
K^{(l)}[t]&=Z^{(l)}[t]\cdot W^{(l)}_{k}\\\ V^{(l)}[t]&=Z^{(l)}[t]\cdot
W^{(l)}_{v},\end{split}$ (3)

$l\in[1,L]$ is the index of attention layer and
$W^{(l)}_{q}\in\mathbb{R}^{d\times d_{k}}$, $W^{(l)}_{k}\in\mathbb{R}^{d\times
d_{k}}$, and $W^{(l)}_{v}\in\mathbb{R}^{d\times d_{v}}$ are the linear
weights, where $d_{k}$ and $d_{v}$ represent the dimensions of the keys and
values, respectively. For brevity, we exclude the notation for multi-head
attention. In our implementation, both $d_{k}$ and $d_{v}$ are set to 64.

Next, the positioning encoding is applied to the embedding vectors in $Z_{n}$
[54] for each modality index $n$. Then, self-attention across time steps is
performed as

$\begin{split}Z^{(l+1)}_{n}&=\mathrm{TemporalAttention}(Z^{(l)}_{n})\\\
&=\mathrm{Softmax}\left(\frac{Q_{n}^{(l)}((K_{n}^{(l)})^{T}V_{n}^{(l)})}{\sqrt{d_{k}}}\right)\,\end{split}$
(4)

where

$\begin{split}Q_{n}^{(l)}&=Z^{(l)}_{n}\cdot V^{(l)}_{q}\\\
K_{n}^{(l)}&=Z^{(l)}_{n}\cdot V^{(l)}_{k}\\\ V_{n}^{(l)}&=Z^{(l)}_{n}\cdot
V^{(l)}_{v},\end{split}$ (5)

$W^{(l)}_{n,q}\in\mathbb{R}^{d\times d_{k}}$,
$W^{(l)}_{n,k}\in\mathbb{R}^{d\times d_{k}}$, and
$W^{(l)}_{n,v}\in\mathbb{R}^{d\times d_{v}}$ are the linear weights.

After $L$ attention layers, the attention values $Z^{(L)}[1],...,Z^{(L)}[T]$
and $Z^{(L)}_{1},...,Z^{(L)}_{N}$ are arranged, concatenated and linearly
projected, resulting in the generation of the final $NT$ features
$C_{1}[1:T],...,C_{N}[1:T]$. Note that the skip connection, originating from
the initial values $Z^{(0)}_{n}$ and $Z^{(0)}[t]$, is incorporated during the
process of generating the final features.

(a) Simultaneous Time-Modality Factorization (FTMT-Sim) (b) Sequential Time-
Modality Factorization (FTMT-Seq)

Figure 2: Factorized Time-Modality Self-attention : (a) FTMT-Sim processes the
embedding features in both time and modality domains concurrently, (b) FTMT-
Seq alternates the encoding process between the time-domain and the modality-
domain in a sequential manner.

####  III-D2 Module 2: Sequential Time-Modality Factorization (FTMT-Seq)

Unlike FTMT-Sim, the Sequential Time-Modality Factorization (FTMT-Seq)
approach applies the self-attention operations in time axis and modality axis
one by one. First, after positional encoding is performed, the time-domain
self-attention operation is performed as

$Y^{(l)}_{n}=\mathrm{TemporalAttention}(Z_{n}^{(l)})$ (6)

Then, the output $Y^{(l)}_{1},...,Y^{(l)}_{N}$ are rearranged to
$Y^{(l)}[1],...,Y^{(l)}[T]$. Then, the modality-domain self-attention follows

$Z^{(l+1)}[t]=\mathrm{ModalityAttention}(Y^{(l)}[t]).$ (7)

After passing through $L$ attention layers and skip connection from the input,
FTMT-Seq produces the final features $C_{1}[1:T],...,C_{N}[1:T]$.

###  III-E Multimodal Constrastive Alignment Network

Utilizing the output of FTMT, $C_{1}[1:T],...,C_{N}[1:T]$, MCANet employs
Weighted Multimodal Feature Fusion that combines the multi-modal features. In
our framework, we let $C_{1}[1:T]$ be the main modality features and the rest
be the sub-modality features. The primary goal of this approach is to
selectively aggregate the information that is pertinent to the main modality
data from the sub-modality data. For each time step $t$, Weighted Multimodal
Feature Fusion combines the multimodal features as

$\displaystyle C_{agg}[t]$
$\displaystyle=C_{1}[t]+\sum_{n=2}^{N}\mathrm{sigmoid}(\mathrm{sim}(C_{1}[t],C_{n}[t]))C_{n}[t],$
(8)

where $\mathrm{sim}(A,B)$ denotes the cosine similarity measure

$\displaystyle\begin{split}\mathrm{sim}(A,B)=\frac{A^{T}\cdot B}{\lVert
A\rVert_{2}\lVert B\rVert_{2}}.\end{split}$ (9)

Note that the sub-modality features $C_{n}[t]$ $n\neq 1$ are weighted
according to their similarity with the main modality features
$\mathrm{sigmoid}(\mathrm{sim}(C_{1}[t],C_{n}[t]))$. Additionally, we employ a
contrastive learning framework to reduce feature discrepancy across different
modalities. By minimizing a contrastive loss that captures the dis-
similarities between the different modalities during training, we can make our
weighted feature aggregation more effective. Details on the contrastive loss
will be discussed in the following subsection.

Finally, MCANet flattens the matrix $[{C}_{agg}[1],...,{C}_{agg}[T]]$ into a
single-dimensional vector. The flattened vector passes through the MLP
followed by softmax layer, generating the final classification scores
$O\in\mathbb{R}^{n_{cls}}$, where $n_{cls}$ is the number of action classes.
Each element in the vector $O$ represents the probability score assigned to a
specific action class. By analyzing these probabilities, the model can
determine the most likely class for a given input.

###  III-F Loss Function

As depicted in Figure 1, the loss function used to train our entire network
consists of the contrastive loss term $\mathcal{L}_{contrast}$ and the cross
entropy loss term $\mathcal{L}_{cls}$, i.e.,

$\mathcal{L}=\mathcal{L}_{cls}+\alpha\mathcal{L}_{contrast},$ (10)

where $\alpha$ is the regularization parameter. In our approach, we set
$\alpha$ to 0.2.

The contrastive loss term $\mathcal{L}_{contrast}$ aims to encourage the
alignment of features among different modalities. It quantifies the
dissimilarity or disparity between the main modality and the $N-1$ sub-
modalities, i.e.,

$\mathcal{L}_{contrast}=-\sum_{n=2}^{N}\log\left(\frac{1}{T}\sum_{t=1}^{T}\mathrm{sim}(C_{1}[t],C_{n}[t])\right)$
(11)

By using the contrastive loss term, the model can generate semantically
consistent representations.

The cross entropy loss term $\mathcal{L}_{cls}$ is given by

$\mathcal{L}_{cls}=-\sum_{i=1}^{n_{cls}}{T_{i}\log(O_{i})},$ (12)

where $O_{i}$ is the $i$th element of $O$ and $T_{i}$ is the $i$th element of
ground truth one hot vector.

##  IV Implementation of Multimodal HAR System

In this section, we present design variants of UCFFormer, specifically
tailored for the sensor configurations provided in two datasets: UTD-MHAD [27]
and NTU RGB+D [28].

###  IV-A Implementation of UCFFormer on UTD-MHAD dataset

UCFFormer is designed to fuse three different modalities including RGB video,
skeleton data and inertial sensor signals provided by UTD-MHAD dataset. RGB
video consists of length-$T$ sequence of video frames, i.e.,
$x_{1}[1:T]\in\mathbb{R}^{T\times H\times W}$, where $H$ and $W$ are the
height and width of a video frame. The skeleton data consists of 3D
coordinates of $J$ joints, i.e., $x_{2}[1:T]\in\mathbb{R}^{T\times J\times
3}$, where $J$ is the number of joints. The data from the inertial sensors is
divided into segments, with each segment being processed individually. The
inertial sensor data is also processed in a similarly way, yielding vectors of
the dimension $S$, $x_{3}[1:T]\in\mathbb{R}^{T\times S}$.

We tried two different camera backbones; the ResNet50 model and the Temporal
Shift Module (TSM) [55]. ResNet50 was used to to encode every frame of the RGB
video while TSM was used to encode sequential RGB video. We specifically
selected ResNet50 to ensure fair comparison with several latest methods [56,
57, 22, 23] that utilized ResNet50 on the UTD-MHAD dataset. We also tried a
stronger backbone, TSM at the cost of higher computational complexity. These
backbone networks take video frames $x_{1}[1:T]$ as input and produces the
sequence of feature maps $f_{1}[1:T]$. We encoded the skeleton data
$x_{2}[1:T]$ using a Spatio-Temporal Graph Convolutional Network (STGCN) [58].
Lastly, a DeepConvLSTM [59] was employed to encode the $T$ segments of
inertial sensor data $x_{3}[1:T]$ and produce the feature vectors
$f_{3}[1:T]$. The remaining steps follow the procedure described previously.
In contrastive learning setup, RGB video acted as a primary modality, while
the remaining data sources served as sub-modalities.

###  IV-B Implementation of UCFFormer on NTU RGB+D dataset

We consider two modalities: RGB images and skeleton data on NTU RGB+D dataset.
The data preprocessing step was performed similarly as in UTD-MHAD dataset. To
encode the RGB image sequence, we employ TSM as a video backbone network. The
samples of NTU RGB+D dataset capture both individual actions and interactions
among multiple individuals. From this perspective, we processed skeleton data
for each of $P$ persons and constructed the tensor,
$x_{2}[t]\in\mathbb{R}^{T\times J\times P\times 3}$. The skeleton data was
processed using the same backbone architecture used for the UTD-MHAD dataset.
In contrastive learning setup, RGB video was selected as a primary modality.

##  V Empirical Results

In this section, we evaluate the performance of the proposed UCFFormer on two
widely used datasets: UTD-MHAD [27] and NTU RGB+D [28].

###  V-A Dataset

UTD-MHAD: The UTD-MHAD dataset comprises 27 human actions performed by eight
subjects. It offers data in four distinct modalities: RGB video, depth video,
skeletal joint positions, and inertial sensor signals. For evaluating
performance, we utilized the Top-1 accuracy metric. In this dataset, we
adopted the cross-subject evaluation method widely used to evaluate the
performance of HAR models [27]. This approach divides subjects into different
groups for training versus testing to assess the model’s ability to generalize
to new, unseen subjects. Specifically, we used a dataset comprising 8 subjects
for this experiment. Among them, 6 subjects were utilized for training, 1
subject for validation, and 1 subject for testing.

NTU RGB+D: The NTU RGB+D dataset is an extensive collection of RGB video,
depth video and skeleton data, consisting of 56,880 action samples performed
by 40 subjects across 60 action classes. In this dataset, two evaluation
methods, namely cross-subject (CS) and cross-view (CV), have been widely
employed [28]. For the CS evaluation method, 40 samples were randomly selected
as training and testing groups, with the training IDs being 1, 2, 4, 5, 8, 9,
13, 14, 15, 16, 17, 18, 19, 25, 27, 28, 31, 34, 35, and 38. The remaining
subjects were reserved for testing, resulting in 40,320 for training and
16,560 samples for testing. On the other hand, for CV evaluation, samples from
cameras 2 and 3 were used for training, while samples from camera 1 were used
for testing. The resulting sets comprised 37,920 for training and 18,960
samples for testing.

###  V-B Experimental Setup

####  V-B1 Implementation Details

We present the implementation details of UCFFormer. The performance of
UCFFormer was evaluated across four different configurations. Based on the two
types of proposed factorized attention, UCFFormer is divided into UCFFormer-
Sim and UCFFormer-Seq. Additionally, when using the Image Backbone ResNet50,
it was denoted as R, and when using the Video Backbone TSM, it was denoted as
T. Therefore, the four tested types were denoted as UCFFormer-Sim-R,
UCFFormer-Seq-R, UCFFormer-Sim-T, and UCFFormer-Seq-T.

UTD-MHAD: In the case of UTD-MHAD, we employed ResNet50 [60] and TSM [55] as
RGB video backbone networks, while we used STGCN [58] for skeleton data and
DeepConvLSTM [59] for inertial data. For the RGB video data, ResNet50 was used
to extract features from each of eight video frames of
$\textrm{height}\times\textrm{width}\times\textrm{channel}=224\times 224\times
3$. TSM was used to extract features from the eight video frames of
$\mathrm{time}\times\mathrm{height}\times\mathrm{width}\times\mathrm{channel}=8\times
224\times 224\times 3$. The STGCN method extracted features from skeleton data
of $\textrm{joint}\times\textrm{spatial position}=20\times 3$ using a window
of size 15 over 8 time steps. For the inertial sensor data with
$\textrm{Accelerometer}\times\textrm{Rotation}=3\times 3$, DeepConvLSTM was
used with a window size of 100 over 8 time steps. The unified Transformer was
configured with parameters, including a dimension of 512 and 8 attention
heads. Each layer was repeated 4 times for both the FTMT-Sim and FTMT-Seq
modules. During training, we employed stochastic gradient descent (SGD) as the
optimizer for the entire network, utilizing a momentum of 0.9 and a learning
rate of 0.0005. The training process for the FTMT-Sim and FTMT-Seq modules
lasted for 16 hours and 300 epochs on the UTD-MHAD dataset. All training and
testing were conducted on a single NVIDIA Geforce GTX 1080Ti GPU, with a batch
size of 8.

NTU RGB+D: In the case of UTD-MHAD, we used TSM [55] to extract the features
from RGB video of size
$\mathrm{time}\times\mathrm{height}\times\mathrm{width}\times\mathrm{channel}=8\times
224\times 224\times 3$. We also used STGCN to extract the features from
skeleton data with dimension
$\mathrm{joint}\times\mathrm{spatial}\,\,\mathrm{position}\times\mathrm{person}=25\times
3\times 2$. We used a window of size 15 over 8 time steps. Both the FTMT-Sim
and FTMT-Seq modules were configured with a dimension of 512 and 16 attention
heads. The Transformer encoder layer was repeated 4 times. For optimization,
we employed stochastic gradient descent (SGD) with a momentum of 0.9 and a
learning rate of 0.003. The training duration was set to 480 hours and 70
epochs for the FTMT-Sim module, and 502 hours and 50 epochs for the FTMT-Seq
module. Training is conducted with two NVIDIA Geforce GTX 1080Ti GPUs, with a
batch size of 8.

TABLE I: Performance comparison on UTD-MHAD dataset: Top-1 Accuracy

Method Year Modality Accuracy (%) DCNN [56] 2016 RGB, Depth 91.20 MCRL [57]
2018 RGB, Depth 93.02 PoseMap [61] 2018 3D Skeleton, Heatmap 94.51 HDM [62]
2019 RGB, Skeleton 92.8 CC [63] 2019 RGB, Inertial 94.87 Gimme⋅ Signals [64]
2020 Skeleton, Inertial, Wifi 93.33 TPSMMs [65] 2020 Skeleton 93.26 LSTM-CNN
[66] 2020 Skeleton 93.2 HAMLET [22] 2020 RGB, Depth, Skeleton, Inertial 95.12
Fusion-GCN [67] 2022 RGB, Skeleton, Inertial 94.42 VSKD [24] 2022 RGB,
Inertial 96.97 MuMu [23] 2022 RGB, Depth, Skeleton, Inertial 97.60 MAVEN [25]
2022 RGB, Depth, Skeleton, Inertial 97.81 Multi-stream CNNs [68] 2023 RGB,
Depth, Skeleton, Inertial 96.95 UCFFormer-Sim-R(Ours) - RGB, Skeleton,
Inertial 99.04 UCFFormer-Sim-T(Ours) - RGB, Skeleton, Inertial 99.99
UCFFormer-Seq-R(Ours) - RGB, Skeleton, Inertial 99.03 UCFFormer-Seq-T(Ours) -
RGB, Skeleton, Inertial 99.60

TABLE II: Performance comparison on NTU RGB+D dataset: Cross-Subject (CS) and
Cross-View (CV)

Method Year Modality CS CV PoseMap [61] 2018 RGB, Skeleton 91.7 95.2 TSMF [69]
2020 RGB, Skeleton 92.5 97.4 HAC [70] 2020 RGB, Skeleton 95.66 98.79 VPN [71]
2020 RGB, Skeleton 95.5 98.0 FUSION [72] 2020 IR, Skeleton 91.8 94.9 InfoGCN
[73] 2021 Skeleton 93.0 97.1 PoseC3D [74] 2021 RGB, Skeleton 97.0 99.6 STAR-
Transformer [75] 2022 RGB, Skeleton 92.0 96.5 UCFFormer-Sim-T(Ours) - RGB,
Skeleton 97.1 99.3 UCFFormer-Seq-T(Ours) - RGB, Skeleton 95.0 98.0

(a) RGB (b) Inertial (c) Skeleton (d) RGB+Inertial

(e) RGB+Skeleton (f) RGB+Inertial+Skeleton

Figure 3: Confusion matrices for demonstrating the effect of multimodal
fusion. The evaluation is conducted using UCFFormer-Sim-R architecture on the
UTD-MHAD dataset.

###  V-C Performance Comparison

The performance of UCFFormer was evaluated in comparison with conventional HAR
methods. Table I presents the Top 1 accuracy results of the HAR models of
interest evaluated on the UTD-MHAD dataset. We note that all UCFFormer
variants achieve significant performance gains over the other HAR methods. In
particular, UCFFormer-Sim-T set a new state-of-the-art performance with an
impressive Top 1 accuracy of 99.99%. It outperforms the latest MAVEN model
[25] by a significant margin of 2.18%. These results show the effectiveness of
UCFFormer in handling the multimodal data. We also observe that both FTMT-Sim
and FTMT-Seq achieve comparable performance.

TABLE III: Performance impact of multimodal fusion evaluated on the UTD-MHAD
Dataset

Method Modalities Accuracy (%) RGB Inertial Skeleton Unimodal ✓ 87.54 ✓ 86.00
✓ 94.72 Multimodal ✓ ✓ 91.35 ✓ ✓ 96.15 ✓ ✓ ✓ 99.04

Table II presents the performance of UCFFormer evaluated on the NTU RGB+D
dataset [28]. UCFFormer exhibits a performance that is on par with the current
state-of-the-art method, PoseC3D [74]. Notably, UCFFormer offers substantial
performance gains over other HAR models, underscoring its competitive
performance in accurately recognizing human activities. It is also worth
mentioning that FTMT-Sim achieves slightly better performance than FTMT-Seq on
NTU RGB+D dataset.

###  V-D Performance Behavior

In this section, we present a comprehensive analysis of performance behavior
of our UCFFormer.

####  V-D1 Effect of Multimodal Fusion

Table III investigates performance gains achieved by multimodal fusion using
UCFFormer-Sim-R on the UTD-MHAD dataset. While the original UCFFormer combined
three modalities including RGB video, skeleton, and inertial data, Table III
provides the performance achieved when only a single modality is used or only
two modalities are combined. When a single modality was used, the embedding
vectors were encoded solely through Temporal Attention, and then the flattened
embedding vectors were used for classification. We observe that the
performance of UCFFormer considerably decreases when employing a single
modality or combining two modalities. When we use RGB, inertial, and skeleton
data individually, the skeleton data yields the highest performance 94.72%,
which is 4.32% lower than the accuracy of three modalities. When only two
modalities are combined, the highest achieved accuracy is 96.15%, exhibiting a
2.89% reduction compared to the accuracy of the original design. Figure 3
presents the confusion matrices obtained with different combinations of
modalities used for UCFFormer-Sim-R. It confirms that the inclusion of an
additional modality leads to substantial improvements in classification
accuracy.

(a) The t-SNE plot with contrastive learning

(b) The t-SNE plot without constrative learning

Figure 4: t-SNE visualization of multimodal feature vectors. We present the
t-SNE plots of the multimodal feature vectors extracted from the RGB video
(represented in red), skeleton (represented in blue), and inertial
(represented in green) modalities. We compare the feature distributions with
contrastive learning versus without contrastive learning.

(a) Performance versus # of Transformer layers (b) Performance versus # of
multi-attention heads (c) Performance versus embedding size

Figure 5: Performance of UCFFormer for different hyperparameter values.

TABLE IV: Ablation study on UTD-MHAD dataset (Top-1 Accuracy)

Settings Accuracy (%) FTMT-Sim FTMT-Fac MCANet 92.03 ✓ 97.87 ✓ 97.13 ✓ 96.15 ✓
✓ 99.04 ✓ ✓ 99.03

####  V-D2 Effect of Contrastive Learning

Figure 4 displays a t-SNE analysis that represents features in a reduced-
dimensional space. We compared the feature distributions when using
contrastive learning and when not using it. We observed that with the
application of contrastive learning, the distribution gap in features across
different modalities is notably reduced as intended.

####  V-D3 Performance versus Hyperparameters

In Figure 5, we evaluate the performance of the UCFFormer when applying
different hyperparameter values including the number of Transformer layers,
embedding size, and attention heads. The default values of the number of
layers, the embedding size, and the number of attention heads are set to 4,
512, and 8, respectively.

Figure 5 (a) presents the performance versus the number of Transformer layers
for both FTMT-Sim and FTMT-Seq. Our observations reveal that the peak
classification accuracy is attained with 4 layers, and increasing the depth
beyond this point adversely impacts performance. Figure 5 (b) provides the
performance versus the number of multi-heads. Performance improvement
diminishes as the number of heads is above 8. Figure 5 (c) provides the
performance of UCFFormer versus embedding size. The performance dramatically
improves as embedding size increases from 16 up to 64 but plateaus beyond 64,
reaching its peak at 512. For all cases, FTMT-Sim consistently outperforms
FTMT-Seq.

####  V-D4 Ablation Studies

Table IV presents an ablation study presenting the contributions made by each
component of UCFFormer to the overall performance. We employed a ResNet-50
backbone and evaluated the performance using the Top-1 accuracy metric on the
UTD-MHAD dataset. The baseline method neither includes FTMT nor MCANet; it
directly inputs the embedding vectors into the Flatten + MLP module. Table IV
demonstrates that the baseline attains an accuracy of merely 92.03%, which
falls short by 7% in comparison to UCFFormer-Sim-R. Upon integrating FTMT-Sim
with the baseline, there is a notable accuracy enhancement of 5.84%. The
inclusion of FTMT-Seq results in a performance improvement of 5.1%. This
demonstrates that the unified Transformer enhances the action semantics
through joint time-modality self-attention. Adding MCANet further improves the
accuracy by 1.17% for FTMT-Sim and by 1.9% for FTMT-Seq. Note that in the
absence of FTMT, the utilization of MCANet alone yields a notable 4.12%
improvement in accuracy compared to the baseline, which demonstrates the
efficacy of MCANet.

TABLE V: Comparison of computational complexity between factorized Self-
Attention versus full self-attention on the UTD-MHAD dataset

Module |  # of layers | Accuracy (%) | Whole Network | Transformer Encoder
---|---|---|---|---
Params(M) |  GFLOPs |  Params(M) |  MFLOPs
Factorized Self-Attention | 2 | 99.04 |  33.72 |  66.96 |  7.69 |  122.8
3 | 99.04 |  36.87 |  67.00 |  10.84 |  173.16
4 | 99.04 |  40.02 |  67.06 |  13.99 |  223.52
5 | 98.07 |  43.17 |  67.06 |  17.14 |  273.88
Self-Attention | 2 | 97.15 |  43.17 |  67.1 |  17.15 |  273.94
3 | 96.12 |  51.05 |  67.22 |  25.03 |  399.88
4 | 95.04 |  58.93 |  67.36 |  32.91 |  525.82
5 | 87.75 |  66.81 |  67.48 |  40.79 |  705.74
|  |  |  |  |  |

Table V compares the performance and computational complexity between the
full-dimenional self-attention versus the factorized self-attention of FTMT-
Sim. Table V highlights the computational advantage of that Factorized Self-
Attention over the conventional Self-Attention mechanism. For each added
layer, Factorized Self-Attention adds approximately 50 million floating-point
operations per second (MFLOPs), while the conventional Self-Attention sees an
increase of 130 MFLOPs. Table V also shows a substantial reduction in the
number of parameters by adopting the Factorized Self-Attention. In spite of
significant reduction in the memory usage and computation time, the factorized
self-attention achieves slightly better accuracy than the full self-attention.

##  VI Conclusions

In this paper, we presented the UCFFormer, an innovative approach to feature-
level fusion designed for HAR task. Employing two core components, FTMT and
MCANet, the UCFFormer established a solid framework for achieving effective
multimodal fusion. UCFFormer incrementally refines the embedding features
extracted from each modality, leveraging both FTMT and MCANet. FTMT captures
the high-level inter-dependencies of embedding features spanning both time and
modality domains. Utilizing the Factorized Time-Modality Self-attention
mechanism, FTMT offers an efficient architecture for encoding multimodal
features. MCANet then further refines these embedding features, employing
contrastive loss to mitigate potential domain discrepancies that might arise
among different modalities. The performance of UCFFormer was evaluated on two
widely used benchmarks. UCFFormer achieved the state-of-the-art performance,
surpassing the latest HAR methods. Ablation studies confirmed the
effectiveness of the ideas applied to UCFFormer. In conclusion, the UCFFormer
presents a robust and adaptable technique for merging various data types to
enhance HAR performance. Beyond HAR, this research holds promise for other
applications where a joint representation of diverse data types is essential
for task execution.

## References

  * [1] R. Liu, A. A. Ramli, H. Zhang, E. Henricson, and X. Liu, “An overview of human activity recognition using wearable sensors: Healthcare and artificial intelligence,” in _Internet of Things–ICIOT 2021: 6th International Conference, Held as Part of the Services Conference Federation, SCF 2021, Virtual Event, December 10–14, 2021, Proceedings_. Springer, 2022, pp. 1–14.
  * [2] N. S. Pathan, M. T. F. Talukdar, M. Quamruzzaman, and S. A. Fattah, “A machine learning based human activity recognition during physical exercise using wavelet packet transform of ppg and inertial sensors data,” in _2019 4th International Conference on Electrical Information and Communication Technology (EICT)_. IEEE, 2019, pp. 1–5.
  * [3] R. Saini and V. Maan, “Human activity and gesture recognition: A review,” in _2020 International Conference on Emerging Trends in Communication, Control and Computing (ICONC3)_. IEEE, 2020, pp. 1–2.
  * [4] L. Fan, P. Delir Haghighi, Y. Zhang, A. R. M. Forkan, and P. P. Jayaraman, “Context-aware human activity recognition (ca-har) using smartphone built-in sensors,” in _Advances in Mobile Computing and Multimedia Intelligence: 20th International Conference, MoMM 2022, Virtual Event, November 28–30, 2022, Proceedings_. Springer, 2022, pp. 108–121.
  * [5] P. Kumari, L. Mathew, and P. Syal, “Increasing trend of wearables and multimodal interface for human activity monitoring: A review,” _Biosensors and Bioelectronics_ , vol. 90, pp. 298–307, 2017.
  * [6] E. Cippitelli, E. Gambi, and S. Spinsante, “Human action recognition with rgb-d sensors,” _Motion Tracking and Gesture Recognition_ , vol. 97, 2017\.
  * [7] G. W. Taylor, R. Fergus, Y. LeCun, and C. Bregler, “Convolutional learning of spatio-temporal features,” in _European conference on computer vision_. Springer, 2010, pp. 140–153.
  * [8] Z. Qiu, T. Yao, and T. Mei, “Learning spatio-temporal representation with pseudo-3d residual networks,” in _proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 5533–5541.
  * [9] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6299–6308.
  * [10] C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 6202–6211.
  * [11] A. Diete, T. Sztyler, and H. Stuckenschmidt, “Vision and acceleration modalities: Partners for recognizing complex activities,” in _2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops)_. IEEE, 2019, pp. 101–106.
  * [12] C. Chen, K. Liu, R. Jafari, and N. Kehtarnavaz, “Home-based senior fitness test measurement system using collaborative inertial and depth sensors,” in _2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society_. IEEE, 2014, pp. 4135–4138.
  * [13] P. Khandnor, N. Kumar _et al._ , “A survey of activity recognition process using inertial sensors and smartphone sensors,” in _2017 International Conference on Computing, Communication and Automation (ICCCA)_. IEEE, 2017, pp. 607–612.
  * [14] Y. Zhao, R. Yang, G. Chevalier, X. Xu, and Z. Zhang, “Deep residual bidir-lstm for human activity recognition using wearable sensors,” _Mathematical Problems in Engineering_ , vol. 2018, pp. 1–13, 2018.
  * [15] M. Al-Naser, H. Ohashi, S. Ahmed, K. Nakamura, T. Akiyama, T. Sato, P. X. Nguyen, and A. Dengel, “Hierarchical model for zero-shot activity recognition using wearable sensors.” in _ICAART (2)_ , 2018, pp. 478–485.
  * [16] W. Gao, L. Zhang, Q. Teng, J. He, and H. Wu, “Danhar: Dual attention network for multimodal human activity recognition using wearable sensors,” _Applied Soft Computing_ , vol. 111, p. 107728, 2021.
  * [17] L. M. Dang, K. Min, H. Wang, M. J. Piran, C. H. Lee, and H. Moon, “Sensor-based and vision-based human activity recognition: A comprehensive survey,” _Pattern Recognition_ , vol. 108, p. 107561, 2020.
  * [18] D. Ravi, C. Wong, B. Lo, and G.-Z. Yang, “Deep learning for human activity recognition: A resource efficient implementation on low-power devices,” in _2016 IEEE 13th international conference on wearable and implantable body sensor networks (BSN)_. IEEE, 2016, pp. 71–76.
  * [19] P. K. Atrey, M. A. Hossain, A. El Saddik, and M. S. Kankanhalli, “Multimodal fusion for multimedia analysis: a survey,” _Multimedia systems_ , vol. 16, pp. 345–379, 2010.
  * [20] M. Moencks, V. De Silva, J. Roche, and A. Kondoz, “Adaptive feature processing for robust human activity recognition on a novel multi-modal dataset,” _arXiv preprint arXiv:1901.02858_ , 2019.
  * [21] A. Shahroudy, T.-T. Ng, Y. Gong, and G. Wang, “Deep multimodal feature analysis for action recognition in rgb+ d videos,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 5, pp. 1045–1058, 2017.
  * [22] M. M. Islam and T. Iqbal, “Hamlet: A hierarchical multimodal attention-based human activity recognition algorithm,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 10 285–10 292.
  * [23] M. M. Islam and T. Iqba, “Mumu: Cooperative multitask learning-based guided multimodal fusion,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 1, 2022, pp. 1043–1051.
  * [24] J. Ni, R. Sarbajna, Y. Liu, A. H. Ngu, and Y. Yan, “Cross-modal knowledge distillation for vision-to-sensor action recognition,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 4448–4452.
  * [25] M. M. Islam, M. S. Yasar, and T. Iqbal, “Maven: A memory augmented recurrent approach for multimodal fusion,” _IEEE Transactions on Multimedia_ , pp. 1–1, 2022.
  * [26] L. Zhou, H. Palangi, L. Zhang, H. Hu, J. Corso, and J. Gao, “Unified vision-language pre-training for image captioning and VQA,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 07, 2020, pp. 13 041–13 049.
  * [27] C. Chen, R. Jafari, and N. Kehtarnavaz, “Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor,” in _2015 IEEE International conference on image processing (ICIP)_. IEEE, 2015, pp. 168–172.
  * [28] A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang, “Ntu rgb+ d: A large scale dataset for 3d human activity analysis,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1010–1019.
  * [29] N. Dawar and N. Kehtarnavaz, “Action detection and recognition in continuous action streams by deep learning-based sensing fusion,” _IEEE Sensors Journal_ , vol. 18, no. 23, pp. 9660–9668, 2018.
  * [30] N. Dawar, S. Ostadabbas, and N. Kehtarnavaz, “Data augmentation in deep learning-based fusion of depth and inertial sensing for action recognition,” _IEEE Sensors Letters_ , vol. 3, no. 1, pp. 1–4, 2018.
  * [31] J. Imran and B. Raman, “Evaluating fusion of rgb-d and inertial sensors for multimodal human action recognition,” _Journal of Ambient Intelligence and Humanized Computing_ , vol. 11, no. 1, pp. 189–208, 2020.
  * [32] H. Zou, J. Yang, H. P. Das, H. Liu, Y. Zhou, and C. J. Spanos, “Wifi and vision multimodal learning for accurate and robust device-free human activity recognition,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, 2019, pp. 426–433.
  * [33] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 4489–4497.
  * [34] X. Long, C. Gan, G. Melo, X. Liu, Y. Li, F. Li, and S. Wen, “Multimodal keyless attention fusion for video classification,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [35] B. Zhou, A. Andonian, A. Oliva, and A. Torralba, “Temporal relational reasoning in videos,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 803–818.
  * [36] J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah, “Signature verification using a” siamese” time delay neural network,” _Advances in neural information processing systems_ , vol. 6, 1993.
  * [37] S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , vol. 1. IEEE, 2005, pp. 539–546.
  * [38] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , vol. 2. IEEE, 2006, pp. 1735–1742.
  * [39] K. Q. Weinberger and L. K. Saul, “Distance metric learning for large margin nearest neighbor classification.” _Journal of machine learning research_ , vol. 10, no. 2, 2009.
  * [40] R. Collobert and J. Weston, “A unified architecture for natural language processing: Deep neural networks with multitask learning,” in _Proceedings of the 25th international conference on Machine learning_ , 2008, pp. 160–167.
  * [41] G. Chechik, V. Sharma, U. Shalit, and S. Bengio, “Large scale online learning of image similarity through ranking.” _Journal of Machine Learning Research_ , vol. 11, no. 3, 2010.
  * [42] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 9729–9738.
  * [43] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” _arXiv preprint arXiv:2003.04297_ , 2020.
  * [44] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [45] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton, “Big self-supervised models are strong semi-supervised learners,” _Advances in neural information processing systems_ , vol. 33, pp. 22 243–22 255, 2020\.
  * [46] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar _et al._ , “Bootstrap your own latent-a new approach to self-supervised learning,” _Advances in neural information processing systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [47] J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of NAACL-HLT_ , 2019, pp. 4171–4186.
  * [48] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [49] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [50] R. Hu and A. Singh, “Unit: Multimodal multitask learning with a unified transformer,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1439–1449.
  * [51] J. Wang, X. Hu, Z. Gan, Z. Yang, X. Dai, Z. Liu, Y. Lu, and L. Wang, “Ufo: A unified transformer for vision-language representation learning,” _arXiv preprint arXiv:2111.10023_ , 2021.
  * [52] K. Li, Y. Wang, G. Peng, G. Song, Y. Liu, H. Li, and Y. Qiao, “Uniformer: Unified transformer for efficient spatial-temporal representation learning,” in _International Conference on Learning Representations_ , 2021.
  * [53] Z. Ma, J. Li, G. Li, and Y. Cheng, “Unitranser: A unified transformer semantic representation framework for multimodal task-oriented dialog system,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2022, pp. 103–114.
  * [54] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [55] J. Lin, C. Gan, and S. Han, “Tsm: Temporal shift module for efficient video understanding,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 7083–7093.
  * [56] J. Imran and P. Kumar, “Human action recognition using rgb-d sensor and deep convolutional neural networks,” in _2016 international conference on advances in computing, communications and informatics (ICACCI)_. IEEE, 2016, pp. 144–148.
  * [57] T. Liu, J. Kong, and M. Jiang, “Rgb-d action recognition using multimodal correlative representation learning model,” _IEEE Sensors Journal_ , vol. 19, no. 5, pp. 1862–1872, 2018.
  * [58] B. Yu, H. Yin, and Z. Zhu, “Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting,” in _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , 2018, pp. 3634–3640.
  * [59] S. P. Singh, M. K. Sharma, A. Lay-Ekuakille, D. Gangwar, and S. Gupta, “Deep convlstm with self-attention for human activity decoding using wearable sensors,” _IEEE Sensors Journal_ , vol. 21, no. 6, pp. 8575–8582, 2020.
  * [60] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [61] M. Liu and J. Yuan, “Recognizing human actions as the evolution of pose estimation maps,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1159–1168.
  * [62] R. Zhao, W. Xu, H. Su, and Q. Ji, “Bayesian hierarchical dynamic model for human action recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7733–7742.
  * [63] B. Peng, X. Jin, J. Liu, D. Li, Y. Wu, Y. Liu, S. Zhou, and Z. Zhang, “Correlation congruence for knowledge distillation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 5007–5016.
  * [64] R. Memmesheimer, N. Theisen, and D. Paulus, “Gimme signals: Discriminative signal encoding for multimodal activity recognition,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 10 394–10 401.
  * [65] Y. Chen, L. Wang, C. Li, Y. Hou, and W. Li, “Convnets-based action recognition from skeleton motion maps,” _Multimedia Tools and Applications_ , vol. 79, pp. 1707–1725, 2020.
  * [66] A. Zhu, Q. Wu, R. Cui, T. Wang, W. Hang, G. Hua, and H. Snoussi, “Exploring a rich spatial–temporal dependent relational model for skeleton-based action recognition by bidirectional lstm-cnn,” _Neurocomputing_ , vol. 414, pp. 90–100, 2020.
  * [67] M. Duhme, R. Memmesheimer, and D. Paulus, “Fusion-gcn: Multimodal action recognition using graph convolutional networks,” in _DAGM German Conference on Pattern Recognition_. Springer, 2021, pp. 265–281.
  * [68] F. Khezerlou, A. Baradarani, M. A. Balafar, and R. G. Maev, “Multi-stream cnns with orientation-magnitude response maps and weighted inception module for human action recognition,” in _2023 3rd International conference on Artificial Intelligence and Signal Processing (AISP)_. IEEE, 2023, pp. 1–5.
  * [69] X. Bruce, Y. Liu, and K. C. Chan, “Multimodal fusion via teacher-student network for indoor action recognition,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 4, 2021, pp. 3199–3207.
  * [70] M. Davoodikakhki and K. Yin, “Hierarchical action classification with network pruning,” in _Advances in Visual Computing: 15th International Symposium, ISVC 2020, San Diego, CA, USA, October 5–7, 2020, Proceedings, Part I 15_. Springer, 2020, pp. 291–305.
  * [71] S. Das, S. Sharma, R. Dai, F. Bremond, and M. Thonnat, “Vpn: Learning video-pose embedding for activities of daily living,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16_. Springer, 2020, pp. 72–90.
  * [72] A. M. De Boissiere and R. Noumeir, “Infrared and 3d skeleton feature fusion for rgb-d action recognition,” _IEEE Access_ , vol. 8, pp. 168 297–168 308, 2020.
  * [73] H.-g. Chi, M. H. Ha, S. Chi, S. W. Lee, Q. Huang, and K. Ramani, “Infogcn: Representation learning for human skeleton-based action recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 20 186–20 196.
  * [74] H. Duan, Y. Zhao, K. Chen, D. Lin, and B. Dai, “Revisiting skeleton-based action recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 2969–2978.
  * [75] D. Ahn, S. Kim, H. Hong, and B. C. Ko, “Star-transformer: A spatio-temporal cross attention transformer for human action recognition,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 3330–3339.