# Deep state-space modeling for explainable representation, analysis, and
generation of professional human poses

Brenda Elizabeth Olivas-Padilla and Sotiris Manitsaris Mines Paris, Université
PSL, Paris 7500, France. E-mail<EMAIL_ADDRESS>
###### Abstract

The analysis of human movements has been extensively studied due to its wide
variety of practical applications, such as human-robot interaction, human
learning applications, clinical diagnosis, and activity monitoring.
Nevertheless, the state-of-the-art still faces scientific challenges while
modeling human movements. Firstly, new models that account for the
stochasticity of human movement and the physical structure of the human body
are required to accurately predict the evolution of full-body motion
descriptors over time. Secondly, the explainability of existing deep learning
algorithms regarding their body posture predictions while generating human
movements still needs to be improved as they lack comprehensible
representations of human movement. This paper addresses these challenges by
introducing three novel approaches for creating explainable representations of
human movement. In this work, full-body movement is formulated as a state-
space model of a dynamic system whose parameters are estimated using deep
learning and statistical algorithms. The representations adhere to the
structure of the Gesture Operational Model (GOM), which describes movement
through its spatial and temporal assumptions. Two approaches correspond to
deep state-space models that apply nonlinear network parameterization to
provide interpretable posture predictions. The third method trains GOM
representations using one-shot training with Kalman Filters. This training
strategy enables users to model single movements and estimate their
mathematical representation using procedures that require less computational
power than deep learning algorithms. Ultimately, two applications of the
generated representations are presented. The first is for the accurate
generation of human movements, and the second is for body dexterity analysis
of professional movements, where dynamic associations between body joints and
meaningful motion descriptors are identified.

###### Index Terms:

Article submission, IEEE, IEEEtran, journal, LaTeX, paper, template,
typesetting.

##  I Introduction

Movement is an essential component of human life. Humans are continually
exchanging information and interacting with their surroundings through their
movements, which result from complex and highly coordinated mechanical
interaction between bones, muscles, ligaments, and joints within the
musculoskeletal system. Through the study of this interaction and its effects,
the structure, function, and motion of human bodies can be examined, and the
resulting knowledge can be then used to improve the quality of human life.

The automatic analysis of human movements based on motion capture (MoCap) data
as a research domain has increased in significance due to the emergence of
numerous applications in health, sports, motion-driven user interfaces, and
intelligent surveillance.

Computer-based movement analysis serves the same purpose as a trainer, or
another specialist who objectively examines motions. In order to do this,
motion data must first be segmented. Then the tracked motion data is mapped
into meaningful motion descriptions that the user can interpret. Depending on
the motion-related application, various techniques such as statistical models
or data-driven approaches, like machine learning or deep learning algorithms,
can be used to model the human motion data. However, it remains complex and
requires overcoming scientific challenges to design an accurate and versatile
automatic analysis tool to describe human motion dynamics based on MoCap data.
A first challenge would be achieving a high level of modeling accuracy. The
system should consider human motion’s stochasticity and body joints’
mediations to accurately predict how human motion descriptors will evolve over
time so that this information can be used proactively. Secondly, the models
should be able to describe its outputs and assist humans in understanding how
specific human motions are performed in order to properly employ this
knowledge in an application.

This paper addresses the previous challenges through the analytical modeling
of human motion dynamics. Accordingly, the following two hypotheses were
formulated. The first hypothesis is that human motion dynamics can be modeled
analytically by taking into account the interdependencies between joints as
well as the dependencies between their prior values. Secondly, the cooperation
of body joints and their contribution during the performance of a human
movement can be learned and represented through interpretable models. To
address these hypotheses, it was first investigated statistical and deep
learning approaches for parameterizing the human motion mathematical
representation. This investigation resulted in the creation of three novel
approaches to represent and simulate human movement.

In this article, Section II reviews relevant research in human motion
modeling. Section III introduces the three novel approaches for modeling human
movements through interpretable mathematical representations. Next, Section VI
presents and discusses the simulation performance of the three approaches with
each dataset used. Section VII details the application of the trained motion
representations for body dexterity analysis. Lastly, Section VIII provides
conclusions of this work.

##  II Related Works

###  II-A Biomechanical modeling

Biomechanical models have generally been used to simulate human movement and
the changes that occur as a result of internal and external action forces.
These models represent the human body as a set of articulated links in a
kinetic chain, with joint torques and forces calculated using anthropometric,
postural, and hand load data [1]. Inverse dynamics is applied to extract
quantitative information about the mechanics of the musculoskeletal system
during the performance of a motor task . In previous studies [2, 3, 4],
optical and inertial MoCap systems were utilized to measure the subject’s
posture as well as the position of an object. Then, along with the
measurements of ground reaction forces with force plates, a kinematic analysis
was done. The outcomes are afterward utilized by the inverse dynamic
algorithm.

Developing precise and noninvasive methods in biomechanical modeling remains
challenging. The need for new methods to analyze motions captured outside of
laboratories arises as laboratory recordings lack authenticity. They are not
done on the actual scenarios where the movements are performed, presumably
leading to inaccurate measurements.

###  II-B Stochastic modeling

Stochastic modeling has been used to learn the random behavior of human
motion. These models utilize the variance information contained in body motion
trajectories to predict and identify human intentions and actions. Among the
most successful methods for dealing with the temporal variations of human
movements are generative models, in which time series are reorganized by
sequential states. A common approach to describing human motion in this way is
by using state-space models (SSMs). Previous research applied SSMs based on
Kalman filters for representing kinematic models to forecast pedestrian
position trajectories [5, 6]. The Kalman filter (KF) was mostly used to track
the position of pedestrians based on their estimated velocity or acceleration.
Caramiaux et al. [7] introduced an adaptive SSM based on particle filtering
that recognizes and continuously tracks the variations of gestures. Hidden
Markov Models (HMMs) have proven successful in modeling the temporal evolution
of gestures globally. Previous works [8, 9, 10] used HMMs to recognize
gestures associated with professional tasks performed in the crafts and
manufacturing industries. In another stochastic approach to model full-body
movements, Wang et al. [11] proposed the Intention-Driven Dynamics Model that
presupposes human dynamics change when actions are motivated by various
intentions. Devanne et al.[12] used a Dynamic Naive Bayes model to capture the
dynamics of motion primitives and continuously segment distinct human
behaviors in extended sequences.

Despite these encouraging advances in recent years, accurate human motion
modeling in unconstrained environments remains challenging. Unresolved
difficulties in the modeling concern spatiotemporal dynamics. For example,
even the same motion done by the same individual can have varying speeds and
starting/ending positions, let alone in scenarios involving multiple
performers. Hybrid biomechanical-stochastic models have been developed to
overcome these challenges.

###  II-C Hybrid biomechanical-stochastic modeling

Biomechanical-stochastic models encapsulate the kinematic correlations among
different skeletal joints besides capturing human motion stochastics.
Biomechanical-stochastic modeling has been effectively used to research human
movement variability and the prevention of a wide range of musculoskeletal
system injuries [13, 14]. Lin et al. [15] proposed a hybrid model to predict
the probability of injury and identify factors contributing to the risk of
non-contact anterior cruciate ligament injuries. In another work, Donnell et
al. [16] used a two-state Markov chain model to represent the survival of
surgical repair from a torn rotator cuff. To simulate human motions and
explain the interaction between joints for achieving the learned motion
trajectories, the Gesture Operational Model (GOM) presented by Manitsaris et
al. [10] has proven successful. Through state-space modeling, GOM’s
assumptions about joints’ dynamic associations are converted into a
simultaneous equation system. The equation system can then simulate human
motion and describe the joints’ interaction by examining the learned
parameters.

A limitation of hybrid models is that their processing requirements grow
exponentially as the number of model parameters increases. They are not
practical for analyzing large datasets or high-dimensional MoCap data as data-
driven approaches, requiring feature engineering. Moreover, to adequately
design the models, it is necessary to have prior knowledge of the training
data. Data-driven approaches do not require this prior knowledge since they
generate their own internal representations based on the training data.

###  II-D Data-driven approaches

The main advantage of data-driven approaches over other conventional
statistical approaches is their great modeling capacity and considerable
flexibility in designing architectures. The underlying similarity of motion
forecasting and sequence-to-sequence prediction tasks has led research in this
domain toward encoder-decoder architectures. Recent works have integrated
skeletal representations to include spatial correlations among joints for the
prediction of full-body human movements. Shu et al. [17] applied a co-
attention mechanism for motion prediction that learned body joint spatial
coherence and temporal evolution. Mao et al. [18] proposed three autoencoder
(AE) with attention, which processed motion on distinct levels: the whole
body, body parts, and single joints. Cai et al. [19] utilized the Discrete
Cosine Transform to transform the motion into the frequency domain. The
frequency components were then processed using a transformer-based
architecture (global attention mechanism) in order to capture spatio-temporal
correlations of the human pose.

Variational autoencoders (VAEs) have been applied for stochastic motion
prediction, where they predict multiple and diverse motion sequences in the
future from a single input sequence. Aliakbarian et al. [20] accomplished this
through conditional VAE, whereas Mao et al. [21] used a VAE to generate the
motion of various body parts in a sequential manner. In order to enhance
advances in probabilistic time series forecasting, Chung et al. [22] and
Fraccaro et al. [23] used Recurrent Neural Networks (RNNs) to build
connections between SSMs and VAEs. This led to deep SSMs, in which RNNs are
used to parameterize the non-linear observation and transition models. For
probabilistic forecasting, Salinas et al. [24] proposed the DeepAR, which uses
auto-regressive RNNs with mean and standard deviation as output. Liu et al.
[25] introduced a deep SSM based on Convolutional Neural Networks that
provided a unified formulation for multiple human motion systems and enabled
the accurate prediction of 3D human postures.

Even though there has been a lot of progress in modeling human motions
recently, with neural networks (NNs) that make impressive predictions and
simulations of human motions, as these approaches get more complicated, they
become harder to understand and interpret their results. They can learn highly
complex non-linear relationships from large datasets and surpass humans and
other methods at many tasks. Nevertheless, their obscurity restricts their
applicability and inspires little confidence among scientists and analysts
who, for example, undertake the prognosis of movement disorders. So, while
complex networks can handle activity recognition and event detection problems
that put predictive accuracy above interpretability, models that can be
intuitively interpreted, like analytical models, are better for applications
that help people learn and improve their skills in handicrafts, industry, or
sports, as well as for medical diagnostic and prognostic tools.

##  III Methodology

This paper hypothesizes that human motion dynamics can be modeled by
analytical models that can be interpreted and whose assumptions take into
account the stochasticity of human motion and physical body structure.
Consequently, given the nature of the hypotheses defined, GOM, a hybrid
stochastic-biomechanical approach based on kinematic descriptors, was selected
to model the dynamics of human movements and create interpretable
representations for human motion trajectories. Representing human motion as a
SSM of a dynamic system provides a simplified mathematical formalization of
the motion phenomenon and approximates it to reality, for instance, through
static or dynamic simulation. In addition, the mathematical representation of
GOM permits a more intuitive description of how body joints cooperate (spatial
dynamics) and evolve over time (temporal dynamics). GOM demonstrated to be
effective at simulating human joint position trajectories. Furthermore, due to
the usage of a transition function, it performs well with observations
obtained from varied environments and subjects without requiring extensive
training datasets [26]. This generalization capability is essential for
applications requiring rapid and accurate analysis of varied human movements.

This paper presents three novel approaches for modeling human movements using
full-body motion representations of GOM. The first method estimates the motion
parameters using statistical modeling, whereas the second and third methods
employ RNNs. Each approach trains time-varying motion representations that can
be utilized to simulate realistic human movements. The simulation performance
achieved with each approach is evaluated and compared with that attained with
the first GOM version proposed in [10], that uses constant motion
representations. The schematic of the methodology followed in this paper is
shown in Fig. 1.

Figure 1: Methodology for the creation and validation of the motion-based
methods to simulate and describe human movement.

##  IV Data collection

Diverse MoCap datasets have been made public for the analysis of human motion
[27, 28, 29, 30, 31]. However, most of these available datasets consist of
ordinary activities and sports motions recorded inside laboratories. For that
reason, seven datasets presented and further detailed in X were used in this
paper to analyze actual operators’ and artisans’ motions.

These datasets are composed of various professional movements captured in
real-world scenarios. The recorded professional movements can be divided into
three categories: industrial and crafts. Three datasets contain industrial
tasks related to television assembly (TVA), packaging (TVP), and airplane
assembly (APA), in total six operators were recorded . There is another
dataset composed of motions of different ergonomic risk levels performed by
ten subjects in a laboratory (ERGD). The last three correspond to motion data
from four craftsmen engaged in glassblowing (GLB), silk weaving (SLW), and
mastic cultivation (MSC).

For the creation of each dataset, the experts and subjects agreed to be
recorded in their actual workplace while wearing the BioMed bundle motion
capture system from Nansense Inc.111Baranger Studios, Los Angeles, CA, USA.
The system is composed of a full-body suit with 52 inertial sensors
strategically positioned across the torso, limbs, and hands. At a rate of 90
frames per second, the sensors measure the orientation and acceleration of
body segments on the articulated spine chain, shoulders, arms, legs, and
fingertips. After a recording, the Euler local joint angles on the X, Y, and Z
axes were automatically calculated through the Nansense Studio’s inverse
kinematics.

##  V Movement representation

The three methods proposed in this paper parametrize the GOM representation of
each motion of the seven datasets. The structure of this representation is
described next.

GOM models full-body movements using an equation system of autoregressive
models. Each model in this system learns the motion trajectory of one joint.
Suppose, human motion is depicted as a sequence of human postures
${P_{t}}=\left[{{P_{1}},{P_{2}},...,{P_{T}}}\right]\in\mathbb{R}^{T\times N}$.
$T$ is the length of the posture sequence and $N=J\times D$, where $J$ is the
number of joints measured, and $D$ is the number of dimensions that the
joint’s motion descriptor is decomposed. The number of models in the equation
system is equal to the number of dimensions associated with a given body joint
($D$), multiplied by the number of body joints ($J$) defined in GOM.

Inside these $N$ models are four different assumptions regarding the dynamic
relationship between body joints and their temporal dependencies:

  * H1

Transitioning: Velocity of the movement.

  * H2

Intra-joint association: Movement of the body joint across the 3D space.

  * H3

Inter-limb synergies: Cooperation and synchronisation between body limbs.

  * H4

Serial and non-serial intra-limb mediations: Cooperation between serially and
nonserially linked joints of the body.

Each assumption consists of a specific set of variables (in this case local
joint angles) that are parametrized and depict a particular relationship
between body joints or a temporal dependency. By examining the generated
coefficients and statistical significance of each variable, it can be gleaned
how relevant these are according to the movement modeled and the predicted
joint angles. Human postures are expressed as 3D Euler joint angles in order
to generate motions with subjects of various morphologies. Unlike joint
positions, Euler joint angles are unaffected by identity-specific body shape.
Moreover, Euler angles can be intuitively interpreted in the analytical model
and provide a more clear illustration of how human movements are conducted.

A diagram of GOM and its assumptions is provided in Fig. 2, along with the
joints measured for the modeling of the full-body movements. For the purposes
of this work, only measurements from 19 inertial sensors were used for the for
the modeling. Discarding MoCap data from the fingers and feet to simplify the
human motion representation. Thus, 57 joint angles were modeled in GOM. An
example of the estimated mathematical representations of $n$ assumptions for
one joint angle with time-varying coefficients $\alpha_{n,t}$ is illustrated
in (1).

$\begin{aligned}
Px_{1,t}=\underbrace{{\alpha_{1,t}}Px_{1,t-1}-{\alpha_{2,t}}Px_{1,t-2}}_{\text{H1}}+\underbrace{{\alpha_{3,t}}Py_{1,t-1}+{\alpha_{4,t}}Pz_{1,t-1}}_{\text{H2}}+\\\
\underbrace{{\alpha_{5,t}}Px_{2,t-1}+\cdots+{\alpha_{n,t}}Px_{j,t-1}}_{\text{H3\
or\ H4}}\end{aligned}$

(1)

The model represents the angle trajectory of the body joint $P_{1,t}$ at time
$t$ on the _X_ -axis ($Px_{1,t}$). The $P_{1,t}$ joint’s motion is decomposed
on _XYZ_ axes ($Px_{1,t}$, $Py_{1,t}$, and $Pz_{1,t}$) and has an association
with $j$ body parts. Furthermore, GOM’s models are second-order autoregressive
models due to the correlation between lag values (auto-correlation) in the
time series. Overall, the GOM structure used in this work comprises 57 models
as (1), each of which represents a joint motion trajectory.

The trained GOM can be utilized for two main applications. First, the joint
angle trajectories of each modeled motion can be simulated by solving the
simultaneous equation system that composes GOM. The second is that, as an
analytical model, the trained models can be used to obtain insights into the
dynamic relationship between body joints during the performance of a movement.
This dexterity analysis corresponds to describing the movement of each joint
based on the learned parameters from its mathematical representation.
Following, it is described the three different approaches to trained the GOM
representation. These are later evaluated in Section VII.

Figure 2: The Gesture Operational Model and its assumptions. The mathematical
representation of GOM is utilized to model the movements of every joint of the
MoCap skeleton. Then, the full-body movement is simulated and explained using
each joint’s motion model.

###  V-A Learning of constant and time-varying GOM representations using one-
shot learning

The first approach trains GOM representations using one-shot training with KF.
The fundamental concept is to formulate each joint angle model as a separate
SSM and then use KF to compute the log-likelihood of the observed joint angle
data for a given set of parameters.

Suppose that the joint angle on the X-axis of a body part, $Px_{1,t}$, is
modeled. Its GOM representation consists of the (1). The observation would be
the real joint angle, $y_{t}=Px_{1,t}$, where $Px_{1,t}\in\mathbb{R}^{1\times
1}$, and the variables from the assumptions H2, H3, and H4 correspond to the
exogenous input $x_{t}$. The following log-likelihood is then maximized with
respect to $\theta$ and $\alpha$, utilizing the KF to calculate the log-
likelihood for each time $t$:

$\ell\left(\theta,\alpha\right)=\sum_{t=1}^{T}\log{p_{\theta}\left(y_{1},\ldots
y_{t-1}|x_{1},\ldots x_{t-1}\right)}$ (2)

$\theta$ corresponds to the tuning parameters of the KF, whereas $\alpha$
constitutes the coefficients of the overall GOM representation,
$\alpha=[A,B]$. $A$ corresponds to the coefficients of assumption H1, and $B$
the coefficients for the rest of the assumptions. The diagram of the iterative
process of the Kalman filter for calculating the likelihood in (2) for every
time $t$ is illustrated in Figure 3. Note that $\hat{y}_{t}$ corresponds to
the joint angle prediction using the motion representation in (1) with the
estimated coefficients $\alpha$.

Figure 3: Flow chart of the iterative process of the KF while doing MLE for
estimating GOM’s coefficients.

The preceding approach is repeated for every joint angle model in GOM. This
results in 57 motion representations that can be used to simulate full-body
movements.

###  V-B Data-driven strategies for estimating time-varying GOM
representations

By taking advantage of the modeling power of NNs, both approaches train all
motion representations of GOM simultaneously. To this end, now the
observations are defined as $y_{t}=P_{t}\in\mathbb{R}^{1\times N}$, meaning
the $N$ joint angles at time $t$ that compose the whole body posture $P_{t}$,
and $x_{t}=[P_{t-1},P_{t-2}]\in\mathbb{R}^{2\times N}$.

The decoders from both frameworks have the full-body GOM mathematical
representations as the output layer. The decoder calculates the coefficients
$\alpha_{t}=\left[\alpha_{1}\cdots{\alpha}_{N}\right]$, which are then
utilized by the GOM equation system to produce the prediction $\hat{y}_{t}$.
Since GOM employs a second-order equation system, each element in $\alpha_{t}$
corresponds to a 2D tensor with the shape $(2,N)$. $N$ because all joint
angles are included in each GOM equation as the assumptions (H2, H3, and H4),
and two vectors as it also computed the coefficients of the transition
assumptions (H1). Thus, being $x_{t}$ and $\alpha_{t}$ tensors of shape
$(2,N)$ and $(N,2,N)$, respectively, the procedure for generating
$\hat{y}_{t}$ utilizing the GOM representations in the decoder is as follows:

$m_{t}=\alpha_{t}\circ\begin{bmatrix}1&0\\\ 0&-1\end{bmatrix}x_{t}$ (3)
$\hat{y}_{t}=\sum_{w=1}^{2}\sum_{k=1}^{N}m_{t,i,w,k}$ (4)

where $\circ$ is an element-wise product, $\hat{y}_{t}\in\mathbb{R}^{1\times
N}$, and $m_{t}\in\mathbb{R}^{N\times 2\times N}$. $m_{t}$ corresponds to GOM
in a matrix form, consisting of the 57 joint angle models as Equation 1.

VAE-RGOM and ATT-RGOM are composed of RNNs for the encoder and decoder,
representing human motions similarly to KF-GOM and KF-RGOM as a state-space
model. These were trained to maximize the following likelihood, approximating
$\hat{y}_{1:T}$ as close as possible to the system’s observed $y_{1:T}$:

$p_{\theta}\left(\hat{y}_{1:T}|x_{1:T}\right)=\int\underbrace{p_{\theta}(\hat{y}_{1:T}|z_{1:T},x_{1:T})}_{\text{Observation
model}}\underbrace{p_{\theta}(z_{1:T}|x_{1:T})}_{\text{Transition
model}}dz_{1:T}$ (5)

where $z_{1:T}$ represents the states of the system, and $x_{1:T}$ is the
input that, with $z_{1:T}$, generates the outputs $\hat{y}_{1:T}$. The
generative model in Equation 5 is composed of an observation model represented
as $p_{\theta}(\hat{y}_{1:T}|z_{1:T},x_{1:T})$ and a transition model
represented as $p_{\theta}(z_{1:T}|x_{1:T})$:

$p_{\theta}(\hat{y}_{1:T}|z_{1:T},x_{1:T})=\prod_{t=1}^{T}p_{\theta}(\hat{y}_{t}|z_{t},x_{t})$
(6)
$p_{\theta}(z_{1:T}|x_{1:T})=\prod_{t=1}^{T}p_{\theta}(z_{t}|z_{t-1},x_{t})$
(7)

The parameters $\theta$ from the observation and transition models are learned
during training by the decoder of each framework.

In each approach, the encoder has a different function, taking advantage of
their very specific encoder-decoder architecture. In VAE-RGOM, the encoder
functions as an inference network, whereas in ATT-RGOM, it initializes the
system’s state as a selected sequence of observed joint angles. However, for
both approaches, the decoder’s goal is to learn the state transition in order
to update the system’s state and produce the desired future posture
$\hat{y}_{t}$.

VAE-RGOM and ATT-RGOM seek to maximize the marginal log-likelihood below:

$Maximum\
\ell(\theta,\varphi)=\sum_{t=1}^{T}\log{p_{\theta}\left(\hat{y}_{1:T}|x_{1:T}\right)}$
(8)

In VAE-RGOM similar to a conventional VAE, the encoder estimates the
stochastic latent states $z$ by approximating $q_{\varphi}(z_{t}|x_{t},y_{t})$
to the true posterior distribution $p_{\theta}(z_{t}|z_{t-1},x_{t})$ defined
by a mean $\mu_{z_{t}}$ and a log covariance $\Sigma_{z_{t}}$. Stochastic
gradient optimization was used to train the networks. This entails first
sampling $z_{t}$, subsequently estimating the ELBO, then the gradients for
$\theta$, $\varphi$, and $\alpha$, and lastly, updating these parameters. The
loss of Equation 8 is thus equivalent to the maximum ELBO with respect to
$\theta$, $\varphi$, and $\alpha$ that results:

$\begin{aligned}
\ell\left(\theta,\varphi,\alpha\right)=\sum_{t=1}^{T}{\underbrace{-\beta_{\text{VAE}}\
KL(q_{\varphi}(z_{t}|z_{t-1},x_{t},y_{t})|\left|p_{\theta}\left(z_{t}|z_{t-1},x_{t}\right)\right)}_{\text{Regularization
loss}}}+\\\
\underbrace{\beta_{\text{GOM}}\mathbb{E}_{q_{\varphi}(z_{t}|z_{t-1},x_{t},y_{t})|}\left[\log{p_{\theta}(y_{t}|z_{t},x_{t})\mathrm{\
}}\right]}_{\text{Prediction loss}}\end{aligned}$

(9)

In Equation 9, KL denotes the Kullback-Leibler divergence that captures the
complexity of the data; the prediction loss measures the accuracy of the model
in the prediction; $\beta_{\text{VAE}}$ and $\beta_{\text{GOM}}$ correspond to
tuning hyperparameters. Instead of directly sampling from $q_{\varphi}$ at
each time step, $z_{t}=\mu_{z_{t}}+\epsilon\odot\Sigma_{z_{t}}$ is re-
parametrized using samples from a normal random variable
$\epsilon\sim\mathcal{N}(0,I)$. Consequently, the gradients relative to the
parameters $\theta$, $\varphi$, and $\alpha$ can be back-propagated through
the encoder via the sampled $z_{t}$. As the prediction loss, the mean absolute
difference of all joint angles is employed:

$\ell_{euler}=\frac{1}{J}\sum_{j=1}^{J}\frac{1}{D}\sum_{d=1}^{D}\left\|P_{t,j,d}-\hat{P}_{t,j,d}\right\|_{1}$
(10)

Figure 4: Overview of the Variational Autoencoder network for estimating GOM’s
coefficients.

ATT-RGOM parameterizes a state-space system by using its LSTM encoder to
initialize the system’s state $z_{t}$ as a context vector $c_{t}$ of the
observed joint angles. The context vector is determined by the sequence of
hidden states $h_{1:T}$ to which the encoder maps the input sequence $x_{1:T}$
of length $T$. The LSTM decoder models the state transition
$p_{\theta}(z_{t}|z_{t-1},c_{t})$ in order to update the system’s state and
generate the future posture $p_{\theta}(y_{t}|z_{t},x_{t})$.

The Luong attention mechanism is integrated in order to capture state
dynamics. The attention mechanism takes previous postures into account and
maps them to attention weights ($w_{t}$), computed using a dot product
alignment. The attention weights determine the degree to which previous hidden
states $h_{1:T}$ influence future state transitions. This influence is
indicated in the context vector, which is the weighted sum of the $h_{i}$:

$c_{t}=\sum_{i=1}^{T}w_{t,i}h_{i}$ (11)

In accordance with the model structure of AE with Luong attention mechanism,
the context vector is first used to compute the attentional hidden state
$\widetilde{s_{t}}$. The decoder then uses this state to generate the GOM’s
coefficients, followed by $\hat{y}_{t}$.

Figure 5: Overview of the Autoencoder with Luong Attention for estimating
GOM’s coefficients.

In order to tune the VAE-RGOM’s and ATT-RGOM’s hyperparameters, a Bayesian
optimization was carried out based on the loss achieved on a validation set
[32]. The best architecture of each network is described in tables II and II.

TABLE I: VAE-RGOM architecture.

Layer | Type |  | Output
---
shape
Activation | Dropout | Rdrop |  | Input
---
layer
1 | Input | (2,57) | - | - | - | -
2 | LSTM | (2,32) | Softsign | 0.2 | 0.2 | 1
3 ($\mu_{z_{t}}$) | FC | 2 | Linear | - | - | 2
4 ($\Sigma_{z_{t}}$) | FC | 2 | Linear | - | - | 2
5 ($z_{t}$) | Sampling | 2 | - | - | - | 3,4
6 | LSTM | (2,32) | Softsign | 0.2 | 0.2 | 5
7 | Dropout | (2,32) | - | 0.8 | - | 6
8 | TFC | (2,3249) | Linear | - | - | 7
9 ($\alpha_{t}$) | Reshape | (57,2,57) | - | - | - | 8
10 (GOM) | Lambda | (1,57) | - | - | - | 1,9

TABLE II: ATT-RGOM architecture.

Layer | Type | Output shape | Activation | Dropout | Rdrop |  | Input
---
layer
1 | Input | (2,57) | - | - | - | -
2 | LSTM | 2.1 Output state:(2,32) | Softsign | 0.2 | 0.2 | 1
2.2 Hidden state: 32
2.3 Cell state: 32
3 | BatchN | 32 | - | - | - | 2.2
4 | BatchN | 32 | - | - | - | 2.3
5 | LSTM | (2,32) | Softsign | 0.2 | 0.2 | 3, 4
6 | Dot | (2,2) | - | - | - | 2.1, 5
7 ($w_{t}$) | Softmax | (2,2) | - | - | - | 6
8 ($c_{t}$) | Dot | (2,32) | - | - | - | 2.1, 7
9 | Concatenate | (2,64) | - | - | - | 5, 8
10 | TFC | (2,3249) | Linear | - | - | 9
11 ($\alpha_{t}$) | Reshape | (57,2,57) | - | - | - | 10
12 (GOM) | Lambda | (1,57) | - | - | - | 1, 11

##  VI Static and dynamic simulation of professional movements

This section evaluates KF-RGOM, VAE-RGOM, and ATT-RGOM in terms of their
ability to simulate realistic human movements. Their performances are compared
with the first approach denoted as KF-RGOM proposed in [10]. In KF-GOM,
constant GOM representations are learned using one-shot training with KF. Each
approach predicted one time step per iteration. After predicting all the time
frames of a movement, the simulated movement was compared with the original
for evaluation. A reference motion was utilized for the training of the KF-
RGOM and KF-GOM. Regarding the training of VAE-RGOM and ATT-RGOM, these were
trained with all seven datasets using an NVIDIA GPU RTX 2060, and applying
80-10-10 sets (80% for training, 10% for validation, and 10% for testing). The
validation set was used to estimate the neural networks’ hyperparameters,
while the test set was utilized for evaluation.

TABLE III: Mean absolute angle errors for each dataset. Dataset | KF-GOM | KF-RGOM | VAE-RGOM | ATT-RGOM
---|---|---|---|---
TVA | 16.830 ($\sigma$: 15.379) | 6.938 ($\sigma$: 0.459) | 0.093 ($\sigma$: 0.016) | 0.191 ($\sigma$: 0.024)
TVP | 7.947 ($\sigma$: 5.278) | 9.867 ($\sigma$: 5.592) | 0.213 ($\sigma$: 0.058) | 0.398 ($\sigma$: 0.085)
APA | 3.312 ($\sigma$: 2.816) | 10.946 ($\sigma$: 0.664) | 0.091 ($\sigma$: 0.019) | 0.203 ($\sigma$: 0.048)
GLB | 19.211 ($\sigma$: 9.981) | 12.916 ($\sigma$: 3.665) | 0.119 ($\sigma$: 0.044) | 0.220 ($\sigma$: 0.066)
SLW | 14.267 ($\sigma$: 7.739) | 9.207 ($\sigma$: 3.307) | 0.115 ($\sigma$: 0.041) | 0.246 ($\sigma$: 0.078)
MSC | 23.313 ($\sigma$: 14.514) | 16.002 ($\sigma$: 5.887) | 0.247 ($\sigma$: 0.101) | 0.457 ($\sigma$: 0.126)
ERGD | 13.461 ($\sigma$: 8.699) | 13.569 ($\sigma$: 4.931) | 0.095 ($\sigma$: 0.052) | 0.198 ($\sigma$: 0.085)

To complement the analysis of the full-body simulations generated with the
deep SSMs, visual comparisons between the quality of the generated sequences
and the ground truth sequence are offered in figures 6 and 7. The red boxes
show variations in the generated motions at different temporal windows.

Figure 6: Visual comparison of generated posture sequences from GLB and its
ground-truth. The glassblower rotates the blowpipe with the left hand while
shaping the glass with the right. Figure 7: Visual comparison of generated
posture sequences from TVP and its ground-truth. The operator places a
television on the third level of a pallet.

The experiments suggest that by solving the simultaneous equations that
compose the GOM, it is possible to accurately simulate diverse human motions
using Euler joint angles as motion descriptors. Overall, time-varying GOM
representations are tolerant of slight variations in human motions and offsets
between motions of the same class produced by varying recording conditions
(different subjects or different recording days). Implementing time-varying
coefficients increased the modeling performance of GOM, especially for motions
with greater variance and longer duration, such as those conducted during
glassblowing. This is due to the fact that coefficients were adapted to the
change in mediations between the dependent variables and their assumptions
throughout the whole time series.

Across all seven datasets, the time-varying parameter models estimated by
data-driven approaches performed the best. Two arguments were deduced as to
why this improvement in performance. First, both deep SSMs used motion data
from all seven datasets for training, which would have allowed them to map
diverse relationships between assumptions and dependent variables and
accurately estimate the optimal coefficients for one-step prediction. The
variability between the training motion and test motions may account for the
errors exhibited by both KF-GOM and KF-RGOM. Accordingly, it can be inferred
that the quality of their simulations depends on the recorded person’s ability
to replicate their movements while repeatedly performing the same activity.

Secondly, the temporal encoder-decoder structure of VAE-RGOM and ATT-RGOM
enables these models to learn a low-dimensional (latent space) manifold of the
data. Ideally, this manifold untangles variation factors across distinct
movements, clusters related motion descriptors, and aids in identifying joint
dynamics across sequences. In the case of VAE-RGOM, it disentangles the
dynamics and postures in terms of the ELBO. For ATT-RGOM, this latent space
allows the attention mechanism to interpret the hidden mechanisms and
connections underlying the motion descriptors sequences.

According to the presented metrics, VAE-RGOM gave the most accurate movement
simulations. VAE-RGOM may outperform ATT-RGOM since it models a probability
distribution over future postures rather than making point estimates.
Particularly, VAE-RGOM yields the highest simulation performance for motions
from the datasets ERGD and TVA. These datasets are the largest ones and
correspond to the simplest movements with low intraclass variability. In
these, the movements were performed in a more controlled setting. For
instance, in ERDG, the subjects performed diverse motions in a laboratory,
receiving constant instructions on how to perform them. In the case of TVA,
the operators were recorded in a production cell performing the same tasks
repeatedly for several hours with little variation in between repeats. In
addition, the movements in TVA primarily involved manipulating objects with
their hands. In contrast to the movements performed, for instance, by the
craftsmen and farmers, who had to employ their entire bodies to perform their
work properly.

The most challenging movements to replicate were those associated with mastic
cultivation. The reason behind this could be a bias in the training data, as
MSC was the smallest dataset and involved motions where the farmer most of the
time moved while kneeling. In the other six datasets, the subjects were mostly
standing while performing their tasks. This may have prevented the networks
from fully learning the dynamics of the legs when they are flexed. Because
when the farmer moved to reach the tree or objects, he usually repositioned
the legs while kneeling to improve balance.

##  VII Body dexterity analysis of expert professionals

###  VII-A Statistical analysis of human motion representations

The following are two examples of models with time-varying coefficients. These
are summarized in tables, displaying the various representations provided by
KF-RGOM, VAE-RGOM, and ATT-RGOM for the same movement. The equation is
presented first with their corresponding assumptions and time-varying
coefficients $\alpha_{i,t}$, where $i$ is the number of coefficients. Next,
the summary of their coefficients and p-values calculated for each time step
is presented. The mean and standard deviation of the coefficients are
provided, together with the range of p-values. The range indicates the highest
and lowest p-values that were calculated over all time steps. Lasly, the
posture sequence of the motion modeled in each equation is also given in a
figure, along with the color annotations.

The first time-varying model is for the motion $\text{TVA}_{1}$, which
consists of an operator grabbing a circuit board from a container. The joint
angle on the Y-axis of the right arm ($RAy$) is modeled and represented in
Equation 12. Tables VI, VI, and VI present the summaries of coefficients and
p-values estimated by each approach. Figure 8 illustrates the posture sequence
and highlights the assumptions shown in Equation 12.

$\displaystyle
RAy_{t}={\alpha_{1,t}}{\color[rgb]{0.9296875,0.48828125,0.19140625}\definecolor[named]{pgfstrokecolor}{rgb}{0.9296875,0.48828125,0.19140625}RAy_{t-1}}+{\alpha_{2,t}}{\color[rgb]{0.9296875,0.48828125,0.19140625}\definecolor[named]{pgfstrokecolor}{rgb}{0.9296875,0.48828125,0.19140625}RAy_{t-2}}$
(12)
$\displaystyle+{\alpha_{3,t}}{\color[rgb]{0.44140625,0.68359375,0.27734375}\definecolor[named]{pgfstrokecolor}{rgb}{0.44140625,0.68359375,0.27734375}RAx_{t-1}}+{\alpha_{4,t}}{\color[rgb]{0.44140625,0.68359375,0.27734375}\definecolor[named]{pgfstrokecolor}{rgb}{0.44140625,0.68359375,0.27734375}RAz_{t-1}}$
$\displaystyle+{\alpha_{5,t}}{\color[rgb]{0,0.69140625,0.94140625}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.69140625,0.94140625}LAy_{t-1}}+{\alpha_{6,t}}{\color[rgb]{0.4375,0.1875,0.62890625}\definecolor[named]{pgfstrokecolor}{rgb}{0.4375,0.1875,0.62890625}RSH1y_{t-1}}$
$\displaystyle+\cdots+{\alpha_{7,t}}{\color[rgb]{0.4375,0.1875,0.62890625}\definecolor[named]{pgfstrokecolor}{rgb}{0.4375,0.1875,0.62890625}RFAy_{t-1}}$

Figure 8: Illustration of the movement performed in $\text{TVA}_{1}$, where
the operator grabs from a container a circuit board. The color annotations are
based on the assumptions of Equation (12).

TABLE IV: KF-RGOM: $\text{TVA}_{1}$. Variable | Coefficients | P-values
---|---|---
$\mu_{\alpha}(\sigma_{\alpha})$ | [min, max]
$\alpha_{1,t}$ | 0.486 (0.014) | [0.001, 0.004]
$\alpha_{2,t}$ | -0.096 (0.009) | [0.078, 0.220]
$\alpha_{3,t}$ | -1.236 (0.007) | [0.001, 0.010]
$\alpha_{4,t}$ | -0.273 (0.002) | [0.001, 0.005]
$\alpha_{5,t}$ | -0.005 (0.03) | [0.001, 0.040]
$\alpha_{6,t}$ | -4.711 (0.010) | [0.001, 0.015]
$\alpha_{7,t}$ | 0.156 (0.020) | [0.026, 0.040]

TABLE V: VAE-RGOM: $\text{TVA}_{1}$. Variable | Coefficients | P-values
---|---|---
$\mu_{\alpha}(\sigma_{\alpha})$ | [min, max]
$\alpha_{1,t}$ | 1.089 (0.012) | [0.001, 0.005]
$\alpha_{2,t}$ | 0.051 (0.006) | [0.001, 0.010]
$\alpha_{3,t}$ | -0.068 (0.003) | [0.001, 0.014]
$\alpha_{4,t}$ | 0.108 (0.002) | [0.001, 0.004]
$\alpha_{5,t}$ | -0.001 (0.001) | [0.001, 0.474]
$\alpha_{6,t}$ | -0.024 (0.002) | [0.001, 0.004]
$\alpha_{7,t}$ | -0.003 (0.003) | [0.001, 0.020]

TABLE VI: ATT-RGOM: $\text{TVA}_{1}$. Variable | Coefficients | P-values
---|---|---
$\mu_{\alpha}(\sigma_{\alpha})$ | [min, max]
$\alpha_{1,t}$ | 0.688 (0.012) | [0.001, 0.004]
$\alpha_{2,t}$ | 0.309 (0.030) | [0.001, 0.008]
$\alpha_{3,t}$ | -0.506 (0.012) | [0.001, 0.011]
$\alpha_{4,t}$ | -0.100 (0.006) | [0.001, 0.004]
$\alpha_{5,t}$ | -0.007 (0.015) | [0.157, 0.499]
$\alpha_{6,t}$ | 0.009 (0.012) | [0.003, 0.496]
$\alpha_{7,t}$ | -0.001 (0.003) | [0.028, 0.499]

According to each table, the time-varying coefficients show a time dependence,
being a low-speed motion; however, at certain periods of the time series, the
values two time steps prior to the prediction were not significant in the KF-
RGOM representation. All estimated representations exhibit an intra-joint
association with the X and Z axes of RA ($RAx$ and $RAz$). There is also an
inter-limb synergy, but according to the coefficients provided by VAE-RGOM and
ATT-RGOM, it was not significant for some periods of the time series. This
indicates that there was not always a relationship between the movements of
the right arm and the left arm, which could be the case if the operator was
simply holding the card with the right hand. Lastly, there is a serial intra-
limb mediation with RSH1 and RFA for KF-RGOM’s and VAE-RGOM’s representation.
For ATT-RGOM, though, the mediation with RSH1 and RFA was not present
throughout the entire time series. Specifically, RSH1 and RFA were not
significant on transitions where the operator is walking toward the container
or just standing and holding the circuit board for a moment.

The second example is a time-varying model of $\text{GLB}_{4}$, which
corresponds to the motion of shaping the glass decanter curves with a wooden
block while turning the blowpipe with the right hand. The Equation 13
represents the left shoulder’s motion along the X-axis ($LSH2x_{t}$). Tables
IX, IX, and IX summarize each approach’s estimated coefficients and their
respective p-values. The posture sequence is illustrated in Figure 9.

$\displaystyle
LSH2x_{t}={\alpha_{8,t}}{\color[rgb]{0.9296875,0.48828125,0.19140625}\definecolor[named]{pgfstrokecolor}{rgb}{0.9296875,0.48828125,0.19140625}LSH2x_{t-1}}+{\alpha_{9,t}}{\color[rgb]{0.9296875,0.48828125,0.19140625}\definecolor[named]{pgfstrokecolor}{rgb}{0.9296875,0.48828125,0.19140625}LSH2x_{t-2}}$
(13)
$\displaystyle+{\alpha_{10,t}}{\color[rgb]{0.44140625,0.68359375,0.27734375}\definecolor[named]{pgfstrokecolor}{rgb}{0.44140625,0.68359375,0.27734375}LSH2y_{t-1}}+{\alpha_{11,t}}{\color[rgb]{0.44140625,0.68359375,0.27734375}\definecolor[named]{pgfstrokecolor}{rgb}{0.44140625,0.68359375,0.27734375}LSH2z_{t-1}}$
$\displaystyle+{\alpha_{12,t}}{\color[rgb]{0,0.69140625,0.94140625}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.69140625,0.94140625}RSH2x_{t-1}}+{\alpha_{13,t}}{\color[rgb]{0.4375,0.1875,0.62890625}\definecolor[named]{pgfstrokecolor}{rgb}{0.4375,0.1875,0.62890625}LAx_{t-1}}$
$\displaystyle+\cdots+{\alpha_{14,t}}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}LFAx_{t-1}}+{\alpha_{15,t}}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}SP3x_{t-1}}$

TABLE VII: KF-RGOM: $\text{GLB}_{5}$. Variable | Coefficients | P-values
---|---|---
$\mu_{\alpha}(\sigma_{\alpha})$ | [min, max]
$\alpha_{8,t}$ | 0.543 (0.001) | [0.001, 0.005]
$\alpha_{9,t}$ | 0.456 (0.009) | [0.001, 0.002]
$\alpha_{10,t}$ | 0.223 (0.017) | [0.001, 0.020]
$\alpha_{11,t}$ | 0.192 (0.026) | [0.001, 0.010]
$\alpha_{12,t}$ | -0.087 (0.037) | [0.018, 0.045]
$\alpha_{13,t}$ | 0.011 (0.002) | [0.001, 0.003]
$\alpha_{14,t}$ | -0.002 (0.001) | [0.053, 0.060]
$\alpha_{15,t}$ | 0.135 (0.001) | [0.045, 0.087]

TABLE VIII: VAE-RGOM: $\text{GLB}_{5}$. Variable | Coefficients | P-values
---|---|---
$\mu_{\alpha}(\sigma_{\alpha})$ | [min, max]
$\alpha_{8,t}$ | 0.483 (0.003) | [0.001, 0.007]
$\alpha_{9,t}$ | 0.343 (0.001) | [0.001, 0.014]
$\alpha_{10,t}$ | 0.060 (0.001) | [0.001, 0.033]
$\alpha_{11,t}$ | -0.094 (0.001) | [0.001, 0.002]
$\alpha_{12,t}$ | 0.016 (0.004) | [0.001, 0.009]
$\alpha_{13,t}$ | 0.066 (0.009) | [0.001, 0.005]
$\alpha_{14,t}$ | -0.042 (0.004) | [0.001, 0.011]
$\alpha_{15,t}$ | 0.006 (0.001) | [0.007, 0.015]

TABLE IX: ATT-RGOM: $\text{GLB}_{5}$. Variable | Coefficients | P-values
---|---|---
$\mu_{\alpha}(\sigma_{\alpha}$) | [min, max]
$\alpha_{8,t}$ | 0.444 (0.008) | [0.001, 0.006]
$\alpha_{9,t}$ | 0.348 (0.011) | [0.002, 0.018]
$\alpha_{10,t}$ | 0.022 (0.034) | [0.134, 0.161]
$\alpha_{11,t}$ | -0.083 (0.010) | [0.001, 0.005]
$\alpha_{12,t}$ | 0.014 (0.003) | [0.002, 0.023]
$\alpha_{13,t}$ | 0.048 (0.003) | [0.001, 0.002]
$\alpha_{14,t}$ | -0.013 (0.005) | [0.021, 0.144]
$\alpha_{15,t}$ | -0.024 (0.017) | [0.029, 0.043]

Figure 9: Illustration of the movement performed in $\text{GLB}_{4}$, where
the expert glassblower shapes the decanter curve with a block and
simultaneously rotates the blowpipe back and forward. The color annotations
are based on the assumptions of Equation (13).

The statistical analysis of each representation reveals a temporal dependency,
indicating that the motion is slow. According to KF-RGOM and VAE-RGOM, there
is an intra-joint association for all time series with $LSH2y$ and $LSH2z$,
except for ATT-RGOM with respect to $LSH2y$. The motion presents an inter-limb
synergy with $RSH2y$, indicated by KF-RGOM, VAE-RGOM, and ATT-RGOM, as it was
as well for KF-GOM. This suggests a collaboration between both arms. As
visualized in Figure 9, the glassblower manipulated the molten glass with one
arm while rotating it with the other. This action requires synchronization
between both arms, which is reflected in the representations of all four
approaches. Again, all methods considered a non-serial intra-limb mediation
with LFA. There is also a mediation with SP3 in all the representations.
Similarly to the representation from KF-GOM, the p-value of SP3 was near the
threshold in the representation of KF-RGOM and ATT-RGOM. Accordingly, the
movement of the upper part of the spine is crucial when performing this
particular movement in glassblowing. This may also be observed in Figure 9,
where the glassblower moves his arms while bending his torso back and forth.

The preceding examples showed how the trained analytical models can be
utilized to explain the physical dexterity of operators, craftsmen, and
laboratory subjects. The models emphasized the key motion descriptors
associated with and contributing to complex whole-body motion. This
information can later be utilized to test skill acquisition strategies. A
novice can learn to make precise motions by minimizing the variability of
their motion representations compared to those of professional artisans or
operators.

###  VII-B Selection of the most significant sensors to maximize recognition
accuracy

This section explains how the best motion descriptors for modeling and
recognizing a set of human movements from each dataset are determined
according to the trained GOM representations. After performing the statistical
analysis, the number of times a motion descriptor (assumption) is
statistically significant for all equations that comprise GOM is counted. Then
for the selection, different combinations of descriptors considered most
frequently significant were utilized for training in an all-shots approach.
Because a single inertial sensor gives three joint angles, all of the sensor’s
joint angles were used for recognition if at least one was among the joint
angles that were more often significant in all motions of a dataset.

For the recognition of human movements utilizing different sensor
combinations, HMMs were trained. In order to properly train the HMMs, a
gesture vocabulary containing the motions with the most iterations was
specified for each dataset. The total number of motion classes for TVA, APA,
and ERGD were four, three, and 28, respectively. The TVP, GLB, and MSC gesture
vocabularies contained only motions with at least seven repetitions.
Therefore, their respective gesture vocabularies included five, seven, and six
classes of motion. Regarding SLW, the gesture vocabulary consisted of only
three classes of silk weaving on a loom.

All gesture vocabularies were subjected to the selection process described
previously with each approach’s representations. Then, the sensor
configurations that achieved the best performance were compared to the
recognition performance obtained by utilizing all motion data from all
sensors. Additionally, the recognition performance using a minimal set of two
sensors was also computed for comparison. This minimal set consisted of two
hand-picked sensors that provided the Euler joint angles of the right forearm
(RFA) and hips (H). The sensor positioned on the right forearm was chosen
since the majority of individuals in all datasets were right-handed, and the
sensor placed on the hips because all spinal movement originates from the
hips. The purpose of these comparisons is to assess the method’s capability to
select the set of sensors that achieves superior recognition performance over
configurations that include all 52 inertial sensors or a manually picked set
of sensors.

The HMM ergodic and left-to-right topologies, along with a different number of
hidden states, were evaluated to determine the best HMM settings for the
gesture vocabulary defined in each dataset. Left-to-right HMM topology
produced the best results for all recognition problems. Concerning the number
of hidden states, it was defined for the HMMs of TVA and ERGD with seven
states, TVP with six states, APA, GLB, and MSW with eight states, and SLW with
three states. Table X illustrates the sensors selected for each dataset based
on the motion representations generated by each approach. Then, Fig. 10
displays the calculated F1-scores for each sensor set per dataset.

TABLE X: Selected sensors for each dataset. Motion | Dataset
---|---
representation | TVA | TVP | APA | GLB | MSC | SLW | ERGD
KF-GOM | LA, SP1, | RSH1, LFA, | RA, LSH1, LSH2, | LSH2, RFA | LSH1, SP3, | RSH1, LSH1, | LA, RSH1,
RUL | SP2 | SP3, SP2, | H,SP3 | LUL, LL | HE, LUL, | LFA, SP2,
|  | LUL, RUL |  |  | RL | RUL
KF-RGOM | LA, RFA, | LSH1, RFA, | RA, LSH1, | RSH1, RFA, | LSH1, SP3, | LA, RA, | RA, LSH1,
SP | SP2 | SP3, LUL | SP1, RUL | LL, LUL | SP, LUL, | RFA, SP1,
|  |  |  |  | RL | LL
VAE-RGOM | LA, SP, | LA, RFA, | LA, SP2, | LA, LSH2, | LA, LSH2, | LA, LSH1, | LA, LSH1,
LL | SP | LL | SP3, LL | SP3, LL | RFA, SP3, | RFA, SP3,
|  |  |  |  | LL | LL
ATT-RGOM | LA, LSH1, | LA, RFA, | LA, LSH1, | LSH1, LSH2, | LSH2, RFA, | RA, LSH1, | RA, LSH1,
H | SP1 | H | RFA, LUL | LUL, LL | RFA, SP1, | RFA, SP1,
|  |  |  |  | LL | LL

TVATVPAPAGLBMSCSLWERGD$0$$0.2$$0.4$$0.6$$0.8$$1$DatasetF1-scoreAll 52
sensorsTwo sensorsKF-GOMKF-RGOMVAE-RGOMATT-RGOM

Figure 10: F1-score of the recognition according to each selected set of
sensors.

The relevance of the sensors selected based on each approach for each dataset
was demonstrated by the superior or similar recognition performance attained
compared to using all sensor data. By observing Fig. 10, the best minimal set
for TVA and SLW was determined by using the representations estimated by ATT-
RGOM. These sensors achieved comparable results to using all sensors’ data,
having the ATT-RGOM set an F1-score of 0.949 for TVA, whereas all sensors set
had an F1-score of 0.966. Then, for SLW, the ATT-RGOM sensor set performed at
least 1% better than all other sensor sets.

VAE-RGOM representations provided the best sensor set for TVP, with an
F1-score of 0.966. When comparing the sensors from the VAE-RGOM and ATT-RGOM
sets, it is observed that the VAE-RGOM set included a sensor from the lower
spine (SP). This sensor enhanced the ability to distinguish between the
motions of placing a box on the first and second levels.

The recognition of motions from APA was better using the data provided by the
sensor sets of KF-RGOM and ATT-RGOM, although the ATT-RGOM sensor set
contained fewer sensors. The set selected using ATT-RGOM representations
attained an F1-score of 0.905, outperforming the set containing all sensor
data. The APA motions were the most challenging to recognize. This may be
because the motions in this vocabulary are more complex and prolonged. In
addition, there is a substantial intra-class variance due to the fact that
just one airplane structure was constructed for this dataset. There were no
repetitions in which the pneumatic hammer was positioned in the same location
more than once.

The glassblowing motions performed in GLB were better recognized using the
data from the sensor sets estimated using VAE-RGOM representations, reaching
an F1-score of 0.923. According to all GOM representations, the shoulders
contribute the most to the execution of glassblowing motions, which is why the
two-sensors configuration performed the worst. In addition, using the motion
data measured from the left calf, which VAE-RGOM included, the recognition
performance improved by at least 5% in the F1-score compared to all other
sets.

The sensors picked using VAE-RGOM representations were also the most effective
at discriminating motions from MSC and ERGD. These were the gesture
vocabularies with the greatest number of classes. Because of this, the
selected sensors for each recognition problem are similar. The VAE-RGOM sensor
set attained an F1-score of 0.913 for MSC. Then, for ERGD, the VAE-RGOM sensor
set achieved an F1-score of 0.926. The poor performance of the two-sensor
configuration for MSC and ERGD may have been caused by its inability to
differentiate between motions that differ only in the posture of the legs, as
the motion data from the hips was insufficient.

##  VIII Conclusion

In determining the optimal approach for modeling human movements, there is a
trade-off to be considered between the accuracy of the modeling and the
computing cost of the training procedure. VAE-RGOM and ATT-RGOM are able to
simulate human motions more accurately and can be scaled to provide a greater
variety of human motions. In addition, these approaches generate the
representations of all full-body motion descriptors simultaneously, unlike KF-
RGOM, which require modeling one motion descriptor at a time, meaning training
separately 57 models (one per descriptor) for simulating full-body movements.
Nevertheless, the training of VAE-RGOM and ATT-RGOM is data-intensive,
necessitating a large volume of data depending on the architecture of the
neural network and a significant amount of computing power. KF-RGOM, on the
other hand, is sufficiently accurate to generate specific human motions using
one-shot training. This training strategy enables users to specify the human
motion and motion descriptors to be analyzed. Then construct their
mathematical representation according to GOM using straightforward procedures
that demand less computational power than data-driven methods.

The results demonstrated the ability of the proposed approaches to explain how
a motion is conducted in accordance with GOM’s assumptions, and identified the
most significant motion descriptors. As potential future work, the proposed
analytical models can be integrated with neurophysiological techniques that,
for example, account for muscle activity and motor cortex activity. This
combination would allow for a more thorough approach that could provide a
neurophysiological roadmap of complex body dexterity. However, because of the
inherent complexity and the sheer amount of data it requires, such a complete
study that considers all these neurophysiological factors hasn’t been done to
this point. Lastly, it is neither feasible nor practicable to employ a full-
body MoCap suit in many human motion analysis applications. Determining the
minimal motion descriptors to measure allows for the adoption of less invasive
technologies, such as smartphones and smartwatches, that could also measure
these motion descriptors.

## Acknowledgments

The research leading to these results has received funding by the EU Horizon
2020 Research and Innovation Programme under Grant Agreement No. 820767,
CoLLaboratE project, and Grant No. 822336, Mingei project.

## References

  * [1] A. Muller, C. Pontonnier, X. Robert-Lachaine, G. Dumont, and A. Plamondon, “Motion-based prediction of external forces and moments and back loading during manual material handling tasks,” _Applied Ergonomics_ , vol. 82, no. August 2019, p. 102935, 2020. [Online]. Available: https://doi.org/10.1016/j.apergo.2019.102935
  * [2] G. S. Faber, C. C. Chang, I. Kingma, J. T. Dennerlein, and J. H. van Dieën, “Estimating 3D L5/S1 moments and ground reaction forces during trunk bending using a full-body ambulatory inertial motion capture system,” _Journal of Biomechanics_ , vol. 49, no. 6, pp. 904–912, 2016. [Online]. Available: http://dx.doi.org/10.1016/j.jbiomech.2015.11.042
  * [3] D. Menychtas, A. Glushkova, and S. Manitsaris, “Analyzing the kinematic and kinetic contributions of the human upper body’s joints for ergonomics assessment,” _Journal of Ambient Intelligence and Humanized Computing_ , pp. 1–23, 2020.
  * [4] F. G. Larsen, F. P. Svenningsen, M. S. Andersen, M. de Zee, and S. Skals, “Estimation of Spinal Loading During Manual Materials Handling Using Inertial Motion Capture,” _Annals of Biomedical Engineering_ , vol. 48, no. 2, pp. 805–821, 2020. [Online]. Available: http://link.springer.com/10.1007/s10439-019-02409-8
  * [5] A. Barth and U. Franke, “Where will the oncoming vehicle be the next second?” in _2008 IEEE Intelligent Vehicles Symposium_ , 2008, pp. 1068–1073.
  * [6] E. Binelli, A. Broggi, A. Fascioli, S. Ghidoni, P. Grisleri, T. Graf, and M. Meinecke, “A modular tracking system for far infrared pedestrian recognition,” in _IEEE Proceedings. Intelligent Vehicles Symposium, 2005._ , 2005, pp. 759–764.
  * [7] B. Caramiaux, N. Montecchio, A. Tanaka, and F. Bevilacqua, “Adaptive Gesture Recognition with Variation Estimation for Interactive Systems,” _ACM Transactions on Interactive Intelligent Systems_ , vol. 4, no. 4, pp. 1–34, 2015\. [Online]. Available: https://dl.acm.org/doi/10.1145/2643204
  * [8] A. Glushkova and S. Manitsaris, “Gesture recognition and sensorimotor learning-by-doing of motor skills in manual professions: A case study in the wheel-throwing art of pottery,” _Journal of Computer Assisted Learning_ , vol. 34, no. 1, pp. 20–31, 2018.
  * [9] A. Malaisé, P. Maurice, F. Colas, F. Charpillet, A. Malaisé, P. Maurice, F. Colas, F. Charpillet, S. I. A. Recog, A. Malais, P. Maurice, F. Colas, and S. Ivaldi, “Activity Recognition With Multiple Wearable Sensors for Industrial Applications,” in _ACHI 2018 - Eleventh International Conference on Advances in Computer-Human Interactions_ , 2018, pp. 1–7.
  * [10] S. Manitsaris, G. Senteri, D. Makrygiannis, and A. Glushkova, “Human movement representation on multivariate time series for recognition of professional gestures and forecasting their trajectories,” _Frontiers in Robotics and AI_ , vol. 7, pp. 1–20, 2020.
  * [11] Z. Wang, K. Mülling, M. P. Deisenroth, H. Ben Amor, D. Vogt, B. Schölkopf, and J. Peters, “Probabilistic movement modeling for intention inference in human-robot interaction,” _International Journal of Robotics Research_ , vol. 32, no. 7, pp. 841–858, 2013.
  * [12] M. Devanne, S. Berretti, P. Pala, H. Wannous, M. Daoudi, and A. Del Bimbo, “Motion segment decomposition of RGB-D sequences for human behavior understanding,” _Pattern Recognition_ , vol. 61, pp. 222–233, 2017. [Online]. Available: http://dx.doi.org/10.1016/j.patcog.2016.07.041
  * [13] P. L. Davidson, D. J. Chalmers†, and B. D. Wilson‡, “Stochastic-rheological Simulation of Free-fall Arm Impact in Children: Application to Playground Injuries,” _Computer Methods in Biomechanics and Biomedical Engineering_ , vol. 7, no. 2, pp. 63–71, apr 2004. [Online]. Available: http://www.tandfonline.com/doi/abs/10.1080/1025584042000206461
  * [14] J. E. Langenderfer, J. E. Carpenter, M. E. Johnson, K.-n. An, and R. E. Hughes, “A Probabilistic Model of Glenohumeral External Rotation Strength for Healthy Normals and Rotator Cuff Tear Cases,” _Annals of Biomedical Engineering_ , vol. 34, no. 3, pp. 465–476, mar 2006. [Online]. Available: http://link.springer.com/10.1007/s10439-005-9045-9
  * [15] C.-F. Lin, H. Liu, M. T. Gros, P. Weinhold, W. E. Garrett, and B. Yu, “Biomechanical risk factors of non-contact ACL injuries: A stochastic biomechanical modeling study,” _Journal of Sport and Health Science_ , vol. 1, no. 1, pp. 36–42, 2012. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S2095254612000129
  * [16] D. M. S. Donnell, J. L. Seidelman, C. L. Mendias, B. S. Miller, J. E. Carpenter, and R. E. Hughes, “A stochastic structural reliability model explains rotator cuff repair retears,” _International Biomechanics_ , vol. 1, no. 1, pp. 29–35, 2014. [Online]. Available: http://dx.doi.org/10.1080/23310472.2014.983166
  * [17] X. Shu, L. Zhang, G.-J. Qi, W. Liu, and J. Tang, “Spatiotemporal Co-Attention Recurrent Neural Networks for Human-Skeleton Motion Prediction,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 6, pp. 3300–3315, 2022. [Online]. Available: https://ieeexplore.ieee.org/document/9321130/
  * [18] W. Mao, M. Liu, M. Salzmann, and H. Li, “Multi-level Motion Attention for Human Motion Prediction,” _International Journal of Computer Vision_ , vol. 129, no. 9, pp. 2513–2535, sep 2021. [Online]. Available: https://link.springer.com/10.1007/s11263-021-01483-7
  * [19] Y. Cai, L. Huang, Y. Wang, T.-J. Cham, J. Cai, J. Yuan, J. Liu, X. Yang, Y. Zhu, X. Shen, D. Liu, J. Liu, and N. M. Thalmann, “Learning progressive joint propagation for human motion prediction,” in _Computer Vision – ECCV 2020_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 226–242.
  * [20] S. Aliakbarian, F. Saleh, L. Petersson, S. Gould, and M. Salzmann, “Contextually Plausible and Diverse 3D Human Motion Prediction,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE, 2021, pp. 11 313–11 322. [Online]. Available: http://arxiv.org/abs/1912.08521https://ieeexplore.ieee.org/document/9710194/
  * [21] W. Mao, M. Liu, and M. Salzmann, “Generating Smooth Pose Sequences for Diverse Human Motion Prediction,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE, oct 2021, pp. 13 289–13 298. [Online]. Available: https://ieeexplore.ieee.org/document/9711405/
  * [22] J. Chung, K. Kastner, L. Dinh, K. Goel, A. Courville, and Y. Bengio, “A recurrent latent variable model for sequential data,” in _Proceedings of the 28th International Conference on Neural Information Processing Systems \- Volume 2_ , ser. NIPS’15. Cambridge, MA, USA: MIT Press, 2015, p. 2980–2988.
  * [23] M. Fraccaro, S. K. Sønderby, U. Paquet, and O. Winther, “Sequential Neural Models with Stochastic Layers,” _Advances in Neural Information Processing Systems_ , pp. 2207–2215, 2016. [Online]. Available: http://arxiv.org/abs/1605.07571
  * [24] D. Salinas, V. Flunkert, and J. Gasthaus, “DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks,” 2017. [Online]. Available: http://arxiv.org/abs/1704.04110
  * [25] X. Liu, J. Yin, H. Liu, and J. Liu, “DeepSSM: Deep State-Space Model for 3D Human Motion Prediction,” _arXiv e-prints_ , 2020. [Online]. Available: http://arxiv.org/abs/2005.12155
  * [26] B. E. Olivas-Padilla, S. Manitsaris, D. Menychtas, and A. Glushkova, “Stochastic-biomechanic modeling and recognition of human movement primitives, in industry, using wearables,” _Sensors_ , vol. 21, no. 7, 2021\.
  * [27] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, “Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 36, no. 7, pp. 1325–1339, 2014.
  * [28] Carnegie Mellon University, “CMU Graphics Lab Motion Capture Database.” [Online]. Available: http://mocap.cs.cmu.edu/
  * [29] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, “AMASS: Archive of motion capture as surface shapes,” in _International Conference on Computer Vision_ , 2019, pp. 5442–5451.
  * [30] L. Sigal, A. O. Balan, and M. J. Black, “HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion,” _International Journal of Computer Vision_ , vol. 87, no. 1-2, pp. 4–27, 2010.
  * [31] S. Ghorbani, K. Mahdaviani, A. Thaler, K. Kording, D. J. Cook, G. Blohm, and N. F. Troje, “MoVi: A large multi-purpose human motion and video dataset,” _PLoS ONE_ , vol. 16, no. 6 June, 2021. [Online]. Available: http://dx.doi.org/10.1371/journal.pone.0253157
  * [32] J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimization of machine learning algorithms,” in _Advances in Neural Information Processing Systems_ , F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds., vol. 25. Curran Associates, Inc., 2012. [Online]. Available: https://proceedings.neurips.cc/paper/2012/file/05311655a15b75fab86956663e1819cd-Paper.pdf