formed by taking single electron excitations from core orbitals and up to two
excitations of valence electrons from the leading configurations. Here we are
inspired by [104] who suggest using single core excitations to treat valence
and core-valence correlations more efficiently. We also expect that by
unfreezing the core, the calculations of short-range matrix elements could be
improved. The results of our calculations that include these additional core-
excited configurations are listed in Tab. 13 as “Core holes”. We see that
generally, the difference between finite field and operator approaches to the
matrix elements is smaller when using core holes, reflecting that the
relaxation of the core is partially included using this method. The exception
is the $\gamma$ transition for which we see a large difference: this
transition directly involves a change in the occupancy of the $4f$ valence
state, and the relaxation of this shell is large. Again we stress that our
core hole approach is used as a measure of the theoretical uncertainty
associated with having frozen cores in our fiducial CI method.

##  Appendix J Determining the evolution of $\delta\langle r^{4}\rangle$ along
the isotope chain from King plot data

Assuming that the leading source of nonlinearity in the King plot is
$\delta\langle r^{4}\rangle$, we can use isotope shift measurements to extract
information on this quantity. We proceed in two steps: Using the precise
nuclear mass and frequency measurements presented in this work, we construct
the quantities

$\displaystyle\hat{\nu}_{\tau}^{a,r}=$
$\displaystyle\nu_{\tau}^{a}-\frac{w^{a}}{w^{r}}\nu_{\tau}^{r}\,,$ (15)

where $\tau$ is the transition index, $r=(174,176)$ denotes the reference
isotope pair, and $a\in\\{(168,170),(170,172),(172,174)\\}$ is an isotope pair
index. Approximating Eq. (1) (with $\alpha\to\tau$) by

$\displaystyle\nu_{\tau}^{a}\,\approx$ $\displaystyle\,F_{\tau}\delta\langle
r^{2}\rangle^{a}+K_{\tau}w^{a}+G_{\tau}^{(4)}\delta\langle
r^{4}\rangle^{a}\,,$ (16)

we obtain

$\displaystyle\hat{\nu}_{\tau}^{a,r}\approx
F_{\tau}D^{a,r}+G_{\tau}^{(4)}Q^{a,r}\,,$ (17)

where the isotope-pair dependent quantity

$\displaystyle D^{a,r}=$ $\displaystyle\delta\langle
r^{2}\rangle^{a}-\frac{w^{a}}{w^{r}}\delta\langle r^{2}\rangle^{r}\,,$ (18)

can be constructed from the charge radius measurements tabulated in Ref. [32],
and

$\displaystyle Q^{a,r}=$ $\displaystyle\delta\langle
r^{4}\rangle^{a}-\frac{w^{a}}{w^{r}}\delta\langle r^{4}\rangle^{r}$ (19)

describes the evolution of the nuclear deformation parameter $\delta\langle
r^{4}\rangle$ along the isotope chain. Following the King-plot approach, we
perform a linear fit in isotope pair space, i.e., we fit $F_{\tau}$ to

$\displaystyle\hat{\boldsymbol{\nu}}_{\tau}^{r}\approx
F_{\tau}\mathbf{D}^{r}\,,$ (20)

where $\mathbf{x}^{r}=(x^{a,r},x^{b,r},x^{c,r},x^{d,r})$,
$x\in\\{\hat{\nu}_{\tau},D\\}$. In this way, the precision of the isotope-
shift measurements and nuclear mass measurements is exploited and field shift
coefficients $F_{\tau}$ with relative uncertainties at the level of $\approx
0.3\,\%$ can be obtained.

Next, we subtract the fit results from the $\hat{\nu}^{a,r}$ and obtain the
residuals

$\displaystyle\hat{\nu}_{\tau}^{a,r}-F_{\tau}D^{a,r}=G_{\tau}^{(4)}Q^{a,r}\,.$
(21)

Making use of the calculated electronic coefficients $G_{\tau}^{(4)}$, which
are listed in Tab. 13 we deduce three independent objects $Q^{a,r}$ per
isotope shift transition $\tau$. Fixing $r=(174,176)$ and
$\tau=\alpha_{\mathrm{PTB}}$ and choosing a value $\delta\langle
r^{4}\rangle^{r}=-7\>\text{fm}^{4}$, we are able to extract $\delta\langle
r^{4}\rangle$ values for the remaining isotope pairs, given in Tab. 14. We
choose the reference value $\delta\langle r^{4}\rangle^{r}=-7\>\text{fm}^{4}$
informed by predictions from ab initio nuclear structure theory and density
functional theory shown in Fig. 8, which all predict this value to lie between
$6-8$ $\text{fm}^{4}$. Beyond this chosen reference value, uncertainties and
covariances are obtained by propagating uncertainties from experimental input
($\nu_{\tau}^{a}$, $w^{a}$, and $\delta\langle r^{2}\rangle^{a}$) and atomic
theory ($G_{\tau}^{(4)}$) assuming independent normal distributions. The
dominant source of uncertainty is $\delta\langle r^{2}\rangle^{(170,168)}$,
and we also understand this uncertainty to lead to correlations in the fit in
Eq. (20), leading to the considerable anti-correlation between $\delta\langle
r^{4}\rangle^{(170,168)}$ and $\delta\langle r^{4}\rangle^{(172,170)}$.

We note that the approach outlined here to extract information on
$\delta\langle r^{4}\rangle$ from a King-plot nonlinearity is unable to give
information on the absolute scale of $\delta\langle r^{4}\rangle$. This is
simply because we are considering only the nonlinearity, but $\delta\langle
r^{4}\rangle$ also produces an isotope shift contribution parallel to the
linear field shift. We are unable to disentangle this contribution from the
dominant field shift, so our approach is only able to make statements about
how $\delta\langle r^{4}\rangle$ changes across the isotopic chain.

These values are compared with predictions for ab initio nuclear structure
theory and density functional theory in Fig. 8. Our extracted $\delta\langle
r^{4}\rangle$ values show a weak decrease between $\delta\langle
r^{4}\rangle^{(172,170)}$ and $\delta\langle r^{4}\rangle^{(174,172)}$ and
otherwise flat trend in $\delta\langle r^{4}\rangle$ across the ytterbium
isotopes studied here. Comparing this to the discussed nuclear structure
calculations, we find that the flatter $\delta\langle r^{4}\rangle$ trends of
ab initio calculations more closely reproduce the extracted trends.

Table 14:  Experimental $\delta\langle r^{4}\rangle^{A,A-2}$ values relative to $\delta\langle r^{4}\rangle^{176,174}=7\>\text{fm}^{4}$ extracted from isotope shifts from the $\alpha_{\text{PTB}}$ transitions using atomic theory with uncertainties and covariances propagated from input uncertainties.  Isotope pair $a$ = $(A,A^{\prime})$ | $\delta\langle r^{4}\rangle{}^{a}$ [$\text{fm}^{4}$] | cov($\delta\langle r^{4}\rangle{}^{a}$, $\delta\langle r^{4}\rangle{}^{b}$) [$\text{fm}^{8}$]
---|---|---
Isotope pair $b$ = $(A,A^{\prime})$
(170,168) | (172,170) | (174,172)
(170,168) | 7.33 (27) |  $-$0.076 |  $-$0.088 |  $-$0.010
(172,170) | 7.53 (32) |  $-$0.088 | 0.104 |  $-$0.006
(174,172) | 6.97 (28) |  $-$0.010 | 0.006 |  $-$0.079
(176,174) | 7 (reference) |  |  |
Figure 8: Experimental $\delta\langle r^{4}\rangle^{A,A-2}$ values relative to
$\delta\langle r^{4}\rangle^{176,174}=7\>\text{fm}^{4}$ extracted from isotope
shifts from the $\alpha$ transitions using atomic theory (fiducial, core
holes) are compared to nuclear theory predictions from our ab initio
calculations [top, 1.8/2.0 (EM), VS1 and VS2;
$\Delta\text{N}^{2}\text{LO}_{\text{GO}}$, VS1] and from density functional
theory calculations [8] [bottom, SV-min, RD-min, UNEDF1, $\text{Fy}(\Delta
r)$] for $A\in\\{170,172,174,176\\}$. The gray bands give estimated
uncertainties of the theory results.  Figure 9: Experimental $\delta\langle
r^{2}\rangle^{A,A-2}$ values [32] are compared to nuclear theory predictions
from our ab initio calculations [top, 1.8/2.0 (EM), VS1 and VS2;
$\Delta\text{N}^{2}\text{LO}_{\text{GO}}$, VS1] and from density functional
theory calculations [8] [bottom, SV-min, RD-min, UNEDF1, $\text{Fy}(\Delta
r)$] for $A\in\\{170,172,174,176\\}$. The gray bands give estimated
uncertainties of the theory results.

##  Appendix K Bounds on the new boson

The spectroscopy bounds in Fig. 3 were derived by constructing generalized
King plots [10], which require the isotope shift and mass measurements of $n$
pairs of stable spinless isotopes on $n-1$ transitions to eliminate $n-2$
nuclear factors from the system of isotope shift equations. In ytterbium, it
is currently the number of suitable isotope pairs, $n=4$, that fixes the
maximal dimension of the generalized King plot: using the frequency
measurements on 3 transitions, the charge radius variance $\delta\langle
r^{2}\rangle$, as well as the leading King nonlinearity, induced by
$\delta\langle r^{4}\rangle$, can be eliminated. The remaining King
nonlinearity is used to set upper bounds on the coupling
$\alpha_{\mathrm{NP}}$ of the new boson to electrons and neutrons. This method
is particularly useful since it is fully data-driven: The subtraction of
additional calculated SM effects such as $\delta\langle r^{2}\rangle^{2}$ from
the remaining King nonlinearity would reduce the precision of the experiment
to the level of the theory uncertainties on said SM terms and hence not
improve the bounds.

The red curve in Fig. 3 shows the bound on the new boson obtained by applying
the generalized King plot method [10] to the frequency and mass measurements
presented in this work, combined with frequency measurements presented in Ref.
[30]. This bound is compared with previous generalized King plot bounds using
Ca+ data [38, 39] and Yb+ data [8, 30]. Note that the same transitions were
used for the orange and red curves. The locations of the characteristic peaks
differ at the upper end of the plotted $m_{\phi}$ values due to an offset
between the $\alpha_{\mathrm{MIT}}$ and $\alpha_{\mathrm{PTB}}$ measurements.
When this offset is taken into account, the slopes of the two curves agree.

The competing $2\sigma$ bounds are shown as exclusion regions. The combination
of the bound on $y_{\mathrm{e}}$ from the measurement of the anomalous
magnetic moment of the electron, $(g-2)_{e}$ [40, 41], and the bound on
$y_{\mathrm{n}}$ [2] from neutron optics [42] and neutron scattering
experiments [43, 44, 45, 42] is indicated in yellow. For masses
$m_{\phi}<10^{2}\,\mathrm{eV}/c^{2}$, fifth force searches via the Casimir
effect provide particularly strong bounds (green) [46, 47]. Beam dump
experiments provide competitive bounds for $2m_{e}\lesssim m_{\phi}$. These
are not shown in Fig. 3 due to their dependence on the lifetime of the new
boson $\phi$, which is a function of all couplings of $\phi$, not only of the
couplings to electrons and neutrons.

Not shown either are the bounds from hydrogen-deuterium (HD) isotope shifts
[105]. For light elements, there are not sufficiently many isotopes to
construct a King plot, but the precision of the theory predictions reaches the
level of the experimental precision, such that the direct comparison of the
two can be used to set a bound. This bound, however, depends on the measured
HD charge radius difference which is determined via electron scattering or via
muonic atom spectroscopy. As the former leads to a weaker bound than the bound
from $(g-2)_{e}$ times neutron scattering, and the latter depends on the new
boson’s coupling to muons, the HD bounds are omitted in Fig. 3.

The blue-shaded areas in Fig. 3 are disfavored by astrophysical
considerations, such as bounds from star cooling in horizontal branch stars
(globular cluster) [48, 49] and from energy loss in the core of the supernova
SN1987a [50], which is subject to an $\mathcal{O}(1)$ uncertainty [106].
Complementary bounds from laboratory experiments would provide valuable
contributions to the search for new bosons also in these regions of parameter
space.

Finally, the green dashed lines in Fig. 3 show the bounds on $y_{e}$ from the
anomalous magnetic moment of the electron [40, 41], $(g-2)_{e}$, multiplied by
the bounds on $y_{n}$ from the kaon decay $K\to\pi+\mathrm{invisible}$, which
can be estimated using the bounds on the new boson couplings to quarks and the
first coefficient of the QCD $\beta$-function [107]. These bounds are highly
dependent on the flavor structure of the couplings of the new boson $\phi$ to
the quarks. For illustration, Fig. 3 shows the bound on
$\alpha_{\mathrm{NP}}/\alpha_{\mathrm{EM}}$ in the presence of a coupling to
top-quarks, $y_{t}$, and the bound in presence of a coupling to up-quarks,
$y_{u}$. They differ by approximately seven orders of magnitude.

This leaves open regions of parameter space where King plots have the
potential of providing the most competitive bounds on $\phi$, that are
agnostic both of its decay widths into dark matter and of the flavor structure
of its couplings to quarks. In the future, the King plot bounds can be further
improved by increasing the dimension of the generalized King plots. This can
be achieved by expanding the experimental program to isotopes with long half-
lives (e.g. 166Yb), or by moving to new atomic systems such as Sn [53] or Xe
[54], both of which have seven, spinless stable isotopes and suitable clock
transitions.


# Adaptive Generation of Phantom Limbs Using Visible Hierarchical Autoencoders

Dakila Ledesma  Yu Liang<EMAIL_ADDRESS>Dalei Wu  Deparment of Computer
Science and Engineering, University of Tennessee at Chattanooga, United States

###### Abstract

This paper proposed a hierarchical visible autoencoder in the adaptive phantom
limbs generation according to the kinetic behavior of functional body-parts,
which are measured by heterogeneous kinetic sensors. The proposed visible
hierarchical autoencoder consists of interpretable and multi-correlated
autoencoder pipelines, which is directly derived from hierarchical network
described in forest data-structure. According to specified kinetic script
(e.g., dancing, running, etc.) and users’ physical conditions, hierarchical
network is extracted from human musculoskeletal network, which is fabricated
by multiple body components (e.g., muscle, bone, and joints, etc.) that are
bio-mechanically, functionally, or nervously correlated with each other and
exhibit mostly non-divergent kinetic behaviors. Multi-layer perceptron (MLP)
regressor models as well as several variations of autoencoder models are
investigated for the sequential generation of missing or dysfunctional limbs.
The resulting kinematic behavior of phantom limbs will be constructed using
virtual reality and augmented reality (VR/AR), actuators, and potentially
controller for prosthesis (an artificial device that replaces a missing body
part). The addressed work aims to develop practical innovative exercise
methods that (1) engage individuals at all ages, including those with chronic
health condition(s) and/or disability, in regular physical activities, (2)
accelerate the rehabilitation of patients, and (3) release users’ phantom limb
pain. The physiological and psychological impact of the addressed work will
critically assessed in future work.

###### keywords:

Visible autoencoder, human musculoskeletal network, phantom limb, virtual
reality, kinetics.

††journal: Neural Network

##  1 Introduction

This paper proposed a hierarchical auto-encoder neural network [1] in the
adaptive phantom limbs generation (denoted as PLG-HAE for simplicity)
according to the kinetic behavior of functional body-parts, which are measured
by variance of kinetic sensors. Clinically, a phantom limb [2] is defined by
the sensation that an amputated or missing limb is still attached [3]. To
counteract phantom limbs, this work employs sensors and actuators, virtual
reality and augment reality, and deep learning to generate missing or
dysfunctional limbs. As a an intelligent, four-dimensional (namely X-Y-Z plus
somatosensory), partial control (e.g., phantom limb generation), and virtual-
reality-enabled rehabilitation modality, this project directly influences the
user’s motivation for movement [4].

Figure 1: Compensation of disabled input channels via Deep Neural Network: (a)
framework; (b) snapshot about the compensated kinetic status of a wheel-
chaired user, who receives “functional phantom legs” (in yellow color) through
disable-channel compensation; (c) walking by moving arms (online video [5])

It is known that various components of human body are correlated functionally,
bio-mechanically, or nervously. As a machine learning strategy, the proposed
work generates phantom limbs from other functional and measurable body parts
by employing above correlations among human body [2]. As illustrated in Figure
1(a), $x_{1}$, $x_{2}$, $x_{3}$, and $x_{4}$ constitute the four input
channels while the lower-limb-related input $x_{4}$ is not valid for a seated
user. Fortunately, we can formulate $x_{4}$ according to the correlation
between $x_{4}$ and other enabled inputs – $x_{1}$, $x_{2}$, and $x_{3}$. The
correlation can be discovered using a supervised deep neural network (DNN)
regression [6, 7], based on which the input $x_{4}$ can be reconstructed as a
function of other known inputs: $x_{4}^{\prime}=DNN(x_{1},x_{2},x_{3})$. Tai-
Chi movement is selected as the script, there the training data comes from the
recorded virtual Tai-Chi master’s kinetic data [8, 4]. As our preliminary
contribution, MLP and autoencoder methods are employed to construct phantom
legs according to arm kinetics.

Figure 1(b) shows a snapshot about the compensated kinetic status of a
wheelchair-bound user, who received “functional legs,” in compensation for the
phantom limbs through compensation of missing data. In this figure, the
compensation of missing data is generated by a neural network, and the
visualization of this generated data is done through a virtual environment,
namely VR. Besides bringing to the forefront improvements to body gesture
recognition and motion generation, missing data compensation has potential
clinical benefit to users, i.e. as a treatment to phantom limb pain [2].

Figure 2: Derivation of visible autoencoder Neural Network: (a)
muscoloskeletal network; (b) sample hierarchical network in forest data
structure; (c) visible autoencoder neural network derived from hierarchical
network.

As illustrated in Figure 2, the proposed visible hierchical autoencoder
consists of interpretable and multi-correlated autoencoder pipelines, which is
directly derived from hierarchical network described in forest data-structure.
According to specified kinetic script (e.g., dancing, running, etc.) and
users’ physical conditions, hierarchical network is extracted from human
musculoskeletal network, which is fabricated by multiple body components
(e.g., muscle, bone, and joints, etc.) that are bio-mechanically,
functionally, or nervously correlated with each other and exhibit mostly non-
divergent kinetic behaviors.

For the purposes of the neural network, the motion similarity between these
correlations are assessed to improve performance. Multi-layer perceptron (MLP)
regressor models, as well as several variations of autoencoder models are
investigated for the sequential compensation of missing or dysfunctional
limbs. The resulting kinematic behavior of phantom limbs will be constructed
using virtual reality and augmented reality (VR/AR), actuator, and potentially
controller for prosthesis (an artificial device that replaces a missing body
part).

The addressed work aims to develop practical innovative exercise methods that
(1) engage individuals at all ages, including those with chronic health
condition(s) and/or disability, in regular physical activities, (2) accelerate
the rehabilitation of patients, and (3) release the phantom limb pain. The
physiological and psychological impact of the addressed work will critically
assessed in future work.

The remainder of this paper is organized as follows. Section 2 overviews the
implementation of the generation system. Section 3 explains the data
acquisition, transmission, and visualization of data. Section 4 investigates
the human musculoskeletal network and some of its correlations to the
intuitions used for the proposed neural network. Section 5 explains the data
preprocessing methods for the autoencoder. Section 6 explains and reports the
neural network architecture. Section 7 reports the the results of using our
proposed network. Lastly, Sections 8 and 9 concludes the paper and defines
future work.

##  2 Implementation of System

The proposed work employs deep learning techniques such as autoencoder to
generate phantom limbs according to the observed kinetic behaviors of other
body parts based on the following hypothesis: (1) The human body consists of
multiple components such as muscle, bones, and joints, which are correlated
with each other mechanically, nervously, or functionally. (2) Because deep
learning techniques such as autoencoder can be used to formulate the pattern
of specific kinetic behavior script such as dancing and sports.

Figure 3: The flowchart of the proposed adaptive phantom limb generation based
on multi-correlation hierarchical autoencoder

Figure 3 shows the implementation of the proposed work, which consists of the
following critical tasks:

  * 1.

Formulating human musculoskeletal network (illustrated in Figure 2(a)) [9]
according the functional, mechanical and nervous correlation between each body
components (muscle, joint, or bone).

  * 2.

Deriving hierarchical network (illustrated in Figure 2(b), in the
configuration of forest data structure) from the human musculoskeletal network
according to the physical status of users, where the phantom limbs will form
the leaves of hierarchical tree.

  * 3.

Building visible autoencoder neural network (illustrated in Figure 2(c))
according to the hierarchical network so that the kinetic behavior can be
constructed according to the kinetic behavior of user’s functional body parts
measured by heterogeneous sensors.

  * 4.

Preprocess the kinetic sensory data using noise removing, data normalization,
incomplete data compensation such as Kalman filter and Spherical linear
interpolation (SLERP)[4], and kinetic signal decomposition such as singular
value decomposition (SVD), and wavelet analysis, etc.

  * 5.

Training the addressed visible autoencoder neural network according to
specific human motions script such as walking, jogging, dancing, or any other
physical activities.

  * 6.

Representing kinematic behavior about phantom limbs using VR/AR, and
actuators, which can directly stimulate users.

##  3 Developing Data Acquisition and Transmission Schemes

Proper data acquisition schemes will be developed to collect 4D kinetic data
with suitable format from various hardware instruments such as a Microsoft
Kinect V2, foot-pressure sensors, actuators, and VR goggles. Realtime
application of two-way communications and edge computing are considered to
facilitate the human computer interaction in the PLG-HAE sytem.

Figure 4: Input&output instruments

###  3.1 Acquisition of 4D Sensory Data

Figure 4 show the basic input and output equipment of phantom-limb generation.
A Microsoft Kinect V@ sensor, foot pressure sensor, and electronic glove
(studded with force-sensitive resistors and vibration motors that can
essentially re-create the experience of kinetic movement) can be used to
acquire kinetic data (or 4D sensory data) of a user. A virtual reality goggle,
such as the Oculus Rift, HTC Vive, or Google Cardboard, motor to drive
prosthetic limb, and a tactile actuator are used as output equipment that work
together to depict 4D feedback to the user.

#####  Acquisition and Polishment of Kinematic Data

The Microsoft Kinect collects the kinematic data of the coach (or physical
therapist, for training purposes) and the user. Through Kinect, we can obtain
joints’ transient position ${\langle x,y,z\rangle}_{t}^{k}$ and corresponding
Quaternion rotation [10]
${\langle\cos(\frac{\theta}{2}),\sin(\frac{\theta}{2})\vec{v}\rangle}_{t}^{k}$,
where $\theta$ is an angle around unit axis $\vec{v}$, $t$ is the time, and
$k$ is joint’s identifier. Quaternions [10] are considered to represent the
rotation of a rigid body in 3D space using four degree-of-freedoms (DOFs).

Quaternions are superior to many other traditional rotation formulation
methods because they completely avoid gimbal-lock [11]. In the addressed
phantom-limb generation system, Quaternions will be used in 4D reconstruction
over Unity3D platform and acquisition of kinetic signal. On the other hand, as
a Quaternion is specified with reference to an arbitrary axis vector it is not
a good choice in rotation recognition. In this work, Euler angles
$\langle\alpha,\beta,\gamma\rangle$, which represent the angles rotating
around axis Z, X, Y respectively (in some literature it is denoted as $\langle
yaw,pitch,roll\rangle$) will be adopted in gesture recognition.

The addressed phantom-limb generation system stores the captured kinetic data
in JavaScript Object Notation (JSON) format, which includes joint position
(${\langle x,y,z\rangle}_{t}^{k}$), quaternion rotation
(${\langle\cos(\frac{\theta}{2}),\sin(\frac{\theta}{2})\vec{v}\rangle}_{t}^{k}$),
tracking status (0: invisible; 1: referred; 2: observable). Potentially, force
load ($\mathbf{f}_{t}^{k}$) and momentum, etc. may be included. Tracking
status indicates whether or not the specific joint is observable by the depth
sensor. The force load and momentum are derived by inverse dynamics analysis.
Besides JSON, a kinetic dataset in Comma-Separated Values (CSV) format is also
provided in PLG-HAE as many data analytics systems do not accept JSON format.

Due to measurement error or unavoidable occlusion, a joint is not always
observable or tractable by the kinetic sensor. Spherical linear interpolation
(commonly abbreviated as SLERP) [12] and Kalman filtering techniques (be
discussed in Section 5) will be employed to compensate the missing data. As
illustrated in our preliminary online video [13], SLERP can effectively
address those short-term missed-tracking joints (namely tracking status=0 or
1).

##### Acquisition of Tactile Data

Besides the Kinect, other acquisition instruments such as an accelerometer,
orientation sensors, and strain gauges [14] will also be considered for the
addressed system. As indicated above, a foot pressure sensor will be used to
obtain the ground reaction force $F_{t}$ for inverse dynamic analysis.
Furthermore, electromyography (EMG) [14] will be selectively employed to
evaluate and record the electrical activity produced by skeletal muscles. The
EMG signal is characterized by a frequency range of several hertz to over 1
kHz and by amplitudes ranging from fractions of a microvolt to a few thousand
microvolts. Electromyogram can be analyzed to detect activation level or to
analyze the biomechanics of users’ movement. For the acquisition of high-
quality EMG signals from localized muscle region, the PIs will focus on
identification of localized muscle region of users, noise reduction and
grounding practices (to eliminate extraneous electrical noise), electrode site
preparation and placement (to minimize the detection of irrelevant
bioelectrical signals) and appropriate differential signal preamplification
and preliminary signal conditioning (to further enhance signal-to-noise
ratio).

##### Reconstruction of 4D data

4D kinetic feedback/instruction is reconstructed through virtual reality and
tactile actuators. For example, HTC Vive can be used as the VR device, as it
is readily supported and available to consumers and does not exhibit poor VR
qualities, such as low refresh rate, that may induce nausea to the user.
Unity3D may be used as the SDK for generating a virtual environment using AR
and VR. To visualize the human body through a virtual reality facility, the
output kinematic data is translated into Quaternion format [10, 11], which is
recognized by Unity3D. The conversion of joint input data to Quaternions
allows for the manipulation of a 3D human model, through the manipulation of a
model’s child objects or joints. The manipulation of a model allows for a
better visual representation of a body in both recorded and real-time Kinect
data.

Equipped with somatosensory stimuli, the phantom limb generator also directly
guides users with somatosensory feedback. Tactile actuators potentially used
in this work include Eccentric Rotating Mass (ERM), Linear Resonant Actuator
(LRA), Piezo, and Electro-Active polymers (EAP). EAP will be investigated in
the development of phantom-limb generation because of its high fidelity of
sensations, and excellent durability.

Figure 5: Edge-computing-enabled phantom-limb generation deployed on commodity
hardware (demo in online video [15])

We will apply deep learning techniques to electric prosthesis with the goal of
developing an agent that could learn and adapt to the needs of the user and
changes in the environment [16]. Deep learning provides a nice set of tools
that allow us to incorporate numerous sensory streams (the kinetic behavior of
other body parts, vision, tactile sensation, inertial measurements, servo
position, etc.) in ways that are computationally tractable and suitable for
online learning. In this work, autoencoder neural network enables the phantom
limbs coordinate with other body parts according to their correlation, some
latency is observed. In the future work, more ”intelligent” deep learning
techniques such as reinforcement learning will be investigated to improve the
performance of prosthetic limb (with much less latency).

###  3.2 Task: Developing Communication and Edge-Computing Protocols

##### Real-time, Two-Way Communication.

Two-way communications are of key importance in the proposed system, since the
information needs to be exchanged in a real-time manner. The challenges of the
communication protocol for the proposed system include: (1) Real-time
communication: Information in the addressed system needs to be conveyed in
real time. If there is a significant delay in the communications,
synchronization between the virtual coach and user will be lost and the user
will experience a disjointed rhythm. (2) High throughput: When there are many
users, all corresponding video and audio need to be conveyed in the network,
thus incurring a substantial requirements for communication bandwidth. (3)
Two-way communications: The communications are between the virtual coach and
users with mutual interactions. Therefore, it could be sub-optimal if one-way
communications are considered separately. (4) Dynamics awareness: The
communications may be optimized together with the physical dynamics of the
virtual coach and users (namely the motions).

To address the above challenges, we will first model phantom-limb generation
as a cyber physical system (CPS) [17, 18] and then analyze the bandwidth
needed for controlling the physical dynamics. Then, the detailed communication
protocol will be designed and evaluated with the whole system.

##### Deployment of Phantom-limb Generation Using Affordable Hardware Based on
Edge Computing.

Edge computing enables real-time knowledge generation and application to occur
at the source of the data close to user device [19, 20], which makes it
particularly suitable for the proposed latency-sensitive system. An edge
server can be adapted to serve multiple users through interaction with their
devices. There are communication and computing trade-offs between the edge
server and each user device. Data could either be locally processed at the
user device or else be transmitted to and processed at the edge server.
Different strategies introduce different communication costs, resulting in a
difference in delay performance. To provide the best quality of experience for
users, the PIs will conduct the following studies for the proposed PLG-HAE
system: (1) Identification and modularization of computing tasks: the
computing tasks of data preprocessing, kinetic movement recognition, the
analysis of individualized movement choreography and the corresponding
computing overheads (CPU cycles, memory) will be determined. (2) Design,
prototyping and enhancement of offloading schemes: Based on the results of
bandwidth and delay analysis as well as delay performance requirement,
computation offloading schemes will be developed to determine which computing
tasks should be performed locally at the user device and which computation
tasks should be offloaded to the edge server.

As illustrated in Figure 5 (b), enabled by edge-computing technology, the
proposed phantom limb generation system can be deployed over commodity
hardware so that it can benefit a broader population. An illustrative concept
demonstration about edge-computing-enabled phantom limb generator is given in
our online video [15].

##  4 Deriving Hierarchical Neural Network from Human Musculoskeletal Network

###  4.1 Structure, function, and control of the human musculoskeletal network

Figure 6: Human musculoskeletal network.

As illustrated in Figure 6, the human body is a complex organism, the gross
mechanical properties of which are enabled by an interconnected
musculoskeletal network [9] controlled by the nervous system. The nature of
musculoskeletal interconnection facilitates stability, voluntary movement, and
robustness to injury. However, a fundamental understanding of this network and
its control by neural systems has remained elusive. Here we address this gap
in knowledge by utilizing medical databases and mathematical modeling to
reveal the organizational structure, predicted function, and neural control of
the musculoskeletal system. We constructed a highly simplified whole-body
musculoskeletal network in which single muscles connect to multiple bones via
both origin and insertion points. We demonstrated that, using this simplified
model, a muscle’s role in this network could offer a theoretical prediction of
the susceptibility of surrounding components to secondary injury. Finally, we
illustrated that sets of muscles cluster into network communities that mimic
the organization of control modules in primary motor cortex. This novel
formalism for describing interactions between the muscular and skeletal
systems serves as a foundation to develop and test therapeutic responses to
injury, inspiring future advances in clinical treatments.

###  4.2 Derivation of hierarchy out of musculoskeletal network

Human-body consists of multiple body-components, which are correlated
functionally, bio-mechanically, or nervously. From the correlation graph, we
can derive a hierarchical sub-graph that can be formulate by neural network in
a relatively straightforward way. In this work, hierarchical clustering
algorithm [21] is employed to formulate the hierarchical network for phantom
limbs out of human muscoloskeletal network.

###  4.3 Visible and hierarchical neural network architecture for real-time
phantom limb generation.

Figure 7: Visible and hierarchical neural network

It is known that any system can be regarded as a hierarchical structure (i.e.,
system $\rightarrow$ subsystem $\rightarrow$ subsubsystem, …). As illustrated
in Figure 7(a), the human body system can be always divided into sub-
components that are mechanically correlated and whose motions are non-
divergent. Inspired by the Bayesian network, we propose a visible and
hierarchical neural network to accurately formulate a system. As illustrated
in Figure 7(b), a sample visible and hierarchical neural network, which is
directly derived from the human body system, is employed to specify the
musculoskeletal kinematic. The visible and hierarchical neural network can be
employed in phantom limb generation, 4D kinetic behavior recognition, and
individualized Tai-Chi choreography (to be discussed in the remaining
sections). Preliminary experimental results demonstrate that is superior to a
classical neural network from the point of view of training speed and
stability.

###  4.4 Related Work

Much of work done in human motion generation focuses on choosing a model
expressing the inter-dependence between joints. Recently, human motion was
generated through the use of auto-regressive functions, such as models relying
on Gated Recurrent Networks (GRU) or Long-Short Term Memory (LSTM) [1].
However, these models are not feasible for use in our applications. Unlike
long-term or short-term prediction, models focusing on real-time generation
must be small and computationally simple to achieve a 24 predictions-per-
second minimum. Other work has focused on minimizing loss by using techniques
such as highway units, custom loss functions, or defining loss parameters.
Generative-Adversarial Networks (GAN) [1] have also been experimented.
However, these models tend to be complex and convergence of the neural network
is not intuitive. Thus, determining ground truth error for these models is
nuanced during training. Due to the inability to use these neural networks,
methods including real-time autoencoders are examined in this paper.

Figure 8: Flowchart of the Preprocessing for the generation of phantom limb
generation

##  5 Developing Data Preprocessing Methods of Autoencoder-enabled Phantom
Limb Generation

Data preprocessing operations play an indispensable role in phantom limb
generation because: (1) Input data is of a heterogeneous nature. For example,
different users have variable sizes; sensors may have various viewing angles;
users will not always be located in a deterministic position; and the two
time-series data sets may not be synchronized. As a result, scaling, rotating,
translating, and dynamic time warping (DTW) are needed to normalize the
original input data. (2) The input data set may be incomplete. For example,
occlusion inevitably leads to missing data; Musculoskeletal force and momentum
exerted over the joints or muscle cannot be directly obtained from the
sensors; some input channels are not enabled (e.g., partial control) for users
with mobility-based chronic conditions (i.e., partial control). In the
implementation of autoencoder-enabled phantom limb generation, Kalman
filtering, inverse dynamics, and time-series prediction are employed to handle
the incomplete data [22, 23]. (3) The measurement-induced noise is
significant.

Figure 8 shows the flowchart of data preprocessing of autoencoder-enabled
phantom limb generation [24, 25, 26, 27]. $\langle\theta,\vec{v}\rangle$
(denoted as
$\langle\cos{\frac{\theta}{2}},\sin{\frac{\theta}{2}}\vec{v}\rangle$ in
Section 3, where $\theta$ is the rotation angle about axis $\vec{v}$ )
indicates a joint’s Quaternion rotation; $\langle x,y,z\rangle$ denotes a
joint’s position; $f_{t}$ indicates a joint’s applied force, which is derived
from inverse dynamics; $\langle\alpha,\beta,\gamma\rangle$ indicates a joint’s
rotation under normalized Joint Coordinate System (JCS) – Euler angle. Its
main implementation techniques include data fusion, inverse dynamics analysis,
spatial normalization, Kalman filtering, and reconstruction of disable input
channels. The kinetic data is stored in JSON format.

###  5.1 Formulating Musculoskeletal Kinetic Features

Inverse dynamics analysis (IDA), which is derived from Newton-Euler equations
[28, 29, 30]), aims to calculate unknown kinetic information (the force and
moments of joints and muscles) from measured kinematic information (e.g.,
position, velocities and accelerations of joints) and measured kinetic
information (e.g., ground reaction force measured by foot pressure sensor). As
illustrated in Figure 8, given joints’ location $\langle
x_{i},y_{i},z_{i}\rangle$, Euler rotation
$\langle\alpha_{i},\beta_{i},\gamma_{i}\rangle$ where $i$ denotes the identity
of a joint, and ground-reaction force $F$, the joints’ force $f_{i}$ and other
musculoskeletal kinetic features can be computed via IDA.

As illustrated in Figure 8, autoencoder-enabled phantom limb generation
employs inverse dynamics to compute internal joint forces and moments with
given ground reaction forces. In this work, the human body is divided into
multiple connected rigid bodies [31, 32] which correspond to relevant
anatomical segments such as the thigh, calf, foot, arm, etc. The model’s
anthropometric dimension (e.g., the mass and momentum inertia) is derived from
statistical analysis. In addition, it is assumed that each joint is
rotationally frictionless.

The proposed methods in Figure 8 can be customized to investigate the bio-
mechanical response of human motion by considering appropriate pathology for
different health issues such as cerebral palsy, poliomyelitis, spinal cord
injury, and muscular dystrophy [30]. This will make the addressed work a more
constructive rehabilitation system.

###  5.2 Spatial Normalization

As addressed in Section 3, we can acquire the joints’ position and rotation,
which are denoted as $\langle x,y,z\rangle$ and $\langle\theta,\vec{v}\rangle$
respectively. Both need to be normalized to ease and boost the gesture
recognition:

##### Normalization of Joints’ Rotation

Spatial rotations in three dimensions can be parametrized using either Euler
angles and unit quaternions. Instead of being denoted as a rotation around an
arbitrary vector such as a unit Quaternion, an Euler angle describes the
orientation of a rigid body with respect to a fixed coordinate system.
Therefore, in kinetic gesture recognition, Euler angle is more constructive
than Quaternion in data analytics. In the proposed research, joints’ rotations
can be normalized by converting the quaternion $\langle\theta,\vec{v}\rangle$
into Euler angle $\langle\alpha,\beta,\gamma\rangle$.

##### Normalization of Joint’s Position

Joints’ position $\langle x,y,z\rangle$ involves many factors irrelevant to
gesture recognition. These include: a user’s orientation to the camera,
location in the room, and body shape and size. To eliminate such irrelevant
factors, This work employs a series of spatial normalization techniques: (1)
bone scaling, which makes uniform the bone length of users [22]; (2) axis-
oriented rotating of view angle; (3) translation of the origin, which makes a
user positioned at the center of a sensor; (4) re-constructing $\langle
x,y,z\rangle$ according to joint rotation [11] ; and (5) polishing the kinetic
curve using a Savitzky-Golay filter. Our preliminary experimental results
demonstrate that the normalization techniques addressed above can greatly
improve the quality of data (less noise and smoother kinetic performance) so
as to achieve higher recognition [22, 25, 24].

###  5.3 Recovering Occlusion-induced Missing Data

During sensory data acquisition, unavoidable occlusion may introduce missing
data or lost-tracking. As addressed in Section 3, SLERP can basically fix the
issues caused by short-term occlusion. Kalman filter [33, 34, 35] can be
employed to fix the missing information (including both position and rotation)
caused by long-term occlusion. A preliminary comparison between the raw and
preprocessed physical rehabilitation kinematic data is available on our online
video [26, 27].

###  5.4 Application of Singular Value Decomposition on Kinetic Data

Singular value decomposition (SVD) is employed to decompose the human kinetics
into multiple of modes, which will be fed into the autoendoer in different
channels. SVD can reduce the nonlinearity degree of kinetic signal so that it
can be formulated by neural network more precisely.

##  6 Neural Network Architecture and Generation of Phantom Limbs

###  6.1 Previous Work

Deep neural networks (DNN) have proven to be significant in feature learning
through their ability to learn representations in a hierarchical manner [36].
Specific to the applications focusing on the generation of human motion data,
DNN models based on recurrent neural networks (RNN), such as ones that employ
Long-Short Term Memory (LSTM) and Gated Recurrent Units (GRU) currently
represent the majority of the techniques proposed. These proposed models, such
as ones that focus on temporal encoding, were more focused on learning
general, generic representations from a large pool of general, non-specific
human motion data. Thus, previous models were more focused on the generation
of new, novel motions.

In contrast, our work is focused on assessing current body language that
results in the representation of missing joints given a deterministic motion.
In addition, unlike previous models that rely on large datasets, we must be
able to train on very small datasets (such as ones with less that 25 samples
per motion). Many architectural modifications are proposed to attain proper
performance given these constraints.

For the recreation of missing data through phantom limbs, two traditional
methods for machine learning were assessed – multilayer perceptron (MLP) and
denoising autoencoders (DAE). In addition, this we proposed a new autoencoder
architecture for the recreation of phantom joints, a multi-correlated
hierchical autoencoder.

###  6.2 Generating phantom legs based on arm movement using visible and
hierarchical autoencoder network.

As our preliminary contribution, a neural network is trained to generate the
kinetic status of hip, knees, and feet according to the kinetic status of
shoulders, elbows, and arms captured by 4D sensors [5]. As illustrated in
figures 9, 10, 11, and 12, four network architectures are investigated in this
research: (a) multiple layer perceptron (MLP); (b) denoising autoencoder (a
classical autoencoder architecture); (c) visible and hierarchical neural
network with two subsystems (VHNN2); and (c) VHNN with four subsystems
(VHNN4).

It can be observed that VHNN splits the input tensor and then feeds the split
tensor into multiple smaller, parallelized autoencoders. Thus, data for each
joint can be calculated in parallel with their own respective autoencoder. The
aforementioned parallelized autoencoder pipelines are simplified stacked
autoencoders, allowing for optimization of specific, key tasks rather than one
large task.

A video playlist of the generation of phantom legs based on VHNN may be found
at [37].

###  6.3 Multilayer perceptron models

Multilayer perceptrons (MLP) were assessed as a preliminary, prototype network
for the inference and generation of phantom joints. Despite their simplicity,
MLP models are able to learn non-linearly separable data, and can learn
features through stochastic approximation. Thus, they are the simplest deep
model available, and was used as a minimum benchmark to the other models we
have thus developed. Rather than feeding input into a logistic regression, MLP
networks work by adding layers that process the input using activation
functions.

Figure 9: Generation of phantom legs from moving arms using a multilayer
perceptron architecture [37])

As seen in Figure 9, our implementation of a MLP Regressor model was built
with 7 hidden layers, with each layer possessing 18 nodes. The MLP model is
also fitted with a regularizer term to prevent over-fitting of our motion
data. However, the complexity of our data proved too difficult for the model
to learn, and thus our implementation of an MLP regression model achieved poor
accuracy [38] as the model struggles to generate proper regression values. The
increase and/or change of hyperparameter values, such as layer number or nodes
per layer, do not significantly benefit accuracy of the results, but do
increase the training time.

###  6.4 Denoising Autoencoder

In generative data applications, autoencoders have had an important influence
for their low computation cost–relative to RNNs–and their high accuracy. For
the generation of phantom limbs, autoencoders (AE) were critically assessed in
their performance for generating human-like motion data whilst being quick in
their predictions/generation of data. Quickness and speed of prediction is
important to the generation of phantom limbs as the predictions must be done
in realtime, i.e. generated over 24 times per second. Whilst the generation of
temporal/sequence data are most normally done through recurrent neural
networks, convolutional neural networks such as autoencoders have found
success in generative tasks [39]. During initial testing, autoencoders
exhibited data bias within the generated movement, causing difficulties in the
neural network converging [37].

For phantom limb generation, autoencoders are quite applicable as the
generation of data does not necessarily have to be over a large time window.
Instead, accuracy during short-time windows and quick reaction speed relative
to the realtime input were more important. In addition, convolutional neural
networks are less susceptible to noisy or substandard data and can better
generalize data compared to the more complex recurrent units.

Figure 10: Generation of phantom legs from moving arms using a classical
denoising autoencoder architecture [37])

As seen in 11, we moved our attention away from MLP and employed a standard
denoising autoencoder to learn the correlations of one’s body language to a
specific set of phantom limbs. Due to the noisy nature of data captured by our
Microsoft Kinect V2, the performance of autoencoders for denoising as well as
generative representation was assessed. Through the use of a stacked denoising
autoencoder, accuracy when viewed in comparison to multi-layer perceptron
models were significantly higher in terms of the motion data generated and
exhibited less noise within the movement [38][37]. In contrast to data
generated by MLP models, autoencoders achieved motion data that was believably
human-like.

####  6.4.1 Autoencoder Data Bias

Despite the improvement of generation of human-like data using autoencoders,
the generated motions were not perfect for all phantom limbs. Using our
implementation of a denoising autoencoder, the minimization of loss presented
bias towards data points that were closer to the starting joints. Our
autoencoder’s input and output tensors exhibit a dimensionality of (n, 18, 3),
representing sample number, joint index, and x, y, z-coordinates respectively.
The generation of motion data found within the first few joints, such as the
first six indices out of 18, exhibited much better human motion data in
comparison to data found in the last twelve data points. Thus, motion
generation data generated for a specific leg (such as the left leg) was
significantly more believable than data generated for the complement leg (such
as the right leg). The exhibition of the bias in optimization is most apparent
during the realtime generation/prediction of data [37].

The exhibition of data bias may due to the hierchical nature of the human
body, e.g. the musculoskeletal network. Due to the divergent motion relation
of different body segments, optimization of the may be compromised when other
joints exhibit opposite or very divergent motion. Thus, the correlations
between one’s body language may be intuitive for joint segment (i.e. one’s
left leg), but entirely opposite or non-intuitive for another joint segment
(i.e. one’s right leg).

###  6.5 Application of Hierarchical Autoencoder

The prior applications of hierchical autoencoders can be found in
classification tasks and natural language processing (NLP) [40]. In
retrospect, these neural networks have been employed to decrease complexity
within ambiguous classifications [41] or used as a means to assess long
sequences of data using CNNs [39]. The benefits of using hierarchical CNNs for
sequential data are two-fold: (1) They are much quicker to train and predict
and (2) They can attain higher accuracy than RNNs in certain applications.

We turned to the parallelization of convolution pipelines for the generation
of phantom limbs in response to the optimization problems by our stacked
denoising autoencoder showed. Thus, the attainment of certain goals was needed
for the new, parallelized autoencoder models: (1) the autoencoder must not too
complex that the generation of data would not be realtime, (2) the autoencoder
must not exhibit signs of optimization bias, and (3) the autoencoder must be
able to allow for multi-correlation of outputs.

Figure 11: Generation of phantom legs from moving arms using a two thread
(subsystem) visible and hierarchical autoencoder neural network (VHNN-AE-2)
architecture [37])

Figure 11 shows the autoencoder architecture used to resolve this inherent
data bias. As seen, the proposed network has its splits each input tensor
according to the joint joint segments that the data is correspondent to for
both input and output, in comparison to figure 10. This resolves biased
optimization by allowing each autoencoder to optimize for a smaller input data
dimensionality, and for motion that is mostly non-divergent. Because of the
reduced dimensionality of input data, the MaxPooling as well as Upsampling of
data does not need to encode and decode as large of data in comparison to a
deeper, larger stacked autoencoder. Thus, despite the seemingly double amount
of layers, the amount of layers per smaller autoencoder is reduced, and
parameters per parallelized autoencoders are also reduced. This results in
training and prediction performance comparable to a standard denoising
autoencoder, but with much higher ground-truth accuracy.

Figure 12: Generation of phantom legs from moving arms using a four thread
(subsystem) visible and hierarchical autoencoder neural network (VHNN-AE-2)
architecture [37])

Whilst the splitting of inputs into two parallel autoencoders performed much
better than a standard stacked autoencoder, the parallelization of
autoencoders has caused a separate problem, in which correlations can only be
found in two distinct joint pairs, i.e. one’s left arm motion can influence
the left leg but not the right leg. Thus, multi-correlation was needed in
order to help give better representation of each input to the generated
output. As seen in Figure 12, this resulted in further splitting of the
autoencoders, essentially doubling the amount of pipelines within this phantom
limb application. This was done to give correlations to both input joints
segments to both output joint segements.

Table 1: Architectures of Autoencoders Assessed and Proposed Classical | VHNN-2 | VHNN-4
---|---|---
Type | Hyperparameters | Type | Hyperparameters | Type | Hyperparameters
conv1D | Filter: 32 | conv1D | Filter: 32 | conv1D | Filter: 32
| Kernel Size: 3 |  | Kernel Size: 3 |  | Kernel Size: 3
| Activation: Linear |  | Activation: Linear |  | Activation: Linear
| Padding: same |  | Padding: same |  | Padding: same
|  | Dropout | Rate: 0.5 | Dropout | Rate: 0.5
MaxPooling | Pool size: 3 | MaxPooling | Pool size: 3 | MaxPooling | Pool size: 3
conv1D | Filter: 32 |  |  |  |
| Kernel Size: 3 |  |  |  |
| Activation: Linear |  |  |  |
| Padding: same |  |  |  |
MaxPooling | Pool size: 2 |  |  |  |
conv1D | Filter: 32 | conv1D | Filter: 32 | conv1D | Filter: 32
| Kernel Size: 3 |  | Kernel Size: 3 |  | Kernel Size: 3
| Activation: Linear |  | Activation: Linear |  | Activation: Linear
| Padding: same |  | Padding: same |  | Padding: same
|  |  |  | Dropout | Rate: 0.2
Upsampling | Size: 2 | Upsampling | Size: 3 | Upsampling | Size: 3
conv1D | Filter: 32 |  |  |  |
| Kernel Size: 3 |  |  |  |
| Activation: Linear |  |  |  |
| Padding: same |  |  |  |
Upsampling | Size: 3 |  |  |  |
conv1D | Filter: 1 | conv1D | Filter: 1 | conv1D | Filter: 1
| Kernel Size: 3 |  | Kernel Size: 3 |  | Kernel Size: 3
| Activation: Linear |  | Activation: Linear |  | Activation: Linear
| Padding: same |  | Padding: same |  | Padding: same
|  | Concatenate | Axis: 1 | Concatenate | Axis: 1
|  |  |  | Concatenate | Axis: 1

Partial control is an important feature of the generation of phantom limbs as
it enables those users with disability to operate freely. Partial control is
implemented by compensating the missing data inferred by disabled input
channels: in the event that several input channels are disabled, the addressed
model is able to construct the void input channels by taking the advantage of
correlation among all inputs. Compensation can normalize the input data so
that VIGOR can achieve higher recognition rate, its psychological and
physiological benefits to users are also under our investigation.

##  7 Results

Table 2: Time-performance of phantom-legs generation using visible and hierarchical autoencoder neural network (VHNN) (Intel Core i9-7900X, 1x NVIDIA GTX1080 Ti, 64GB RAM; MLP does not employ GPU) NN architecture | Training time | Training time | convergence | ground | online
---|---|---|---|---|---
| per step | for 1K epochs | in epoch | truth error | video
MLP | NA | NA | 250 | 9515.51 | [38]
Denoising Autoencoder | $108-110\mu s$ | 9m 15s | 1000 | 2107.46 | [42]
VHNN-Autoencoder-2 | $30-31\mu s$ | 2m 44s | 500 | 276.68 | [43]
VHNN-Autoencoder-4 | $52-57\mu s$ | 4m 37s | 800 | 366.15 | [44]

As illustrated in Table 2, the proposed VHNN architecture has proven to have
overall superior results compared to previous work. Decreased training time
compared to previous autoencoders architectures can be observed due to the
parallelization of simpler autoencoders, increasing efficiency by easing
optimization. This is done by allowing autoencoders to train on specific
gestures in a whole movement. In addition, it does not exhibit data-hungry
tendencies that state-of-the-art models exhibit, allowing it to be trained on
small amounts of data.

Lower ground truth error can be seen in the VPNN-AE-2 versus VPNN-AE-4. This
is due to training data having no anomalies of data that real-time can
exhibit. While VHNN-AE-2 with single-correlation works better when testing
against ground truth data, VPNN-AE-4 with double-correlation works better in
real-time as the patient may not follow the Tai-Chi movements correctly. This
causes worse ground truth error as the added complexity of the architecture
increases noise during output, but enables better patient-error tolerance.
Because of this additional noise produced of VHNN-AE-4, improvements such as
larger training datasets, more sophisticated pre- and post-processing of data,
as well as improvements in NN architecture will be investigated.

The neural network addressed in our preliminary work is single-modality, which
generates a single type of kinematic output, either joints’ location $\langle
x,y,z\rangle$ or joints’ rotation in Euler angle
$\langle\alpha,\beta,\gamma\rangle$. There might be potential issues for
single-modality network. As illustrated in our preliminary work [4, 8],
divergence is commonly observed in an XYZ-oriented network; Euler-angle-
oriented network suffer from $0-360degree$ issue [4]. A multi-modality neural
network will be studied for intelligent choreography generation.

Figure 13: Choreography using multi-modality network

As illustrated in Figure 13, multiple heterogeneous kinetic data will be
generated simultaneously as the output of neural network so that a cross-
validation can be made.

##  8 Conclusion and Future Work

As seen from the reported results, the proposed multi-correlation visible
hierarchical autoendoder such as VHNN-4 exhibits much better realtime
generated data in comparison to MLP or classical autoencoders. Thus, the
segmentation of joint data through human musculoskeletal network as well as
non-divergent motion are used as effective methods to improve neural network
optimization and prediction quality. In addition, the proposed work is able to
generate phantom limbs with very scarce amounts of data, in contrast to
previous models. Given the flexible nature of parallelized autoencoders,
adding pipelines as well as the modification of individual pipelines for other
applications may be easily executed.

In the future, Generative Adversarial Networks (GAN) may be assessed for the
prediction of longer time-horizons in comparison to the short time-horizons
predicted by our proposed autoencoder. In addition, more pre-processing and
post-processing methods may be assessed, such as singular value decomposition
(SVD), in order to further aid accuracy and performance. For improvements in
data acquisition, better depth sensors relative to the Microsoft Kinect V2 may
be assessed for less noisy, precise data.

Future work also includes evaluating performance and user experience. A
variety of user experience data and clinical benefit data will be generated
during system assessment. Analytical results about the extent to which the
addressed work demonstrates superior physical and psychological benefits and
user experience over those of existing systems will also be derived

## Acknowledgments

This work is jointly sponsored by the National Science Foundation (award No:
1240734), UTC THEC/CEACSE 2016 Grant Program, and 2015 UTC CRISP program.
Special thanks also go to Mr. Mr. Samuel Clark for his contributions in the
paper.

## References

## References

  * [1] I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016.
  * [2] M. Ortiz-Catalan, R. A. Guomundsdottir, M. B. Kristoffersen, A. Zepeda-Echavarria, K. Caine-Winterberger, K. Kulbacka-Ortiz, C. Widehammar, K. Eriksson, A. Stockselius, C. Ragnö, Z. Pihlar, H. Burger, L. Hermansson, Phantom motor execution facilitated by machine learning and augmented reality as treatment for phantom limb pain: a single group, clinical trial in patients with chronic intractable phantom limb pain, The Lancet 388 (10062) (2016) 2885–2894.  doi:10.1016/S0140-6736(16)31598-7.
  * [3] V. S. Ramachandran, W. Hirstein, The perception of phantom limbs. The D. O. Hebb lecture., Brain 121 (9) (1998) 1603–1630.  arXiv:http://oup.prod.sis.lan/brain/article-pdf/121/9/1603/17864034/1211603.pdf, doi:10.1093/brain/121.9.1603.
URL https://dx.doi.org/10.1093/brain/121.9.1603

  * [4] Y. Liang, D. Wu, D. Dakila, C. Davis, R. Slaughter, Z. B. Guo, Virtual tai-chi system, a smart connected modality for rehabilitation, Elsevier: Smart Health 9–10 (2018) 232–249.
  * [5] D. Ledesma, Y. Liang, [online] vtcs: Construct phantom legs according to arms’ movement using varius autoencoder strategies, https://youtu.be/ZmTobUjRX64, cite November 08, 2018.
  * [6] Y. Bengio, Learning deep architectures for ai, fundamental Trends Machine Learning 2 (1) (2009) 1–127.
  * [7] A. Sperduti, A. Starita, Supervised neural networks for the classification of structures, IEEE Transactions on Neural Networks 8 (3) (1995) 714–735.  doi:10.1109/72.572108.
  * [8] Y. Liang, D. Wu, D. Dakila, C. Davis, R. Slaughter, Z. B. Guo, Virtual taiji system, a smart-connected modality for rehabilitation, in: The Third IEEE/ACM Conference on Connected Health: Applications, Systems, and System Technologies, Washington, D.C., 2018.
  * [9] A. C. Murphy, S. F. Muldoon, D. Baker, A. Lastowka, B. Bennett, M. Yang, D. S. Bassett, Structure, function, and control of the human musculoskeletal network. plos biology 16 (1).  doi:10.1371/journal.pbio.2002811.
  * [10] R. W. Farebrother, J. Grob, S.-O. Troschke, Matrix representation of quaternions, Linear Algebra and its Applications 362 (2003) 251–255.  doi:doi:10.1016/s0024-3795(02)00535-9.
  * [11] G. ES, S. WJ, A joint coordinate system for the clinical description of three-dimensional motions: application to the knee, J Biomech Eng. 105 (2) (1983) 136–44.
  * [12] K. Shoemake, Animating rotation with quaternion curve, SIGGRAPH ’85 proceedings, Computer Graphics 19 (3) (1985) 245–255.
  * [13] Y. Liang, D. Ledesma, D. Wu, [online] vtcs: virtuality coaching based on unity 3d, https://www.youtube.com/watch?v=OQySt2i8dzo&feature=youtu.be, cited May 8, 2018.
  * [14] M. Tiwana, S. Redmond, N. Lovell, A review of tactile sensing technologies with applications in biomedical engineering, Sensors and Actuators A: Physical 179 (2012) 17–31.  doi:10.1016/j.sna.2012.02.051.
  * [15] D. Ledesma, W. Baker, Y. Liang, [online] vtcs: Deployment of vtcs over commodity hardware, https://youtu.be/_bB8K627JZk, cite July 04, 2018.
  * [16] G. Vasan, P. M. Pilarski, Learning from demonstration: Teaching a myoelectric prosthesis with an intact limb via reinforcement learning, in: 2017 International Conference on Rehabilitation Robotics (ICORR), 2017, pp. 1457–1464.  doi:10.1109/ICORR.2017.8009453.
  * [17] E. Lee, Cyber physical systems: Design challenges, in: Proc. of IEEE International Symposium on Object Oriented Real-time Distributed Computing (ISORC), 2008.
  * [18] L. Sha, S. Gopalakrishnan, X. Liu, Q. Wang, Cyber-physical systems: A new frontier, in: Proc. of IEEE International Conference on Sensor Networks, Ubiquitous and Trustworthy Computing, 2008.
  * [19] Y. Hu, M. Patel, D. Sabella, N. Sprecher, V. Young, Mobile edge computing a key technology towards 5g, European Telecommunications Standards Institute (ETSI) White Paper.
  * [20] O. Mkinen, Streaming at the edge: Local service concepts utilizing mobile edge computing, in: The 9th International Conference on Next Generation Mobile Applications, Services and Technologies, 2015.
  * [21] P. Langfelder, S. Horvath, Wgcna: an r package for weighted correlation network analysis, BMC Bioinformatics 9 (2008) 1471–2105.  doi:10.1186/1471-2105-9-559.
  * [22] N. Alharbi, Y. Liang, D. Wu, Extended-kalman-filter preprocessing technique for gesture recognition, in: 2nd IEEE/ACM CHASE, Philadelphia, USA, 2017.
  * [23] D. Ledesma, D. Ledesma, R. Slaughter, D. Wu, Z. Guo, Y. Liang, Kinetic gesture recognition and choreography, in: IEEE BigDataService 2018, Bamberg, Germany, 2018\.
  * [24] Y. Liang, Z. B. Guo, D. L. Wu, N. Fell, A. Clark, Virtual taiji system - an innovative modality for rehabilitation, in: Annual BSEC Conference at Oak Ridge National Laboratory Collaborative Biomedical Innovations, 2015.
  * [25] C. Davis, D. Ledesma, R. Slaughter, D. Wu, Z. Guo, Y. Liang, Kinetic data processing for gesture recognition, in: IEEE BigDataService 2018, Bamberg, Germany, 2018.
  * [26] Y. Liang, D. Ledesma, C. Davis, R. Slaughter, D. Wu, [online] vtcs: preprocessed and raw kinematic data (front view), https://www.youtube.com/watch?v=KFBV_EBTEQw, cited March 2, 2018.
  * [27] Y. Liang, D. Ledesma, C. Davis, R. Slaughter, D. Wu, [online] vtcs: preprocessed and raw kinematic data (front and back view), https://youtu.be/Ee31SdhXxXc, cited March 2, 2018.
  * [28] J. Ambrosio, A. Kecskemethy, Multibody dynamics of biomechanical models for human motion via optimization, Multibody Dynamics: Computational Method and Application, J.C. Garcia Orden et al. (Eds) 6 (2007) 245–272.  doi:10.1007/978-1-4020-5684-0-12.
  * [29] E. Eich-Soellner, C. Führer, Numerical Methods in Multibody Dynamics, Teubner, Stuttgart, 2008.
  * [30] P. Moreira, U. Lugrís, J. Cuadrado, P. Flores, Biomechanical models for human gait analyses using inverse dynamics formulation, Sociedade Portuguesa de Biomecânica, 2013.
URL http://hdl.handle.net/1822/23057

  * [31] Y. Liang, M. Szularz, L. T. Yang, Finite-element-wise domain decomposition iterative solvers based on polynomial preconditioning, Mathematical and Computer Modeling 58 (1-2) (2013) 421–437.  doi:10.1016/j.mcm.2012.11.017.
  * [32] J. Wittenburg, Dynamics of Multibody Systems, Berlin, Springer, 2008.
  * [33] R. Kalman, A new approach to linear filtering and prediction problems, Journal of Basic Engineering 80 (1960) 35.  doi:doi:10.1115/1.3662552.
  * [34] G. Einicke, Smoothing, Filtering and Prediction: Estimating the Past, Present and Future, Rijeka, Croatia: Intech, 2012.
  * [35] P. Zarchan, H. Musoff, Fundamentals of Kalman Filtering: A Practical Approach, American Institute of Aeronautics and Astronautics, Incorporated, 2000.
  * [36] J. Butepage, M. J. Black, D. Kragic, H. Kjellstrom, Deep representation learning for human motion prediction and classification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6158–6166.
  * [37] D. Ledesma, Y. Liang, [online] playlist of phantom legs induced by moving arms, https://www.youtube.com/playlist?list=PL8VO3nxhbh0IngDo5ij0PkKxK5uFbtXE0, cite December 08, 2018.
  * [38] D. Ledesma, Y. Liang, [online] vtcs: Construct phantom legs according to arms’ movement using 7-layer mlp, https://youtu.be/BFW3JVI0dKM, cite December 08, 2018.
  * [39] X. Qiao, C. Peng, Z. Liu, Y. Hu, Word-character attention model for chinese text classification, International Journal of Machine Learning and Cybernetics (2019) 1–17.
  * [40] J. Li, M.-T. Luong, D. Jurafsky, A hierarchical neural autoencoder for paragraphs and documents, arXiv preprint arXiv:1506.01057.
  * [41] Y. Seo, K. Shin, Hierarchical convolutional neural networks for fashion image classification, Expert Systems with Applications 116 (2019) 328–339.
  * [42] D. Ledesma, Y. Liang, [online] vtcs: Construct phantom legs according to arms’ movement using autoencoder, https://youtu.be/1aHJQyST9k0, cite December 08, 2018.
  * [43] D. Ledesma, Y. Liang, [online] vtcs: Construct phantom legs according to arms’ movement using two-thead vhnn autoencoder, https://youtu.be/pVtzw0dJduA, cite December 08, 2018.
  * [44] D. Ledesma, Y. Liang, [online] vtcs: Construct phantom legs according to arms’ movement using four-thead vhnn autoencoder, https://youtu.be/dGPm4z7vibI, cite December 08, 2018.