# Field dynamics inference for local and causal interactions

###### Abstract

Inference of fields defined in space and time from observational data is a
core discipline in many scientific areas. This work approaches the problem in
a Bayesian framework. The proposed method is based on a Gaussian process prior
for fields and demonstrates how to reconstruct the field together with its
prior correlation structure from data. The prior model of the correlation
structure is described in a non-parametric fashion and solely builds on
fundamental physical assumptions such as space-time homogeneity, locality, and
causality. These assumptions are sufficient to successfully infer the field
and its prior correlation structure from noisy and incomplete data of a single
realization of the process as demonstrated via multiple numerical examples.

###### keywords:

Information theory, Probability theory, Stochastic processes, Data analysis,
Time series analysis, Bayesian methods

Philipp Frank* Reimar Leike Torsten A. Enßlin

Ph. Frank, R. Leike, PD. Dr. T. A. Enßlin
Max-Planck Institut für Astrophysik, Karl-Schwarzschild-Straße 1, 85748,
Garching, Germany
Ludwig-Maximilians-Universität München, Geschwister-Scholl-Platz 1, 80539,
München, Germany
<EMAIL_ADDRESS>
##  1 Introduction

Modeling as well as inferring quantities defined in space and time on the
basis of observational data thereof has always been at the very core of many
scientific areas. In recent years, astrophysical imaging began to become
sensitive to the temporal dimension, in addition to the spatial ones. This is
due to the fact that although large astrophysical objects such as galaxies
appear to be static on observational timescales, small objects such as stars
and binary black holes exhibit periodic modulations of the emission on
observable timescales.

In addition, the spatio-temporal correlation structure of non-astronomical
systems plays a central role in the calibration of modern telescopes. In
particular the temporal variability of these systems is used in order to
identify and distinguish them from the typically static astronomical object of
interest. As a prominent example, modern radio telescopes such as LOFAR [1] or
the upcoming SKA are limited in resolution by the deflection of incoming radio
signals due to the ionosphere. The strength of these distortions ultimately
depends on the electron density of the ionosphere. As this density is not
known for all observed locations $x$ at time $t$, it has to be inferred along
with the incoming flux. Typically, the electron density is probed via
observing a calibration target with known flux at location $x^{\prime}$ and
time $t^{\prime}$. Therefore, it is also necessary to make a statement about
the correlation structure of the electron density in order to extrapolate the
information gained at $(t^{\prime},x^{\prime})$ to the space-time location
$(t,x)$ where the actual observation is made.

In order to tackle these as well as other inference problems of this kind in
space and time, we have to perform inference of continuous quantities, or
fields, from a finite set of measurement data. This problem is in general ill-
posed, as we aim to constrain infinite degrees of freedoms (dofs) on finite,
usually also noisy, measurements. Consequently we rely on Bayesian inference,
more precisely on Information Field Theory (IFT) [2], and use this language to
encode prior knowledge about the system under consideration. Typically, this
prior knowledge is incomplete, meaning that there exist a set of unknown
hyper-parameters, such as the spatio-temporal prior correlation structure,
which have to be inferred along with the field of interest. We will outline in
this paper how physically motivated concepts such as spatio-temporal
homogeneity, locality as well as causality can be encoded into the prior
correlation structure. Furthermore, we will demonstrate that the resulting
prior is restrictive enough to perform inference on the basis of noisy and
incomplete data of a single realization of the process, while still being
flexible enough to capture complicated correlation structures.

Traditionally there exist two different pictures on random fields in space-
time, one results in space and time being treated separately [3, 4], while the
other models the field as defined over a single space, namely space-time [5,
6] (see e.g. [7] for an extensive discussion). In this work we rely on the
latter picture. Consequently the corresponding inference problems can be
regarded as the task of inferring a field defined on a single space, given a
finite amount of measurements in this space.

To this end, in section 2 we start with a brief introduction to IFT and the
notation used in this work. In section 3 we discuss how to encode our prior
knowledge about the field and its correlation structure into a joint prior
distribution thereof. This prior is then used in section 4 in order to solve
the corresponding inference problem and the performance of the resulting
algorithm is demonstrated in section 5. Ultimately, in section 6, we conclude
the paper with a brief summary of the proposed concepts.

##  2 Information Field Theory

In this work, we deal with fields $s^{x}$ defined on some domain $\Omega$ with
$\Omega\subset\mathds{R}^{n+1}$, where $n\in\mathds{N}$ denotes the number of
spatial dimensions and $x\equiv(t,\mathbf{x})$ labels the space-time
coordinates of $\Omega$. In order to construct a probability distribution for
$s$ we define a scalar product for fields as

$a^{\dagger}b\equiv\int_{\Omega}a^{*}_{x}\ b^{x}\ \mathrm{d}x\ .$ (1)

Consequently, applications of linear operators are denoted as

$b^{x}=(Oa)^{x}=O^{x}_{\;x^{\prime}}\
a^{x^{\prime}}=\int_{\Omega}O^{x}_{\;x^{\prime}}\ a^{x^{\prime}}\
\mathrm{d}x^{\prime}\ ,$ (2)

where we also introduced the continuous version of the Einstein sum
convention. This can be used in order to define for example a Gaussian
distribution with mean $m$ and covariance $D$ for a field $s$ as

$\displaystyle P(s)$ $\displaystyle=\mathcal{G}(s-m,D)$
$\displaystyle\equiv\frac{1}{\left|2\pi D\right|^{\frac{1}{2}}}\
e^{-\frac{1}{2}(s-m)^{\dagger}D^{-1}(s-m)}\ ,$ (3)

where $|\bullet|$ denotes the functional determinant. (For further details see
e.g. [2]). These definitions are sufficient in order to define probability
distributions for fields. To perform inference, however, we additionally need
to introduce the concept of a measurement. Before we turn to the actual
inference problem of reconstructing a field together with its correlation
structure, we outline the simpler problem of inferring a field, given its
correlation structure first. For the sake of simplicity, in addition, we stick
to linear measurements of fields throughout this work. However a
generalization to non-linear measurements is possible (see e.g [8]).

###  2.1 Linear Measurements

Consider a linear measurement of $s$ with additive Gaussian distributed noise
$n$, defined as

$d^{i}=R^{i}_{x}s^{x}+n^{i}\ .$ (4)

Here the measurement response $R$ maps from the continuous space $\Omega$,
called signal space, to the discrete space of the measurements $d$, called
data space. If we assume $s$ to be a priori distributed as

$P(s)=\mathcal{G}(s,S)\ ,$ (5)

with zero mean and known covariance $S$ and furthermore denote $P(n)$ as

$P(n)=\mathcal{G}(n,N)\ ,$ (6)

with known covariance $N$, the resulting inference problem can be solved
analytically as the posterior distribution is again a Gaussian distribution
and given as

$P(s|d)=\mathcal{G}(s-m,D)\ ,$ (7)

with

$\displaystyle D$ $\displaystyle=(R^{\dagger}N^{-1}R+S^{-1})^{-1}\ ,$ (8)
$\displaystyle m$ $\displaystyle=Dj=D\ R^{\dagger}N^{-1}d\ .$ (9)

This resembles the solution of the famous Wiener filter equations [9], applied
to a field theoretical setting.

Note that although the posterior solution is available in closed form
analytically, for many interesting combinations of $R,N,S$ the posterior
covariance $D$ cannot be calculated analytically due to the operator
inversion. Consequently the application of $D$ is ultimately approximated
numerically on a discretized grid, and via linear solvers such as the
conjugate gradient method. All results in this work are implemented using the
Python package NIFTy [10].

###  2.2 Useful coordinate transformation

From a theoretical perspective the above linear inference problem is solved
completely. In practice, however, an equivalent reformulation of the above
turns out to be typically more efficient numerically, and also provides a
different perspective on the inference problem.

Consider a generic coordinate transformation of the form

$s=F[\xi_{s}]\ ,$ (10)

with $F$ invertible. Note that this is a coordinate transformation in the
infinite dimensional configuration space of $s$. Using the transformation rule
for probabilities

$P(s|d)\ \mathcal{D}s=P(\xi_{s}|d)\ \mathcal{D}\xi_{s}\ ,$ (11)

we get that

$P(\xi_{s}|d)=P(s|d)|_{s=F[\xi_{s}]}\left|\frac{\partial
F[\xi_{s}]}{\partial\xi_{s}}\right|\ .$ (12)

Furthermore if we consider a linear transformation

$s=G\ \xi_{s}\ ,$ (13)

with $G$ such that

$S=GG^{\dagger}\ ,$ (14)

we find after some calculation that the transformed posterior $P(\xi_{s}|d)$
reads

$P(\xi_{s}|d)=\mathcal{G}(\xi_{s}-\tilde{m},\tilde{D})\ ,$ (15)

with

$\displaystyle\tilde{D}$
$\displaystyle=(G^{\dagger}R^{\dagger}N^{-1}RG+\mathds{1})^{-1}\ ,$ (16)
$\displaystyle\tilde{m}$ $\displaystyle=Dj=\tilde{D}\
G^{\dagger}R^{\dagger}N^{-1}d\ .$ (17)

Furthermore, the transformed prior distribution $P(\xi_{s})$ turns out to be

$P(\xi_{s})=\mathcal{G}(\xi_{s},\mathds{1})\ .$ (18)

The first observation is that the spectrum of the transformed posterior
covariance $\tilde{D}$ is bounded from above by $1$. This renders numerical
inversion of this operator to be faster and also typically more stable (see
[11] for further details). In addition, the transformed dofs $\xi_{s}$ are
independent and identically distributed (i.i.d.), and eq. (13) defines a
generative process for $s$. In a space time setting, this allows for an
interpretation of $\xi_{s}$ as an external force (excitation) applied to a
system, while $G$ models the dynamic response (Green’s function) of the system
to these excitations. As our goal is to construct a physically motivated prior
for $(s,S)$, this formulation allows to directly encode our prior belief about
how a typical response of a dynamic system to excitations should look like. We
will therefore proceed to formulate a prior for $(\xi_{s},G)$ rather than for
$(s,S)$ as it appears to be a more convenient choice. Note that the original
quantities $(s,S)$ can at any point be recovered using eqs. (13) and (14).

##  3 Prior

In the introduction we proposed three different concepts we aim to encode into
a prior for $G$ in order to reduce its complexity. These are

  * •

Statistical space-time homogeneity

  * •

(Spatio-) temporal causality

  * •

Locality.

The first concept implies that $G$ (and consequently also $S$) is the same for
all space-time locations. I.e.

$G^{x}_{x^{\prime}}=g(x-x^{\prime})\ .$ (19)

From the Wiener-Khinchin theorem [9] we get that $G$ is diagonal in Fourier
space and its eigen-spectrum is given via the Fourier transformed $g$. This
reads

$G^{k}_{k^{\prime}}\propto\delta(k-k^{\prime})\ (\mathcal{F}g)^{k}\ ,$ (20)

with

$\mathcal{F}^{k}_{x}\equiv e^{ikx}\ ,$ (21)

where $k=(\omega,\mathbf{k})$ denotes the coordinates of the Fourier space
corresponding to $\Omega$. This assumption greatly reduces the complexity of
the inference problem since we have to infer two fields $\xi_{s}$ and $g$,
rather than one field and one linear operator.

The second concept, causality, is introduced via an additional constraint on
$G$. As $G$ should model the response of the system to the excitations
$\xi_{s}$, it should not contain a response at times before an excitation
happens. This can be formulated mathematically as

$G^{x}_{x^{\prime}}=\Theta(t-t^{\prime})\ g(x-x^{\prime})\ ,$ (22)

where $\Theta$ denotes a step function in time. This trivially ensures that no
response happens before an excitation.

In space-time, the physical concept of causality also implies a maximal finite
propagation speed of interactions. This leads to propagation within a light
cone, as depicted in figure 1. Therefore, given a maximal propagation speed
$c$, we can restrict the reconstruction to Greens functions that are non-zero
only within the light cone. We can implement this constraint by extending eq.
22 as

$G^{x}_{x^{\prime}}=\Theta(t-t^{\prime})\ \Theta(l^{2})\ g(x-x^{\prime})\ ,$
(23)

where

$l^{2}=(t-t^{\prime})^{2}-(\mathbf{x}-\mathbf{x}^{\prime})^{\dagger}\mathbf{C}^{-1}(\mathbf{x}-\mathbf{x}^{\prime})\
.$ (24)

This ensures that the propagator is only non-zero within the light cone. In
general we might expect different maximal propagation speeds in different
directions, and consequently $\mathbf{C}$ becomes a symmetric tensor. In case
propagation is isotropic in space with speed $c$ we have
$\mathbf{C}_{ij}=c^{2}\delta_{ij}$. Note that even in cases where we do not
know $\mathbf{C}$, for example if the medium in which the interaction is
realized is unknown, such a constraint can also be useful if we elevate
$\mathbf{C}$ to be an additional unknown parameter that has to be inferred.
Since $\mathbf{C}$ is assumed to be the same for all scales, an inference
algorithm can use large scale information, where the signal to noise ration
(SNR) is usually higher, to determine $\mathbf{C}$, which then effectively
increases the SNR for smaller scales. A detailed description how $\mathbf{C}$
enters the reconstruction can be found in appendix A.

In order to encode the last concept, locality, we first have to revisit some
properties of Green’s functions. Consider a system that can be described via

$\mathcal{L}s=0\ ,$ (25)

where $\mathcal{L}$ is a linear differential operator. The response $G$ of
such a system to an external force has to fulfill

$\left(\mathcal{L}G\right)^{x}_{x^{\prime}}=\delta(x-x^{\prime})\ .$ (26)

In order to give rise to a homogeneous $G$, $\mathcal{L}$ also has to be
homogeneous and therefore can be written as

$\mathcal{L}^{x}_{x^{\prime}}=q(\partial_{x^{\mu}})\ \delta(x-x^{\prime})\ ,$
(27)

where $q(\partial_{x^{\mu}})$ is a differential operator encoding function,
and $\mu$ is an index of the coordinates $x=(t,\mathbf{x})$. Its Fourier space
representation takes the form

$\mathcal{L}^{k}_{k^{\prime}}=f^{k}\ \delta(k-k^{\prime})\ ,$ (28)

where we also introduced the complex valued operator encoding function
$f_{k}\equiv q(ik)$. Consequently $G$ takes the form

$G^{k}_{k^{\prime}}=\frac{\delta(k-k^{\prime})}{f^{k}}\ ,$ (29)

and therefore the eigen-spectrum of $G$ reads

$g^{k}=\frac{1}{f^{k}}\ .$ (30)

It turns out that locality can be encoded more intuitively in terms of a prior
for $f$. As we can see in Figure 2, the locality of a response of the system
to an excitation is related to the order in the derivatives of the
differential operator $\mathcal{L}$. Low order derivatives result in an almost
instantaneous response of the system while higher order derivatives lead to an
apparent non-local response. Therefore we seek to formulate a prior for $f$
such that lower order derivatives are a priori favoured. Nevertheless it
should be possible that $f$ can deviate from this assumption if there is
enough evidence in the data to support this. What we aim to do is to impose a
certain degree of smoothness for $f$. We can construct a prior for $f^{k}$
such that it punishes second and higher order derivatives of $k$. One way to
do so is to propose a Gaussian process prior

$P(f)=\mathcal{G}(f,T)=\frac{1}{\left|2\pi
T\right|^{\frac{1}{2}}}e^{-\frac{1}{2}f^{\dagger}T^{-1}f}\ ,$ (31)

where the exponential factor reads

$-\frac{1}{2}f^{\dagger}T^{-1}f=-\frac{1}{2\sigma^{2}}\int|\triangle_{k}f^{k}|^{2}dk\
,$ (32)

where $\Delta_{k}$ denotes the Laplace operator and $\sigma$ is an overall
scaling parameter in order to steer the strength of this prior. For further
details see [12, 13]. Consequently an $f$ being a differentiable function of
its arguments is a priori favoured, and therefore the most local configuration
that is consistent with the data is the one selected during reconstruction.
This is perfectly motivated physically, since we know that for direct
(classical) interactions, locality is a fundamental concept. Consequently
apparent non-localities in the response indicate additional hidden layers
within the system, since the excitation does not influence the observed
quantity $s$ directly, but couples to an additional unobserved quantity which
then couples to $s$. Such a configuration is of course possible and perfectly
realizable, but if the system can also be described via a more direct
interaction, the corresponding model is favoured.

All in all, we can combine the above concepts to end up with a generative
description of our prior which takes the form

$s=G\ \xi_{s}=\mathcal{F}^{-1}\ \widehat{g}\ \mathcal{F}\ \xi_{s}\ ,$ (33)

where $\widehat{g}$ denotes a diagonal operator in Fourier space with $g_{k}$
on its diagonal. Furthermore

$g=\mathcal{F}\ \widehat{\Theta(t)}\ \widehat{\Theta(l^{2})}\
\mathcal{F}^{-1}\frac{1}{f}\ ,$ (34)

with $f$ being a priori distributed according to eq. (31) while here the
$\widehat{\Theta}$ are diagonal operators in position space.

###  3.1 Prior distributions for excitations

Since the set of Gaussian distributions is closed under linear maps, we get
that if $\xi_{s}$ is a priori Gaussian distributed, the prior for $s$ is also
a Gaussian. In particular for

$P(\xi_{s})=\mathcal{G}(\xi_{s},\mathds{1})\ ,$ (35)

we got that $P(s)$ is a Gaussian with zero mean and covariance
$S=GG^{\dagger}$ (see eq. (14)).

From the perspective of $s$ being the result of a dynamic response $G$ to
external excitations $\xi_{s}$ it is not necessary that $\xi_{s}$ is Gaussian
distributed. In fact, in many applications it might be more realistic to
define a different prior for the excitations. In this case, the prior
statistics of $s$ is also non-Gaussian, but can be calculated from
$P(\xi_{s})$. In order to demonstrate the implications of a non-Gaussian
prior, in section 5, we will show results for the inference problem in cases
where $\xi_{s}$ is distributed according to an inverse-gamma distribution at
each location. This prior is typically used as a sparsity prior, and in our
case results in a system that is subject to sparse external excitations. Of
course, many other prior distributions are also reasonable and important, but
for the sake of simplicity we will stick to $\xi_{s}$ being either Gaussian or
inverse-gamma distributed in the examples.

##  4 Inference

In section 2.1 we already discussed the inference problem in the case of a
linear measurement and a Gaussian prior with known covariance. Now, with the
appropriate prior for the covariance at hand, we can set up the task of
inferring the covariance together with the field, given observational data
about the field. To this end, consider again a linear measurement as defined
in eq. (4) which in the transformed coordinates $(G,\xi_{s})$ takes the form

$d=R\ G_{f}\ \xi_{s}+n\ ,$ (36)

where $G_{f}$ denotes the Green’s function as a function of $f$, with an
eigen-spectrum as defined in eq. (34). If we again assume $n$ being Gaussian
distributed with zero mean and covariance $N$ we get that the joint
distribution of $(d,\xi_{s},f)$ reads

$P(d,\xi_{s},f)=\mathcal{G}(d-R\ s(f,\xi_{s}),N)\ \mathcal{G}(f,T)\
P(\xi_{s})\ ,$ (37)

where we also introduced $s(f,\xi_{s})\equiv G_{f}\ \xi_{s}$ to indicate the
dependency of $s$ on the quantities of interest. The posterior
$P(\xi_{s},f|d)$ is proportional to the joint distribution up to a factor that
only depends on $d$ since

$P(d,\xi_{s},f)=P(\xi_{s},f|d)\ P(d)\propto P(\xi_{s},f|d)\ .$ (38)

It turns out that this posterior is intractable, due to the non-linear
dependency of $s$ on $(f,\xi_{s})$. Consequently the corresponding inference
problem cannot be solved analytically and we have to rely on a numerical
approximation. In this work, we use a variational approximation algorithm
called Metric Gaussian Variational Inference (MGVI) [14] where we approximate
the true posterior with a Gaussian distribution in order to get an estimate
for the mean and the covariance of the posterior.

###  4.1 Variational Inference

In general, variational inference can be described as the task of
approximating one probability distribution $P(x)$ for some quantity $x$ with
another distribution $Q_{\sigma}(x)$ which, in addition, is defined up to a
set of parameters $\sigma$. Approximation is then achieved via minimizing the
Forward Kullbach-Leibler divergence (KL) with respect to $\sigma$. The KL is
defined as

$\displaystyle\mathrm{KL}(Q_{\sigma},P)$ $\displaystyle\equiv\int
Q_{\sigma}(x)\log\left(\frac{Q_{\sigma}(x)}{P(x)}\right)\mathrm{d}x$
$\displaystyle=\left<H_{P}\right>_{Q_{\sigma}}-\left<H_{Q_{\sigma}}\right>_{Q_{\sigma}}\
,$ (39)

where we also introduced the so called Information Hamiltonian $H$ defined as

$H_{P}=-\log(P)\ .$ (40)

In our case, the approximate distribution $Q$ is chosen to be a Gaussian
distribution in $a=(f,\xi_{s})$ with mean $m$ and covariance $A$. For many
relevant inference problems, and also for the one studied in this work, this
approximation cannot be performed analytically as typically
$\left<H_{P}\right>_{Q_{\sigma}}$ cannot be calculated analytically.
Therefore, similar to the linear problem studied in section 2.1, we choose an
appropriate discretization for the space-time domain $\Omega$, and perform a
variational approximation of the corresponding discrete problem. It turns out
that in order to achieve a reasonable resolution in space-time, the inference
problems can become very high dimensional. As an example, in section 5, we
show an application for a discretized 1+1 dimensional space-time with a
resolution of $256\times 200$ pixels. This yields that the number of dofs in
$a$ is $2\times 256\times 200\propto 10^{5}$ and consequently the number of
entries in $A$ are $\propto 10^{10}$ which renders an explicit representation
of $A$ on a computer to be inefficient generally. Therefore, following [14],
we avoid an explicit representation by setting it to be equal to the inverse
Fisher Metric $\mathcal{M}^{-1}$ [15] of the posterior, evaluated at $m$. For
the posterior distribution as defined in eqs. (37) and (38), together with a
Gaussian prior for $\xi_{s}$ with zero mean and unit covariance, we get that
$\mathcal{M}$ takes the form

$\displaystyle\mathcal{M}=$
$\displaystyle\left[\left(\begin{matrix}\frac{\partial s}{\partial\xi_{s}}\\\
\frac{\partial s}{\partial
f}\end{matrix}\right)N^{-1}\left(\begin{matrix}\frac{\partial
s}{\partial\xi_{s}}&\frac{\partial s}{\partial
f}\end{matrix}\right)+\left(\begin{matrix}\mathds{1}&0\\\
0&T^{-1}\end{matrix}\right)\right]_{(\xi_{s},f)=m}\ .$ (41)

Using the definition of $s$ (eq. (33)) we get that

$\displaystyle\frac{\partial s}{\partial\xi_{s}}$
$\displaystyle=\mathcal{F}^{-1}\ \widehat{g}\ \mathcal{F}\ ,$ (42)
$\displaystyle\frac{\partial s}{\partial f}$
$\displaystyle=\mathcal{F}^{-1}\widehat{\mathcal{F}\xi_{s}}\mathcal{F}\
\widehat{\Theta(t)}\ \widehat{\Theta(l^{2})}\
\mathcal{F}^{-1}\widehat{\nicefrac{{-1}}{{f^{2}}}}\ .$ (43)

Therefore, together with the definition of $T$ (eq. (32)) we see that
$\mathcal{M}$ has an implicit representation, i.e. it can be applied solely
using Fourier transformations and diagonal operations, avoiding the explicit
storage of this matrix at any point. Consequently the application of $A$ can
be achieved via linear solvers. In addition, the structure of $\mathcal{M}$
allows for an efficient sampling of the approximate distribution $Q$. This is
another important property since the KL involves expectation values w.r.t $Q$,
which can be approximated via samples from $Q$. Ultimately, the Fisher metric
is also a measure for the local curvature of the KL and therefore enables us
to use second order optimization schemes to solve the corresponding
optimization problem in $m$.

As discussed in the previous section, in some cases space-time causality can
only be imposed if we also infer the propagation speed $\mathbf{C}$. To do so,
we notice that given a prior for $\mathbf{C}$, the above still applies with
the extension that $s$ is now also a function of $\mathbf{C}$. Therefore the
Fisher metric gets an additional entry with the same structure as in eq.
(4.1), for the corresponding gradient $\nicefrac{{\partial
s}}{{\partial\mathbf{C}}}$ (See appendix A for further details).

All in all, the solution strategy as defined in MGVI, starting from an random
initialized $m$, can be summarized as:

  * •

Using $\mathcal{M}$, evaluated at the current approximate mean $m$, draw a set
of samples $\\{a^{*}\\}$ from the approximate distribution $Q$.

  * •

With these samples, calculate an estimation for the current value of the KL
and its gradient.

  * •

Together with the metric $\mathcal{M}$ perform a second order Newton
minimization step in order to get a new estimate for $m$.

  * •

Repeat this procedure by re-evaluating everything using the updated $m$, until
convergence.

##  5 Application

In order to demonstrate the applicability of our method we apply it to several
synthetic data examples. First, we demonstrate the performance of the
algorithm in a one dimensional setting, where we only aim to infer the Green’s
function of the system, given the excitations and data. In the second example
we perform a reconstruction of the excitations as well as the Green’s function
from data alone, in a 1+1 dimensional setting. In the last example we add
another level of complexity via non-Gaussian statistics in the excitations.
Hereby we perform source detection, the task of inferring sparse excitations
at various locations in space-time, in a case where also the Green’s function
is unknown.

All examples were implemented using the NIFTy software package in its version
5 [16]. We included an implementation of the here introduced prior structure
for the Green’s function in this publicly available package.

###  5.1 Temporal evolution

In our first application, shown in figure 3, we aim to demonstrate the
inference of the dynamics encoding field $f$ alone in a one dimensional
setting where there is only a temporal evolution to be reconstructed. The
excitation field $\xi_{s}$ is known during inference. We generate synthetic
data according to eq. 36, with $R$ being the identity. The excitations were
drawn from an inverse gamma distribution to model known, sparse excitations of
the system (e.g. in a laboratory setting where the unknown system is driven
via sparse excitations). The synthetic signal $s$ as well as corresponding
data $d$ is shown in the top-left panel of figure 3. The dynamic operator used
to generate signal and data is of the form:

$\mathcal{L}=(\partial_{t}^{2}+m^{2}+\gamma\partial_{t})^{5}\
(\partial_{t}^{2}+\tilde{m}^{2}+\tilde{\gamma}\partial_{t})\ ,$ (44)

with $(m,\gamma,\tilde{m},\tilde{\gamma})=(0.6,0.23,0.2,0.03)$. The goal is
now to reconstruct the differential operator encoding field $f$ given $d$ and
$\xi_{s}$. The results are shown in figure 3 as well.

We see that the reconstruction of the Green’s function behaves as expected and
is in agreement with the ground truth in the temporal domain, within
uncertainties. Due to the fact that the reconstructed dynamics is uncertain,
the recovered signal also has uncertainty although the excitations are known.
The reconstructed Green’s function indicates that it is indeed possible to
reconstruct an apparent non-local response of the system (due to higher order
derivatives in this setup) since the true propagator as well as its
reconstruction show oscillations that grow in the beginning of the propagator
before decaying exponentially. We also notice that there is relatively high
uncertainty in the first timesteps of the response $G_{t}$. This is caused by
the low initial response to excitations of the true propagator. The initial
part of the reconstructed propagator is purely dominated by noise and thus
only constrained up to the noise level (the standard deviation of the noise
$\sigma_{n}$ is set to be $\sigma_{n}=20$ and the temporal domain is
discretized via 512 equidistant pixels).

We also notice that the posterior solution for the spectrum levels out for
high-frequency modes (large $\omega$) below the signal to noise ratio, while
the true spectrum continues to decay (ultimately also the true spectrum levels
out in the numerical example due to the finite size of the considered space).
The uncertainty increases in this region, but not enough to capture the true
solution. Here we notice the limitations of the variational inference, which
provides a local approximation of the posterior with a Gaussian. Consequently
the true uncertainty might be underestimated, as in this case. However this
deviation occurs orders of magnitudes below the peak of the spectrum and
therefore has only a barely visible effect on the reconstruction of the
signal. One way to allow for a better extrapolation to higher frequencies
would be to provide a more restrictive prior for the dynamics encoding field
e.g. by defining it on a polynomial basis which is a more suitable basis for
this particular setup as the true dynamics is also described in terms of a
polynomial. However we aim to provide a general and less restrictive approach
here capable of also reconstructing non-polynomial dynamics encoding functions
and consequently being less able to extrapolate to regions where we have no
information provided by data.

To further quantify the reconstruction error, we also investigate the residual
as well as the corresponding posterior uncertainty for the signal $s_{t}$ and
the time representation of the Greens function $G_{t}$ (see figure 4). The
residual is defined as the difference between the true solution and the
posterior mean of the reconstruction. In addition to the case of
$\sigma_{n}=20$, we also show the residuals and uncertainties for various
other noise levels. For a better comparison, no other changes where made
during reconstruction. In particular also the random number generator used to
generate the synthetic data as well as the approximate posterior samples
during reconstruction was seeded with the same random seed for all runs. We
notice that the posterior uncertainty appears to be on a reasonable scale as
the residual is within the one or two sigma confidence interval for almost all
cases. In addition, we notice that the posterior uncertainty of the signal $s$
is particularly high in regions right after a strong excitation happened. This
is due to the fact that the reconstruction of the Green’s function $G$, which
is the response to these excitations, is most uncertain for the first
timesteps. This uncertainty propagates into the uncertainty of $s$.

###  5.2 Space-time evolution

In our second example, shown in figure 5, we aim to reconstruct the dynamics
as well as the excitations in a spatio-temporal (1+1 dimensional) setting from
incomplete and noisy observations. We aim to infer the dynamical field $f$,
the propagation speed $c$, and the excitations $\xi$ from noisy and incomplete
measurements $d$ of the field $s$ alone. The dynamical system used for the
generation of synthetic data is a product of a damped harmonic oscillator and
an advection-diffusion generating term. This reads

$\mathcal{L}=(\partial_{t}^{2}-c^{2}\partial_{x}^{2}+m^{2}+\alpha\partial_{t}-\beta\partial_{x})\
(\partial_{t}+\tilde{m}^{2}-\gamma\partial_{x}^{2}+\tilde{\beta}\partial_{x})\
.$ (45)

Furthermore, the excitations are Gaussian distributed with zero mean and unit
covariance from which a single realization was drawn and convolved with the
synthetic Green’s function corresponding to $\mathcal{L}$. The signal $s$, the
data $d$ and the reconstruction of $s$ are shown in figure 5 for a case with
$(c,m,\alpha,\beta,\tilde{m},\gamma,\tilde{\beta})=(0.4,0.16,-0.19,-0.05,0.1,0.5,0.2)$.
We generate the data according to eq. 36 with a linear measurement response
which partially ($\approx 25\%$) masks the observed region, and Gaussian
distributed noise with $\sigma_{n}=10$. The space-time is discretized via a
regular grid with $256\times 200$ pixels, respectively.

The reconstruction algorithm is capable of reconstructing the signal in
regions where we have observations thereof, while being relatively blind in
unobserved regions. Consequently the posterior uncertainty is higher there. In
addition, we notice that unlike the posterior mean, posterior samples
consistently fill unobserved regions. Although in these regions the samples
deviate strongly from the true signal, they obey the same statistical
properties, which is important for posterior analysis. The results indicate
that the statistical properties (and ultimately the dynamics encoding field)
are correctly reconstructed.

As a validation, in figures 7 and 7, we compare the reconstructed propagator
as well as the corresponding spectrum with the underlying ground truth. The
spectrum is comparable to the ground truth in regions with sufficient SNR
while it levels out in regions where we have no information given via data. In
addition, the reconstructed propagator also shows oscillations consistent with
the true propagator. However we notice that modes that propagate “downwards”
are reconstructed well while the weaker “upwards” propagating modes are not
reconstructed due to the fact that they are below the noise level. In
addition, we see that deviations in posterior samples of the propagator only
occur within a “cone” and remain zero outside. This is due to the fact that we
also reconstruct the maximal propagation speed of the process, which is
reconstructed to be $c\approx 0.45\ (\pm 0.08)$ enclosing the correct value of
$c=0.4$.

###  5.3 Source detection

In our last example we aim to perform source detection in the excitation
field, in a case where also the dynamic response is unknown. To demonstrate
this scenario we again generate a 1+1-dimensional synthetic example where in
this case we assume that we are only able to measure the temporal evolution of
the system at several locations. In particular we measure the temporal
evolution at 50 randomly selected locations of the space under consideration.
This results in $\approx 80\%$ of the discrete space-time being unobserved, as
the resolution is the same as in the previous example. As before, we assume
that the measurements are subject to additive Gaussian noise with
$\sigma_{n}=0.3$ and also assume the system to be at rest at $t=0$. The
resulting data is shown in the top-left panel of figure 8. The unknown
excitations are inverse gamma distributed to model strong but sparse
excitations. We infer those from measurements of the system at multiple
locations together with the dynamics encoding function $f$. The system used to
generate the data exhibits damped traveling waves described by

$\mathcal{L}=\partial_{t}^{2}-c^{2}\partial_{x}^{2}+m^{2}+\alpha\partial_{t}-\beta\partial_{x}\
,$ (46)

with $(c,m,\alpha,\beta)=(1.2,0.04,-0.013,0.005)$. From an information
theoretical point of view this setup is very similar to the previous one since
we only changed the measurement response to describe measurements of the
temporal evolution at several locations, as well as the prior for excitations
to be an inverse gamma prior. Consequently also the inference can be performed
in the same way as before.

The setup as well as the reconstruction of the field evolving in space-time
are shown in figure 8. We see that the reconstruction recovers many sources
and the corresponding propagation. The algorithm uses information from the
response of strong excitations to reconstruct the Green’s function of the
system. Due to the assumed homogeneity in space-time, this information helps
to improve the overall reconstruction in other regions. The quality of the
reconstruction of single excitations depends on the surrounding measurement
scenario.

In figures 10 and 10 we depict the dynamic propagator as well as the spectrum
and reconstructions. We can validate that the Green’s function was indeed
reconstructed correctly, within uncertainties. We conclude that the task of
source detection is possible even in cases where the underlying dynamics is
unknown, as long as the assumptions of spatio-temporal homogeneity, causality,
and locality hold.

##  6 Conclusion

In this work we considered the problem of reconstructing a Gaussian random
field $s$, defined in space and time, together with its correlation structure
$S$ from noisy and incomplete data $d$ about $s$. We have shown that this
inference problem can be reformulated to a (theoretically) equivalent problem
where we aim to infer an excitation field $\xi_{s}$ as well as the dynamic
response $G$. Ultimately the eigen-spectrum of $G$ was encoded in the dynamics
encoding field $f$ and the propagation-speed encoding parameter $\mathbf{C}$
which, together with $\xi_{s}$, denote the quantities of interest. Furthermore
we proposed a Gaussian process prior for $f$ which gives rise to a non-
parametric description of the dynamic response $G$. In addition, together with
a Gaussian prior for $\xi_{s}$, this gives rise to a Gaussian process prior
for $s$. As the proposed method is also applicable for non-Gaussian prior
distributions for $\xi_{s}$ it can also give rise to a variety of other prior
distributions for $s$. This flexibility is discussed for the example of an
inverse gamma distribution for the excitations.

To demonstrate its applicability, the proposed method is then applied to
several synthetic data examples. These include a one dimensional example where
the excitations were known and only $G$ had to be inferred, a 1+1 dimensional
example with unknown Gaussian distributed excitations as well as an example
with inverse gamma distributed $\xi_{s}$ and a measurement response that is
sparse in the spatial domain.

As we restricted the prior assumptions for $G$ to the physically motivated
concepts of space-time homogeneity, locality, and causality, the method
appears to be applicable in a wide range of problems. One particular strength
is the non-parametric formulation of the Green’s function. This becomes
important in scenarios where physical models cannot provide a simple
parametric description of evolution so far, to describe the Green’s function.
In addition, a probabilistic description of excitations is sufficient for
inference. Consequently the method is still applicable in cases where external
influences cannot be described completely.

All in all, we believe that the proposed method is capable of dealing with
current as well as upcoming inference problems involving fields defined over
space and time, arising from the context of astrophysical imaging.

##  Appendix A Light cone prior on a discretized space

As discussed in section 3 the concept of causality in space-time results in a
restriction of propagation within a light cone. In a continuous description,
this restriction is realized via a convolution with a step function of the
form

$\Theta(l^{2})=\Theta\left(t^{2}-\mathbf{x}^{\dagger}\mathbf{C}^{-1}\mathbf{x}\right)\
.$ (47)

Due to the fact that our calculations are ultimately performed on a finite
grid, this definition appears to be somewhat problematic, as it introduces
boundary effects along the edges of the step function, when realized on a
discretized space. In addition, if we aim to elevate $\mathbf{C}$ to be an
unknown parameter of the problem that has to be inferred, gradient based
methods are no longer applicable due to the fact that the gradient is zero
almost everywhere in space-time (or not defined on the boundary). Therefore we
seek to find a way to relax the sharp boundary introduced via the cone,
without loosing its useful properties. To do so we borrow an idea from quantum
field theory where it turns out that these sharp boundaries are “smeared” out
when considering them on a quantum scale. In our case the “quantum” scale can
be regarded as the resolution of the discrete representation of space-time,
although this analogy is purely artificial.

To achieve this relaxation, consider the following quantity

$\Delta=\sqrt{-l^{2}}=\sqrt{-\left(t^{2}-\mathbf{x}^{\dagger}\mathbf{C}^{-1}\mathbf{x}\right)}\
.$ (48)

It has two useful properties: For causal (including time-like as well as
light-like) points the real part of $\Delta$, $\Re(\Delta)$, is zero as the
square root is taken from a negative number. For non-causal (space-like)
points the real part becomes positive. Furthermore, for fixed $t$,
$\Re(\Delta)$ is asymptotically linear in $x$. Therefore, if we consider a
Gaussian in $\Re(\Delta)$,

$G(\Delta)\equiv\exp\left(-\frac{1}{2\sigma^{2}}\Re(\Delta)^{2}\right)\ ,$
(49)

we notice that this quantity remains one within the light cone, while it
asymptotically falls off like a Gaussian in $x$, for fixed $t$. Here $\sigma$
stands for an optional scaling parameter which controls the width of the
Gaussian. In all applications of this paper it was set to a size of a few
pixels.

Acknowledgements

We would like to thank Jakob Knollmüller, Philipp Arras and Margret Westerkamp
for fruitful discussions, as well as Martin Reinecke for his contributions to
NIFTy.

Conflict of interest

The authors declare no conflicts of interest.

## References

  * [1] M. P. van Haarlem, M. W. Wise, A. W. Gunst, G. Heald, J. P. McKean, J. W. T. Hessels, A. G. de Bruyn, R. Nijboer, J. Swinbank, R. Fallows,  _A &A_ 2013, _556_ A2.
  * [2] T. A. Enßlin,  _Annalen der Physik_ 2019, _531_ , 3 1800127.
  * [3] P. E. Protter,  In _Stochastic integration and differential equations_ , 249–361. Springer, 2005.
  * [4] W. Grecksch, P. E. Kloeden,  _Bulletin of the Australian mathematical society_ 1996, _54_ , 1 79.
  * [5] C. R. Doering,  _Physics Letters A_ 1987, _122_ , 3-4 133.
  * [6] A. J. Roberts,  _ANZIAM Journal_ 2003, _45_ 1\.
  * [7] L. Arnold,  _New York_ 1974.
  * [8] T. A. Enßlin, M. Frommert, F. S. Kitaura,  _Phys. Rev. D_ 2009, _80_ , 10 105005.
  * [9] N. Wiener,  _Extrapolation, interpolation, and smoothing of stationary time series, with engineering applications._ ,  Number ix, 163 p. in Stationary time series. Technology Press of the Massachusetts Institute ofTechnology, 1950.
  * [10] T. Steininger, J. Dixit, P. Frank, M. Greiner, S. Hutschenreuter, J. Knollmüller, R. Leike, N. Porqueres, D. Pumpe, M. Reinecke,  _Annalen der Physik_ 2019, _531_ , 3 1800290.
  * [11] J. Knollmüller, T. A. Enßlin,  _arXiv e-prints_ 2018, arXiv:1812.04403.
  * [12] N. Oppermann, M. Selig, M. R. Bell, T. A. Enßlin,  _Phys. Rev. E_ 2013, _87_ , 3 032136.
  * [13] P. Frank, T. Steininger, T. A. Enßlin,  _Phys. Rev. E_ 2017, _96_ , 5 052104.
  * [14] J. Knollmüller, T. A. Enßlin,  _arXiv e-prints_ 2019, arXiv:1901.11033.
  * [15] S.-i. Amari, H. Nagaoka,  _Methods of information geometry_ , volume 191,  American Mathematical Soc., 2007.
  * [16] P. Arras, M. Baltac, T. A. Ensslin, P. Frank, S. Hutschenreuter, J. Knollmueller, R. Leike, M.-N. Newrzella, L. Platz, M. Reinecke,  NIFTy5: Numerical Information Field Theory v5, 2019.

Figure 1: Left: Causal response of a system defined as a combination of two
damped harmonic oscillators with different masses. Right: Anti-causal response
of the system where one of the oscillations travels backwards in time. Figure
2: Excitation $\xi$ and responses of various systems defined in terms of
powers of $f$, with $f_{t}=\partial_{t}+\gamma$ and $\gamma$ small. We see
that higher order derivatives result in apparent non-local responses. Since
time derivatives are the generators of temporal translation, the case where
the response is the exponential of $-\Delta t\ f$ leads to translations by
$\Delta t$. Figure 3: Top: On the left we depict the signal (red line), the
data (blue dots) as well as 50 over-plotted posterior samples (gray). The
right panel shows the synthetic propagator (Greens function) in the temporal
domain (red) as well as corresponding posterior samples. Bottom: Left:
Excitation field used to generate the signal. Right: Natural logarithmic
spectrum of the synthetic propagator (red) and corresponding posterior
samples. Figure 4: Left: Residuals between the true signal $s_{t}$ and the
posterior mean of the reconstruction (blue line) as well as the one and two
sigma confidence intervals of the corresponding posterior uncertainty. We show
those for various different noise standard deviations $\sigma_{n}$ starting
with the highest noise level at the top to the lowest at the bottom. Right:
Corresponding residuals and confidence intervals for the temporal
representation of the dynamic Green’s function $G_{t}$ again for various noise
levels. In all inference runs, we seeded the random number generator used for
data generation and during reconstruction with the same random seed such that
the only difference in these reconstructions is a different $\sigma_{n}$.
Figure 5: Top: The left panel shows the spatio-temporal masked and noisy data,
drawn from the synthetic signal (middle panel) and the corresponding signal
reconstruction (right panel). Bottom: Natural logarithmic one-sigma posterior
uncertainty (left panel), natural logarithmic residual between the true signal
and the posterior mean (middle panel), as well as an approximate posterior
sample (right panel).

Figure 6: Top: True Green’s function (left) and corresponding posterior mean
(right). Bottom: Residual of the true Green’s function and the reconstruction
(left) and a posterior sample for the Green’s function (right).

Figure 7: Natural logarithmic spectrum of the true Green’s function (left) as
well as the corresponding posterior mean (right). Figure 8: Top: The left
panel shows the sparse and noisy measurement data, drawn from the synthetic
signal (middle panel) and the corresponding reconstruction (right panel).
Bottom: Natural logarithmic one-sigma posterior uncertainty (left panel),
natural logarithmic residual of the true signal and the corresponding
posterior mean (middle panel), as well as an approximate posterior sample
(right panel).

Figure 9: Top: True Green’s function of the process defined in eq. 46 (left)
and corresponding posterior mean (right). Bottom: Residual of the true Green’s
function and the reconstruction (left) and a posterior sample for the Green’s
function (right).

Figure 10: Natural logarithmic spectrum of the true Green’s function (left) as
well as the corresponding posterior mean (right).

Table of Contents

This work presents a Bayesian inference method to infer space-time homogeneous
stochastic processes from noisy and incomplete observations of a single
realization of the process. It relies on a reformulation in terms of external
random exitations and the dynamic response of the system and performs a joint
inference thereof by means of a Variational approximation of the posterior.


# Enhanced Robot Arm at the Edge with NLP and Vision Systems

Pascal Sikorski1, Kaleb Yu1, Lucy Billadeau2, Flavio Esposito1, Hadi
AliAkbarpour1 and Madi Babaiasl2 1 Pascal Sikorski and Kaleb Yu are
undergraduate research assistants and Flavio Esposito and Hadi Akbarpour are
associate and assistant professors, respectively with the Department of
Mathematics & Computer Science, Saint Louis University, St. Louis, MO
631032Lucy Billadeau is an undergraduate research assistant and Madi Babaiasl
is an assistant professor with the Aerospace & Mechanical Engineering
Department, Saint Louis University, St. Louis, MO 63103<EMAIL_ADDRESS>
###### Abstract

This paper introduces a ”proof of concept” for a new approach to assistive
robotics, integrating edge computing with Natural Language Processing (NLP)
and computer vision to enhance the interaction between humans and robotic
systems. Our ”proof of concept” demonstrates the feasibility of using large
language models (LLMs) and vision systems in tandem for interpreting and
executing complex commands conveyed through natural language. This integration
aims to improve the intuitiveness and accessibility of assistive robotic
systems, making them more adaptable to the nuanced needs of users with
disabilities. By leveraging the capabilities of edge computing, our system has
the potential to minimize latency and support offline capability, enhancing
the autonomy and responsiveness of assistive robots. Experimental results from
our implementation on a robotic arm show promising outcomes in terms of
accurate intent interpretation and object manipulation based on verbal
commands. This research lays the groundwork for future developments in
assistive robotics, focusing on creating highly responsive, user-centric
systems that can significantly improve the quality of life for individuals
with disabilities.

##  I INTRODUCTION

Figure 1: The architecture of the proposed system. Initially, a user’s spoken
input is captured and processed by VOSK Speech to Text. This text is then fed
into GPT API, which interprets the text and generates a response in JSON
format. Subsequently, this JSON output is translated into a ROS (Robot
Operating System) Command, which acts as a bridge between high-level commands
and robotic control. Finally, the ROS Command triggers the desired Robot
Action, completing the process from human speech to robotic operation.

The advent of assistive robotics [1] has marked a significant milestone in
leveraging technology for augmenting human capabilities, especially for
individuals with disabilities [2]. These robotic systems offer an
unprecedented potential to assist in daily activities [3, 4], thereby
enhancing the quality of life and fostering greater independence for people
facing physical challenges. The integration of robotics in assistive
technologies ranges from sophisticated prosthetic limbs [5] to autonomous
wheelchairs and robotic arms [6] designed for home and workplace environments.
The primary objective behind these innovations is to bridge the gap between
human intentions and the physical execution of tasks, making everyday
activities more accessible to those with limited mobility or dexterity.

Despite considerable advancements in mechanical design and sensor technology,
one of the persistent challenges in assistive robotics is developing intuitive
and natural interaction interfaces [7] based on the specific needs of the
users. Traditional control mechanisms, such as joysticks [8], or touch screens
[9], often require a degree of fine motor control that might not be feasible
for all users. Furthermore, these control schemes can be cumbersome and
unintuitive, particularly for complex tasks that involve multiple steps or
precise movements [10]. As an example, users of the assistive robotic arms
mounted on a powered wheelchair must alternate between various control
settings to operate a 7-degree-of-freedom (DOF) arm and its preset positions
using a 2-axis wheelchair joystick [11, 12]. Voice control is also implemented
in some assistive robotic arms mounted on wheelchairs but user studies
conducted by Chung et. al. [13] showed that voice control, while beneficial
for hands-free operation, often struggles with accuracy in noisy environments
and can be slow for executing a series of commands, leading to frustration
among users. These limitations underscore the need for more adaptive and user-
friendly methods of communication between humans and assistive robotic
systems.

Natural Language Processing (NLP) emerges as a promising solution to this
challenge, offering a more accessible and intuitive means for users to
interact with robotic systems [14]. The rise of large language models (LLMs)
like GPT (Generative Pre-trained Transformer) [15] further enhances the
potential of NLP in robotics [16, 17]. These models, trained on vast amounts
of text data, can understand and generate human-like text, allowing for more
nuanced and complex interactions between humans and robots. Different research
groups have started to use LLMs to control robots and autonomous vehicles
[18]. Despite these advancements, research [19] showed that one of the main
challenges in assistive robotics is the translation of natural language into
robot actions that are both precise and contextually appropriate. On the other
hand, the specific needs of users have often not been the central focus in the
design of NLP-powered robots. There remains a significant opportunity to
tailor these technologies to better serve individuals with diverse
requirements, particularly in assistive applications. By prioritizing user-
centric development, future research can bridge the gap between technological
capability and practical usability, ensuring that assistive robots are not
only advanced in their understanding and execution of natural language
commands but also finely attuned to the nuanced needs and preferences of their
human users.

Edge computing [20, 21] represents a significant advancement in data
processing for robotics [22, 23], particularly in the realm of assistive
technologies. By localizing computation to data sources—such as robots and
sensors—it not only minimizes latency, essential for the safety and efficacy
of assistive robots responding to instant commands, but also enables these
systems to operate independently of internet connections, enhancing autonomy
and safety that research [24] has shown are important factors in designing
assistive technology. The integration of edge computing with advanced
technologies like LLM and NLP facilitates the creation of highly responsive,
personalized robotic systems capable of understanding and executing complex
natural language commands in real-time. This approach not only meets the
specific needs and preferences of users, making assistive devices more
intuitive and effective for individuals with disabilities but also propels the
field towards more adaptive and autonomous solutions.

This paper serves as a ”proof of concept” that combines edge computing with
LLM, NLP, and computer vision to control robots with the hope of eventually
advancing the field of assistive robotics towards highly responsive and user-
centric assistive robotic systems. In this paper, we have implemented our
proposed method on a robotic arm as a ”proof of concept” and current ongoing
research involves implementing our method on a wheelchair-mounted assistive
robot arm (Gen3 robot arm) and collaborating with Kinova, a leading assistive
robotics company, and a neuromuscular neurologist to conduct user studies to
tailor and enhance our technology based on direct feedback, targeting the
unique needs and preferences of users. Emphasizing a user-centric design
philosophy is crucial in our project to develop assistive robots that
significantly improve the quality of life for individuals with disabilities,
leading to more personalized and impactful robotic aids.

##  II Methods

###  II-A Architecture

Figure 1 shows the architecture of the proposed system where spoken language
is converted to robot actions. Speech is first transcribed into text, then
interpreted by GPT API into JSON formatted commands. These commands are
translated into ROS (Robot Operating System) instructions, which ultimately
drive the robot to perform the corresponding actions.

###  II-B Experimental Setup

We used 4 Degrees of Freedom (DOF) PincherX 100 robotic arm and the vision kit
(Intel D415 RealSense Depth Camera, which features $1920\times 1080$
resolution at 30fps with a $65^{o}\times 40^{o}$ field of view and an ideal
range of 0.5m to 3m) from Trossen Robotics, IL, USA for our poof-of-concept
experiments. This robot arm is controlled by Robot Operating System (ROS2)
Humble running on Ubuntu 22.04 operating system. The computer used is a Dell
Precision 3660 Tower Core i7 with 32GB DDR5 up to 4400MHz UDIMM non-ECC memory
and Nvidia RTXA4000 GPU. For audio capture, a Blue Yeti X USB Microphone is
used to ensure high-quality input for the speech-to-text conversion process.
Figure 2 shows the experimental setup.

Figure 2: For our experiments, we used a 4 DOF PincherX 100 robotic arm and an
Intel D415 RealSense Depth Camera from Trossen Robotics, IL, USA. The robot
arm, controlled via ROS2 Humble on Ubuntu 22.04, was connected to a Dell
Precision 3660 with an i7 processor, 32GB RAM, and Nvidia RTXA4000 GPU. Audio
is captured using a Blue Yeti X USB Microphone.

###  II-C Screw Theory-based Numerical Inverse Kinematics of the Robot Arm
Using Newton-Raphson Iterative Method

Inverse kinematics is used to map the desired position of the end-effector of
the robot arm to joint positions. We have used numerical inverse kinematics
based on Screw Theory in Robotics [25, 26] using Newton-Raphson iterative
method. The following algorithm is implemented to find the numerical inverse
kinematics of our robot arm:

  1. 1.

Initialization: Given $T_{sd}$ (transformation of the desired frame with
respect to space frame) and an initial guess of joint variables
$q_{o}\in\mathbb{R}^{n}$, set $i=0$.

  2. 2.

Set $[{\mathcal{V}}_{b}]=log(T_{sb}^{-1}(q_{i})T_{sd})$. While the algorithm
is not converged:

     * •

Set $q_{i+1}=q_{i}+J^{\dagger}_{b}(q_{i}){\mathcal{V}}_{b}$.

     * •

Increment $i$,

where $q_{i}$ is the current joint angles guess, $\mathcal{V}_{b}$ is the body
twist calculated from the error between the current end-effector configuration
and the desired configuration, and $J_{b}^{\dagger}(q_{i})$ is the
pseudoinverse of the body Jacobian matrix evaluated at the current joint
angles $q_{i}$ [27].

For our robot arm that we use for the experiments, the screw axis of each
joint expressed in the base frame in the robot’s zero position (see Figure 3
for link lengths and screw axes assignments) are as follows:

$\displaystyle\mathcal{S}_{1}$
$\displaystyle={\begin{bmatrix}0&0&1&0&0&0\end{bmatrix}}^{T},$ (1)
$\displaystyle\mathcal{S}_{2}$
$\displaystyle={\begin{bmatrix}0&1&0&-0.08945&0&0\end{bmatrix}}^{T},$
$\displaystyle\mathcal{S}_{3}$
$\displaystyle={\begin{bmatrix}0&1&0&-0.18945&0&0.035\end{bmatrix}}^{T},$
$\displaystyle\mathcal{S}_{4}$
$\displaystyle={\begin{bmatrix}0&1&0&-0.18945&0&0.135\end{bmatrix}}^{T},$

Figure 3: Robot arm depicted in home position with screw axes and link lengths
assignments that are essential to finding the forward kinematics. The lengths
depicted on the bottom photo are: $L_{1}=0.08945$, $L_{2}=0.1$, $L_{m}=0.035$,
$L_{3}=0.1$, and $L_{4}=0.08605$

and thus, $T_{sb}(q)$ (the pose of the end-effector frame $\\{b\\}$ in the
base frame $\\{s\\}$) can be written as the following equation using the
Product of Exponentials PoE formula:

$T_{sb}(q)=e^{[\mathcal{S}_{1}]q_{1}}e^{[\mathcal{S}_{2}]q_{2}}e^{[\mathcal{S}_{3}]q_{3}}e^{[\mathcal{S}_{4}]q_{4}}M,$
(2)

where, $q_{1},q_{2},q_{3},\text{ and }q_{4}$ are joint angles, and $M\in
SE(3)$ is the end-effector configuration when the robot is at its zero
position and for our robot arm it is represented by the following matrix:

$M=\begin{pmatrix}1&0&0&0.22105\\\ 0&1&0&0\\\ 0&0&1&0.18945\\\
0&0&0&1\end{pmatrix}.$ (3)

The Space Jacobian of this robot, $J_{s}(q)$ can be expressed as:

$J_{s}(q)=\begin{bmatrix}J_{s1}&J_{s2}&J_{s3}&J_{s4}\end{bmatrix},$ (4)

where, $J_{s1}$, $J_{s2}$, $J_{s3}$, and $J_{s4}$ can be calculated using the
following equations:

$\begin{array}[]{l}J_{s1}=\mathcal{S}_{1}\\\
J_{s2}=[Ad_{e^{[\mathcal{S}_{1}]q_{1}}}]\mathcal{S}_{2}\\\
J_{s3}=[Ad_{e^{[\mathcal{S}_{1}]q_{1}}e^{[\mathcal{S}_{2}]q_{2}}}]\mathcal{S}_{3}\\\
J_{s4}=[Ad_{e^{[\mathcal{S}_{1}]q_{1}}e^{[\mathcal{S}_{2}]q_{2}}e^{[\mathcal{S}_{3}]q_{3}}}]\mathcal{S}_{4}\end{array}$
(5)

in which, $\mathcal{S}_{i}$ is an expression for the screw axis describing the
ith joint axis in terms of the fixed frame with the robot in its zero position
given above, and $[Ad_{T_{i-1}}]\mathcal{S}_{i}$ is the screw axis describing
the ith joint axis, but after it undergoes the rigid body displacement $T_{i}$
instead of being at zero position. In other words, it is the Adjoint map of
the screw axis for when the robot is no longer in zero position. Note that if
$T=(R,p)\in SE(3)$ is a transformation matrix where $R$ is the rotation matrix
and $p$ is the position vector, then its adjoint representation $[Ad_{T}]$ can
be calculated by:

$[Ad_{T}]=\begin{pmatrix}R&o\\\ [p]R&R\end{pmatrix}\in\mathbb{R}^{6\times 6}.$
(6)

Note that the bracket notation $[p]$ is the $3\times 3$ skew-symmetric matrix
representation of the position vector $p$. The body Jacobian can then be
calculated from the space Jacobian using the Adjoint transformation:

$J_{b}(q)=[Ad_{T_{bs}}]J_{s}(q),$ (7)

where, $T_{bs}=T^{-1}_{sb}$ is the inverse of the homogeneous matrix $T_{sb}$
that was derived from forward kinematics earlier.

###  II-D Vision‐Language-Aided Inverse Kinematics Control of the Robot Arm

####  II-D1 Vision-Aided Control

The objective here is to merge the numerical inverse kinematics with the
robot’s perception package to create a vision-aided inverse kinematics motion
planner. The robot’s perception pipeline uses a combination of depth data and
color information from the RealSense camera to create a point cloud for 3D
object detection and manipulation. The depth camera first captures the
AprilTag [28] attached to the robot’s arm to help ROS figure out the
homogeneous transformation of the robot’s base frame relative to the camera
and vice versa. This helps convert the camera’s depth readings of scene
objects to homogeneous transformations with respect to the robot’s base.
Later, the inverse kinematics module maps these transformations to joint angle
set-points to make the robot manipulate the objects. Stereo vision technology
alongside the infrared sensor helped estimate the 3D structure of the scene.

####  II-D2 Speech-to-Text Integration

Our program incorporates an Open Source speech recognition program called VOSK
by Alpha Cephei Inc. VOSK operates offline and runs entirely on the local
machine. This is crucial for the program as it no longer requires voice
activation to require internet connectivity to transcribe the speech. Once the
speech is captured and processed by VOSK, the program takes over to analyze
the transcribed text. This analysis involves parsing the text output to
understand the commands and translating them into specific actions that
control the robot.

####  II-D3 ChatGPT API Integration

OpenAI provides API access to their LLM and for this program, we used GPT-3.5
and later GPT-4 (for comparison with GPT-3.5 tests) as our models. The API was
fine-tuned to comprehend and transcribe natural spoken language, significantly
enhancing accuracy and context relevance for the process. The user’s speech
that has previously been parsed by VOSK is now processed by the LLM. We
leveraged the OpenAI API to develop a program that takes the text input and
converts it into a JSON-formatted list of instructions. This output gave us
certain key objectives such as the action, the object, the direction, and so
forth. For example, when given a phrase such as ”it’s opposite day, now move
to the right”, the API integration can understand key terms ”opposite” and
”right” leading the robot to move to the left; thus interpreting intent rather
than solely reading ”right” and moving to the right as is common in voice
control devices. This implementation produces an increase in language the
robot can interpret, allowing the user to speak in a natural manner.

####  II-D4 Language-Vision-aided Control

In our approach to create a language-vision-aided control system for robotic
manipulation, we integrated the LLM with advanced computer vision
technologies. The method involved first processing verbal commands through the
LLM to interpret the user’s intent accurately. These commands were then
translated into specific parameters for the vision system, guiding the robot
to identify and interact with objects based on their characteristics, such as
color. Our experimental setup included objects of three distinct colors (red,
blue, and green) placed within the robot’s workspace. The system’s objective
was to accurately decode spoken commands like ”Pick up the red cube,” and
through a seamless integration of NLP and computer vision, enable the robot to
visually locate, identify, and manipulate the specified object. This method
aimed to test the robustness of combining language understanding with visual
perception in executing complex tasks.

####  II-D5 Enhanced Control at the Edge

We hypothesized that an entirely offline speech-to-text and LLM system would
offer a more secure, private, and faster alternative to online API reliance.
To demonstrate the speed, we evaluated the execution time of our program’s
speech-to-text component, transitioning from an online service to VOSK for
local processing. Although we have not implemented an offline LLM in this
phase, the shift to a local setup with VOSK allows us to fully govern our data
and inputs, ensuring robust data protection and enabling the program’s use in
network-restricted environments.

##  III RESULTS

###  III-A NLP-Aided Control Efficacy

####  III-A1 Intent Interpretation Accuracy Using GPT-3.5

The model using GPT-3.5 was consistent with simple commands. However,
introducing more commands or alterations of those commands resulted in the LLM
struggling to interpret into our instruction format. The most erroneous
results were when the instructions were incorrect such as if it was meant to
move the arm left and it ended up moving right. We classified lesser errors as
those that yielded results that were not the intended outcome but were also
not incorrect from a linguistic perspective. For example, if we told the robot
to not move left, the LLM would give the command to move forward resulting in
an unintended outcome, but one that satisfies the condition. The program was
successfully able to identify the correct instructions for simple and basic
sentences such as ”move to the left” or ”move to the opposite of right”.
However, upon being introduced to longer or more complex tasks that require
logic such as ”Pretend it’s opposite day, move left”, the language model
struggles to create the correct course of instructions. These results are
shown in Table I.

TABLE I: Intent Interpretation Analysis for GPT Versions 3.5 and 4

| GPT-3.5 | GPT-4
---|---|---
Command* | Trial 1 | Trial 2 | Trial 3 | Trial 1 | Trial 2 | Trial 3
1 | PASS | PASS | PASS | PASS | PASS | PASS
2 | PASS | PASS | PASS | PASS | PASS | PASS
3 | PASS | PASS | PASS | PASS | PASS | PASS
4 | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL
5 | PASS | PASS | FAIL | PASS | PASS | PASS
6 | PASS | PASS | PASS | PASS | PASS | PASS
7 | PASS | FAIL | FAIL | PASS | PASS | PASS
8 | FAIL | FAIL | FAIL | PASS | PASS | PASS
9 | FAIL | FAIL | FAIL | PASS | PASS | PASS
10 | PASS | PASS | PASS | PASS | PASS | PASS
11 | PASS | PASS | PASS | PASS | PASS | PASS

  * *

Command 1: Move to the right, Command 2: Move to the left, Command 3: Your
favorite direction is left. Now decide, and move to where you want to be,
Command 4: I am to your left, now I have turned you around 180 degrees. Now
move towards me, Command 5: Move to the opposite of left, Command 6: Move to
the opposite of right, Command 7: Pretend it’s opposite day, now move to the
right, Command 8: Pretend it’s opposite day, now move to the left, Command 9:
The spider is to your left, but you are scared of it, so move to where you
won’t be scared, Command 10: You want to take a nap and you are on the left
side of the room, but your bed is on the other side. Move to where your bed
is, and Command 11: There’s 20 dollars on the right, but 50 dollars on the
left. Move to the best side.

TABLE II: Comparative Analysis of Task Execution Times (Speech to Text
Conversion): Cloud Computing vs. Edge Computing

| Measured Time Edge | Measured Time Cloud
---|---|---
Command/Prompts* | Trial 1 | Trial 2 | Trial 3 | AVG | STDEV | Trial 1 | Trial 2 | Trial 3 | AVG | STDEV
1 | 5.17 | 5.34 | 4.30 | 4.94 | 0.56 | 5.15 | 3.73 | 3.30 | 4.06 | 0.97
2 | 4.61 | 4.56 | 4.72 | 4.63 | 0.08 | 4.52 | 4.03 | 3.82 | 4.12 | 0.36
3 | 8.51 | 8.87 | 8.52 | 8.63 | 0.21 | 8.11 | 8.41 | 8.54 | 8.35 | 0.22
4 | 9.95 | 9.92 | 9.28 | 9.72 | 0.38 | 9.86 | 9.43 | 10.05 | 9.78 | 0.32
5 | 6.15 | 6.02 | 5.98 | 6.05 | 0.09 | 4.68 | 5.03 | 4.79 | 4.83 | 0.18
6 | 5.74 | 5.92 | 5.37 | 5.68 | 0.28 | 4.99 | 5.84 | 6.36 | 5.73 | 0.69
7 | 6.32 | 6.34 | 6.85 | 6.50 | 0.30 | 5.49 | 7.23 | 6.68 | 6.47 | 0.89
8 | 6.62 | 6.36 | 6.33 | 6.44 | 0.16 | 5.99 | 5.94 | 5.26 | 5.73 | 0.41
9 | 9.21 | 8.89 | 8.54 | 8.88 | 0.34 | 8.62 | 8.32 | 8.86 | 8.60 | 0.27
10 | 9.61 | 9.68 | 9.99 | 9.76 | 0.20 | 10.48 | 10.49 | 10.42 | 10.46 | 0.04
11 | 8.82 | 9.14 | 8.84 | 8.93 | 0.18 | 8.75 | 8.91 | 9.03 | 8.90 | 0.14

  * *

The commands/prompts are the same as Table I. p-value from the paired t-test
across all commands is 0.105 that suggests that there is no statistically
significant difference in execution time between the offline and online model.

TABLE III: Analysis of Robot Task Execution Success across Trials

Command* | Trial 1 | Trial 2 | Trial 3 | Trial 4 | Trial 5
---|---|---|---|---|---
1 | PASS | PASS | PASS | PASS | PASS
2 | PASS | PASS | PASS | PASS | PASS
3 | PASS | PASS | PASS | PASS | PASS

  * *

Command 1: Pick up the Red Cube, Command 2: Pick up the Blue Cube, and Command
3: Pick up the Green Cube

####  III-A2 Response Time

In our comparative study between the offline VOSK speech-to-text program and
an online speech-to-text model from Google, we aimed to evaluate the
efficiency and speed of command processing. Initially, we hypothesized that
processing commands offline with VOSK would be significantly faster and more
reliable, given its independence from internet connectivity. However, our
empirical tests yielded unexpected results (see Table II).

The paired t-test across all commands/prompts yielded a t-statistic of 1.784
and a p-value of 0.105. This result suggests that, overall, there is no
statistically significant difference in execution time between the offline and
online models across all commands/prompts at the conventional alpha level of
0.05. The findings suggest that the online model resulted in a wider range of
task execution times compared to the offline model. This suggests that
although the processing power of the online model might be greater than the
local machine, its reliance on the network connection results in inconsistent
execution times.

It’s important to note that while the execution times were similar, the
offline model provided consistent performance, which is crucial for real-time
applications in environments with unreliable internet access. This consistency
supports the case for further exploring and enhancing offline processing
capabilities for robotic control, particularly in assistive technology
scenarios where reliability and quick response are paramount [24].

####  III-A3 Improved LLM Intent Interpretation Accuracy

In testing LLMs for intent interpretation in robotic control, we transitioned
from GPT-3.5 to GPT-4 to assess potential gains in accuracy and contextual
understanding. Our initial experiences with GPT-3.5 revealed a consistent
performance for straightforward commands. However, as the complexity and
linguistic nuances of the commands increased, the model’s limitations became
apparent, leading to inaccuracies and misinterpretations.

The upgrade to GPT-4 marked a significant improvement in our system’s ability
to process and accurately interpret complex commands (See Table I). This newer
iteration of the LLM demonstrated a remarkable capability to understand
nuanced instructions, converting them into actionable tasks with higher
precision. Notably, commands that previously confounded GPT-3.5, such as those
involving conditional logic or abstract concepts, were handled more adeptly by
GPT-4, leading to successful task execution in a wider array of test
scenarios.

Despite this, certain complex commands continued to pose challenges.
Instructions that required the LLM to infer actions based on hypothetical or
reversed conditions were not always successfully interpreted. This was
exemplified in our test command involving a 180-degree turn and subsequent
movement, which both versions of the LLM struggled to accurately process. This
particular case underscores the existing gaps in current LLMs’ ability to
fully grasp and respond to context-heavy or conceptually intricate commands
that need further exploration in future research.

These findings show the potential of advanced LLMs in enhancing robotic
control systems, particularly in their ability to understand and act upon a
broader spectrum of natural language commands. However, the observed
limitations also highlight the need for ongoing refinement of these models,
especially in their handling of complex linguistic structures and contextual
interpretation.

###  III-B Vision-Language-Aided Control Efficacy

####  III-B1 Object Manipulation Success Rate

The implementation of our language-vision-aided control system yielded
acceptable results in the object manipulation experiments. Demonstrating a
100% success rate (see Table III), the robotic arm correctly picked up the
red, blue, and green cubes as instructed across multiple trials. The LLM
accurately interpreted the user’s intent from the spoken commands, while the
vision system provided object detection and localization, facilitating precise
manipulation by the robot. These experiments validate our method’s
effectiveness in creating a responsive and accurate robotic control system
that leverages the synergistic potential of language processing and visual
perception.

##  IV Conclusion and Future Work

This study has presented a ”proof of concept” for an assistive robot arm
system enhanced with edge computing, NLP, and computer vision capabilities.
Our results demonstrate the feasibility and effectiveness of using LLMs with
vision systems to interpret and execute complex commands through natural
language processing. The integration of these technologies has shown promise
in improving the intuitiveness and accessibility of assistive robotic systems,
making them versatile to the nuanced needs of users with disabilities. Our
experimental findings highlight the significant potential of combining LLMs
with computer vision to accurately interpret user intents and manipulate
objects based on verbal commands. This synergy between language understanding
and visual perception paves the way for more responsive and versatile
assistive robots that can perform a wider array of tasks with greater
precision and reliability.

Despite the successes, this study has identified several areas for future
development. One of the limitations identified is the lack of feedback control
in our current system. Future work will focus on integrating feedback
mechanisms to allow the robot to adjust its actions in real-time based on
environmental feedback, enhancing task execution accuracy and reliability.
Building on the current proof of concept, future iterations include
implementing our system on a more advanced Kinova 6 DOF assistive robot arm.
This will allow us to explore the capabilities of our system in handling more
complex tasks and environments. To ensure our system meets the needs of its
intended users, comprehensive user studies will be conducted. These studies
will involve individuals with disabilities to gather direct feedback on the
system’s performance, usability, and areas for improvement. The current study
has revealed certain limitations in the LLM’s ability to interpret complex or
context-heavy commands. Ongoing research will be dedicated to refining the
LLM’s understanding and processing capabilities, possibly through the
development of a dedicated, localized LLM tailored specifically for robotic
control applications. To enhance the system’s reliability and accessibility,
further development will aim at achieving full network independence. By
localizing all processing tasks, we aim to ensure functionality in
environments with limited or no internet access, increasing versatility.
Upgrading the vision pipeline, potentially to more advanced frameworks like
YOLO, will be explored to overcome current limitations in object detection and
interaction.

## References

  * [1] B. Abdulrazak and M. Mokhtari, “Assistive robotics for independent living,” _The engineering handbook of smart technology for aging, disability, and independence_ , pp. 355–374, 2008.
  * [2] S. Gushi, Y. Shimabukuro, and H. Higa, “A self-feeding assistive robotic arm for people with physical disabilities of the extremities,” in _2020 5th International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS)_. IEEE, 2020, pp. 61–64.
  * [3] R. J. López-Sastre, M. Baptista-Ríos, F. J. Acevedo-Rodríguez, S. Pacheco-da Costa, S. Maldonado-Bascón, and S. Lafuente-Arroyo, “A low-cost assistive robot for children with neurodevelopmental disorders to aid in daily living activities,” _International Journal of Environmental Research and Public Health_ , vol. 18, no. 8, p. 3974, 2021.
  * [4] L. Fiorini, M. De Mul, I. Fabbricotti, R. Limosani, A. Vitanza, G. D’Onofrio, M. Tsui, D. Sancarlo, F. Giuliani, A. Greco, _et al._ , “Assistive robots to improve the independent living of older persons: Results from a needs study,” _Disability and Rehabilitation: Assistive Technology_ , vol. 16, no. 1, pp. 92–102, 2021.
  * [5] J. Ten Kate, G. Smit, and P. Breedveld, “3d-printed upper limb prostheses: a review,” _Disability and Rehabilitation: Assistive Technology_ , vol. 12, no. 3, pp. 300–314, 2017.
  * [6] M. S. H. Sunny, M. I. I. Zarif, I. Rulik, J. Sanjuan, M. H. Rahman, S. I. Ahamed, I. Wang, K. Schultz, and B. Brahmi, “Eye-gaze control of a wheelchair mounted 6dof assistive robot for activities of daily living,” _Journal of NeuroEngineering and Rehabilitation_ , vol. 18, no. 1, pp. 1–12, 2021.
  * [7] F. Robinson and G. Nejat, “An analysis of design recommendations for socially assistive robot helpers for effective human-robot interactions in senior care,” _Journal of Rehabilitation and Assistive Technologies Engineering_ , vol. 9, p. 20556683221101389, 2022.
  * [8] I. Rulik, M. S. H. Sunny, J. D. Sanjuan De Caro, M. I. I. Zarif, B. Brahmi, S. I. Ahamed, K. Schultz, I. Wang, T. Leheng, J. P. Longxiang, _et al._ , “Control of a wheelchair-mounted 6dof assistive robot with chin and finger joysticks,” _Frontiers in Robotics and AI_ , vol. 9, p. 885610, 2022.
  * [9] L. Wu, R. Alqasemi, and R. Dubey, “Development of smartphone-based human-robot interfaces for individuals with disabilities,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5835–5841, 2020.
  * [10] Y.-S. L.-K. Cio, M. Raison, C. L. Menard, and S. Achiche, “Proof of concept of an assistive robotic arm control using artificial stereovision and eye-tracking,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 27, no. 12, pp. 2344–2352, 2019.
  * [11] C.-S. Chung, H. W. Ka, H. Wang, D. Ding, A. Kelleher, and R. A. Cooper, “Performance evaluation of a mobile touchscreen interface for assistive robotic manipulators: A pilot study,” _Topics in spinal cord injury rehabilitation_ , vol. 23, no. 2, pp. 131–139, 2017.
  * [12] H. W. Ka, C.-S. Chung, D. Ding, K. James, and R. Cooper, “Performance evaluation of 3d vision-based semi-autonomous control method for assistive robotic manipulator,” _Disability and Rehabilitation: Assistive Technology_ , vol. 13, no. 2, pp. 140–145, 2018.
  * [13] C.-S. Chung, B. Styler, E. L. Wang, and D. Ding, “Robotic assistance in action: Examining control methods for long-term owners of wheelchair-mounted robotic arms,” in _ASSISTIVE TECHNOLOGY_ , vol. 36, no. 1. TAYLOR & FRANCIS INC 530 WALNUT STREET, STE 850, PHILADELPHIA, PA 19106 USA, 2024, pp. 85–86.
  * [14] Z. Li, Y. Mu, Z. Sun, S. Song, J. Su, and J. Zhang, “Intention understanding in human–robot interaction based on visual-nlp semantics,” _Frontiers in Neurorobotics_ , vol. 14, p. 610139, 2021.
  * [15] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [16] J. Wang, Z. Wu, Y. Li, H. Jiang, P. Shu, E. Shi, H. Hu, C. Ma, Y. Liu, X. Wang, _et al._ , “Large language models for robotics: Opportunities, challenges, and perspectives,” _arXiv preprint arXiv:2401.04334_ , 2024.
  * [17] F. Zeng, W. Gan, Y. Wang, N. Liu, and P. S. Yu, “Large language models for robotics: A survey,” _arXiv preprint arXiv:2311.07226_ , 2023.
  * [18] S. Sharan, F. Pittaluga, M. Chandraker, _et al._ , “Llm-assist: Enhancing closed-loop planning with language-based reasoning,” _arXiv preprint arXiv:2401.00125_ , 2023.
  * [19] C. Keroglou, I. Kansizoglou, P. Michailidis, K. M. Oikonomou, I. T. Papapetros, P. Dragkola, I. T. Michailidis, A. Gasteratos, E. B. Kosmatopoulos, and G. C. Sirakoulis, “A survey on technical challenges of assistive robotics for elder people in domestic environments: The aspida concept,” _IEEE Transactions on Medical Robotics and Bionics_ , 2023.
  * [20] A. Samanta, F. Esposito, and T. G. Nguyen, “Asap: Adaptive and scalable microservice provisioning for edge-iot networks,” in _2023 18th Wireless On-Demand Network Systems and Services Conference (WONS)_. IEEE, 2023, pp. 80–87.
  * [21] F. Esposito, “Digital telepathology and virtual control of a microscope using edge computing,” Nov. 8 2022, uS Patent 11,494,901.
  * [22] A. Barnawi, M. Alharbi, and M. Chen, “Intelligent search and find system for robotic platform based on smart edge computing service,” _IEEE Access_ , vol. 8, pp. 108 821–108 834, 2020.
  * [23] S. Wan, Z. Gu, and Q. Ni, “Cognitive computing and wireless communications on the edge for healthcare service robots,” _Computer Communications_ , vol. 149, pp. 99–106, 2020.
  * [24] S. Poirier, F. Routhier, and A. Campeau-Lecours, “Voice control interface prototype for assistive robots for people living with upper limb disabilities,” in _2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR)_. IEEE, 2019, pp. 46–52.
  * [25] K. M. Lynch and F. C. Park, _Modern robotics_. Cambridge University Press, 2017.
  * [26] R. M. Murray, Z. Li, and S. S. Sastry, _A mathematical introduction to robotic manipulation_. CRC press, 2017\.
  * [27] M. Babaiasl, “Modern robotics course wiki,” https://github.com/madibabaiasl/modern-robotics-course/wiki, accessed: 02/21/2024.
  * [28] University of Michigan, “Apriltag: A robust and flexible visual fiducial system,” https://april.eecs.umich.edu/software/apriltag, accessed: 2024-02-21.