# Deep Active Latent Surfaces for Medical Geometries

Patrick M. Jensen
<EMAIL_ADDRESS>&Udaranga Wickramasinghe
<EMAIL_ADDRESS>&Anders B. Dahl
<EMAIL_ADDRESS>&Pascal Fua∗
<EMAIL_ADDRESS>&Vedrana A. Dahl111
<EMAIL_ADDRESS>
###### Abstract

Shape priors have long been known to be effective when reconstructing 3D
shapes from noisy or incomplete data. When using a deep-learning based shape
representation, this often involves learning a latent representation, which
can be either in the form of a single global vector or of multiple local ones.
The latter allows more flexibility but is prone to overfitting. In this paper,
we advocate a hybrid approach representing shapes in terms of 3D meshes with a
separate latent vector at each vertex. During training the latent vectors are
constrained to have the same value, which avoids overfitting. For inference,
the latent vectors are updated independently while imposing spatial
regularization constraints. We show that this gives us both flexibility and
generalization capabilities, which we demonstrate on several medical image
processing tasks.

11footnotetext: Equal supervision.

##  1 Introduction

3D shape reconstruction from noisy or incomplete data usually benefits from
the judicious use of shape priors. When using a deep-learning based shape
representation, this usually means searching for an appropriate latent
representation under regularization losses. This latent vector representation
can take the form of either a single global latent vector per shape or a grid
of latent vectors.

In either case, it is a challenge to balance regularization against quality of
fit to the data. With too much regularization the recovered shapes are too
smooth and fine details are lost. With too little, robustness to noise and
generalization capabilities will suffer. Our insight is that, when using
multiple latent vectors, instead of imposing regularity constraints directly
on the surface, we can impose them on the latent vectors. This is effective
because, if a latent vector has been trained to model a sharp feature,
requiring that this vector be similar to its neighbors will not detract from
that. To this end, we represent surfaces as triangulated meshes and advocate
using a separate latent vector at each vertex while imposing smoothness
constraints on these vectors. To exploit this, we borrow the regularization
idea from early approaches to 3D shape modeling [24, 57, 56] that predate deep
learning and apply it to the latent vectors instead of the vertex positions.

Figure 1: DALS overview. (top) Given a set of training shapes, we use an auto-
decoding approach to learning a latent space of shapes. (bottom left) To each
training shape is associated a single latent vector $\mathbf{z}$ that is used
to compute a translation for each vertex of sphere to minimize the distance
between the deformed sphere and the training shape. (bottom right) At
inference time, the latent vectors
$\mathbf{z_{1}},\mathbf{z_{2}},\ldots,\mathbf{z_{N}}$ at each one the $N$
sphere vertices are allowed to change independently to minimize a weighted sum
of a data loss function and a regularization term that prevents neighboring
latent vectors from being too different from each other.

More specifically, at training time, we use an auto-decoding approach [44] to
jointly learn the network weights along with one common latent vector for all
vertices of each training sample. By contrast, at inference time, we allow the
latent vectors to be different at each vertex while enforcing consistency of
the vectors by minimizing a regularization loss. Fig. 1 depicts our Deep
Active Latent Surfaces (DALS) approach. It makes our model both easy to train
from relatively small datasets and very expressive: Because, we learn a single
latent vector per shape, we do not require the training set to be huge. At
model fitting time, we do not need to find an individual vector that
accurately represents a whole shape, which can be difficult when the shape is
complex. Instead, we can smoothly blend a number of vectors to model such a
shape, while preserving sharp features. Furthermore, explicitly using a
triangulated mesh instead of a grid of latent vectors makes our model more
compact and simpler to train because it does not have to waste capacity on
modeling empty regions.

##  2 Related Work

#### Active Surface Models

Active contour models are used to refine contours according to local image
properties while remaining smooth. They were first introduced in [24] for
interactive delineation and then extended for many different purposes [12].
Active surface models operate on the same principle [56, 57] but replace the
contours by triangulated meshes to model 3D surfaces. They have proved very
successful for medical [19, 36] and cartographic applications [13], among
others, and are still being improved [22, 28, 48]. In recent years, Deep
Neural Networks (DNNs) have been used to evaluate the energy that the active
contours minimize [17, 34] and, in [30, 31, 45], they are used to directly
predict vertex offsets. In [58], active surface models are embedded in special
purpose network layers that regularize surface meshes using the same semi-
implicit scheme as the original active contours [24].

Correctly balancing the relative influence of the data and regularization
terms to avoid over-smoothing while being robust to noise remains a challenge
for all these approaches. In [30] smoothing is only added as a loss during
training but not during inference. In [58], smoothing is made adaptive to
allow sharp edges. Recently, it has been proposed to replace smoothing with
preconditioned gradient descent of the external energy [42]. In all these
approaches, the regularization tends to flatten sharp geometric features of
the mesh geometry, which almost always results in over-smoothing, despite the
goodness of fit at a global level. In our work, we side step these issues by
focusing the smoothing on the latent space.

#### Neural Shape Modeling

Deep-learning methods are now routinely used to model 3D shapes. Most methods
rely on auto-encoders or auto-decoders to produce latent vectors that
parameterize the target shapes in terms of triangulated meshes [16, 32, 40],
tetrahedral meshes [14, 49], surface patches [15], point clouds [1, 47], voxel
grids [6, 11], occupancy functions [8, 38, 46], signed and unsigned distance
fields [9, 23], and neural splines [59].

These methods can be classified as those that use a single latent vector to
represent a complete shape and those that use multiple ones. Among those that
use a single one are the approaches of [8, 23, 32, 44]. They are effective but
accurately representing all the details may require more than one latent
vector as in the methods of [9, 21, 37, 46] that rely on a grid of latent
vectors and a shared decoder to represent the signed distance function of a
complete shape. Since each latent vector only has to describe a small part of
the complete shape, this greatly increases the model flexibility. It also
allows for a much smaller decoder network, which makes inference faster.
However, storing a full grid of latent vectors means that some latent vectors
are wasted on representing empty space. This increases memory use and training
time. The approach of [7, 33] avoids this by using a sparse grid from which
unused grid cells are removed. In [35, 54], a tree structures is used to
construct sparse multiscale representations. However, these methods either
require prior knowledge about the surface or periodic updates of the sparse
data structure as it deforms, which makes training more complex. In [41],
these issues are alleviated by using a spatial hash encoding which allows the
model to implicitly allocate more capacity to regions near the surface.
However, this method is designed to represent single shapes and would require
non-trivial extensions for model fitting purposes. In our work, we also rely
on multiple latent vectors but require neither a priori knowledge about the
surface nor complex adaptation of a data structure.

##  3 Deep Active Latent Surfaces

We now describe our Deep Active Latent Surface (DALS) approach, which is
illustrated by Fig. 1. We represent watertight 3D shapes by triangulated
spheres with a latent vector at each vertex. This latent vector along with the
vertex coordinates is fed to a decoder $\mathcal{D}_{\theta}$ that generates
an offset vector that is then used to translate the vertex to its final
position. Once all the vertices have been translated, we have the final shape
such as the one shown on the top right part of Fig. 1. At training time, we
use the same latent vector for all vertices whereas, at model fitting time, we
allow them to be different but impose spatial consistency on the vectors. This
does not preclude the modeling of sharp features because such features can be
predicted by individual latent vectors.

###  3.1 Training Scheme

Formally, let $\mathcal{D}_{\Theta}$ be a neural network with weights $\Theta$
that takes as input a $d$-dimensional latent vector $\mathbf{z}$ and a 3D
location $\mathbf{x}$ and returns an offset
$\mathcal{D}_{\Theta}(\mathbf{z},\mathbf{x})$. Given a triangulated sphere
with $V$ vertices and $F$ facets, we denote by
$\mathcal{M}_{\theta}(\mathbf{z})$ the deformed mesh we obtain by translating
each vertex $\mathbf{x}_{v}$ by
$\mathcal{D}_{\Theta}(\mathbf{z},\mathbf{x}_{v})$ for all $v$ between 1 and
$V$.

Let us further assume we are given a set of $N$ training shapes
$S=\\{S_{1},\ldots,S_{N}\\}$. As in [44], we can simultaneously learn $\Theta$
and a $\mathbf{z}_{i}$ for each $S_{i}$ by looking for

$\displaystyle\Theta^{*},\mathbf{z}_{1}^{*},\ldots,\mathbf{z}_{N}^{*}$
$\displaystyle=\operatorname*{\arg\min}_{\Theta,\mathbf{z}_{1},\ldots,\mathbf{z}_{N}}\sum_{i=1}^{N}\mathcal{L}_{\text{dat}}(\Theta,\mathbf{z}_{i},S_{i})\;,$
(1) $\displaystyle\mathcal{L}_{\text{dat}}(\Theta,\mathbf{z},S_{i})$
$\displaystyle=L_{\text{cf}}(\mathcal{M}_{\theta}(\mathbf{z}),S_{i})+\lambda_{\text{reg}}L_{\text{reg}}(\mathcal{M}_{\theta}(\mathbf{z}))+\lambda_{\text{n}}\|\mathbf{z}\|^{2}\;,$

where $L_{\text{cf}}$ is the Chamfer distance [52], $L_{\text{reg}}$ is shape
regularization term, and $\lambda_{\text{reg}}$ and $\lambda_{\text{n}}$ are
weighting constants.

In practice, we take $\mathcal{D}_{\Theta}$ to be an MLP with three hidden
layers of size $724$, $724$, and $362$, which takes as input a concatenation
of $\mathbf{x}$ and $\mathbf{z}$. We use ReLU activations for the hidden
layers and none for the last layer. Before each ReLU activation we use layer
normalization [3]. Our initial spherical triangulation is a subdivided
icosahedron. We use a pointwise MLP instead of a mesh based decoder because we
want to learn a mapping from the surface of a sphere conditioned on a latent
vector rather than a mapping from a specific template mesh. To this end, we
also randomly rotate the template

$\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\boldsymbol{\checkmark}$$\mathbin{\leavevmode\hbox
to14.29pt{\vbox to14.29pt{\pgfpicture\makeatletter\hbox{\hskip
0.89296pt\lower-0.89296pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{}{}\pgfsys@setlinewidth{1.78593pt}\pgfsys@invoke{
}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\pgfsys@color@rgb@stroke{1}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,0,0}{}{{}}{} {}{}{}{{}}{}
{}{}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{12.50159pt}{12.50159pt}\pgfsys@moveto{0.0pt}{12.50159pt}\pgfsys@lineto{12.50159pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}$$\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\boldsymbol{\checkmark}$$\mathbin{\leavevmode\hbox
to14.29pt{\vbox to14.29pt{\pgfpicture\makeatletter\hbox{\hskip
0.89296pt\lower-0.89296pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{}{}\pgfsys@setlinewidth{1.78593pt}\pgfsys@invoke{
}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\pgfsys@color@rgb@stroke{1}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,0,0}{}{{}}{} {}{}{}{{}}{}
{}{}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{12.50159pt}{12.50159pt}\pgfsys@moveto{0.0pt}{12.50159pt}\pgfsys@lineto{12.50159pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}$ Figure 2: Behavior
of $L_{\mathrm{reg}}$. Minimizing $L_{\mathrm{reg}}$ will leave the regular
meshes on the left unchanged, even though the first one exhibits a sharp
crease. In contrast, doing so with the meshes on the right, will increase the
regularity of the triangles and smooth out the isolated outlier.

during training to specific vertex placement. This enables us to use any
template mesh without changing the decoder. We will take advantage of this at
inference time by progressively increasing the mesh resolution.

The $L_{\text{reg}}$ term in Eq. 1 is intended to encourage the generation of
high quality meshes. We could have expressed it in terms of the Laplacian as
is often done but we found experimentally that it tends to result in meshes
that are too smooth. Instead, as in [4], we take it to be

$L_{\text{reg}}(\mathcal{M})=1-\frac{4\sqrt{3}}{|\mathcal{F}|}\sum_{f\in\mathcal{F}}\frac{A_{f}}{a_{f}^{2}+b_{f}^{2}+c_{f}^{2}}\;,$
(2)

where $\mathcal{F}$ stands for the mesh facets, $a_{f},b_{f},c_{f}$ for the
lengths of the three edges of facet $f$, and $A_{f}$ for its area. It is easy
to compute and has favorable properties for numerical optimization [50]. In
practice, minimizing $L_{\text{reg}}$ promotes regularity without directly
penalizing high frequency features, as illustrated by Fig. 2.

###  3.2 Fitting Scheme

Let $\mathcal{D}=\mathcal{D}_{\Theta^{*}}$ be the network we trained in
Section 3.1 and let $\mathbf{Z}\in\mathbb{R}^{V\times d}$ whose $V$ rows are
latent vectors, one for each vertex of our triangulated sphere. We now denote
by $\mathcal{M}(\mathbf{Z})$ the mesh we obtain by shifting each vertex
$\mathbf{x}_{v}$ by $\mathcal{D}(\mathbf{Z}[v])$. In other words, we now
assign to each vertex a _different_ latent vector and use it to compute the
corresponding translation from the initial sphere to where it should be. To
fit our model to data, we look for

$\displaystyle\textbf{Z}^{*}=$
$\displaystyle\operatorname*{\arg\min}_{\textbf{Z}}\mathcal{L}_{\text{task}}(\mathcal{M}(\mathbf{Z}))\,+$
(3)
$\displaystyle\lambda_{\text{reg}}L_{\text{reg}}(\mathcal{M}(\mathbf{Z}))+\lambda_{\text{dir}}L_{\text{dir}}(\mathbf{Z})\;,$

where $\mathcal{L}_{\text{task}}$ is a task-specific loss function,
$L_{\text{reg}}$ is the geometric regularization loss of Eq. 2,
$L_{\text{dir}}$ is a new regularization term designed to enforce consistency
of the latent vectors across the surface, and $\lambda_{\text{reg}}$ and
$\lambda_{\text{dir}}$ are weighting constants. In practice,
$\mathcal{L}_{\text{task}}$ can be expressed in terms of Chamfer distance to
fit points, slice Chamfer to fit curve annotations, and SDF gradients for
segmentation purposes as discussed in Section 4.

Inspired by active surfaces [24, 56, 57, 58], we use Dirichlet energy [42, 53]
to define $L_{\text{dir}}$. We write

$L_{\text{dir}}(\mathbf{Z})=\text{Tr}(\mathbf{Z}^{T}\textbf{L}^{p}\mathbf{Z})\;,$
(4)

where L is the uniform Laplacian matrix and $p$ is an integer power. Note that
$\nabla L_{\text{dir}}(\textbf{Z})=\textbf{L}^{p}\mathbf{Z}$, meaning that a
gradient step corresponds to $p$ iterations of Laplacian smoothing of the
latent vectors. We found $p=2$ to work well.

Weight $\lambda_{\text{dir}}$ controls how constrained the fitting is by the
prior information contained in the latent space parameterization. When
$\lambda_{\text{dir}}\to\infty$, all latent vectors will have the same value
and our approach reverts to a global approach with one single latent vector.
For small values of $\lambda_{\text{dir}}$, the model becomes much more
flexible as the values of the latent vector can more easily change from vertex
to vertex.

One could also allow independant training vectors during training. However, in
our experience, this causes the decoder to produce severely self-intersecting
meshes as it has too much freedom.

##  4 Experiments

We demonstrate the benefits of DALS on several medical image processing tasks.
We train all models to learn a latent representation of livers and spleens
using data from the Medical Segmentation Decathlon (MSD) [2] (CC BY-SA 4.0
license). To create ground-truth meshes, we first resampled the annotated
images so that their voxel size is $1\times$1$\times 1$ mm and then used
marching cubes [29] to extract isosurfaces. We standardize the surfaces to
have zero mean and be contained in the unit sphere.

The datasets contains 111 livers and 41 spleens. We train a model on the first
71 livers and hold out the last 40 for evaluation. We also train another model
on the first 31 spleens with the last 10 held out for evaluation. We augment
the training data using the recent PointWOLF algorithm [25] to create 100 new
shapes for each training shape. PointWOLF applies a smoothly varying non-rigid
transformation to mesh vertices and yields diverse and realistic
augmentations.

We use 128 dimensional latent vectors and an icosahedron subdivided 3 times
for training and 4 times for fitting as a template for the decoder. To learn
these vectors and the decoder weights we solve the minimization problem of Eq.
1 with $\lambda_{\mathrm{reg}}=10^{-4}$ and $\lambda_{\mathrm{n}}=10^{-3}$. To
this end, we use the ADAM optimizer [26]. We set the learning rate to 0.002,
the momentum terms to $\beta_{1}=0.9,\beta_{2}=0.999$ and train for 24 hours
on a single NVIDIA Tesla V100 GPU (ca. 7,500 epochs). If the loss does not
improve for 100 epochs we reduce the learning rate by a factor 2, down to a
minimum of $10^{-5}$.

#### Baselines

We compare our model against the following baselines: DeepSDF [44],
SIREN+DeepSDF where the training and inference method is as in DeepSDF but
with a SIREN [51] based decoder, and the DUAL-MLP approach of [37]. We also
compare to DASM, the active surfaces of [58], along with an improved version
that we dub DASM+R because it adds a re-meshing step during fitting to avoid
self-intersections. These are the only two baselines that do not rely on a
learned shape prior and simply promote smoothness.

As the DUAL-MLP authors did not release code, we implemented two separate
versions of it, one that uses one single latent vector per shape (global) and
one that uses several (local). All these methods were trained as recommended
in the relevant papers.

###  4.1 Shape Reconstruction from 3D Point Clouds

#### Experimental Setup.

We test the ability of the latent vector models to reconstruct unknown shapes
from a given class, here the liver and the spleen, by randomly and uniformly
sampling 2,500 points across the test surface and attempting to reconstruct
from them by minimizing the loss of Eq. 3. For DALS, DASM, and DASM+R that use
a mesh-based representation, we take $\mathcal{L}_{\mathrm{task}}$ to be the
Chamfer distance. For the other methods, we take it to be the mean absolute
SDF value at the sample points. For all methods, we use the ADAM optimizer to
minimize their fitting losses. For this task, we set
$\lambda_{\mathrm{reg}}=0.001$ and $\lambda_{\mathrm{dir}}=0.2$. Finally, for
DASM+R and DALS we post process the results using five iterations of Botsch-
Kobbelt remeshing [5].

To evaluate the reconstructions, we use the Chamfer distance, the Hausdorff
distance, and the F-score [55, 27] at 1% and 2% of the surface’s bounding
sphere diameter. We also evaluate the mesh quality of the reconstructions
using the quality measure of Eq. 2 and the percentage of self-intersecting
faces.

#### Results.

(a) Ground truth.

(b) DeepSDF.

(c) SIREN+DeepSDF.

(d) DASM.

(e) DASM+R.

(f) DUAL-MLP (global).

(g) DUAL-MLP (local).

(h) DALS (Ours).

Figure 3: Reconstruction of a previously unseen liver from 2500 3D points. For each method, we present the full 3D volume and a version of it cut in the middle. The red outline denotes the ground-truth section and the black one that of the reconstructed organ. Note that only ours is smooth while still following closely the ground-truth one. Table 1: Quantitative results for reconstructing unseen livers from unoriented points. For each metric we report the mean and standard deviation over the reconstructed shapes. DALS concistently produces better reconstructions while still having very good mesh quality. |  Chamfer*$\downarrow$ |  Hausdorff$\downarrow$ |  F@1%$\uparrow$ |  F@2%$\uparrow$ |  Quality$\uparrow$ |  %self. ints.$\downarrow$
---|---|---|---|---|---|---
DeepSDF [44] |  40.7 $\pm$ 23.7 |  0.21 $\pm$ 0.06 |  31.3 $\pm$ 13.0 |  63.8 $\pm$ 15.6 | 0.98 $\pm$ 0.00 | 0.00 $\pm$ 0.00
SIREN+DeepSDF [44, 51] |  36.2 $\pm$ 34.1 |  0.20 $\pm$ 0.04 |  51.4 $\pm$ 8.68 |  78.4 $\pm$ 9.27 | 0.98 $\pm$ 0.00 | 0.00 $\pm$ 0.00
DASM [58] |  17.0 $\pm$ 10.0 |  0.23 $\pm$ 0.50 |  87.7 $\pm$ 3.25 |  92.9 $\pm$ 2.57 |  0.74 $\pm$ 0.03 |  7.40 $\pm$ 4.57
DASM+R [58] |  8.8 $\pm$ 7.53 |  0.19 $\pm$ 0.07 |  94.6 $\pm$ 2.57 |  96.8 $\pm$ 1.97 | 0.98 $\pm$ 0.00 | 0.00 $\pm$ 0.00
DUAL-MLP (global) [37] |  13.6 $\pm$ 5.67 |  0.16 $\pm$ 0.05 |  71.7 $\pm$ 6.39 |  91.4 $\pm$ 3.48 | 0.98 $\pm$ 0.00 | 0.00 $\pm$ 0.00
DUAL-MLP (local) [37] |  161.7 $\pm$ 39.6 |  0.43 $\pm$ 0.04 |  44.5 $\pm$ 4.05 |  62.1 $\pm$ 4.08 | 0.98 $\pm$ 0.00 | 0.00 $\pm$ 0.00
DALS (Ours) | 2.4 $\pm$ 1.04 | 0.11 $\pm$ 0.04 | 95.4 $\pm$ 2.06 | 99.0 $\pm$ 0.76 | 0.98 $\pm$ 0.00 |  0.20 $\pm$ 0.40
*multiplied with 10,000

We report comparative results in Tab. 1. DALS consistently outperforms the
other approaches, in part because it can model sharp features more accurately,
as can be seen in the qualitative results of Fig. 3. Note especially the left
side point and the concavity in the lower middle part of the liver. DALS also
produces excellent mesh quality and keeps the number of intersecting triangles
very low although not zero. In future work, we will add an additional loss
term to eliminate them.

Table 2: Quantitative results for reconstructing unseen spleens. Metrics are
reported as in Tab. 1.

|  DASM+R [58] |  DALS (Ours)
---|---|---
Chamfer*$\downarrow$ | 1.6 $\pm$ 0.05 |  2.5 $\pm$ 0.89
Hausdorff$\downarrow$ | 0.05 $\pm$ 0.02 |  0.06 $\pm$ 0.01
F@1%$\uparrow$ | 97.8 $\pm$ 1.75 |  92.9 $\pm$ 2.73
F@2%$\uparrow$ | 100  $\pm$ 0.05 |  99.9 $\pm$ 0.11
Quality$\uparrow$ | 0.98 $\pm$ 0.00 | 0.98 $\pm$ 0.00
%self. ints.$\downarrow$ | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00
*multiplied with 10,000

|
---|---
(a) DASM+R | (b) DALS (Ours)

Figure 4: Reconstruction of previously unseen spleens from 2500 3D points. The
inset shows the ground truth in red.

We repeated the experiment on the much simpler and smoother spleen shapes. We
focused on DALS and DASM+R because they delivered the best results on the
liver. As can be seen in Tab. 2 and Fig. 4 DASM+R delivers very slightly
better metrics but a qualitatively worse reconstruction because it overfits to
the staircase artifacts on the ground-truth shape. In contrast, DALS yields an
organic shape which still fits the data well and can therefore be viewed as a
more realistic result. This effect also exists in the liver dataset but did
not affect the metrics as obviously because the original images were of higher
resolution and the artifacts had much less of an effect.

###  4.2 Shape Reconstruction from Planar Curve Annotations

#### Experimental Setup.

When annotating medical images, a common time-saving practice is to only
annotate three orthogonal 2D slices instead of the entire 3D image. We test
the ability of our model to reconstruct shapes from such weak annotations by
fitting to the 2D planar boundary curves extracted from the 2D slice
annotations. We compute the intersection curves between each held out liver
and three orthogonal axis-aligned planes and sample 5,000 points randomly on
each curve. As can be seen in Fig. 5(e), this represents a very sparse set of
data points and the quality of the shape priors embedded in the models is key
to obtaining good results.

For DALS, DASM, and DASM+R we take $\mathcal{L}_{\mathrm{task}}$ to be a
modified Chamfer distance that relies on distances within the annotation
planes (see supplementary for details). For the other methods, we again use
the mean absolute SDF value to compute $\mathcal{L}_{\mathrm{task}}$. We again
use the ADAM optimizer and Botsch-Kobbelt remeshing as in the previous
section. We set $\lambda_{\mathrm{reg}}=0.01$ and $\lambda_{\mathrm{dir}}=100$
as we want to rely heavily on the shape prior in this task.

#### Results.

We report comparative results in Tab. 3 and qualitative results in Fig. 5. To
generate these results, we only used annotations in the three orthogonal axis-
aligned planes. However, we can also annotate additional planes to provide
further information. In Fig. 6, we plot the same quality metrics as in Tab. 3
as a function of the number of annotated planes for one of the livers. Not
only are DASM results consistently better, but they improve almost
monotonically with the number of planes we provide, which is a very desirable
behavior in clinical practice.

Table 3: Quantitative results for reconstructing unseen livers from planar curve annotations. For each metric we report the mean and standard deviation over the reconstructed shapes. DALS consistently outperforms the baselines while retaining excellent mesh quality. |  Chamfer*$\downarrow$ |  Hausdorff$\downarrow$ |  F@1%$\uparrow$ |  F@2%$\uparrow$ |  Quality$\uparrow$ |  %self. ints.$\downarrow$
---|---|---|---|---|---|---
DeepSDF [44] |  4.36 $\pm$ 2.35 |  0.22 $\pm$ 0.06 |  41.5 $\pm$ 12.1 |  69.5 $\pm$ 12.2 | 0.98 $\pm$ 0.00 | 0.00 $\pm$ 0.00
SIREN+DeepSDF [44, 51] |  3.82 $\pm$ 2.24 | 0.21 $\pm$ 0.05 |  47.0 $\pm$ 7.67 |  74.4 $\pm$ 8.25 | 0.98 $\pm$ 0.00 | 0.00 $\pm$ 0.00
DASM+R [58] |  27.41 $\pm$ 7.91 |  0.47 $\pm$ 0.09 |  14.8 $\pm$ 5.99 |  29.5 $\pm$ 9.99 | 0.98 $\pm$ 0.00 |  0.02 $\pm$ 0.07
DUAL-MLP (global) [37] |  4.05 $\pm$ 1.58 |  0.23 $\pm$ 0.05 |  49.9 $\pm$ 6.17 |  74.4 $\pm$ 6.08 | 0.98 $\pm$ 0.00 | 0.00 $\pm$ 0.00
DUAL-MLP (local) [37] |  15.55 $\pm$ 4.53 |  0.39 $\pm$ 0.05 |  28.3 $\pm$ 2.87 |  48.3 $\pm$ 4.20 | 0.98 $\pm$ 0.00 | 0.00 $\pm$ 0.00
DALS (Ours) | 3.27 $\pm$ 1.48 | 0.21 $\pm$ 0.05 | 52.1 $\pm$ 6.56 | 77.2 $\pm$ 6.87 | 0.99 $\pm$ 0.00 | 0.00 $\pm$ 0.00
*multiplied with 1,000

(a) Ground truth.

(b) DeepSDF.

(c) SIREN+DeepSDF.

(d) DASM+R.

(e) Annotations.

(f) DUAL-MLP (global).

(g) DUAL-MLP (local).

(h) DALS (Ours).

Figure 5: Reconstruction of a previously unseen liver from outlines in three
different planes. The outlines are shown in the bottom left panel. Again, our
reconstruction is smooth while matching the outlines very accurately.

Figure 6: Reconstruction metrics for a liver as a function of the number of
annotated planes. Unlike those of other approaches, DALS results, shown in
purple, consistently improve as more planes are added. However, they tend to
saturate after 6 or 7.

###  4.3 3D Image Segmentation with Little Training Data

#### Experimental setup

A common medical image analysis task is to segment objects with very few
annotations available. Here, we use DALS to refine a voxel segmentation
produced by a CNN backbone network trained on a few 2D slice annotations. As a
backbone, we use the standard U-Net [10] and V-Net [39], along with the more
recent nn-U-Net [20] and UNETR [18].

For this experiment, we only use the 40 liver images we held off for testing
in the previous experiments. We use 20 of them to train the backbone and the
other 20 for testing purposes. This simulates a realistic scenario in which we
have few training images to train the segmentation network and they have not
been used to learn the shape priors.

#### Results.

Table 4: Quantitative results for segmentation refinement. For each metric we report the mean and standard deviation over the 20 reconstructed shapes. Refinement with DALS consistently improves the segmentations for all backbones and metrics. |  Dice$\uparrow$ |  Hausdorff$\downarrow$ |  Chamfer*$\downarrow$
---|---|---|---
| Raw | w/ DALS | Raw | w/ DALS | Raw | w/ DALS
U-Net [10] |  0.81 $\pm$ 0.08 | 0.83 $\pm$ 0.08 |  26.6 $\pm$ 10.2 | 20.9 $\pm$ 6.32 |  44.9 $\pm$ 51.0 | 26.0 $\pm$ 22.4
V-Net [39] |  0.79 $\pm$ 0.16 | 0.80 $\pm$ 0.16 |  28.5 $\pm$ 12.5 | 25.4 $\pm$ 9.29 |  56.4 $\pm$ 74.8 | 43.3 $\pm$ 54.8
nn-U-Net [20] |  0.84 $\pm$ 0.09 | 0.85 $\pm$ 0.07 |  25.2 $\pm$ 11.3 | 19.5 $\pm$ 8.08 |  38.7 $\pm$ 48.4 | 22.3 $\pm$ 20.4
UNETR [18] |  0.74 $\pm$ 0.13 | 0.75 $\pm$ 0.17 |  39.5 $\pm$ 24.2 | 26.4 $\pm$ 12.5 |  164.5 $\pm$ 266 | 52.7 $\pm$ 57.0
*multiplied with 10,000

(a) U-Net.

(b) V-Net.

(c) nn-U-Net.

(d) UNETR.

Figure 7: Comparison of raw (top) and refined (bottom) segmentations. The
black rings highlight examples of how refinement with DALS corrects
segmentation mistakes.

Because the models are not sufficiently well trained — a common occurrence in
medical imaging — the ‘raw’ segmentations reported in Tab. 4 are not
particularly good. However, to refine them by enforcing shape priors, we can
treat them as noisy data to which we fit a DALS model. To this end, we
initialize the shape at the center and scale predicted by the raw
segmentation. We then fit DALS to an unsigned distance function computed from
the segmentation binary image (full details in supplementary). We use
$\lambda_{\mathrm{reg}}=0$ and $\lambda_{\mathrm{dir}}=\infty$ to heavily rely
on the model’s learned prior.

As can be seen in Fig. 7 and Tab. 4, this yields much improved segmentations
both in terms of visual appearance and quantitative metrics. Note that DALS
removes spurious growths _and_ recovers concavities that were missed in the
raw segmentations. In other words, DALS does not simply smooth. It really
enforces geometric priors.

###  4.4 Ablation experiments

(a) Many latent vectors
and $\lambda_{\mathrm{reg}}=0$.

(b) Single latent vector
and $\lambda_{\mathrm{reg}}=0$

(c) Many latent vector
and $\lambda_{\mathrm{reg}}\neq 0$.

(d) Single latent vector
and $\lambda_{\mathrm{reg}}\neq 0$.

0.01.00.20.40.60.8

Figure 8: Reconstructions of the ablated models of Tab. 5. Facets are colored
according to their quality in terms of the measure of Eq. (2): yellow is high
and blue is low. The insets show the reconstruction by DALS including re-
meshing.

We perform an ablation study to investigate how using a single or multiple
latent vectors and our triangle quality loss $L_{\mathrm{reg}}$ affect
performance. To use a single vector, we constrain all latent vectors to be the
same during fitting. To remove $L_{\mathrm{reg}}$ we set
$\lambda_{\mathrm{reg}}$ to 0 during training and fitting.

Table 5: Quantitative results of the ablation study. Local inference results
in a large boost to reconstruction accuracy and the $L_{\mathrm{reg}}$ loss
significantly improves triangle quality. Adding remeshing further boosts the
triangle quality at some expense to reconstruction accuracy.

Local | $L_{\mathrm{reg}}$ |  Chamfer*$\downarrow$ |  Quality$\uparrow$
---|---|---|---
|  |  7.89 $\pm$ 2.75 |  0.75 $\pm$ 0.03
✓ |  |  1.89 $\pm$ 0.63 |  0.72 $\pm$ 0.02
| ✓ |  5.41 $\pm$ 1.77 |  0.83 $\pm$ 0.02

Local | $L_{\mathrm{reg}}$ | Remeshing |  Chamfer*$\downarrow$ |  Quality$\uparrow$
---|---|---|---|---
✓ | ✓ |  |  1.50 $\pm$ 0.49 |  0.87 $\pm$ 0.01
✓ | ✓ | ✓ |  2.41 $\pm$ 1.04 |  0.98 $\pm$ 0.00

As shown in Tab. 5, both our local latent vector approach and
$L_{\mathrm{reg}}$ loss significantly improves reconstruction and mesh
quality, even more so when combined. This is also apparent in Fig. 8 as the
reconstructions with $L_{\mathrm{reg}}$ are smoother and fewer triangles are
severely distorted. Finally, adding remeshing results in excellent mesh
quality at some cost to reconstruction accuracy. We chose to prioritise the
mesh quality for our results in this work.

##  5 Conclusion

We have shown that we train a latent vector model to represent a complex
surface by a triangulated sphere and deformations at each one of its vertices
that are the output of a decoder that takes as input the vertex coordinates
and a latent vector. At training time, we use an auto-decoder approach to
learn a single latent vector per training shape. However, at model-fitting
time, we allow the latent vectors to be different at each vertex but we
enforce consistency of these vectors across the triangulation. This enables us
to learn the model from a relatively small training set while giving the
necessary flexibility to model complex 3D shapes without over-smoothing. A key
ingredient is that we impose regularization constraints on the latent vectors
but not on the vertex 3D locations.

In this work, we have focused on organic shapes represented as watertight
surfaces and demonstrated the effectiveness of our approach on liver and
spleen reconstruction. In future work, we will extend our approach to non-
watertight surfaces by dynamically updating the template mesh, that is, by
removing faces that are predicted to be unused as in [43]. We will also
integrate re-meshing into DALS as we did for DASM+R to further improve mesh
quality.

Finally, as the purpose of our model is to incorporate prior knowledge into
medical image processing tasks, it is important to mention that our model may
introduce unwanted biases if the training data is heavily skewed towards
certain genders, ages, or other groups. The local nature of our model makes
this bias less direct, but representative training data is still a must.

## References

  * [1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas and Leonidas Guibas  “Learning representations and generative models for 3d point clouds”  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2018, pp. 40–49
  * [2] Michela Antonelli et al.  “The medical segmentation decathlon”  In _arXiv preprint arXiv:2106.05735_ , 2021
  * [3] Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E Hinton  “Layer normalization”  In _arXiv preprint arXiv:1607.06450_ , 2016
  * [4] R.. Bhatia and Kent L. Lawrence  “Two-dimensional finite element mesh generation based on stripwise automatic triangulation”  In _Computers & Structures_ 36.2, 1990, pp. 309–319
  * [5] Mario Botsch and Leif Kobbelt  “A remeshing approach to multiresolution modeling”  In _Proceedings of Eurographics_ , 2004, pp. 185–192
  * [6] Andrew Brock, Theodore Lim, James M Ritchie and Nick Weston  “Generative and discriminative voxel modeling with convolutional neural networks”  In _arXiv preprint arXiv:1608.04236_ , 2016
  * [7] Rohan Chabra et al.  “Deep local shapes: Learning local sdf priors for detailed 3d reconstruction”  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020, pp. 608–625
  * [8] Zhiqin Chen and Hao Zhang  “Learning implicit fields for generative shape modeling”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 5939–5948
  * [9] Julian Chibane, Thiemo Alldieck and Gerard Pons-Moll  “Implicit functions in feature space for 3d shape reconstruction and completion”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 6970–6981
  * [10] Özgün Çiçek et al.  “3D U-Net: learning dense volumetric segmentation from sparse annotation”  In _Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)_ , 2016, pp. 424–432  Springer
  * [11] Angela Dai, Charles Ruizhongtai Qi and Matthias Nießner  “Shape completion using 3d-encoder-predictor cnns and shape synthesis”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 5868–5877
  * [12] P. Fua  “Model-Based Optimization: Accurate and Consistent Site Modeling”  In _Congress of International Society for Photogrammetry and Remote Sensing_ , 1996
  * [13] P. Fua and Y.. Leclerc  “Object-Centered Surface Reconstruction: Combining Multi-Image Stereo and Shading”  In _International Journal of Computer Vision (IJCV)_ 16, 1995, pp. 35–56
  * [14] Jun Gao et al.  “Learning deformable tetrahedral meshes for 3d reconstruction”  In _Neural Information Processing Systems (NeurIPS)_ 33, 2020, pp. 9936–9947
  * [15] Thibault Groueix et al.  “A papier-mâché approach to learning 3d surface generation”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 216–224
  * [16] Rana Hanocka, Gal Metzer, Raja Giryes and Daniel Cohen-Or  “Point2Mesh: a self-prior for deformable meshes”  In _ACM Transactions on Graphics (TOG)_ 39.4, 2020, pp. 126–1
  * [17] A. Hatamizadeh, D. Sengupta and D. Terzopoulos  “End-To-End Trainable Deep Active Contour Models for Automated Image Segmentation: Delineating Buildings in Aerial Imagery”  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020
  * [18] Ali Hatamizadeh et al.  “UNETR: Transformers for 3d medical image segmentation”  In _Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2022, pp. 574–584
  * [19] Lei He et al.  “A comparative study of deformable contour methods on medical image segmentation”  In _Image and Vision Computing (IVC)_ 26.2, 2008, pp. 141–163
  * [20] Fabian Isensee et al.  “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation”  In _Nature methods_ 18.2, 2021, pp. 203–211
  * [21] Chiyu Jiang et al.  “Local implicit grid representations for 3d scenes”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 6001–6010
  * [22] Anne Jorstad et al.  “NeuroMorph: a toolset for the morphometric analysis and visualization of 3D models derived from electron microscopy image stacks”  In _Neuroinformatics_ 13.1, 2015, pp. 83–92
  * [23] Kristine Aavild Juhl et al.  “Implicit Neural Distance Representation for Unsupervised and Supervised Classification of Complex Anatomies”  In _Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)_ , 2021, pp. 405–415
  * [24] Michael Kass, Andrew Witkin and Demetri Terzopoulos  “Snakes: Active contour models”  In _International Journal of Computer Vision (IJCV)_ 1.4, 1988, pp. 321–331
  * [25] Sihyeon Kim et al.  “Point Cloud Augmentation with Weighted Local Transformations”  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2021, pp. 548–557
  * [26] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2015
  * [27] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou and Vladlen Koltun  “Tanks and temples: Benchmarking large-scale scene reconstruction”  In _ACM Transactions on Graphics (TOG)_ 36.4, 2017, pp. 1–13
  * [28] M.. Leventon, W.. Grimson and O. Faugeras  “Statistical Shape Influence in Geodesic Active Contours”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2000, pp. 316–323
  * [29] Thomas Lewiner, Hélio Lopes, Antônio Wilson Vieira and Geovan Tavares  “Efficient implementation of marching cubes’ cases with topological guarantees”, 2003, pp. 1–15
  * [30] Justin Liang et al.  “Polytransform: Deep polygon transformer for instance segmentation”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 9131–9140
  * [31] Huan Ling et al.  “Fast interactive object annotation with curve-gcn”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 5257–5266
  * [32] Or Litany, Alex Bronstein, Michael Bronstein and Ameesh Makadia  “Deformable shape completion with graph convolutional autoencoders”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 1886–1895
  * [33] Lingjie Liu et al.  “Neural sparse voxel fields”  In _Neural Information Processing Systems (NeurIPS)_ , 2020, pp. 15651–15663
  * [34] Diego Marcos et al.  “Learning deep structured active contours end-to-end”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 8877–8885
  * [35] Julien N.P. Martel et al.  “ACORN: Adaptive Coordinate Networks for Neural Representation”  In _ACM Transactions on Graphics (TOG)_ , 2021
  * [36] T. Mcinerney and D. Terzopoulos  “A Dynamic Finite Element Surface Model for Segmentation and Tracking in Multidimensional Medical Images with Application to Cardiac 4D Image Analysis”  In _Computerized Medical Imaging and Graphics_ 19.1, 1995, pp. 69–83
  * [37] Ishit Mehta et al.  “Modulated periodic activations for generalizable local functional representations”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 14214–14223
  * [38] Lars Mescheder et al.  “Occupancy networks: Learning 3d reconstruction in function space”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 4460–4470
  * [39] Fausto Milletari, Nassir Navab and Seyed-Ahmad Ahmadi  “V-net: Fully convolutional neural networks for volumetric medical image segmentation”  In _Proceedings of the International Conference on 3D Vision (3DV)_ , 2016, pp. 565–571  IEEE
  * [40] Luca Morreale et al.  “Neural Convolutional Surfaces”  In _arXiv preprint arXiv:2204.02289_ , 2022
  * [41] Thomas Müller, Alex Evans, Christoph Schied and Alexander Keller  “Instant Neural Graphics Primitives with a Multiresolution Hash Encoding”  In _arXiv preprint arXiv:2201.05989_ , 2022
  * [42] Baptiste Nicolet, Alec Jacobson and Wenzel Jakob  “Large steps in inverse rendering of geometry”  In _ACM Transactions on Graphics (TOG)_ 40.6, 2021, pp. 1–13
  * [43] Junyi Pan et al.  “Deep mesh reconstruction from single rgb images via topology modification networks”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 9964–9973
  * [44] Jeong Joon Park et al.  “Deepsdf: Learning continuous signed distance functions for shape representation”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 165–174
  * [45] Sida Peng et al.  “Deep snake for real-time instance segmentation”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 8533–8542
  * [46] Songyou Peng et al.  “Convolutional occupancy networks”  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020, pp. 523–540
  * [47] Songyou Peng et al.  “Shape As Points: A Differentiable Poisson Solver”  In _Neural Information Processing Systems (NeurIPS)_ , 2021
  * [48] R. Prevost et al.  “Incorporating Shape Variability in Image Segmentation via Implicit Template Deformation”  In _Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)_ , 2013, pp. 82–89
  * [49] Tianchang Shen et al.  “Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis”  In _Neural Information Processing Systems (NeurIPS)_ , 2021
  * [50] Jonathan Shewchuk  “What is a good linear finite element? interpolation, conditioning, anisotropy, and quality measures (preprint)”  In _University of California at Berkeley_ , 2002
  * [51] Vincent Sitzmann et al.  “Implicit neural representations with periodic activation functions”  In _Neural Information Processing Systems (NeurIPS)_ 33, 2020, pp. 7462–7473
  * [52] Edward J. Smith, Scott Fujimoto, Adriana Romero and David Meger  “GEOMetrics: Exploiting Geometric Structure for Graph-Encoded Objects”  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2019, pp. 5866–5876
  * [53] Justin Solomon, Keegan Crane and Etienne Vouga  “Laplace-Beltrami: The Swiss army knife of geometry processing”  In _Symposium on Geometry Processing Graduate School (Cardiff, UK, 2014)_ 2, 2014
  * [54] Towaki Takikawa et al.  “Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021
  * [55] Maxim Tatarchenko et al.  “What do single-view 3d reconstruction networks learn?”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3405–3414
  * [56] Demetri Terzopoulos, Andrew Witkin and Michael Kass  “Constraints on deformable models: Recovering 3D shape and nonrigid motion”  In _Artificial Intelligence_ 36.1 Elsevier, 1988, pp. 91–123
  * [57] Demetri Terzopoulos, Andrew Witkin and Michael Kass  “Symmetry-seeking models and 3D object reconstruction”  In _International Journal of Computer Vision (IJCV)_ 1.3 Springer, 1988, pp. 211–221
  * [58] Udaranga Wickramasinghe, Pascal Fua and Graham Knott  “Deep active surface models”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 11652–11661
  * [59] Francis Williams et al.  “Neural Fields as Learnable Kernels for 3D Reconstruction”, 2021  arXiv:2111.13674 [cs.CV]

We provide additional details on how we fit our models to planar curves and to
voxel segmentations, as described in Sections 4.2 and 4.3 respectively.

##  Appendix A Fitting DALS to Planar Curve Annotations

When annotations are only provided in 2D planes, we only wish to evaluate the
reconstruction in these planes. This is similar to how plane annotations are
handled for 3D voxel segmentations [10].

Figure 9: Gradient propagation for samples on the intersection between a
triangle mesh and a plane. The blue curve is the intersection of the mesh and
the plane, and the red curve is the ground truth boundary curve. The
highlighted triangle has vertices u, v, and w and intersects the plane in the
line segment spanned by $\textbf{x}_{1}$ and $\textbf{x}_{2}$.

Formally, assume we are given a plane $\mathcal{P}$ and a triangle mesh
$\mathcal{M}$. To differentially sample points on the intersection between
$\mathcal{P}$ and $\mathcal{M}$ we first find the intersection between the
plane and each triangle facet. The intersection of a plane and triangle is
either empty or a line segment spanned by two points $\textbf{x}_{1}$ and
$\textbf{x}_{2}$. We ignore the degenerate cases where the intersection is the
entire triangle or only one of its vertices. We can then sample a point p on
the intersecting line segment as
$\textbf{p}=r\textbf{x}_{1}+(1-r)\textbf{x}_{2}$, where $r\in U(0,1)$. Let
$(\alpha_{1},\beta_{1},\gamma_{1})$ and $(\alpha_{2},\beta_{2},\gamma_{2})$ be
the barycentric coordinates of, respectively, $\textbf{x}_{1}$ and
$\textbf{x}_{2}$. We can then write p in terms of the triangle vertices u, v,
and w as

$\textbf{p}=\begin{bmatrix}\textbf{u}&\textbf{v}&\textbf{w}\end{bmatrix}\begin{bmatrix}\alpha_{1}&\alpha_{2}\\\
\beta_{1}&\beta_{2}\\\ \gamma_{1}&\gamma_{2}\end{bmatrix}\begin{bmatrix}r\\\
1-r\end{bmatrix}\,.$ (5)

As a result, p is a linear combination of the triangle vertices and $r$ is an
independent stochastic term. Therefore, we can propagate a gradient from the
point p to the triangle vertices u, v, and w [52], see Fig. 9.

Now, let $\mathcal{S}_{\mathcal{P}}(\mathcal{M})$ denote a set of $M$ points
sampled differentially on the intersection of $\mathcal{M}$ and $\mathcal{P}$.
Further, let $\mathcal{T}_{\mathcal{P}}$ be a set of $N$ points sampled
uniformly on the planar curve annotations for plane $\mathcal{P}$. In this
work we use $M=5000$ sample points. The loss for plane $\mathcal{P}$ is then

$L_{\text{cf}}(\mathcal{S}_{\mathcal{P}}(\mathcal{M}),\mathcal{T}_{\mathcal{P}})=\frac{1}{M}\sum_{\textbf{q}\in\mathcal{T}_{\mathcal{P}}}\min_{\textbf{p}\in\mathcal{S}_{\text{P}}(\mathcal{M})}\left\lVert\textbf{p}-\textbf{q}\right\rVert_{2}^{2}+\frac{1}{M}\sum_{\textbf{p}\in\mathcal{S}_{\text{P}}(\mathcal{M})}\min_{\textbf{q}\in\mathcal{T}_{\mathcal{P}}}\left\lVert\textbf{q}-\textbf{p}\right\rVert_{2}^{2}\,.$
(6)

Note that the above is the Chamfer loss between the plane sample points [52].

Finally, given a collection of planes
$\mathcal{P}_{1},\mathcal{P}_{2},\ldots,\mathcal{P}_{P}$, the fitting loss,
$\mathcal{L}_{\text{task}}$, is

$\mathcal{L}_{\text{task}}(\mathcal{M})=\frac{1}{P}\sum_{i=1}^{P}L_{\text{cf}}(\mathcal{S}_{\mathcal{P}_{i}}(\mathcal{M}),\mathcal{T}_{\mathcal{P}_{i}})\,.$
(7)

##  Appendix B Fitting DALS to Voxel Segmentations

To fit DALS to a binary 3D voxel image $\textbf{B}\in\mathbb{R}^{W\times
H\times D}$ we first use the Euclidean distance transform to create a new
image $\textbf{U}\in\mathbb{R}^{W\times H\times D}$ where each voxel contains
the unsigned distance to the segmentation boundary. Given a mesh $\mathcal{M}$
with $V$ vertices, the fitting loss is then given by

$\mathcal{L}_{\text{task}}(\mathcal{M})=\frac{1}{V}\sum_{v=1}^{V}\textbf{U}(\textbf{x}_{v})\,,$
(8)

where $\textbf{U}(\textbf{x}_{v})$ is trilinear interpolation of U at vertex
position $\textbf{x}_{v}$.

To optimize the latent vectors $\textbf{Z}\in\mathbb{R}^{V\times d}$ we
require the gradient of U at $\textbf{x}_{v}$. To get a robust estimate, we
use a Sobel operator to pre-compute a gradient image
$\textbf{G}\in\mathbb{R}^{W\times H\times D\times 3}$ which contains the
gradient of U at each voxel position. We then use trilinear interpolation to
evaluate the gradient of U as
$\nabla\textbf{U}(\textbf{x}_{v})=\textbf{G}(\textbf{x}_{v})$.

We also attempted to fit DALS directly to the binary segmentation or the
softmax outputs of the CNN backbone. In practice, we found that the distance
field gradients made it easier for the model to fit the images.


# Language-Grounded Control for Coordinated Robot Motion and Speech

Ravi Tejwani1, Chengyuan Ma1, Paco Gomez-Paz2, Paolo Bonato3 and H. Harry
Asada4 1Ravi Tejwani and Chengyuan Ma are with Dept. of Electrical Engineering
and Computer Science (EECS), Massachusetts Institute of Technology, Cambridge,
MA 02142 USA {tejwanir, macy404<EMAIL_ADDRESS>Gomez-Paz is with Dept. of
Mathematics, Massachusetts Institute of Technology, Cambridge, MA 02142 USA
pjgomez@@mit.edu3Paolo Bonato is with Harvard Medical School Department of
Physical Medicine and Rehabilitation at Spaulding Rehabilitation Hospital,
Charlestown, MA 02129 USA<EMAIL_ADDRESS>Harry Asada is with Dept.
of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge,
MA 02142 USA<EMAIL_ADDRESS>
###### Abstract

Recent advancements have enabled human-robot collaboration through physical
assistance and verbal guidance. However, limitations persist in coordinating
robots’ physical motions and speech in response to real-time changes in human
behavior during collaborative contact tasks. We first derive principles from
analyzing physical therapists’ movements and speech during patient exercises.
These principles are translated into control objectives to: 1) guide users
through trajectories, 2) control motion and speech pace to align completion
times with varying user cooperation, and 3) dynamically paraphrase speech
along the trajectory. We then propose a Language Controller that synchronizes
motion and speech, modulating both based on user cooperation. Experiments with
12 users show the Language Controller successfully aligns motion and speech
compared to baselines. This provides a framework for fluent human-robot
collaboration.

##  I INTRODUCTION

Robots have been enabled to collaborate with humans by providing physical
assistance as well as verbal guidance during collaborative tasks. Research on
robots providing physical assistance has shown robots assisting with heavy
lifting and materials handling in warehouses and factories [1]; handling
payloads, reducing physical strain on human workers [2]; and helping turn and
lift patients, freeing up human nurses for other critical care tasks [3]. In
addition to providing physical assistance, robots have also been used to
provide verbal instructions and dialog interaction in human-robot
collaboration [4, 5]. Using natural language capabilities, robots understand
commands, ask clarifying questions, and provide guidance to human partners
[6]. Research suggests that language-enabled robots lead to higher perceived
collaboration quality compared to silent robot partners [7].

Human-robot collaboration remains limited by the lack of natural coordination
between physical interactions and verbal communication. Simply combining
robot’s motions and its speech in parallel cannot achieve natural, fluent
coordination. An open challenge remains to develop adaptive control frameworks
that closely coordinate a robot’s physical motions and speech utterances,
dynamically modulating both based on real-time changes in human behavior.

We propose a Language Controller that dynamically aligns the robot’s motion
and speech under changing user cooperation so both end at the same time. It
does so by varying admittance parameters, audio pace, and adaptive
paraphrasing. The controller is inspired by principles derived from analysis
of human-human physical interactions, specifically from an observational study
of a physical therapist collaborating with a patient during therapeutic
exercises at a rehabilitation center. From observations of the therapist’s
physical and verbal guidance, we identify core principles and translate them
into formal control objectives for the controller (details in section III).

We make the following contributions:

  1. 1.

Formalize core principles to derive control objectives for natural human-robot
collaboration – adaptive pacing, aligned speech-motion timing, correlating
speech complexity with motion speed – from observations of human-human
physical therapy interactions;

  2. 2.

Language Robot Controller from the derived control objectives in order to
align the robot motion with the verbal speech in the human-robot interaction;

  3. 3.

Extensive human experiments that validate the controller and demonstrate its
ability to align the pace of the robot motion with its speech.

##  II RELATED WORK

###  II-A Language Grounding for Robot Instructions

Prior work has focused on grounding natural language instructions to enable
robots to follow commands, including techniques for mapping instructions to
internal representations and actions [8, 9, 10]. Other efforts have targeted
collaborative grounding of language between humans and robots for situated
dialog and interactions [11, 12]. While enabling planning and collaboration,
integrating robot physical motions and speech grounded in real-time human
responses still remains an open problem. Our work aims to address this gap.

###  II-B Variable Impedance and Admittance Control

Research has explored variable impedance and admittance control for safer and
adaptive human-robot interaction, including dynamic modulation based on
cooperation [13], adaptive admittance using EEG feedback [14], and online
impedance variation for performance/safety trade-offs [15]. We incorporate
admittance methods for compliant motion, but extend standard admittance
frameworks by explicitly coupling the modulation of control parameters to the
speech state. This ties motion control to verbal communication.

###  II-C Language and Motion Integration

Recent works have combined language understanding with robotic planning and
control, including mapping commands to executable specifications [16, 17] and
leveraging implicit information to improve plan execution [18, 19]. However,
physical motions of the robot and concurrent speech have not been integrated
based on mutual understanding and real-time bidirectional communication. Our
Language Controller addresses this by developing a control framework that
coordinates motions and utterances grounded in human responses.

##  III PRINCIPLES AND CONTROL OBJECTIVES

Through the observational study of therapist-patient exercises at Spaulding
Rehabilitation Hospital (Fig. 2), we identified the following core principles:

  * •

The therapist planned the trajectories for each session, demonstrating the
path before starting an exercise;

  * •

The therapist adapted the pace of motions based on patient responses. When the
patient struggled, she slowed down and gently guided them along the
trajectory;

  * •

The therapist aligned her speech with physical actions. She began verbal
guidance at the start of motions and finished speaking around the end;

  * •

Her speech rate and sentence length correlated with her physical motion speed
- slower motions had slower, longer speech; faster motions had faster, shorter
speech.

We derived the following formal control objectives from these principles. The
robot must:

  1. 1.

Guide the user through a predefined trajectory while modulating its velocity
in response to user cooperation. High cooperation (low resistance) must lead
to faster motion and Low cooperation (high resistance) must lead to slower
motion;

  2. 2.

control the pace of its speech to maximize the alignment with its motion while
adapting to varying user cooperation. We define alignment as the robot
concluding its speech simultaneously as it completes motion.

  3. 3.

paraphrase (choice of words) its speech dynamically along the trajectory,
adapting to changing user cooperation. It must use shorter sentences under
faster motion and longer sentences under slower motion.

Figure 1: Physical therapist is seen performing ’shoulder external rotation’
therapy on to the patient with varying levels of physical resistance. The
physical motions and speech data was recorded across different
sessions.222Detailed therapy sessions videos can be seen at https://language-
playback-robot-controller.github.io/therapy-sessions/ Figure 2: Control scheme
of Language Controller. External force $F_{\text{ext}}$ (from user) and
virtual force $F_{\text{virtual}}$ (based on position to guide user through a
predefined trajectory) are passed to the virtual dynamics $Y_{v}$ to generate
reference velocity $v_{\text{ref}}$ for velocity controller $C$. $C$ outputs a
force that, together with $F_{\text{ext}}$, acts on robot dynamics $Y_{r}$.
The resulting motion is given by $\dot{x}$. Motion time-to-completion
estimator $S$ uses $\dot{x}$ to estimate the time-to-completion (ETC) of the
trajectory $\hat{t}_{x}$. Speech module $G$ reports an ETC of the speech,
$\hat{t}_{a}$, and cooperation model $Y_{c}$ computes cooperation $c$ from
$F_{\text{ext}}$. $\hat{t}_{x},\hat{t}_{a}$ and $c$ are fed to audio-motion
aligner $A$ to update Physical Pace $p$ and Audio Pace $a$. Physical Pace $p$
changes the pace of the robot’s motion by varying $Y_{v}$’s parameters. Audio
Pace $a$ changes the pace of audio through a phase vocoder in $G$.

Building upon the outlined control objectives, we formalize the Language
Controller.

##  IV LANGUAGE CONTROLLER

###  IV-A Overview

Language Controller (Fig. 2) employs variable admittance control for robot
motion and modulates the pace of motion and speech to maximize the alignment.
The controller is designed to align the motion of the robot with its speech.
It does so by updating “Physical Pace” and “Audio Pace” from estimated time-
to-completions for both the robot’s trajectory and its speech. The controller
dynamically updates the paces based on real-time user response. Furthermore,
the controller incorporates adaptive paraphrasing to modulate the speech
content. It traverses a phrase graph representation to select appropriate
wording and phrase length that matches the expected duration of motion.

###  IV-B Admittance Model

Admittance and Impedance Control are the two primary control schemes used in
human-robot interactions [20, 21]. Humans actively control their limbs and
resist unexpected movements, which positions humans as an impedance,
necessitating robots to be treated as an admittance. We thus use Admittance
Control, which converts external force $F_{\text{ext}}$ into desired velocity
$v_{\text{ref}}$ via a virtual dynamics model:

$M_{0}{\dot{v}_{\text{ref}}}+D_{0}v_{\text{ref}}=F_{\text{ext}}.$ (1)

where $v_{\text{ref}}$ is the desired velocity. To lead the user through a
predefined trajectory $T$, we extend the virtual dynamics model above with a
virtual force $F_{\text{virtual}}$,

$M_{0}{\dot{v}_{\text{ref}}}+D_{0}v_{\text{ref}}=F_{\text{ext}}+F_{\text{virtual}}.$
(2)

$F_{\text{virtual}}$ is dependent on the end effector position $x$ and its
closest point on the trajectory ($x_{d}$), defined as follows,

$x_{d}=T(d),\text{ where }\textstyle d=\operatorname*{arg\,min}_{0\leq d\leq
1}\|T(d)-x\|,$ (3)

where we see the trajectory $T$ as a directed curve $[0,1]\mapsto S$ where $S$
is the state space. $F_{\text{virtual}}$ consists of two parts:

$\displaystyle F_{\text{virtual}}$
$\displaystyle=\underbrace{K(x_{d}-x)}_{F_{\text{guide}}}+\underbrace{\textstyle\|F_{\text{propell}}\|\mathbf{b}}_{F_{\text{propell}}},$
(4) $\displaystyle\text{where }\mathbf{b}$ $\displaystyle=\lim_{d^{\prime}\to
d^{+}}\frac{T(d^{\prime})-x_{d}}{\|T(d^{\prime})-x_{d}\|},$ (5)

where $F_{\text{guide}}$ leads the user back on track if they deviate and
$F_{\text{propell}}$ leads the user to complete the trajectory. This achieves
our first objective that our robot should lead the user through the
trajectory. In frequency domain, (2) can be expressed as

$v_{\text{ref}}=A\cdot(F_{\text{ext}}+F_{\text{virtual}})\text{ where
}A=\frac{1}{M_{0}s+D_{0}}.$ (6)

We elaborate in IV-D how to vary $A$ with Physical Pace $p$ to tune the
behavior of this admittance model.

The virtual admittance model produces $v_{\text{ref}}$ that is fed to the
velocity controller $C$, which produces a driving force $F_{\text{drive}}$
with actuators. The equation of motion of our robot is:

$M_{\text{robot}}\ddot{x}=F_{\text{ext}}+F_{\text{drive}}=F_{\text{ext}}+C(v_{\text{ref}}-\dot{x}).$
(7)

###  IV-C Parameters

Aligning the robot’s motion and its speech requires us to be able to control
their pace. In our controller, this is done by varying the Audio Pace $a$ and
the Physical Pace $p$.

Audio Pace ($a$) is the pace at which the robot’s speech is played. E.g.,
$a=1.2$ means the audio is played $20\%$ faster than normal and $a=0.8$ means
playing $20\%$ slower. We pass $a$ to a _Phase Vocoder_[22] which time-scales
the prerecorded speech audio with Short-time Fourier Transform. We empirically
constrain $a\in(0.6,1.4)$ to avoid incongruity arising from over-stretching or
shrinking of the speech audio.

Physical Pace ($p$) is a variable in our admittance controller. It can be
thought of as a “speed knob” with which we vary the controller. We design $p$
to achieve the following effect: assuming constant $F_{\text{ext}}$, $p=1.2$
should cause our controller to complete a trajectory in $20\%$ less time than
a fixed admittance controller following (2) and likewise $p=0.8$ should cause
it to run $20\%$. We empirically constrain $p\in(0.6,1.4)$ so the admittance
parameters do not deviate too much from their base values. Physical Pace ($p$)
must not be confused with end effector velocity.  The role of $p$ is not to
directly modulate velocity, but to adjust the admittance parameters. This
ensures that the controller operates within the safety and compliance
boundaries set by the admittance control framework. Directly multiplying the
end effector velocity by $p$ could lead to unsafe conditions, as it would
bypass these regulatory mechanisms. Therefore, $p$ should be understood as a
rate constant that modifies the admittance parameters to indirectly influence
end effector velocity, maintaining safety and compliance even under varying
external forces. The formal definition of Physical Pace $p$ is in IV-D.

Estimated time-to-complete (ETC) for audio and speech We define alignment of
motion and speech as the two ending at the same time. Therefore, to modulate
the paces, our controller naturally needs to continuously estimate when the
motion and speech will end. Concretely, our controller computes the ETC for
audio ($\hat{t}_{a}$) and motion ($\hat{t}_{x}$). $\hat{t}_{a}$ and
$\hat{t}_{x}$ are computed 333$\hat{t}_{a}$ is computed as the sum of audio
length on a path constructed by repeated use of (20) from the current vertex
in the phrasing graph (minus the duration played for the current audio ).
$\hat{t}_{x}$ is computed assuming $F_{\text{ext}}=0$ (i.e., fully cooperative
user) with simulation of (7) at 500Hz.  _under base pace $a=p=1$_444Both paces
are defined relative to a base pace of $1$. $a=1$ means the audio is played at
the recorded rate (free of distortions); $p=1$ means the trajectory is being
run with the default / intended admittance parameters. We consider $1$ to be
the most natural/ideal pace..

Computing Paces $p$ and $a$ We compute the ideal paces, $p^{*}$ and $a^{*}$,
from the following optimization:

$\text{minimize }(p-p_{\text{natural}})^{2}+(a-a_{\text{natural}})^{2},\text{
s.t. }\frac{\hat{t}_{x}}{p}=\frac{\hat{t}_{a}}{a},$ (8)

where we set $p_{\text{natural}}=a_{\text{natural}}=1$. $\hat{t}_{x}/p$ and
$\hat{t}_{a}/a$ are the ETC for motion and audio considering the current pace,
and we equate them above to express the intent that both should end at the
same time, i.e., aligned. The solution to (8) is

$p^{*}=\frac{s+1}{s^{2}+1},\quad a^{*}=\frac{s^{2}+s}{s^{2}+1}\text{ where
}s=\frac{\hat{t}_{x}}{\hat{t}_{a}}.$ (9)

We further update $p$ and $a$ following the equation below:

$\displaystyle\dot{p}=k_{p}(p^{*}-p),\quad\dot{a}=k_{a}(a^{*}-a).$ (10)

This control equation makes both paces converge exponentially to their optimal
values555We do not directly set $p=p^{*}$ and $a=a^{*}$, which could lead to
abrupt change of pace if $p^{*}$ and $a^{*}$ deviate from current values too
much..

User Cooperation We define Cooperation ($0<c<1$) as

$c(t)=1-\int_{0}^{t}\alpha^{t-\tau}\frac{\left\|F_{\text{ext}}\right\|}{\|F\|_{\text{max}}}d\tau,$
(11)

where $\|F\|_{\text{max}}$ is the maximum magnitude of resisting force and
$\alpha$ is the decay factor. In practice, we apply a deadband filter to
$F_{\text{ext}}$ first to filter out sensor noises and friction.

We then extend (10) to

$\displaystyle\dot{p}=k_{p}(p^{*}-p),\quad\dot{a}=k_{a}(a^{*}-a)-k_{c}(1-c).$
(12)

This allows our controller to slow down the speech when the user does not
cooperate (high resistance) and resume ideal speech pace when the user fully
cooperates (low resistance). (12) achieves our second objective that the
robot’s motion and audio must be aligned under varying user cooperation.

###  IV-D Varying Admittance Model with Physical Pace $p$

We now give a formal definition of the Physical Pace $p$ and integrate it into
the virtual dynamics defined in (2). Let $v_{\text{ref}}^{*}(t)$ be the
reference velocity generated by an admittance controller per (2) (without
$p$), and let $v_{\text{ref}}(t)$ be the reference velocity from our
controller (with pace $p$). Assuming fixed
$(F_{\text{ext}}+F_{\text{virtual}})$, we want

$v_{\text{ref}}(t)=pv_{\text{ref}}^{*}(pt)$ (13)

which, after integrating both sides, implies that a controller with pace $p$
would reach a reference position in $1/p$ the time of a controller without
$p$. E.g., when $p=2$, a controller without $p$ would take twice amount the
time to reach the same position as a controller with $p$. More generally, pace
$p$ varies with time. Let $p_{t}$ be the pace at time $t$, we want

$v_{\text{ref}}(t)=p_{t}v_{\text{ref}}^{*}(\Phi),\text{ where
}\Phi={\textstyle\int_{0}^{t}p_{\tau}\,d\tau}.$ (14)

Here time $\Phi$ generalizes $pt$ for time-varying $p$ in (13). The assumption
of fixed $(F_{\text{ext}}+F_{\text{virtual}})$ in the definition is critical:
$p$ generally is not a scale factor to the end effector velocity but instead
modulates the admittance parameters. This approach is safer as $p$ effects the
velocity only indirectly through the admittance model, which ensures the
safety and compliance of our controller under varying forces.

We achieve (14) with the variable admittance model:

$\displaystyle v_{\text{ref}}$
$\displaystyle=A(p_{t})\cdot(F_{\text{ext}}+F_{\text{virtual}}),$ (15)
$\displaystyle\text{ where }A(p_{t})$
$\displaystyle=\frac{1}{\frac{1}{p_{t}^{2}}M_{0}s+\frac{1}{p_{t}}D_{0}-\frac{\dot{p}_{t}}{p_{t}^{3}}M_{0}}.$
(16)

###### Proof:

Differentiate (14) & multiply both sides by $M_{0}$,

$\displaystyle M_{0}\dot{v}_{\text{ref}}(t)$
$\displaystyle=p_{t}^{2}M_{0}\dot{v}_{\text{ref}}^{*}(\Phi)+\dot{p}_{t}M_{0}v_{\text{ref}}^{*}(\Phi).$
(17) Expanding $M_{0}\dot{v}_{\text{ref}}^{*}(t)$ by (2): $\displaystyle
M_{0}\dot{v}_{\text{ref}}(t)$
$\displaystyle=p_{t}^{2}(F_{\text{ext}}+F_{\text{virtual}})-p_{t}^{2}D_{0}v_{\text{ref}}^{*}(\Phi)+\dot{p}_{t}M_{0}v_{\text{ref}}^{*}(\Phi).$
Substitute $v_{\text{ref}}^{*}(\Phi)=v_{\text{ref}}(t)/p_{t}$ by (14)
$\displaystyle M_{0}\dot{v}_{\text{ref}}(t)$
$\displaystyle=p_{t}^{2}(F_{\text{ext}}+F_{\text{virtual}})-(p_{t}D_{0}-{\textstyle\frac{\dot{p}_{t}}{p_{t}}M_{0}})v_{\text{ref}}.$
(18)

whose simplification then leads to (16). ∎

_E. Passivity Guarantees_

We show that our controller defined in (16) is passive. A passive system is a
system that is constrained in such a way that it does not inject excessive
energy or instability into the interaction [23]. Formally, a system is passive
w.r.t. an input-output pair $(u(t),y(t))$ if and only if there exists a
positive definite storage function $V$ over the system such that:

$V(t)-V(0)\leq\int_{0}^{t}u(t)^{T}\cdot y(t)dt\quad\forall t>0$ (19)

Theorem 1: Consider a controller of the form outlined in (16) operating with
linear trajectory $T$. If $K$ is orthogonal and positive definite, $D_{0}$ is
positive definite, and $p$ is lower-bounded by a positive value, then the
system is passive with respect to the force-velocity
($F_{\text{ext}},v_{\text{ref}}$) input-output pair.

(Proof provided in the Online Supplementary Material 666 Online Supplemental
Proof: https://language-playback-robot-controller.github.io/proofs/.)

Figure 3: Example phrasing graph for our experiments where users were asked to
retract their arms. Paths ending at red nodes represent different phrasings of
the instruction, e.g., “I want you to retract your arm” (short) or “I want you
to move your arm back towards your body.” (long)  Figure 4: A user participant
is seen interacting with the UR5 robot on a desired trajectory, which was
inspired by the therapy session (“shoulder external rotation”) and predefined
in our controller.888Results for all the users are available in our online
appendix https://language-playback-robot-controller.github.io/user-sessions/.
Figure 5:  States and metrics of our controller from a single user session
through a predefined intended trajectory. At $t=3$ (red shaded timespan),
cooperation dropped, causing end effector speed to decrease. This shows that
our motion controller is compliant. Audio pace also dropped and returned back
up at $t=4$, which shows our controller adapts the pace of the robot’s speech
to changing user cooperation. Throughout the session, our controller kept the
estimated time-to-completion (ETC) of speech and trajectory close (the
estimated misalignment is around 0). This demonstrates that our controller
successfully aligns the robot’s speech with its motion. Around $t=7$, ETC for
trajectory $\hat{t}_{x}$ dropped, making the controller select a shorter path
on the phrasing graph, as shown by an abrupt drop in $\hat{t}_{a}$. However,
the temporary dip in $\hat{t}_{x}$ was caused by inaccuracies of estimation,
so the controller reverted to its original planned next phrase shortly after.
As resistance dropped around $t=9$, the controller eventually paraphrased and
chose the shorter path. The paraphrase shortened the ETC to less than the
trajectory, so the controller re-aligns by reducing audio pace and increasing
physical pace. Overall, this demonstrates that our controller is able to align
the robot’s speech with its motion while adapting to changing user
cooperation.66footnotemark: 6 101010Step-like patterns for $\hat{t}_{a}$
around $t=5$ and $t=7$ is due to minor imperfection in our audio code when
transitioning between different phrases at the time of the study. The steps
should be lines of the same slope as the immediately preceding line and don’t
affect the correctness of our analysis.

###  IV-F Adaptively Paraphrasing the Robot Speech

To enable the robot to dynamically paraphrase its speech so the speech length
matches that of the robot’s motion (our third objective), we represent of
speech as a phrasing graph. A phrasing graph is a Directed Acyclic Graph (DAG)
where vertices denote sequences of words or phrases and a directed edge from
vertex $u$ to $v$ denotes that the $v$’s phrase could follow $u$’s in speech.
Phrasing graph captures the various alternative ways to express similar
meanings (see Fig. 3).

When the controller finishes saying the phrase on a vertex, it chooses the
next vertex/phrase based on how long it expects the trajectory to last.
Formally, it does so by choosing a next vertex $u$ from the graph, such that:

$u=\operatorname*{arg\,min}_{\text{next node
}u}\left|\hat{t}_{x}-\frac{\hat{t}_{\text{min}}(u)+\hat{t}_{\text{max}}(u)}{2}\right|$
(20)

where $\hat{t}_{x}$ is the expected time-to-completion (ETC) of the trajectory
(IV-C). $t_{\text{min}}(u),t_{\text{max}}(u)$ are the minimum & maximum time-
to-completion of speeches starting at vertex $u$111111 $t_{\text{min}}(u)$ and
$t_{\text{max}}(u)$ are pre-computed by storing at each vertex the expected
time to say its phrase, and iterating in reverse topological order..

We remark that (a) (a) (20) causes the controller to select longer paraphrases
when the user resists more. Higher resistance slows the motion, increasing
trajectory time $\hat{t}_{x}$. (b) the graph traversal does not depend on $a$
or $p$ to avoid compound effects121212If we consider $p$ here, i.e., we choose
next vertex $v$ that minimizes $|\hat{t}_{x}/p-\ldots|$ instead of
$|\hat{t}_{x}-\ldots|$, a brief episode of low $p$ induces a longer speech
which requires even lower $p$ to align – a vicious cycle..

Figure 6: Violin plot of the distributions of metrics for 3 controllers across
all 12 user sessions. Blue bars indicate min, median, and max of a
distribution. Consistent distributions of cooperation for all controllers show
that all controllers deliver similar physical experience to the users; On this
ground, LC exhibits less audio-motion misalignment (both actual and predicted)
than LC-noAR which exhibits significantly less variation in misalignment than
the AC baseline. Adaptive paraphrasing allows LC to maintain a more natural
speed of speech most of the time than LC-noAR, as shown by a more concentrated
peak around the default rate of 1 in audio rate distribution. Overall, LC best
aligns the robot’s speech with motion66footnotemark: 6.

##  V EXPERIMENTATION

###  V-A User Study

The user study had 12 participants (7 males, 5 females, mean age 23). The UR5
robot [24] guided users through a predefined trajectory demonstrated by the
therapist. Users sat beside the robot, placed their hand on the desk. The
robot guided their hand along the trajectory with speech instructions while
users varied resistance arbitrarily.

###  V-B Control Schemes Evaluated and Compared

  1. 1.

Admittance Controller with Decoupled Audio (AC): Our baseline is a pure
admittance controller with dynamics of (2). The audio is a single prerecorded
audio file that starts simultaneously with the trajectory. Physical and Audio
Paces are not modulated at all.

  2. 2.

Language Controller without Adaptive Paraphrasing (LC-noAP): Our controller
but without the ability to adaptively paraphrase the speech.

  3. 3.

Language Controller (LC): Our controller with adaptive paraphrasing by
traversing a phrase graph (IV-F).

###  V-C Evaluation Metrics

  1. 1.

Cooperation ($c$), defined in (11), quantifies user cooperation. $c$ should
not significantly vary between control schemes.

  2. 2.

Audio Pace ($a$) is the speed at which we deliver the speech audio. A pace
closer to 1 implies less distortion from audio processing and more natural
speech.

  3. 3.

Physical Pace ($p$) is the state variable through which we vary the admittance
parameters of our controller. $p$ closer to 1 implies the robot operating
closer to its most natural admittance parameters. Always 1 for AC.

  4. 4.

Actual Misalignment is defined as the difference between the audio and motion
completion times. A positive value indicates the motion finished before the
audio; A negative value indicates otherwise. A smaller absolute value suggests
better audio-motion alignment.

  5. 5.

Estimated Misalignment ($\text{EM}=\hat{t}_{a}-\hat{t}_{x}$) is the real-time
estimate of Actual Misalignment (AM). A smaller absolute value suggests better
audio-motion alignment; EM differs from AM in that EM is a time series
computed throughout the session whereas the AM is a scalar obtained after the
session ends.

###  V-D Analysis of Language Controller on a User Session

We analyze our controller’s behavior for one of the user sessions in Fig. 10
and present a deep dive analysis. Around $t=3$, cooperation dropped, slowing
end effector speed. Audio pace dropped then recovered, aligning with motion.
Throughout, estimated time to completion (ETC) of speech and trajectory were
kept close, showing speech-motion alignment. Around $t=7$, a temporary
inaccurate ETC drop caused paraphrasing to a shorter phrase path, which was
committed once resistance dropped around $t=9$. Overall, Language Controller
adapts speech pace to follow motion changes from user cooperation variations,
while paraphrasing aligns speech ETC to trajectory ETC.

###  V-E Comparing Controllers across User Sessions

We compare all three control schemes across all users and present the results
in Fig. 6. AC exhibited broad misalignment (-4 to 3 seconds). Given short
session length, this indicates notable lack of alignment. Adding pace control
(LC-noAP) reduced misalignment, and adaptive paraphrasing (LC) further
improved to under 1 second. Both LC finish speech after motion. LC controllers
showed bimodal audio pace distribution - a peak around base rate 1, another at
lower end – corresponding to two behaviors: slowing speech during high
resistance to align with slower motion, and aiming for natural rate
without/low resistance. LC has more concentrated peak at 1, as adaptive
paraphrasing matches speech content length to motion, enabling natural rate
delivery.

The experiments show that our controller met all of our defined objectives. It
guided the users through the intended trajectory while adapting to the
changing user resistance, and it controlled the pace and content of the
robot’s speech to maximize its alignment with the robot’s motion.

##  VI LIMITATIONS

  1. 1.

We employed variable admittance controller for motion control. But the idea
generalizes to other controllers;

  2. 2.

Speech length can be varied with filler words and pauses, which our controller
does not yet implement;

  3. 3.

For richer interactions, large language models could automatically create
phrasing graphs, beyond our therapist-recorded interactions;

  4. 4.

Our controller assumes prerecorded audio phrases. An extension is using text-
to-speech to generate audio from the phrases;

  5. 5.

Formalized for physical therapy, Language Controller principles can be extend
to manufacturing, with robots assisting in lifting, handling, and assembly;
and space exploration, helping astronauts recover from falls.

##  VII CONCLUSION

We present a Language Controller that aligns robot motion and speech for human
collaboration, formalizing principles from analyzing therapist interactions
into control objectives. This enables compliant trajectories while modulated
speech rate and paraphrasing align content length. Experiments validate
motion-speech alignment of the controller over baselines. Future work expands
its capabilities and applies to assistive manufacturing, healthcare, and space
explorations.

## References

  * [1] R. Müller, M. Vette-Steinkamp, and K. Chatzilygeroudis, “Safe lifelong deployment of autonomous mobile robots in changing warehouses,” _Nature Machine Intelligence_ , vol. 4, no. 3, pp. 234–243, 2022.
  * [2] K. Haninger, C. Viol, and K. Fischer, “Towards fluid human-robot handovers of arbitrary objects,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 1197–1204, 2022.
  * [3] K. Barhydt and H. H. Asada, “A high-strength, highly-flexible robotic strap for harnessing, lifting, and transferring humans,” _IEEE Robotics and Automation Letters_ , vol. 8, no. 4, pp. 2110–2117, 2023.
  * [4] N. S. Selby, J. Ng, G. S. Stump, G. Westerman, C. Traweek, and H. H. Asada, “Teachbot: Towards teaching robotics fundamentals for human-robot collaboration at work,” _Heliyon_ , vol. 7, no. 7, 2021.
  * [5] T. Fong, C. Thorpe, and C. Baur, “Collaboration, dialogue, human-robot interaction,” in _Robotics research: The tenth international symposium_. Springer, 2003, pp. 255–266.
  * [6] J. Thomason, S. Zhang, R. Mooney, and P. Stone, “Learning to interpret natural language commands through human-robot dialog,” _Proceedings of the 24th International Conference on Artificial Intelligence_ , pp. 1923–1929, 2015.
  * [7] D. Szafir, “The good, the bad, and the neutral: A meta-analytic review of human-robot collaboration,” _Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction_ , pp. 49–58, 2021.
  * [8] S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy, “Understanding natural language commands for robotic navigation and mobile manipulation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 25, no. 1, 2011, pp. 1507–1514.
  * [9] C. Matuszek, L. Bo, L. Zettlemoyer, and D. Fox, “Learning language for robot control: Evaluating natural language understanding techniques,” in _Experimental robotics_. Springer, 2013, pp. 439–452.
  * [10] J. He, J. Chen, X. He, J. Gao, L. Li, L. Deng, and M. Ostendorf, “Deep reinforcement learning with a natural language action space,” _arXiv preprint arXiv:1511.04636_ , 2015.
  * [11] J. Y. Chai, R. Fang, C. Liu, and L. She, “Collaborative language grounding toward situated human-robot dialogue,” _AI Magazine_ , vol. 37, no. 4, pp. 32–45, 2016.
  * [12] V. V. Unhelkar, S. Li, and J. A. Shah, “Decision-making for bidirectional communication in sequential human-robot collaborative tasks,” in _Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction_ , 2020, pp. 329–341.
  * [13] F. Dimeas, N. Koenig, C. Savvaki, P. Asvestas, P. Sidiropoulos, E. Mavragani, N. Tsagarakis, and P. Artemiadis, “Online human-robot cooperation identification and adaptation to optimize robotic rehabilitation treatment,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 24, no. 11, pp. 1260–1270, 2016.
  * [14] L. Peternel, N. Tsagarakis, and A. Ajoudani, “Adaptive human-robot interaction via online action space adaptation,” in _2017 International Conference on Rehabilitation Robotics (ICORR)_. IEEE, 2017, pp. 1340–1345.
  * [15] D. Gopinath, S. Jain, and B. D. Argall, “Human-aware manipulator path planning using iterative improvement optimal control synthesis,” _IEEE Robotics and Automation Letters_ , vol. 2, no. 4, pp. 2194–2201, 2017.
  * [16] C. Wang, C. Ross, Y.-L. Kuo, B. Katz, and A. Barbu, “Learning a natural-language to ltl executable semantic parser for grounded robotics,” in _Conference on Robot Learning_. PMLR, 2021, pp. 1706–1718.
  * [17] Y.-L. Kuo, B. Katz, and A. Barbu, “Deep compositional robotic planners that follow natural language commands,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 4906–4912.
  * [18] O. A. Can, P. Z. D. Martires, A. Persson, J. Gaal, A. Loutfi, L. De Raedt, D. Yuret, and A. Saffiotti, “Learning from implicit information in natural language instructions for robotic manipulations,” _arXiv preprint arXiv:1904.13324_ , 2019.
  * [19] Y. Bisk, D. Yuret, and D. Marcu, “Natural language communication with robots,” in _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2016, pp. 751–761.
  * [20] A. Q. Keemink, H. van der Kooij, and A. H. Stienen, “Admittance control for physical human–robot interaction,” _The International Journal of Robotics Research_ , vol. 37, no. 11, pp. 1421–1444, 2018.
  * [21] N. Hogan, “Impedance control: An approach to manipulation,” in _1984 American control conference_. IEEE, 1984, pp. 304–313.
  * [22] J. L. Flanagan and R. M. Golden, “Phase vocoder,” _Bell System Technical Journal_ , vol. 45, no. 9, pp. 1493–1509, 1966.
  * [23] J. Wyatt, L. Chua, J. Gannett, I. Goknar, and D. Green, “Energy concepts in the state-space theory of nonlinear n-ports: Part i-passivity,” _IEEE transactions on Circuits and Systems_ , vol. 28, no. 1, pp. 48–61, 1981.
  * [24] U. Robots, “Ur5 robot,” https://www.universal-robots.com/products/ur5-robot/.