# Uncertainty-Aware Boosted Ensembling in Multi-Modal Settings

Utkarsh Sarawgi§ MIT Media Lab
Massachusetts Institute of Technology
Cambridge, USA
<EMAIL_ADDRESS>Rishab Khincha§ MIT Media Lab
Massachusetts Institute of Technology
BITS Pilani, K K Birla Goa Campus
Bangalore, India
<EMAIL_ADDRESS>Wazeer Zulfikar§ McGovern Institute for Brain Research
Massachusetts Institute of Technology
Cambridge, USA
<EMAIL_ADDRESS>Satrajit Ghosh  McGovern Institute for Brain Research
Massachusetts Institute of Technology
Harvard Medical School
Cambridge, USA
<EMAIL_ADDRESS>Pattie Maes  MIT Media Lab
Massachusetts Institute of Technology
Cambridge, USA
<EMAIL_ADDRESS>
###### Abstract

Reliability of machine learning (ML) systems is crucial in safety-critical
applications such as healthcare, and uncertainty estimation is a widely
researched method to highlight the confidence of ML systems in deployment.
Sequential and parallel ensemble techniques have shown improved performance of
ML systems in multi-modal settings by leveraging the feature sets together. We
propose an uncertainty-aware boosting technique for multi-modal ensembling in
order to focus on the data points with higher associated uncertainty
estimates, rather than the ones with higher loss values. We evaluate this
method on healthcare tasks related to Dementia and Parkinson’s disease which
involve real-world multi-modal speech and text data, wherein our method shows
an improved performance. Additional analysis suggests that introducing
uncertainty-awareness into the boosted ensembles decreases the overall entropy
of the system, making it more robust to heteroscedasticity in the data, as
well as better calibrating each of the modalities along with high quality
prediction intervals. We open-source our entire codebase at
https://github.com/usarawgi911/Uncertainty-aware-boosting.

§§footnotetext: Equal contribution

##  I Introduction

Rapid developments in machine learning (ML) across a variety of tasks have
advanced its deployment in real-world settings [1]. However, recent works have
shown how these models are usually overconfident at predicting probability
estimates representative of the true likelihood, and can lead to confident
incorrect predictions [2]. This is particularly detrimental in real-world
domains as the distribution of the observed data may shift and eventually be
very different once a model is deployed in practice, leading to models
exhibiting unexpectedly poor behaviour upon deployment [3].

As such, generating confidence intervals or uncertainty estimates along with
the predictions is crucial for reliable and safe deployment of machine
learning systems in safety-critical settings (such as healthcare) [4, 5, 6,
7]. It is critical to understand what a model does not know when building and
deploying machine learning systems to help mitigate possible risks and biases
in decision making [8]. This can also help in designing reliable human-
assisted AI systems for improved and more transparent decision making as the
human experts in the process can account for the confidence measures of the
models for a final decision.

Our experience of the world is multi-modal; data tend to exist with multiple
modalities such as images, audio, text, and more in tandem. Interpreting these
signals together by designing models that can process and relate information
from multiple sources can help leverage different feature sets together for
better understanding and decision making [9]. Parallel and sequential
techniques are widely used to improve performance by ensembling weak learners
trained with a single data modality [10, 11, 12, 13, 14]. Similarly, base
learners trained with different input modalities can be ensembled together for
performance improvements [9, 15, 16].

Some works have briefly discussed uncertainty estimation in multi-modal
settings [17, 18, 19, 20]. We propose a notion of uncertainty-awareness with
sequentially boosted ensembling in multi-modal settings. Particularly, we
design an ‘uncertainty-aware boosted ensemble’ for a multi-modal system where
each of the base learners correspond to the different modalities. The ensemble
is trained in a way such that the base learners are sequentially boosted by
weighing the loss with the corresponding data point’s predictive uncertainty
(Section III).

The motivation is to sequentially boost the data points for which a particular
base learner is more uncertain about its prediction. Multi-modal data, in
nature, can be more prone to noise in particular modalities due to various
reasons (such as the stochastic data generation process at the source). With
uncertainty estimation, the noisy modalities will exhibit high uncertainty
with the predictions. In such situations, having the base learners pay more
attention to such uncertain predictions can help design a more robust ensemble
learner. This mechanism decreases the overall entropy of the multi-modal
system while generating uncertainty estimates, thus making it more reliable.

We evaluate our method on multi-modal speech and text datasets on healthcare
tasks related to Dementia and Parkinson’s disease using different machine
learning models (Neural Networks and Random Forests) and uncertainty
estimation techniques (Gaussian target distribution [21] and Infinitesimal
Jackknife method [22]). Our analysis shows an increased reduction in the
entropy as we sequentially move from the first base learner to the last base
learner of the ensemble, further demonstrating the significance of introducing
uncertainty-awareness into the ensemble. The model becomes more robust to
heteroscedasticity in the data while also well calibrating each of the
individual modalities along with high quality prediction intervals (Section
IV-D). To the best of our knowledge, we are the first to explore such a
boosting mechanism in a multi-modal ensemble using predictive uncertainty
estimates.

###  I-A Summary of Contributions:

  * •

We first propose and formulate an ‘uncertainty-aware boosted ensemble’
(Section III), as also depicted in the process diagram in Fig. 1.

  * •

We then evaluate our method on multi-modal speech and text datasets on
healthcare tasks related to Dementia and Parkinson’s disease using different
ML models and uncertainty estimation techniques (Section IV).

  * •

We also perform entropy, calibration, and prediction interval analyses to
highlight the significance of introducing uncertainty-awareness into the
ensemble (Section IV).

Figure 1: Process diagrams of 1) ‘vanilla ensemble’ (left), and 2)
‘uncertainty-aware (UA) ensemble’ / ‘UA ensemble (weighted)’ (right). The
symbols in the diagrams follow from the notation and setup in Section III-A.

##  II Related Work

###  II-A Uncertainty Estimation

Numerous works have proposed a variety of both Bayesian and non-Bayesian
methods to model the heteroscedasticity introduced by the stochastic data
generation process for predicting the uncertainty estimates along with a
model’s predictions. Bayesian approximation techniques such as dropout-based
VI [23, 24], expectation propagation [25], variational inference (VI) [26,
27], deterministic VI [28], neural networks as Gaussian processes [29],
approximate Bayesian ensembling [30], and Bayesian model averaging in low-
dimensional parameter subspaces [31] have been shown to be quite useful in
modelling the uncertainties in neural networks. Non-Bayesian approaches [32,
21, 33, 34, 17] that involve bootstrapping and ensembling multiple
probabilistic neural networks have shown performances comparable to Bayesian
methods with reduced computational costs and modifications to the training
procedure. Additionally, there is a breadth of other theoretical, empirical,
and review works on estimating predictive uncertainties with neural networks
[35, 36, 37, 38, 39, 40, 41].

Uncertainty estimation in trees and random forests has been studied in the
past, with multiple methods being proposed for both classification and
regression tasks. NGBoost generalizes gradient boosting to probabilistic
regression by treating the parameters of the conditional distribution as
targets for a multiparameter boosting algorithm [42]. Ensemble methods have
been proposed to study the uncertainty in gradient boosted models [43].
Recently, various methods such as the infinitesimal jackknife [22], monte
carlo based approaches [44], and using the decision tree as a probabilistic
predictor [45] have been suggested to predict the uncertainty in random
forests. Ashukha et al. [46] performed a broad study of ensembling techniques
in context of uncertainty estimation.

###  II-B Ensemble techniques

Parallel and sequential ensembling techniques have been widely studied and
shown to improve performance of models in a variety of tasks [10, 11, 12, 13,
14, 9, 15, 16]. Some works combine latent embeddings from different input
modalities (feature fusion) and train the entire model together in a joint
fashion [47]. Boosting algorithms such as Adaptive Boost [12], Gradient
Boosting [13], and XG Boost [14] are common sequential ensembling techniques.
These methods are often used to improve performance by ensembling weak base
learners trained with a single data modality [10, 11, 12, 13, 14]. Similarly
in multi-modal settings, base learners trained with different input data
modalities can be ensembled together to help leverage different feature sets
together [9, 15, 16].

###

There has been some work on incorporating uncertainty with ensembling methods.
Kendall et al. [48] learns multiple tasks by using the uncertainty predicted
as weights for the losses of each of the models, thus outperforming the
individual models trained on each task. Chang et al. [49] uses uncertainty
estimates and prefers to learn the data points predicted incorrectly with
higher uncertainty in different mini-batches of SGD. Some other works have
addressed the advantages of incorporating uncertainty estimates in multi-modal
settings [17, 18, 19, 20].

Our work particularly designs boosted ensemble techniques that are
uncertainty-aware in multi-modal settings, explores how these can be really
helpful in real-world settings, and motivates future research directions.

##  III Uncertainty-Aware Boosted Ensemble

###  III-A Notation and Setup

Let $\mathbf{x}$ represent the multi-modal input feature set and
$y\in\mathbb{R}$ denote the real-valued label for regression. We let
$\mathbf{x}^{j}\in\mathbb{R}^{d}$ represent a set of $d$-dimensional input
features for the $j^{th}$ modality, with $j=1$ to $k$, where $k$ is the total
number of modalities.

Let $\\{h^{j}\\}_{j=1}^{k}$ represent the corresponding base learner for the
$j^{th}$ modality. The term base learner is just an abstraction for any learnt
functions that maps an input to an output, for example, SVM, random forest,
neural network, etc.

We subsequently have a training dataset
$\\{(\mathbf{x}^{j}_{n},y_{n})\\}_{n=1}^{N}$ consisting of $N$ i.i.d. samples
for the $j^{th}$ modality i.e.

$h^{j}:\mathbf{x}^{j}_{n}\longrightarrow y_{n}$ (1)

###  III-B Defining Uncertainty-Aware Boosted Ensemble

We first define a ‘vanilla ensemble’ for a fair comparison with our proposed
approach. We then define our ‘uncertainty-aware ensemble’ referred to as ‘UA
ensemble’, and its variation referred to as ‘UA ensemble (weighted)’. Fig. 1
shows the process diagrams of vanilla ensemble, UA ensemble, and UA ensemble
(weighted).

####  III-B1 Vanilla ensemble

This makes use of loss values, i.e. mean squared error (MSE) values for
regression, to weight the loss function during training while sequentially
boosting across the base learners. This means that the MSE values
corresponding to the predictions from the ${j}^{th}$ base learner are used to
weight the loss function for the corresponding training samples while training
the $(j+1)^{th}$ base learner. Then, the ensemble computes an average of the
predictions $\\{\hat{y}_{h^{j}}\\}_{j=1}^{k}$ of all the (boosted) base
learners for the final prediction $\hat{y}$.

####  III-B2 UA ensemble

This makes use of predicted uncertainty estimates $\sigma_{h^{j}}$ to weight
the loss function during training while sequentially boosting across the base
learners. This means that the uncertainty estimates $\sigma_{h^{j}}$
corresponding to the predictions from the ${j}^{th}$ base learner are used to
weight the loss function for the corresponding training samples while training
the $(j+1)^{th}$ base learner. Then, the ensemble computes an average of the
predictions $\\{\hat{y}_{h^{j}}\\}_{j=1}^{k}$ of all the (boosted) base
learners for the final prediction $\hat{y}$.

####  III-B3 UA ensemble (weighted)

We experiment with a variation to the final averaging of the predictions in
the UA ensemble discussed above. Here, for the final prediction $\hat{y}$, the
ensemble computes a weighted average of the predictions
$\\{\hat{y}_{h^{j}}\\}_{j=1}^{k}$ of all the (boosted) base learners, where
the weights used are the inverse of the respective predicted uncertainty
estimates $\sigma_{h^{j}}$. Equation (LABEL:eq:ua_voting) mathematically
formulates this, where $\hat{y}(\mathbf{x}_{n})$ is the final prediction
corresponding to $n^{th}$ data point, and $k$ is the total number of
individual modalities as defined in Section III-A.

$\displaystyle\begin{split}{\hat{y}(\mathbf{x}_{n})}=\frac{\sum_{j=1}^{k}\frac{1}{\sigma_{h^{j}}(\mathbf{x}_{n})}\hat{y}_{h^{j}}(\mathbf{x}_{n})}{\sum_{j=1}^{k}\frac{1}{\sigma_{h^{j}}(\mathbf{x}_{n})}}\\\
\end{split}$ (2)

###

Most of the previously proposed boosting methods sequentially boost across
different base learners using the same set of total input features. However,
UA ensembles sequentially boost through different base learners, with each
base learner corresponding to a different input modality. This is because we
want to best leverage each of the modality-wise features while deriving a
strong multi-modal learner using individual modality-wise base-learners
together. It is important to note that unlike other boosting techniques [12,
13, 14], the base learners here need not be weak learners.

##  IV Experiments and Results

We test and evaluate our proposed methods on two speech and language-based
multi-modal datasets in healthcare tasks related to Dementia and Parkinson’s
disease. We make use of different types of machine learning models (Neural
Networks and Random Forests) and uncertainty estimation techniques (Gaussian
target distribution [21] and Infinitesimal Jackknife method [22]) for the two
datasets.

###  IV-A Datasets

####  IV-A1 Dementia

We use the standardized and benchmark ADReSS (Alzheimer’s Dementia Recognition
through Spontaneous Speech) dataset111ADReSS dataset can be downloaded from
https://dementia.talkbank.org/ along with an email to obtain the password for
access. [50]. This dataset consists of speech samples (WAV format) and
transcripts (CHA format), and their corresponding ‘MMSE’ (Mini-Mental State
Examination) scores as labels for regression. MMSE scores (ranging from 0 to
30 and widely used in clinical practice) offer a way to quantify cognitive
function, as well as screening for cognitive loss by testing the individuals’
attention, recall, language, and motor skills [51].

Figure 2: Entropy analysis, using kernel density estimation plots, of the base
learners in a vanilla ensemble (left) and UA ensemble (right). UA ensemble
shows a decrease in the overall entropy of the system. The increased reduction
in the entropy as we sequentially move from the first base learner to the last
base learner of the ensemble further indicates the significance of introducing
uncertainty-awareness into the ensemble (Section IV-D1).

The dataset consists of 156 data points, each from a unique subject, matched
for age and gender. A standardized train-test split of around 70%-30% (108 and
48 subjects) is provided by the dataset. We further split the train set into
80%-20% train-val sets. The test set was held out for all experimentation
until final evaluation.

####  IV-A2 Parkinson’s Disease

We use the publicly available Parkinson’s Telemonitoring dataset222Parkinson’s
Telemonitoring dataset can be downloaded from
https://archive.ics.uci.edu/ml/datasets/Parkinsons+Telemonitoring. [52]. This
dataset consists of a range of 16 biomedical voice measurements and their
corresponding ‘Total UPDRS’ (Unified Parkinson Disease Rating Scale) scores as
labels for regression. Total UPDRS scores (ranging from 0 to 199 and widely
used as a measure of severity of the Parkinson’s disease (PD)) offer a way to
quantify the course of PD in patients by testing the individuals’ mentation,
behaviour, mood, daily-life activities, and motor examination [53].

The dataset consists of 5,875 data points from 42 subjects with early-stage PD
recruited to a six-month trial of a telemonitoring device for remote symptom
progression monitoring. Since a standardized train-test split is not provided
by the dataset, we use a 5-fold cross validation with consistent folds across
the different methods for a fair evaluation.

###  IV-B Multi-modal Feature Extraction

####  IV-B1 Dementia

For a fair comparison with the state-of-the-art, we extract multi-modal
acoustic, cognitive and linguistic features from the available speech samples
and their corresponding transcripts, by using the feature engineering pipeline
as developed by Sarawgi et al. [15]. This results in three input modalities,
namely ‘Disfluency’, ‘Interventions’, and ‘Acoustic’ (following the same
terminology as Sarawgi et al. [15]). The details of features from each of the
three modalities can be found in our Supplementary material333Please see
https://tinyurl.com/16osjsix for the Supplementary material..

####  IV-B2 Parkinson’s Disease

The dataset consists of features related to amplitude and frequency.
Accordingly, we extract two input modalities from the available data,
referring to them as the ‘Amplitude’ and ‘Frequency’ modalities. Consequently,
the list of features in the two input modalities are as below:

  * •

‘Amplitude’: Shimmer, Shimmer(dB), Shimmer:APQ3, Shimmer:APQ5, Shimmer:APQ11,
Shimmer:DDA, NHR, HNR, RPDE, DFA

  * •

‘Frequency’: Jitter(%), Jitter(Abs), Jitter:RAP, Jitter:PPQ5, Jitter:DDP, PPE

###  IV-C Model Architecture and Training

####  IV-C1 Dementia

Following from Section IV-B, we have three input modalities here i.e.
$j=1,2,3$ and $k=3$. Now, with a training dataset
$\\{(\mathbf{x}^{j}_{n},y_{n})\\}_{n=1}^{N}$ consisting of $N$ i.i.d. samples
for each of the three modalities, we model the probabilistic predictive
distribution $p_{h^{j}}(y|\mathbf{x}^{j})$ using a neural network (NN) with
parameters $h^{j}$.

For a fair comparison with the state-of-the-art, we use almost the same NN
architecture as used by Sarawgi et al. [15] for each of the three input
modalities. The Disfluency and Acoustic models make use of multi-layer
perceptrons (MLPs), while the Interventions model makes use of LSTM, along
with regularizers. The exact model architecture, along with regularizers for
each of the three models (base learners) can be found in our Supplementary
material.

For uncertainty estimation, each of the models predicts a target distribution
instead of a point estimate to account for the heteroscedasticity in data and
yields predictive uncertainties along with the predicted mean value [21, 40,
17]. The target distribution is modelled as a Gaussian distribution
$p_{h^{j}}(y_{n}|\mathbf{x}^{j}_{n})$ parameterized by the mean $\mu_{h^{j}}$
and the standard deviation $\sigma_{h^{j}}$, predicted at the final layer of
the models i.e.
$y_{n}\thicksim\mathcal{N}\left(\mu_{h^{j}},\sigma_{h^{j}}^{2}\right)$. It is
important to note here that the prediction $\hat{y}_{h^{j}}$ is the predicted
mean $\mu_{h^{j}}$, and the predicted uncertainty estimate is the predicted
standard deviation $\sigma_{h^{j}}$. The exact illustration of the
implementation can be found in our Supplementary material.

Each of the three base learners is trained with their corresponding input
modality features $\mathbf{x}^{j}$ and ground truth labels $y$ using a proper
scoring rule. We optimize for the negative log-likelihood (NLL) of the joint
distribution $p_{h^{j}}({y}_{n}|\mathbf{x}^{j}_{n})$ according to the equation
below (IV-C1).

$\displaystyle-\log\left(p_{h^{j}}(y_{n}|\mathbf{x}^{j}_{n})\right)$
$\displaystyle=\frac{\log\left(\sigma^{2}_{h^{j}}\right)}{2}+\frac{\left(y-\mu_{h^{j}}\right)^{2}}{2\sigma_{h^{j}}^{2}}$
$\displaystyle+\text{constant}$ (3)

We use the boosting methods explained in Section III-B to train an ensemble
with the three base learners (Disfluency, Acoustic, and Interventions). Each
training run used a batch size of 32 and an Adam optimizer with a learning
rate of 0.00125 to minimize the NLL.

####  IV-C2 Parkinson’s Disease

Following from Section IV-B, we have two input modalities here i.e. $j=1,2$
and $k=2$. Now, with a training dataset
$\\{(\mathbf{x}^{j}_{n},y_{n})\\}_{n=1}^{N}$ consisting of $N$ i.i.d. samples
for each of the two modalities, we model the probabilistic predictive
distribution $p_{h^{j}}(y|\mathbf{x}^{j})$ using a random forest (RF)
regressor with parameters $h^{j}$. Each of the RFs makes use of 300 decision
tree estimators. This was decided upon sweeping the number of decision trees,
from 100 to 1000, as a hyperparameter.

The two base learners are trained with their corresponding input modality
features $\mathbf{x}^{j}$ and ground truth labels $y$ using a mean squared
error (MSE) loss. The uncertainty estimates $\sigma_{h^{j}}$ of each of the
data point is estimated as the confidence interval using the Infinitesimal
Jackknife method [22, 54].

We use the boosting methods explained in Section III-B to train an ensemble
with the two base learners (Amplitude and Frequency).

###  IV-D Results

####  IV-D1 Dementia

For robustness, we repeat every training and test-set evaluation $5$ times and
report the mean and variance of the root mean squared error (RMSE) results
across the five runs. We first evaluate each of the modalities (i.e. base
learners) individually and then compare them with the vanilla and uncertainty-
aware ensembles. The order of sequential boosting for the propagation of the
uncertainties is chosen in the order of the test set performance of the
individual modalities. We observe that the uncertainty-aware ensembles perform
better than the vanilla ensemble and the individual modalities (Table I).

TABLE I: Comparison of individual modalities i.e. base learners and ensemble methods on test set results of the ADReSS dataset. Model | RMSE
---|---
Disfluency | 5.71 $\pm$ 0.39
Interventions | 6.41 $\pm$ 0.53
Acoustic | 6.66 $\pm$ 0.30
Vanilla Ensemble | 5.17 $\pm$ 0.27
UA Ensemble | 5.05 $\pm$ 0.53
UA Ensemble (weighted) | 4.96 $\pm$ 0.49

We also compare our uncertainty-aware ensemble methods with current state-of-
the-art results on the ADReSS test set. Table II shows that the best of 5 runs
of UA Ensemble is competitive, and that of the UA Ensemble (weighted)
outperforms other methods.

TABLE II: Comparison of uncertainty-aware ensemble methods with state-of-the-art results on the ADReSS test set. Model | RMSE
---|---
Pappagari et al. [55] | 5.37
Luz et al. [50] | 5.20
Sarawgi et al. [15] | 4.60
Searle et al. [56] | 4.58
Balagopalan et al. [57] | 4.56
Rohanian et al. [58] | 4.54
Sarawgi et al. [17] | 4.37
UA Ensemble | 4.35
UA Ensemble (weighted) | 3.93

The use of uncertainty awareness can improve the robustness of the ensemble
with uncertain data points (i.e subjects). To highlight this, we evaluate the
entropy of the base learners in the ensemble methods while sequentially
boosting in the vanilla ensemble and the UA ensemble. Upon comparison, we see
a decrease in the overall entropy of the system when the ensemble is
uncertainty-aware (Fig. 2). The increased reduction in the entropy, as we
sequentially move from the first base learner to the last base learner of the
ensemble, further indicates the significance of introducing uncertainty-
awareness into the ensemble.

TABLE III: 5-times repeated test set results of Mean Prediction Interval Width (MPIW) and Prediction Interval Coverage Probability (PICP) for the ensemble techniques on the ADReSS dataset. We report PICP results with the prediction interval ($\Delta$) equal to 1, 2, and 3 times the standard deviation (i.e. $1\sigma$, $2\sigma$, and $3\sigma$). The uncertainty-aware boosting results in tighter bounds for the confidence intervals, along with higher PICP values, and high quality prediction intervals as desired (Section IV-D1). Model | Modality | MPIW | PICP ($\%$)
---|---|---|---
$\Delta=1\sigma$ | $\Delta=2\sigma$ | $\Delta=3\sigma$
Vanilla Ensemble | Disfluency | 4.47 $\pm$ 0.39 | 61.66 $\pm$ 8.29 | 95.83 $\pm$ 2.63 | 97.50 $\pm$ 0.83
Interventions | 7.27 $\pm$ 0.58 | 87.50 $\pm$ 5.43 | 99.17 $\pm$ 1.02 | 100.00 $\pm$ 1.18
Acoustic | 4.50 $\pm$ 0.73 | 59.58 $\pm$ 12.54 | 94.58 $\pm$ 2.12 | 98.75 $\pm$ 1.02
UA Ensemble | Disfluency | 6.29 $\pm$ 0.81 | 82.91 $\pm$ 6.37 | 97.91 $\pm$ 1.31 | 100.00 $\pm$ 0.00
Interventions | 5.46 $\pm$ 1.57 | 73.75 $\pm$ 14.47 | 93.33 $\pm$ 5.17 | 97.91 $\pm$ 1.86
Acoustic | 5.31 $\pm$ 1.30 | 75.41 $\pm$ 11.21 | 96.25 $\pm$ 3.06 | 99.16 $\pm$ 1.02
UA Ensemble (weighted) | Disfluency | 6.29 $\pm$ 0.81 | 83.33 $\pm$ 6.58 | 97.91 $\pm$ 1.31 | 100.00 $\pm$ 0.00
Interventions | 5.46 $\pm$ 1.57 | 76.25 $\pm$ 13.85 | 92.50 $\pm$ 5.98 | 96.66 $\pm$ 3.11
Acoustic | 5.31 $\pm$ 1.30 | 75.83 $\pm$ 10.59 | 95.00 $\pm$ 3.86 | 99.16 $\pm$ 1.02

Figure 3: Calibration curves for the ensemble techniques on the ADReSS
dataset. The two rows of plots use the same data to just visualize the
comparison differently (ensemble-wise and modality-wise respectively). The
plots show that the boosted modalities are better-calibrated in case of
uncertainty-aware boosting (Section IV-D1).

We also analyse our approach on the Mean Prediction Interval Width (MPIW) and
Prediction Interval Coverage Probability (PICP), two widely used metrics for
evaluating uncertainty in regression. PICP is the percentage of the times the
prediction interval contains the actual regression value, while MPIW is the
average size of all prediction intervals. Pearce et al. [59] discusses that
high-quality prediction intervals should be as narrow as possible, whilst
capturing some specified proportion of data points. Accordingly, it is
desirable to have low MPIW values and PICP $\geq(1-\alpha)$, a common choice
of $\alpha$ being $0.05$.

Table III shows the 5-times repeated test set results on MPIW and PICP metrics
with a comparison between the vanilla ensemble and the uncertainty-aware
ensembles. It is important to note that the comparison holds value for the
results corresponding to the modalities that have been boosted (in this case,
the Interventions and Acoustic modalities). UA ensemble and its variation UA
ensemble (weighted) observe reduced MPIW compared to vanilla ensemble for the
Interventions modality. While the vanilla ensemble observes a reduced MPIW for
the Acoustic modality, the uncertainty-aware ensembles observe a gradual
decrease in the MPIW values while sequentially boosting across the modalities.
The uncertainty-aware boosting thus results in tighter bounds for the
confidence intervals, along with higher PICP values, and high quality
prediction intervals as desired [59].

We further use the 65-95-99.7 rule (also called the empirical rule) to obtain
calibration curves for a comprehensive analysis of calibration [21, 17]. To
plot these curves, we first compute the $x\%$ prediction interval for each
data point under evaluation based on Gaussian quantiles using the prediction
value and variance. We then calculate the fraction of data points under
evaluation with true values that fall within this prediction interval. For a
well-calibrated model, the observed fraction should be close to the $x\%$
calculated earlier. To see how our models perform in this setting, we sweep
from $x=10\%$ to $x=90\%$ in steps of 10. A line lying close to the line
($y=x$) would indicate a well-calibrated model.

Fig. 3 shows the calibration curves of the three modalities in case of vanilla
ensemble, UA ensemble, and UA ensemble (weighted). We observe that in the case
of UA ensemble and UA ensemble (weighted), the Interventions and Acoustic
modalities become better-calibrated compared to the Disfluency modality.
However in case of vanilla ensemble, the calibration of the Interventions and
Acoustic modalities become worse when compared to the Disfluency modality.
Consequently, it follows that the boosted modalities are better-calibrated in
case of uncertainty-aware boosting, as desired in real-world settings. This
highlights the significance of introducing the notion of uncertainty-awareness
in ensembles to obtain better-calibrated models.

####  IV-D2 Parkinson’s Disease

Figure 4: Calibration curves for the ensemble techniques on the Parkinson’s
Telemonitoring dataset. The two rows of plots use the same data to just
visualize the comparison differently (ensemble-wise and modality-wise
respectively). Also in the second row, the green and orange plots almost
overlap (and hence, might not be clearly visible). The plots show that the
boosted modalities are better-calibrated in case of uncertainty-aware boosting
(Section IV-D2).

Since there is no standardised train-test split available, we perform a
$5$-fold cross validation and report the mean and variance of the RMSE results
across the five folds. We first evaluate each of the modalities (i.e. base
learners) individually and then compare them with the vanilla and uncertainty-
aware ensembles. Again, the order of sequential boosting for the propagation
of the uncertainties is chosen in the order of the cross-validation
performance of the individual modalities. We observe that the uncertainty-
aware ensembles perform better than the vanilla ensemble and the individual
modalities (Table IV).

TABLE IV: Comparison of individual modalities i.e. base learners and ensemble methods on 5-fold cross validation results of the Parkinson’s Telemonitoring dataset. Model | RMSE
---|---
Amplitude | 3.21 $\pm$ 0.06
Frequency | 3.32 $\pm$ 0.10
Vanilla Ensemble | 3.18 $\pm$ 0.05
UA Ensemble | 3.04 $\pm$ 0.04
UA Ensemble (weighted) | 3.05 $\pm$ 0.05

Table V shows the 5-fold cross-validation results on MPIW and PICP metrics
with a comparison between the vanilla ensemble and the uncertainty-aware
ensembles. It is important to note that the comparison holds value for the
results corresponding to the modalities that have been boosted (in this case,
the Frequency modality). UA ensemble and its variation UA ensemble (weighted)
observe reduced MPIW compared to vanilla ensemble, indicating that the
uncertainty-aware boosting results in tighter bounds for the confidence
intervals, along with higher PICP values, and high quality prediction
intervals as desired [59].

Fig. 4 shows the calibration curves of the two modalities in case of vanilla
ensemble, UA ensemble, and UA ensemble (weighted). We observe that in the case
of UA ensemble and UA ensemble (weighted), the Frequency modality becomes
better-calibrated compared to the Amplitude modality. However in case of
vanilla ensemble, the calibration of the Frequency modality becomes worse when
compared to the Amplitude modality. Consequently, as also observed with the
ADReSS dataset, it follows that the boosted modality is better-calibrated in
case of uncertainty-aware boosting, as desired in real-world settings. This
again highlights the significance of introducing the notion of uncertainty-
awareness in ensembles to obtain better-calibrated models.

TABLE V: 5-fold cross-validation results of Mean Prediction Interval Width (MPIW) and Prediction Interval Coverage Probability (PICP) for the ensemble techniques on the Parkinson’s Telemonitoring dataset. We report PICP results with the prediction interval ($\Delta$) equal to 1, 2, and 3 times the standard deviation (i.e. $1\sigma$, $2\sigma$, and $3\sigma$). The uncertainty-aware boosting results in tighter bounds for the confidence intervals, along with higher PICP values, and high quality prediction intervals as desired (Section IV-D2). Model | Modality | MPIW | PICP ($\%$)
---|---|---|---
$\Delta=1\sigma$ | $\Delta=2\sigma$ | $\Delta=3\sigma$
Vanilla Ensemble | Amplitude | 6.79 $\pm$ 1.28 | 84.56 $\pm$ 1.46 | 98.51 $\pm$ 0.58 | 99.89 $\pm$ 0.12
Frequency | 8.69 $\pm$ 0.59 | 74.17 $\pm$ 8.25 | 94.28 $\pm$ 3.37 | 98.60 $\pm$ 1.18
UA Ensemble | Amplitude | 6.50 $\pm$ 1.76 | 74.09 $\pm$ 9.15 | 93.70 $\pm$ 4.11 | 98.23 $\pm$ 1.47
Frequency | 6.91 $\pm$ 0.85 | 77.90 $\pm$ 5.28 | 95.64 $\pm$ 2.40 | 99.33 $\pm$ 0.51
UA Ensemble (weighted) | Amplitude | 6.50 $\pm$ 1.76 | 74.24 $\pm$ 8.59 | 93.71 $\pm$ 4.13 | 97.97 $\pm$ 1.66
Frequency | 6.91 $\pm$ 0.85 | 77.65 $\pm$ 5.67 | 95.45 $\pm$ 2.56 | 99.18 $\pm$ 0.70

##  V Discussion and Future Work

We proposed an uncertainty-aware boosted ensembling method in multi-modal
settings. Such an ensemble improves the performance when compared to
individual modalities and ensembles boosted using loss values. By focusing
more on data points with higher uncertainty, through uncertainty-weighting of
the loss function (UA Ensemble) and the predictions as well (UA Ensemble
(weighted)), we showed how our ensemble outperforms the results of state-of-
the-art methods. More importantly, our discussion in Section I highlight how
such an ensemble system can help design a more robust learner by having the
base learners pay more attention to uncertain prediction corresponding to
noisy data modalities. Our experiments showed that the propagation of the
uncertainty sequentially through the base learners of every modality aids the
multi-modal system to decrease the overall entropy in the system, making it
more reliable when compared to vanilla ensembles. Additionally, the modalities
indeed become well calibrated along with high quality prediction intervals
when boosted using uncertainty values, rather than loss values.

Such characteristics are significantly desired in real-world settings where
data tends to exist in multiple modalities together. Understanding what a
machine learning model does not know is crucial in safety-critical
applications. Access to such information helps with designing a more reliable
and aware decision-making system [4, 5, 6, 7, 8]. Furthermore, the
availability of predictive uncertainties corresponding to each modality adds a
level of transparency to the machine learning system. This can assist the user
in making more informed decisions, thereby nurturing the synergy between
humans and AI.

There are a lot of interesting possible future research directions to this
work. One could definitely expand the proposed method itself to account for
uncertainty values, as well as loss values, while boosting the base learners.
Our current experiments make use of speech and text data with neural networks
and random forests as the base learners. This can be extended to other forms
of machine learning systems, making use of other Bayesian and non-Bayesian
uncertainty estimation techniques and data modalities. Additionally, we
encourage the community to further evaluate such techniques in other safety-
critical tasks and applications, as well as assess the longitudinal
performance and attributes of these systems, especially in the presence of
noisy data and/or when the observed data distribution tends to shift over time
and eventually becomes very different. This also opens up avenues to
potentially design adaptive systems which could actively learn from the
uncertainty estimates at deployment time.

## References

  * [1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [2] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 1321–1330.
  * [3] A. D’Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman _et al._ , “Underspecification presents challenges for credibility in modern machine learning,” _arXiv preprint arXiv:2011.03395_ , 2020.
  * [4] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané, “Concrete problems in ai safety,” _arXiv preprint arXiv:1606.06565_ , 2016\.
  * [5] K. R. Varshney and H. Alemzadeh, “On the safety of machine learning: Cyber-physical systems, decision sciences, and data products,” _Big data_ , vol. 5, no. 3, pp. 246–255, 2017.
  * [6] A. Kumar, P. S. Liang, and T. Ma, “Verified uncertainty calibration,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 3787–3798.
  * [7] J. J. Thiagarajan, B. Venkatesh, P. Sattigeri, and P.-T. Bremer, “Building calibrated deep models via uncertainty matching with auxiliary interval predictors.” in _AAAI_ , 2020, pp. 6005–6012.
  * [8] Y. Gal, “Uncertainty in deep learning,” _University of Cambridge_ , vol. 1, p. 3, 2016.
  * [9] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 2, pp. 423–443, 2018.
  * [10] L. Breiman, “Bagging predictors,” _Machine learning_ , vol. 24, no. 2, pp. 123–140, 1996.
  * [11] Y. Freund and R. E. Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting,” _Journal of computer and system sciences_ , vol. 55, no. 1, pp. 119–139, 1997.
  * [12] ——, “A decision-theoretic generalization of on-line learning and an application to boosting,” _J. Comput. Syst. Sci._ , vol. 55, no. 1, p. 119–139, Aug. 1997. [Online]. Available: https://doi.org/10.1006/jcss.1997.1504
  * [13] J. H. Friedman, “Greedy function approximation: A gradient boosting machine,” _Annals of Statistics_ , vol. 29, pp. 1189–1232, 2000.
  * [14] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , ser. KDD ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 785–794. [Online]. Available: https://doi.org/10.1145/2939672.2939785
  * [15] U. Sarawgi, W. Zulfikar, N. Soliman, and P. Maes, “Multimodal inductive transfer learning for detection of alzheimer’s dementia and its severity,” _arXiv preprint arXiv:2009.00700_ , 2020.
  * [16] X. Zhang and S. Mahadevan, “Ensemble machine learning models for aviation incident risk prediction,” _Decision Support Systems_ , vol. 116, pp. 48–63, 2019.
  * [17] U. Sarawgi, W. Zulfikar, R. Khincha, and P. Maes, “Why have a unified predictive uncertainty? disentangling it using deep split ensembles,” _arXiv preprint arXiv:2009.12406_ , 2020.
  * [18] S. Oviatt, P. Cohen, L. Wu, L. Duncan, B. Suhm, J. Bers, T. Holzman, T. Winograd, J. Landay, J. Larson _et al._ , “Designing the user interface for multimodal speech and pen-based gesture applications: State-of-the-art systems and future research directions,” _Human-computer interaction_ , vol. 15, no. 4, pp. 263–322, 2000.
  * [19] M. K. Sen and P. L. Stoffa, “Bayesian inference, gibbs’ sampler and uncertainty estimation in geophysical inversion 1,” _Geophysical Prospecting_ , vol. 44, no. 2, pp. 313–350, 1996.
  * [20] D. L. Hill, D. J. Hawkes, N. A. Harrison, and C. F. Ruff, “A strategy for automated multimodality image registration incorporating anatomical knowledge and imager characteristics,” in _Biennial International Conference on Information Processing in Medical Imaging_. Springer, 1993, pp. 182–196.
  * [21] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _Advances in neural information processing systems_ , 2017, pp. 6402–6413.
  * [22] S. Wager, T. Hastie, and B. Efron, “Confidence intervals for random forests: The jackknife and the infinitesimal jackknife,” _Journal of machine learning research : JMLR_ , vol. 15, pp. 1625–1651, 05 2014.
  * [23] D. Kingma, T. Salimans, and M. Welling, “Variational dropout and the local reparameterization trick,” 06 2015.
  * [24] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _international conference on machine learning_ , 2016, pp. 1050–1059.
  * [25] J. M. Hernández-Lobato and R. Adams, “Probabilistic backpropagation for scalable learning of bayesian neural networks,” in _International Conference on Machine Learning_ , 2015, pp. 1861–1869.
  * [26] A. Graves, “Practical variational inference for neural networks,” in _Advances in Neural Information Processing Systems 24_ , J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2011, pp. 2348–2356.
  * [27] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight uncertainty in neural network,” in _Proceedings of the 32nd International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, F. Bach and D. Blei, Eds., vol. 37. Lille, France: PMLR, 07–09 Jul 2015, pp. 1613–1622. [Online]. Available: http://proceedings.mlr.press/v37/blundell15.html
  * [28] A. Wu, S. Nowozin, E. Meeds, R. E. Turner, J. M. Hernández-Lobato, and A. L. Gaunt, “Deterministic variational inference for robust bayesian neural networks,” _arXiv preprint arXiv:1810.03958_ , 2018.
  * [29] J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein, “Deep neural networks as gaussian processes,” _arXiv preprint arXiv:1711.00165_ , 2017.
  * [30] T. Pearce, F. Leibfried, and A. Brintrup, “Uncertainty in neural networks: Approximately bayesian ensembling,” in _International conference on artificial intelligence and statistics_. PMLR, 2020, pp. 234–244.
  * [31] P. Izmailov, W. J. Maddox, P. Kirichenko, T. Garipov, D. Vetrov, and A. G. Wilson, “Subspace inference for bayesian deep learning,” in _Uncertainty in Artificial Intelligence_. PMLR, 2020, pp. 1169–1179.
  * [32] I. Osband, “Risk versus uncertainty in deep learning : Bayes , bootstrap and the dangers of dropout,” 2016.
  * [33] M. W. Dusenberry, D. Tran, E. Choi, J. Kemp, J. Nixon, G. Jerfel, K. Heller, and A. M. Dai, “Analyzing the role of model uncertainty for electronic health records,” in _Proceedings of the ACM Conference on Health, Inference, and Learning_ , 2020, pp. 204–213.
  * [34] S. Jain, G. Liu, J. Mueller, and D. Gifford, “Maximizing overall diversity for improved uncertainty estimates in deep ensembles.” in _AAAI_ , 2020, pp. 4264–4271.
  * [35] D. J. C. MacKay, “A practical bayesian framework for backpropagation networks,” _Neural Comput._ , vol. 4, no. 3, p. 448–472, May 1992. [Online]. Available: https://doi.org/10.1162/neco.1992.4.3.448
  * [36] J. W. Kay, D. M. Titterington _et al._ , _Statistics and neural networks: advances at the interface_. Oxford University Press on Demand, 1999.
  * [37] M. Welling and Y. W. Teh, “Bayesian learning via stochastic gradient langevin dynamics,” in _Proceedings of the 28th International Conference on International Conference on Machine Learning_ , ser. ICML’11. Madison, WI, USA: Omnipress, 2011, p. 681–688.
  * [38] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” in _Advances in neural information processing systems_ , 2017, pp. 5574–5584.
  * [39] K. Shridhar, F. Laumann, and M. Liwicki, “Uncertainty estimations by softplus normalization in bayesian convolutional neural networks with variational inference,” _arXiv preprint arXiv:1806.05978_ , 2018.
  * [40] J. Snoek, Y. Ovadia, E. Fertig, B. Lakshminarayanan, S. Nowozin, D. Sculley, J. Dillon, J. Ren, and Z. Nado, “Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 13 969–13 980.
  * [41] X. Qiu, E. Meyerson, and R. Miikkulainen, “Quantifying point-prediction uncertainty in neural networks via residual estimation with an i/o kernel,” in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=rkxNh1Stvr
  * [42] T. Duan, A. Avati, D. Y. Ding, K. K. Thai, S. Basu, A. Y. Ng, and A. Schuler, “Ngboost: Natural gradient boosting for probabilistic prediction,” 2020.
  * [43] A. Malinin, L. Prokhorenkova, and A. Ustimenko, “Uncertainty in gradient boosting via ensembles,” in _International Conference on Learning Representations_ , 2021. [Online]. Available: https://openreview.net/forum?id=1Jv6b0Zq3qi
  * [44] J. Coulston, C. Blinn, V. Thomas, and R. Wynne, “Approximating prediction uncertainty for random forest regression models,” _Photogrammetric Engineering & Remote Sensing_, vol. 82, pp. 189–197, 03 2016.
  * [45] M. H. Shaker and E. Hüllermeier, “Aleatoric and epistemic uncertainty with random forests,” 2020.
  * [46] A. Ashukha, A. Lyzhov, D. Molchanov, and D. Vetrov, “Pitfalls of in-domain uncertainty estimation and ensembling in deep learning,” _arXiv preprint arXiv:2002.06470_ , 2020.
  * [47] K. Nakamura, S. Levy, and W. Y. Wang, “r/fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection,” _arXiv preprint arXiv:1911.03854_ , 2019.
  * [48] A. Kendall, Y. Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” 2018.
  * [49] H.-S. Chang, E. Learned-Miller, and A. Mccallum, “Active bias: Training a more accurate neural network by emphasizing high variance samples,” 04 2017.
  * [50] S. Luz, F. Haider, S. de la Fuente, D. Fromm, and B. MacWhinney, “Alzheimer’s dementia recognition through spontaneous speech: The adress challenge,” _arXiv preprint arXiv:2004.06833_ , 2020.
  * [51] T. N. Tombaugh and N. J. McIntyre, “The mini-mental state examination: a comprehensive review,” _Journal of the American Geriatrics Society_ , vol. 40, no. 9, pp. 922–935, 1992.
  * [52] A. Tsanas, M. Little, P. Mcsharry, and L. Ramig, “Accurate telemonitoring of parkinson’s disease progression by noninvasive speech tests,” _IEEE transactions on bio-medical engineering_ , vol. 57, pp. 884–93, 11 2009.
  * [53] M. D. S. T. F. on Rating Scales for Parkinson’s Disease, “The unified parkinson’s disease rating scale (updrs): status and recommendations,” _Movement Disorders_ , vol. 18, no. 7, pp. 738–750, 2003.
  * [54] S. Wager, “randomForestCI,” Sep. 2016. [Online]. Available: https://github.com/swager/randomForestCI
  * [55] R. Pappagari, J. Cho, L. Moro-Velazquez, and N. Dehak, “Using state of the art speaker recognition and natural language processing technologies to detect alzheimer’s disease and assess its severity,” 2020.
  * [56] T. Searle, Z. Ibrahim, and R. Dobson, “Comparing natural language processing techniques for alzheimer’s dementia prediction in spontaneous speech,” _arXiv preprint arXiv:2006.07358_ , 2020.
  * [57] A. Balagopalan, B. Eyre, F. Rudzicz, and J. Novikova, “To bert or not to bert: Comparing speech and language-based approaches for alzheimer’s disease detection,” _arXiv preprint arXiv:2008.01551_ , 2020.
  * [58] M. Rohanian, J. Hough, and M. Purver, “Multi-modal fusion with gating using audio, lexical and disfluency features for alzheimer’s dementia recognition from spontaneous speech,” 2020.
  * [59] T. Pearce, A. Brintrup, M. Zaki, and A. Neely, “High-quality prediction intervals for deep learning: A distribution-free, ensembled approach,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. Stockholmsmässan, Stockholm Sweden: PMLR, 10–15 Jul 2018, pp. 4075–4084. [Online]. Available: http://proceedings.mlr.press/v80/pearce18a.html


∎

11institutetext: Jiaming Qi 22institutetext: Harbin Institute of Technology,
Harbin, 150001, China.
Tel.: +86-18646086707
22email<EMAIL_ADDRESS>
# Towards Latent Space Based Manipulation of Elastic Rods using Autoencoder
Models and Robust Centerline Extractions ††thanks: J. Qi, G. Ma and Y. Lyu are
with the Harbin Institute of Technology, Harbin, 150001, China. ††thanks: P.
Zhou and D. Navarro-Alarcon are with the Hong Kong Polytechnic University,
Hung Hom, KLN, Hong Kong. Corresponding author<EMAIL_ADDRESS>††thanks: Haibo
Zhang is with the Beijing Institute of Control Engineering, Beijing, 100190,
China. National Key Laboratory of Science and Technology on Space Intelligent
Control, Beijing, 100190, China.

Jiaming Qi  Guangfu Ma
Peng Zhou  Haibo Zhang  Yueyong Lyu
David Navarro-Alarcon∗

(Received: date / Accepted: date)

###### Abstract

The automatic shape control of deformable objects is a challenging (and
currently hot) manipulation problem due to their high-dimensional geometric
features and complex physical properties. In this study, a new methodology to
manipulate elastic rods automatically into 2D desired shapes is presented. An
efficient vision-based controller that uses a deep autoencoder network is
designed to compute a compact representation of the object’s infinite-
dimensional shape. An online algorithm that approximates the sensorimotor
mapping between the robot’s configuration and the object’s shape features is
used to deal with the latter’s (typically unknown) mechanical properties. The
proposed approach computes the rod’s centerline from raw visual data in real-
time by introducing an adaptive algorithm on the basis of a self-organizing
network. Its effectiveness is thoroughly validated with simulations and
experiments.

###### Keywords:

Robotics Visual Servoing Deformable Objects Autoencoder Self-Organizing
Network Model Predictive Control

##  1 Introduction

Controlling the shape of soft objects automatically with robot manipulators is
highly valuable in many applications, such as food processing
tokumoto2002deformation , robotic surgery abolmaesumi2002image , cable
assembly tang2018framework , and household works sun2019general . Although
great progress has been achieved in recent years, shape control remains an
open problem in robotics navarro2014visual . One of the most crucial issues
that hamper the implementation of these types of controls is the difficulty to
obtain a meaningful and efficient feedback representation of the object’s
configuration in real-time. However, given the intrinsic high-dimensional
nature of deformable objects, standard vision-based control algorithms (e.g.,
based on simple point features) cannot be used as they cannot properly capture
the objects’ state. In this work, a solution is provided to this problem.

The configuration of rigid objects can be fully described by six degrees of
freedom. However, representing the configuration of soft objects is difficult
as they have infinite-dimensional geometric information. Therefore, a simple
and effective feature extractor that can characterize these objects in an
efficient (i.e., compact) manner should be designed cretu2011soft . At
present, traditional methods are roughly divided into two categories: local
and global descriptors. Local descriptors may use centroids, distances,
angles, curvatures navarro2016automatic  to describe geometric
characteristics. However, these features must be “hard-coded.” In turn, they
can only provide a fixed type of representation. Global descriptors produce a
generic representation of the object’s shape. An example method under this
category is the Point Feature Histogram (PFH) reported in rusu2008persistent .
PFH forms a multi-dimensional histogram to represent the overall shape of a
soft object. Subsequent efforts developed PFH into the Fast Point Feature
Histograms (FPFH), which reduces computation time 5152473 ; hu20193 . A method
based on linearly parameterized (truncated) Fourier series was also proposed
to represent the object’s contour navarro2017fourier . This parameterization
idea was generalized in qi2020adaptive , where more shape representations were
analyzed and implemented.

Learning-based solutions have received considerable attention due to their
potential to learn (in latent space) shape representations of virtually any
type of object from data observations only yeo2005colour . Force and position
measurements of a three-finger gripper manipulating a soft object were used in
cretu2011soft  as input to a network, which produced and predicted the
object’s contour (even for unknown objects). A coarse-to-fine shape
representation was also proposed on the basis of spatial transformer networks,
which allowed it to obtain good generalization properties without expensive
ground truth observations yan2020self . Growing neural gas was used in
valencia2019toward  to represent deformable shapes. In 2019Convolutional , a
feature extractor based on the convolutional autoencoder was developed. This
method was used to obtain a low-dimensional latent space from tactile sensing
data.

Traditional methods for manipulating soft objects henrich2000robot  typically
need to identify the complex physical properties of the object. This
requirement hinders their application in practice. Algorithms based on latent
spaces present a feasible solution, as they can effectively extract low-
dimensional features from a high-dimensional shape space. For example,
convolutional neural networks were used to build the inverse kinematics of a
rope nair2017combining , learn the physical model of a soft object in the
latent space without any prior knowledge of the object ebert2018visual , and
estimate the rope’s state and combine it with model predictive control
yan2020self . However, none of these works has been used to establish an
explicitly shaped servo-loop with a latent space representation. This idea has
not been sufficiently explored in the soft manipulation literature.

In the current work, a new solution to the manipulation problem of the elastic
rod is proposed. The novel contributions of this study are listed as follows.

  1. 1.

A centerline extraction algorithm based on self-organizing maps (SOM) is
presented for slender elastic rods.

  2. 2.

A shape feature extraction algorithm is designed using the deep autoencoder
network (DAE). The proposed method is used to represent the elastic rod with
finite-dimensional feature vectors.

  3. 3.

Detailed simulations and experiments are conducted to validate the
effectiveness of the proposed method.

To the best of the authors’ knowledge, this work is the first attempt wherein
a shape servo-controller uses DAE to establish an _explicit_ shape servo-loop.
The remainder of this study is organized as follows. The preliminaries are
presented in Section 2. The overall deformation control implementation process
is discussed in Section 3. Various visually servoed deformation tasks of
elastic rods are shown in Sections 4 and 5. Conclusions and future work are
provided in Section 6.

##  2 PRELIMINARIES

_Notation._ Column vectors are denoted with bold small letters $\mathbf{v}$
and matrices with bold capital letters $\mathbf{M}$. Time evolving variables
are represented as $\mathbf{m}_{k}$, where the subscript $k$ denotes the
discrete time instant. $\mathbf{E}_{n}$ is an $n\times n$ identity matrix.

The deformation control scheme of a robot manipulating the elastic rod based
on visual servoing is investigated. The following conditions are provided to
foster an understanding among readers:

  * •

A fixed camera is used to measure the centerline of the elastic rod, namely,
eye-to-hand configuration (depicted in Fig. 1). The coordinates obtained are
denoted by:

$\begin{array}[]{*{20}{c}}{\bar{\mathbf{c}}={{\left[{\mathbf{c}_{1}^{T},\dots,\mathbf{c}_{N}^{T}}\right]}^{T}}\in{\mathbb{R}^{2N}}}&{{\mathbf{c}_{i}}={{\left[{{u_{i}},{v_{i}}}\right]}^{T}}\in{\mathbb{R}^{2}}}\end{array}$
(1)

where $N$ represents the number of points that make up the centerline, $u_{i}$
and $v_{i}$ represents the coordinates of the $i$th $(i=1,\cdots,N)$ point in
the image frame.

  * •

Before the experiment, the robot has tightly grasped the elastic rod; that is,
object grasping is not the research field of this article. Measurement loss is
also not a problem during the manipulation process.

  * •

The robot supports velocity control mode, which can accurately execute the
given desired kinematic commands $\Delta\mathbf{r}_{k}\in\mathbb{R}^{q}$
siciliano1990kinematic  and satisfy the incremental position motions
$\mathbf{r}_{k}=\mathbf{r}_{k-1}+\Delta\mathbf{r}_{k}$.

  * •

The robot manipulates the elastic rod at low speeds, so the shape is uniquely
determined by elastic potential energy.

Problem Statement. Without any prior physical characteristics of elastic rods,
design a model-free vision-based controller which commands the robot to deform
the elastic rod into the desired shape in the 2D image space.

Figure 1: Schematic diagram of the elastic rod shape deformation. The camera
is utilized to determine shape feature $\mathbf{s}$ in real time, and within
the designed controller the robot automatically deform the real-time shape
denoted by $\bar{\mathbf{c}}$ of elastic rods into the target shape
$\bar{\mathbf{c}}^{*}$.

##  3 Methods

Figure 2: Schematic diagram of SOM-based centerline extraction. The white area
in the left side represents the area of elastic rod (clustering area), and the
red points in the right side represent the obtained centerline points
(clustering points) (in this figure, $N=30$).

###  3.1 Robust SOM-based Centerline Extraction Algorithm

Slender elastic rods whose lengths are much larger than their diameters are
used as the research object. Therefore, the centerline describes the shape of
the elastic rods. Given that the centerline generally comprises center-points
for elastic rods, it should be fixed-length, ordered, and equidistant for
subsequent feature extraction and controller design. Although some centerline
extraction algorithms are used in the literature, e.g., _OpenCV/thinning_ ,
they cannot meet the above requirements and need pre-processing of data, which
will deteriorate the system’s real-time performance.

In this article, SOM is utilized to achieve real-time 2D centerline extraction
of elastic rods without artificial marker points. SOM is a neural network
trained in an unsupervised learning manner kohonen1982self , which is
originally used for dimensionality reduction of high-dimensional data. Here,
it is used as a clustering algorithm. It generates a fixed number of
clustering points from the image data of the elastic rods. Finally, the
centerline is composed of the clustering points. The input of SOM is the white
area where the elastic rod is located in the binary image, as shown in Fig. 2.
The points in the white area are defined by
$\bar{\mathbf{m}}={{\left[{\mathbf{m}_{1}^{T},\dots,\mathbf{m}_{M}^{T}}\right]}^{T}}\in\mathbf{R}^{2M}$,
$\mathbf{m}_{i}\in\mathbf{R}^{2}$ represents coordinates of each point in the
image frame, and $M\gg N$. With the clustering nature of SOM, a fixed-length
equidistant centerline can be obtained, namely, $SOM:2M\to 2N$.

###### Remark 1

The proposed SOM-based centerline extraction is only used in the experiment
and not for simulation. The centerline extracted by SOM is not guaranteed to
be ordered, so the sorting algorithm qi2020adaptive  is utilized to reorder
the centerline. This process will not take too much time because $N$ is small.

Figure 3: Structure of DAE with the centerline $\overline{\textbf{c}}$ as the
input, and s is defined as the shape feature used for deformation Jacobian
matrix approximation and controller design.

###  3.2 Feature Extraction

A controller that can deform the real-time shape $\bar{\mathbf{c}}$ of elastic
rods into the target shape $\bar{\mathbf{c}}^{*}$ can be designed using the
centerline extracted by SOM. However, the centerline cannot be directly
inputted into the system. Given its high dimensionality, it will make the
system run slowly and may even cause many adverse effects, e.g., loss of
control. Thus, designing a shape feature extraction algorithm for elastic rods
to reduce the feature dimension and represent the centerline effectively is
necessary.

In this article, DAE is used to extract shape features
$\mathbf{s}\in\mathbb{R}^{p}$ from the high-dimensional centerline
$\bar{\mathbf{c}}\in\mathbb{R}^{2N}$. DAE is an artificial neural network
trained in an unsupervised-learning manner, which can automatically learn
latent features from unlabeled data zhou2020lasesom . DAE comprises three
parts, an Encoder that projects the input into the hidden layer, a hidden
layer describing the latent feature $\mathbf{s}$, and a Decoder that
reconstructs the latent feature into the original input. Formally, the
centerline $\bar{\mathbf{c}}\in\mathbb{R}^{2N}$ is fed into DAE and mapped to
the hidden layer through the nonlinear transformation
$\mathbf{s}=\mathbf{f}_{\theta_{1}}\left(\bar{\mathbf{c}}\right)=sig(\mathbf{W}_{1}\bar{\mathbf{c}}+\mathbf{b}_{1})$,
where parameter set
$\theta_{1}=\left\\{\mathbf{W}_{1},\mathbf{b}_{1}\right\\}$. $\mathbf{W}_{1}$
is a $k\times 2N$ weight matrix, $\mathbf{b}_{1}$ is a vector of bias and sig
is a $sigmoid$ activation function,
$s\left({\bar{c}}\right)=\frac{1}{{1+{e^{-\bar{c}}}}}$. The latent feature
$\mathbf{s}$ is input into the Decoder to generate a reconstruction
$\hat{\bar{\mathbf{c}}}$ with $2N$ dimensions through the deterministic
equation
$\hat{\bar{\mathbf{c}}}=\mathbf{g}_{\theta_{2}}\left(\mathbf{s}\right)=sig(\mathbf{W}_{2}\mathbf{s}+\mathbf{b}_{2})$,
with $\theta_{2}=\left\\{\mathbf{W}_{2},\mathbf{b}_{2}\right\\}$. The
parameters of $\theta_{1}$ and $\theta_{2}$ of the DAE are designed to
minimize the average error of reconstruction, which is defined as:

$\left\\{{\theta_{1}^{*},\theta_{2}^{*}}\right\\}=\mathop{\arg\min}\limits_{{\theta_{1}},{\theta_{2}}}\sum\limits_{k=1}^{N}{L\left({{\mathbf{c}_{i}},{g_{{\theta_{2}}}}\left({{f_{{\theta_{1}}}}\left({{\mathbf{c}_{i}}}\right)}\right)}\right)}$
(2)

where $\theta_{1}^{*}$ and $\theta_{2}^{*}$ are the ideal parameters, and $L$
is usually a mean square error. Once the Autoencoder is trained, the
centerline $\bar{\mathbf{c}}$ is input into the network, and the low-
dimensional shape feature $\mathbf{s}\in\mathbb{R}^{p}$ can be obtained
through nonlinear transformations $\mathbf{f}_{\theta_{1}}$. The workflow of
DAE is shown in Fig. 3.

For DAE, the reconstruction output $\hat{\bar{\mathbf{c}}}$ is not the focus,
and only the Encoder is utilized to provide the shape feature
$\mathbf{s}\in\mathbb{R}^{p}$ once the DAE is trained. At present, DAE has
various forms. In this paper, multilayer perceptron (MLP) is used, given its
ability to handle 2D data efficiently. The size of shape feature dimension $p$
can also be selected due to a trade-off balance. A small $p$ will improve the
system’s controllability, e.g., $p<q$. However, a large $p$ will enhance the
representation accuracy of centerlines. In the simulation and experiment, the
effect of various $p$ on the shape representation capabilities is presented.

###  3.3 Approximation of the Local Deformation Model

Given that regular (i.e., mechanically well-behaved) elastic objects are
considered, the centerline $\bar{\mathbf{c}}$ is extremely dependent on the
robot command $\mathbf{r}\in\mathbb{R}^{q}$ can be defined as the joint angles
or end-effector’s pose in this study. The relationship between
$\bar{\mathbf{c}}$ and $\mathbf{r}$ can be represented by the following
unknown function (3).

$\bar{\mathbf{c}}=\mathbf{h}\left(\mathbf{r}\right)$ (3)

Following (3), the overall kinematics model from robot command $\mathbf{r}$ to
shape feature $\bar{\mathbf{c}}$ can be constructed as follows:

$\mathbf{s}=\mathbf{f}_{\theta_{1}}\left(\mathbf{h}\left(\mathbf{r}\right)\right)$
(4)

Differentiating (4) concerning time variable $t$ yields:

$\dot{\mathbf{s}}=\mathbf{J}\left(t\right)\dot{\mathbf{r}}$ (5)

where
$\mathbf{J}\left(t\right)=\partial\mathbf{s}/\partial\mathbf{r}\in\mathbb{R}^{p\times
q}$ represents a Jacobian-like matrix that describes the mapping between the
feature change speed $\dot{\mathbf{s}}$ and the velocity command
$\dot{\mathbf{r}}$. The properties of elastic rods are unknown, so the
analytical form of $\mathbf{J}(t)$ cannot be obtained. Discretizing (5) yields
the first-order format as follows:

$\mathbf{s}_{k}=\mathbf{s}_{k-1}+\mathbf{J}_{k}\Delta\mathbf{r}_{k}$ (6)

where $\Delta\mathbf{r}_{k}=\mathbf{r}_{k}-\mathbf{r}_{k-1}\in\mathbf{R}^{q}$.
The application of DAE as feature extraction is the focus of this study.
Accordingly, the simple Broyden algorithms are used to computes local
approximations of $\mathbf{J}_{k}$ in real-time. Define the following
differential signal:

$\begin{array}[]{*{20}{c}}{{\mathbf{y}_{k}}={\mathbf{s}_{k}}-{\mathbf{s}_{k-1}}}&{{\mathbf{u}_{k}}=\Delta\mathbf{r}_{k}}={\mathbf{r}_{k}}-{\mathbf{r}_{k-1}}\end{array}$
(7)

Broyden algorithms are as follows:

  1. 1.

R1 update formula broyden1965class :

$\hat{\mathbf{J}}_{k}=\hat{\mathbf{J}}_{k-1}+\frac{{\left({{\mathbf{y}_{k}}-\hat{\mathbf{J}}_{k-1}{\mathbf{u}_{k}}}\right)\mathbf{u}_{k}^{T}}}{{\mathbf{u}_{k}^{T}{\mathbf{u}_{k}}}}$
(8)

This form has a simple structure and fast calculation speed.

  2. 2.

SR1 update formula broyden1965class :

$\hat{\mathbf{J}}_{k}=\hat{\mathbf{J}}_{k-1}+\frac{{\left({{\mathbf{y}_{k}}-\hat{\mathbf{J}}_{k-1}{\mathbf{u}_{k}}}\right){{\left({{\mathbf{y}_{k}}-\hat{\mathbf{J}}_{k-1}{\mathbf{u}_{k}}}\right)}^{T}}}}{{{\mathbf{u}_{k}}{{\left({{\mathbf{y}_{k}}-\hat{\mathbf{J}}_{k-1}{\mathbf{u}_{k}}}\right)}^{T}}}}$
(9)

The structure of SR1 is similar to R1, but the calculation accuracy is higher.

  3. 3.

DFP update formula nocedal1980updating :

$\displaystyle\hat{\mathbf{J}}_{k}$
$\displaystyle=\hat{\mathbf{J}}_{k-1}+\frac{{\left({{\mathbf{y}_{k}}-\hat{\mathbf{J}}_{k-1}{\mathbf{u}_{k}}}\right)\mathbf{y}_{k}^{T}+{\mathbf{y}_{k}}{{\left({{\mathbf{y}_{k}}-\hat{\mathbf{J}}_{k-1}{\mathbf{u}_{k}}}\right)}^{T}}}}{{{\mathbf{u}_{k}}\mathbf{y}_{k}^{T}}}$
(10)
$\displaystyle-\frac{{\mathbf{y}_{k}^{T}{\mathbf{y}_{k}}}}{{\left\|{{\mathbf{u}_{k}}\mathbf{y}_{k}^{T}}\right\|}}\left({{\mathbf{y}_{k}}-\hat{\mathbf{J}}_{k-1}{\mathbf{u}_{k}}}\right)\mathbf{u}_{k}^{T}$

DFP is a rank two quasi-Newton method, which is efficient for solving
nonlinear optimization.

  4. 4.

BFGS update formula dennis1974characterization :

$\hat{\mathbf{J}}_{k}=\hat{\mathbf{J}}_{k-1}-\frac{{\hat{\mathbf{J}}_{k-1}{\mathbf{u}_{k}}\mathbf{u}_{k}^{T}\hat{\mathbf{J}}_{k-1}^{T}}}{{{\mathbf{u}_{k}}\mathbf{u}_{k}^{T}\hat{\mathbf{J}}_{k-1}^{T}}}+\frac{{{\mathbf{y}_{k}}\mathbf{y}_{k}^{T}}}{{{\mathbf{u}_{k}}\mathbf{y}_{k}^{T}}}$
(11)

It is recognized with the best numerical stability.

When a new data pair $\left(\mathbf{y}_{k},\mathbf{u}_{k}\right)$ enters the
system, the deformation Jacobian matrix $\hat{\mathbf{J}}_{k}$ can be updated
with the above estimators.

###### Remark 2

The robot is assumed to manipulate elastic rods at low speed. Thus, the
deformation of the elastic rods is relatively slow. On the basis, the
deformation Jacobian matrix $\mathbf{J}_{k}$ can be estimated online as the
formula (4) is assumed to be smooth.

Figure 4: The block diagram of the proposed real-time deformation control
strategy.

###  3.4 Shape Servoing Controller

At the discrete-time instant $k$, the deformation Jacobian matrix
$\hat{\mathbf{J}}_{k}$ has been assumed to be exactly approximated by
(8)(9)(10)(11), so that the shape-motion difference model is satisfied:

${\mathbf{s}_{k}}={\mathbf{s}_{k-1}}+{\hat{\mathbf{J}}_{k}}\cdot{\mathbf{u}_{k}}$
(12)

A model predictive controller ouyang2018robust  is utilized to minimize the
shape deformation error $\mathbf{e}_{k}={\mathbf{s}^{*}-\mathbf{s}_{k}}$
between the measured feature $\mathbf{s}_{k}$ and a constant target feature
$\mathbf{s}^{*}$. With the estimated deformation Jacobian matrix
$\hat{\mathbf{J}}_{k}$ and (12), the predicted deformation output at time
$k+w$ is shown below:

$s_{k+w}^{p}={s_{k}}+\hat{\mathbf{J}}_{k}\cdot{\mathbf{u}_{k+w}}$ (13)

where $w\in\left[{0,H}\right]$ represents the length of prediction horizon,
and $\mathbf{u}_{k+w}=\mathbf{r}_{k+w}-\mathbf{r}_{k}$. The reference
deformation trajectory at time $k+w$ is calculated to ensure smooth
deformation of elastic rods and the estimation accuracy of deformation
Jacobian matrix as follows:

$\mathbf{s}_{k+w}^{r}={\mathbf{s}^{*}}-{e^{-\rho w}}\cdot{\mathbf{e}_{k}}$
(14)

where $\rho$ is a positive constant. Error $\varepsilon$ between the reference
and the prediction deformation at instant $k+w$ is defined as follows:

${\varepsilon_{k+w}}=\mathbf{s}_{k+w}^{r}-\mathbf{s}_{k+w}^{p}=\left({1-{e^{-\rho
w}}}\right){\mathbf{e}_{k}}-{\hat{\mathbf{J}}_{k}}{\mathbf{u}_{k+w}}$ (15)

Velocity command $\mathbf{u}_{k}$ is assumed to remain constant from $k$ to
$k+w$ and can be calculated by minimizing $\varepsilon$ from $k$ to $k+H$, as
shown below:

$\min\frac{1}{2}\left({\sum\limits_{w=0}^{H}{{\alpha^{w}}{{\left\|{\left({1-{e^{-\rho
w}}}\right){\mathbf{e}_{k}}-w{\hat{\mathbf{J}}_{k}}{\mathbf{u}_{k}}}\right\|}^{2}}}+\mathbf{u}_{k}^{T}\mathbf{Q}{\mathbf{u}_{k}}}\right)$
(16)

where $0<\alpha\leq 1$, and $\mathbf{Q}$ is a symmetric and positive definite
matrix used to adjust the command $\mathbf{u}_{k}$. When the command
$\mathbf{u}_{k}$ is too large, it will cause the robot to move too fast and
the manipulated object will oscillate. In turn, the estimation accuracy of the
deformation Jacobian matrix will be affected. Taking derivative of (16) with
respect to $\mathbf{u}_{k}$, the gradient $\nabla$ is calculated as follows:

$\nabla=\sum\limits_{w=0}^{H}{-w{\alpha^{w}}\hat{\mathbf{J}}_{k}^{T}\left({\left({1-{\beta^{w}}}\right){\mathbf{e}_{k}}-w{\hat{\mathbf{J}}_{k}}{\mathbf{u}_{k}}}\right)}+\mathbf{Q}{\mathbf{u}_{k}}$
(17)

where $\beta=e^{-\rho}$. By setting $\nabla=0$, the velocity command
$\mathbf{u}_{k}$ is derived:

$\displaystyle\mathbf{u}_{k}$
$\displaystyle={\left({a\hat{\mathbf{J}}_{k}+{\hat{\mathbf{J}}^{T+}}_{k}\mathbf{Q}}\right)^{+}}\left({b-c}\right)\mathbf{e}_{k}$
$\displaystyle a$
$\displaystyle=\left({{H^{2}}{\alpha^{H}}-2b}\right)/\ln\alpha$ $\displaystyle
b$
$\displaystyle=\left({H{\alpha^{H}}\ln\alpha-{\alpha^{H}}+1}\right)/{\ln^{2}}\alpha$
(18) $\displaystyle c$
$\displaystyle=\left({H{{\left({\alpha\beta}\right)}^{H}}\ln\left({\alpha\beta}\right)-{{\left({\alpha\beta}\right)}^{H}}+1}\right)/{\ln^{2}}\left({\alpha\beta}\right)$

Thus, at each time instant, the incremental position command is calculated as
follows:

$\mathbf{r}_{k}=\mathbf{r}_{k-1}+\mathbf{u}_{k}$ (19)

Following (12), it yields:

${\mathbf{e}_{k}}-{\mathbf{e}_{k-1}}=-{\hat{\mathbf{J}}_{k}}\Delta{\mathbf{r}_{k}}$
(20)

$\hat{\mathbf{J}}_{k}$ is assumed to be a full column rank, and substituting
(3.4) into (20) yields:

$\left({a\mathbf{E}_{n}+{\hat{\mathbf{J}}_{k}^{T+}}\mathbf{Q}\hat{\mathbf{J}}_{k}^{+}}\right)\left({{\mathbf{e}_{k}}-{\mathbf{e}_{k-1}}}\right)+\left({b-c}\right){\mathbf{e}_{k}}=0$
(21)

As $a>0,b-c>0$, and
${\hat{\mathbf{J}}_{k}^{T+}}\mathbf{Q}\hat{\mathbf{J}}_{k}^{+}$ is a positive-
definite matrix, the error $\mathbf{e}_{k}$ asymptotically converges to error,
namely,
$\mathop{\lim}\limits_{t\to\infty}{\mathbf{s}_{k}}=\mathbf{s}_{k}^{*}$.
However, when the reachability of the desired goal $\mathbf{s}_{k}^{*}$ is not
satisfied, $\hat{\mathbf{J}}_{k}$ may not be a column full-rank matrix. The
feedback error $\|\mathbf{e}_{k}\|$ may only converge to the neighborhood near
the origin. For such under-actuated visual servo control tasks, guaranteeing
the global asymptotic convergence is challenging hutchinson2006visual . The
block diagram of the proposed real-time deformation control strategy is shown
in Fig. 4.

###### Remark 3

The velocity controller (3.4) and deformation Jacobian estimators
(8)(9)(10)(11) only require visual feedback data without any additional
sensors, prior knowledge of the system model, and the requirement to calibrate
the camera.

Figure 5: Structure of DAE comprised of MLPs as the basic blocks of Encoder
and Decoder. The centerline $\bar{\mathbf{c}}$ is fed into the trained DAE to
generate shape feature denoted by $\mathbf{s}$.

##  4 SIMULATION RESULTS

The following case is considered: one end of an elastic rod is rigidly grasped
by a planar robot (2DOF) and the other end is static. For brevity, the robot
is not shown in the figures. The cable simulator is simulated as in
wakamatsu1995modeling  by using the minimum energy principle hamill2014student
, and publicly available at
https://github.com/q546163199/shape_deformation/tree/master/python/package/shape_simulator.
All numerical simulations are implemented in Python.

Figure 6: Average shape reconstruction error comparison between DAE and PCA
among 200 shape sets in the simulation.

###  4.1 Feature Extraction Comparison

In this section, 40,000 samples ($N=100$) utilized to train the DAE are
generated by randomly moving the robot. As previously mentioned in Section
3.2, DAE comprises MLPs, as shown in Fig. 5. DAE is implemented on PyTorch and
trained by adopting an ADAM optimizer with an initial learning rate of 0.001
and a batch size of 500. RELU activation functions are adopted in the Encoder
and Decoder.

In Fig. 6, the reconstruction error between the simulated shape
$\bar{\mathbf{c}}$ and the shape $\hat{\bar{\mathbf{c}}}$ obtained from DAE or
PCA is denoted by $\left\|\bar{\mathbf{c}}-\hat{\bar{\mathbf{c}}}\right\|$.
$k$ and $p$ determine the dimension of shape feature $\mathbf{s}$ obtained
from PCA zhu2020vision  and DAE, respectively. For the fairness of
competition, $k$ is set to be equal to $p$ for validating the feature
extraction and reconstruction performance of PCA and DAE under the same shape
feature dimension. The result shows that in each case, the reconstruction
performance of DAE is better than that of PCA. For DAE, the results show that
$p=4$ has the best reconstruction performance, followed by $p=6$ and $p=2$.
This finding indicates that $p$ is too low to represent the elastic rod fully.
Considering the trade balance of system controllability and shape
representation performance, DAE with ${p}=4$ is used in the following
sections.

###  4.2 Validation of the Jacobian Estimation

In this section, four deformation Jacobian estimators, namely, R1, SR1, DFP,
and BFGS, are considered, and their effectiveness is evaluated. The planar
robot grasps one end of the simulated rod and conducts a counterclockwise
circular motion with center (0.4, 0.4), as shown in Fig. 7a. Different from
the simple initialization of $\hat{\mathbf{J}}_{0}$, e.g., the identity
matrix, the robot moves in the initial sampling area (the motions are ensured
not to be collinear and close to the starting point) to initialize
$\hat{\mathbf{J}}_{0}$. The initialization accuracy of deformation Jacobian
matrix is improved using this method. This method can also reduce the
possibility of singular problems under the deformation Jacobian matrix in the
manipulation process. In turn, the safety of operations is enhanced. Two error
criteria (22) are utilized to compare each deformation Jacobian estimator
qualitatively.

$\begin{array}[]{*{20}{c}}{{T_{1}}=\left\|{{\mathbf{s}_{k}}-{{\hat{\mathbf{s}}}_{k}}}\right\|}&{{T_{2}}=\left\|{\Delta{\mathbf{s}_{k}}-{{\hat{\mathbf{J}}}_{k}}\Delta{\mathbf{r}_{k}}}\right\|}\end{array}$
(22)

where $\mathbf{s}_{k}$ is feedback shape feature generated by DAE,
$\hat{\mathbf{s}}_{k}$ is calculated by (23).

$\hat{\mathbf{s}}_{k}=\hat{\mathbf{s}}_{k-1}+\hat{\mathbf{J}}_{k}\Delta\mathbf{r}_{k}$
(23)

The deformation Jacobian estimators and the shape reconstruction accuracy of
DAE are verified, as depicted in Fig. 7b. The results show that the shape
reconstruction accuracy of DAE ($p=4$) is well, proving the effectiveness of
DAE in the shape representation. The plots of $T_{1}$ and $T_{2}$ during the
circular motion are demonstrated in Fig. 8. For the $T_{1}$ curve, all the
four deformation Jacobian estimators can accurately update the deformation
Jacobian matrix $\hat{\mathbf{J}}_{k}$, and the average error of BFGS is the
smallest. For $T_{2}$, BFGS has no apparent fluctuations, and the estimation
accuracy is the best. DFP is second-best, and R1 and SR1 share a similar
pattern, consistent with the theoretical analysis. The above analyses prove
that, when starting deformation, BFGS can calibrate and update the deformation
Jacobian matrix in time to identify the pseudo-physical parameters of the
elastic rods. Specifically, BFGS can estimate the change direction of shape
feature $\mathbf{s}$ in the latent space.

(a)

(b)

Figure 7: Deformation Jacobian matrix $\hat{\mathbf{J}}_{k}$ validation
framework. (a) Motion trajectory of robot’s end-effector. (b) Comparison
between the simulated cable profile (black solid line) and its reconstruction
shape obtained by DAE (red dashed line) with $p=4$. Figure 8: Profiles of the
criteria $T_{1}$ and $T_{2}$ that are computed along the circular trajectory
around the center (0.4, 0.4).

###  4.3 Manipulation of Elastic Rods

In this section, the robot is commanded by the velocity controller (3.4) to
deform the elastic rods into the desired constant shape
$\bar{\mathbf{c}}^{*}$, corresponding to $\mathbf{s}^{*}$. The error criterion
(24) is utilized to assess the deformation performance.

${T_{3}}=\left\|{{{\bar{\mathbf{c}}}_{k}}-{{\bar{\mathbf{c}}}}_{k}^{*}}\right\|$
(24)

The progress of the cable deformation under the velocity command (3.4) on the
basis of R1, SR1, DFP and BFGS is depicted in Fig. 9. The curve of $T_{3}$ and
the velocity command $\Delta\mathbf{r}_{k}$ are shown in Fig 10, and the
detailed time comparison is provided in Table 1. Both figures show that BFGS
is the best method with the shortest convergence time and smallest deformation
error, followed by DFP, and the effects of R1 and SR1 are similar.

Table 1: Results among R1, SR1, DFP and BFGS | R1 | SR1 | DFP | BFGS
---|---|---|---|---
Steps | 58 | 48 | 32 | 22
Time (second) | 33.64 | 27.84 | 18.56 | 12.76

(a) R1 result (b) SR1 result (c) DFP result (d) BFGS result

Figure 9: Profiles of of the shape deformation simulation among R1, SR1, DFP
and BFGS (red dashed curves represent the initial and transitional
trajectories, and the black solid curve represents the target shape
$\bar{\mathbf{c}}^{*}$ with shape feature $\mathbf{s}^{*}$). Figure 10:
Profiles of the criterion $T_{3}$ and velocity command $\Delta\mathbf{r}_{k}$
among R1, SR1, DFP and BFGS within manipulation. Figure 11: Experimental setup
comprised of two UR5 which support velocity control mode.

##  5 EXPERIMENTAL RESULTS

Various experiments with two UR5 that support velocity control mode are
conducted, as shown in Fig. 11.
$\Delta\mathbf{r}=[\Delta\mathbf{r}_{1}^{T},\Delta\mathbf{r}_{2}^{T}]^{T}\in\mathbb{R}^{6}$.
$\Delta r_{i1}$ and $\Delta r_{i2}$, $i=1,2$ represent the linear velocity of
end-effector along x-axis and y-axis of each UR5 in the world frame. $\Delta
r_{i3}$, $i=1,2$ represents the angular velocity of the sixth joint of each
UR5 along the direction parallel to the z axis in the world frame. A Logitech
C270 camera is used to capture the rod’s image and combined with OpenCV to
process on the Linux PC at 30 fps. The deformation trajectories display once
every two frames to compare the convergence effects of each algorithm. An
experimental video can be downloaded here
https://github.com/q546163199/experiment_video/raw/master/paper2/video.mp4.

Table 2: Comparison results among three centerline extraction algorithms with $N=50$ | Reference qi2020adaptive  | CL park2000centroid  | SOM
---|---|---|---
Time (Second) | 1.68 | 0.98 | 0.38

###  5.1 Image Processing

This section verifies the proposed SOM-based centerline extraction algorithm
and describes the image processing steps.

First, the SOM-based method proposed in this article is compared with two
other centerline extraction algorithms. The first one is based on the
_OpenCV/thinning_ developed in Reference qi2020adaptive , and the second one
is based on CL, which is the traditional clustering method park2000centroid .
For the fairness of competition, all algorithms need to provide an ordered,
fixed-length $N=50$, equidistant centerline. As the CL-based and SOM-based
algorithms only generate an unordered fixed-length centerline, the sorting
algorithm qi2020adaptive  is used to sort unordered centerlines. For SOM, an
open-source toolbox provided by vettigliminisom  is utilized. The SOM-based
algorithm is the fastest, and it can efficiently perform centerline
extraction, as shown in Table 2. One reason is that the provided SOM toolbox
is already highly optimized. Another reason is that the centerline produced by
CL-based and SOM-based clustering algorithms has the advantage of fixed-length
and equidistant-sampling. This method saves more time than qi2020adaptive ,
which sorts all the points and then perform down-sampling. The proposed SOM-
based algorithm and qi2020adaptive  have similar centerline extraction
accuracy, and CL-based has the worst performance, as shown in Fig. 12. Thus,
in terms of accuracy, considering that the processing speed of SOM is the
fastest, the SOM-based centerline extraction algorithm is used.

(a) Reference qi2020adaptive

(b) CL park2000centroid

(c) Proposed SOM

Figure 12: Comparison of three centerline extraction algorithms, including
reference qi2020adaptive , CL-based park2000centroid  and the proposed SOM-
based.

Second, the relevant image processing for centerline extraction is provided.
The overall process (shown in Fig. 13) is as follows:

  1. 1.

First, segment the red areas nearby the Gripper1 and Gripper2 on the basis of
HSV color space and mark them as two green marker points. Then, segment the
region of the interest (ROI) containing the rod following both green marker
points (see Fig. 13a).

  2. 2.

Next, identify the rod in ROI, remove the noise, and obtain a binary image of
the rod using OpenCV morphological opening algorithm (see Fig. 13b).

  3. 3.

Subsequently, use the proposed SOM-based algorithm to get an unordered
centerline with $N=50$ (see Fig. 13c).

  4. 4.

Finally, apply the sorting algorithm qi2020adaptive  to get an ordered
centerline (see Fig. 13d). The starting point is the closest point to the
right marker point on the centerline.

(a) ROI selection

(b) Thresholding

(c) Centerline extraction (SOM)

(d) Coordinate sorting

Figure 13: Image processing steps. Figure 14: Average shape reconstruction
error comparison between DAE and PCA among 200 shape sets in the experiment.

###  5.2 Feature Extraction Comparison

Similar to Section 4.1, 40,000 samples are generated in the same way. The
structure of DAE is similar with Section 4.1, as shown in Fig. 5. Batch-
Normalization-1D (BN) is added after each layer. From the comparison results
shown in Fig. 14, the reconstruction accuracy of DAE is still better than PCA.
For DAE, the accuracy of ${p}=4$ is the best, and $p=5$ is the second best.
This finding is consistent with the simulation results. The results prove the
effectiveness of the proposed DAE in the shape feature extraction. Thus DAE
with $p=4$ is used in the following sections.

(a)

(b)

Figure 15: Deformation Jacobian matrix $\hat{\mathbf{J}}_{k}$ validation
framework. (a) Motion trajectory of robot’s end-effector. (b) Comparison
between the visually measured cable profile (green dot line) and its
reconstruction shape obtained by DAE (red dot line) with $p=4$. Figure 16:
Profiles of the criteria $T_{1}$ and $T_{2}$ that are computed along the
circular trajectory.

###  5.3 Validation of the Jacobian Estimation

Similar to Section 4.2, two UR5 are commanded to move along a fixed circular
trajectory, as depicted in Fig. 15a. The shape reconstruction performance of
DAE is accurate in the experiment, as shown in Fig. 15b. Thus, whether for
simulation or experiment, DAE can be applied in the shape representation. The
comparison results depicted in Fig. 16 show that BFGS has a minimal
deformation Jacobian approximation error, which validates its effectiveness
and adaptability in different regions.

###  5.4 Manipulation of Elastic Rods

Similar to Section 4.3, a desired shape $\bar{\mathbf{c}}^{*}$ should be given
in advance. The following steps are conducted to obtain the feasible target
shape.

  * •

First, the robot is moved to a position while avoiding the singular shapes,
e.g., straight.

  * •

Second, the current shape of the elastic rod is recorded by the camera and
denoted by $\bar{\mathbf{c}}^{*}$ as the target shape.

  * •

Third, the target shape $\bar{\mathbf{c}}^{*}$ is fed into the trained DAE to
get target shape feature denoted by $\mathbf{s}^{*}$.

  * •

Fourth, the robot moves back to the initial position and starts the
deformation with the given $\mathbf{s}^{*}$.

Considering safety, the saturation of $\Delta\mathbf{r}$ is set to,
$\left|{\Delta{r_{i1}}}\right|\leq 0.01m/s$,
$\left|{\Delta{r_{i2}}}\right|\leq 0.01m/s$ and
$\left|{\Delta{r_{i3}}}\right|\leq 0.1rad/s,i=1,2,3$, respectively. Four
experiments with different initial and target shapes are conducted to verify
the proposed algorithm’s effectiveness.

In the proposed algorithm, the elastic rod can be deformed using two UR5 to
the desired shape accurately without damaging the object during the
deformation process, as depicted in Fig 17. The profiles of $T_{3}$ and the
velocity command $\Delta\mathbf{r}_{k}$ are shown in Fig. 18. Corresponding to
the simulation results, BFGS still has the most significant control effect,
the fastest convergence speed, without apparent fluctuations (large
instantaneous deformation). Thus, BFGS has excellent adaptability and
robustness to various conditions in the shape deformation issue. In this
study, the experiment uses two UR5. Overall, the results prove the
effectiveness of the proposed algorithm for the multi-manipulator shape
deformation issue.

(a) Experiment1-R1

(b) Experiment2-R1

(c) Experiment3-R1

(d) Experiment4-R1

(e) Experiment1-SR1

(f) Experiment2-SR1

(g) Experiment3-SR1

(h) Experiment4-SR1

(i) Experiment1-DFP

(j) Experiment2-DFP

(k) Experiment3-DFP

(l) Experiment4-DFP

(m) Experiment1-BFGS

(n) Experiment2-BFGS

(o) Experiment3-BFGS

(p) Experiment4-BFGS

Figure 17: Initial (black solid line), transition (green solid line) and
target (red solid line) configurations in the four shape deformation
experiments which have a variety of different initial and target shape with
dual-UR5 robot among R1, SR1, DFP and BFGS.

(a) Experiment1 result (b) Experiment2 result (c) Experiment3 result (d)
Experiment4 result

Figure 18: Profiles of the criterion $T_{3}$ and velocity command
$\Delta\mathbf{r}_{k}$ among R1, SR1, DFP and BFGS within four shape
deformation experiments.

##  6 Conclusions

A framework for the deformation control of elastic rods is proposed without
any prior physical knowledge. It includes shape feature extraction,
deformation Jacobian matrix estimation, and a robust SOM-based centerline
extraction algorithm. First, new shape features based on DAE are utilized to
represent the elastic rod’s centerline in the low-dimensional latent space.
Second, the performance of four deformation Jacobian estimators (R1, SR1, DFP,
and BFGS) is evaluated. Third, the velocity controller is derived and the
system stability is proven. Finally, the effectiveness and feasibility of the
proposed algorithm are validated by the numerical and experimental results.

DAE is used in this study to map the high-dimensional geometric information of
elastic rods flexibly into a low-dimension latent space. The proposed feature
extraction algorithm has better shape representation capabilities than the
traditional PCA. It also does not require any artificial markers, making it
widely applicable to practical situations. Broyden algorithms are used to
approximate the deformation Jacobian matrix in real-time. In this way, the
physical parameters and camera models are not identified. From the results,
BFGS has the advantages of simple structure, fast calculation speed, and
accurate approximation performance. Simultaneously, a robust SOM-based
centerline extraction algorithm with a fast calculation speed and high
extraction accuracy is designed. The overall system is completely calculated
from the visual feedback data, without any prior physical characteristics of
the elastic rod and the requirement to calibrate the camera.

The proposed method also has some limitations. First, the manipulated object
is only soft elastic objects, e.g., carbon fiber rod. Thus the proposed
algorithm is not suitable for inelastic items, e.g., plasticine and rope.
Second, although DAE has a good shape representation ability, it needs an
extensive and rich-enough dataset to train itself, which has particular
difficulties in practical applications. Third, the approximation of
deformation Jacobian matrix based on Broyden algorithms is easy to fall into
the local optimum, which may generate the destructive operation, such as over-
tension and over-compression in the manipulation process.

In the future, 3D deformation tasks will be included to manipulate more
complex shapes, e.g., M-shaped and spiral. Moreover, the existing DAE needs to
be improved to be suitable for different scenarios and materials. Path
planning should be considered to avoid possible destructive operations during
the manipulation process.

###### Acknowledgements.

This work was supported in part by the Germany/Hong Kong Joint Research Scheme
sponsored by the Research Grants Council of Hong Kong and the German Academic
Exchange Service under grant G-PolyU507/18, in part by the Research Grants
Council of Hong Kong under grant number 14203917, in part by the Key-Area
Research and Development Program of Guangdong Province 2020 under project 76.

## References

  * (1) Abolmaesumi, P., Salcudean, S.E., Zhu, W.H., Sirouspour, M.R., DiMaio, S.P.: Image-guided control of a robot for medical ultrasound.  IEEE Transactions on Robotics and Automation 18(1), 11–23 (2002)
  * (2) Broyden, C.G.: A class of methods for solving nonlinear simultaneous equations.  Mathematics of computation 19(92), 577–593 (1965)
  * (3) Cretu, A.M., Payeur, P., Petriu, E.M.: Soft object deformation monitoring and learning for model-based robotic hand manipulation.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42(3), 740–753 (2011)
  * (4) Dennis, J.E., Moré, J.J.: A characterization of superlinear convergence and its application to quasi-newton methods.  Mathematics of computation 28(126), 549–560 (1974)
  * (5) Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A., Levine, S.: Visual foresight: Model-based deep reinforcement learning for vision-based robotic control.  arXiv preprint arXiv:1812.00568 (2018)
  * (6) Hamill, P.: A student’s guide to Lagrangians and Hamiltonians.  Cambridge University Press (2014)
  * (7) Henrich, D., Wörn, H.: Robot Manipulation of Deformable Objects (2000)
  * (8) Hu, Z., Han, T., Sun, P., Pan, J., Manocha, D.: 3-d deformable object manipulation using deep neural networks.  IEEE Robotics and Automation Letters 4(4), 4255–4261 (2019)
  * (9) Hutchinson, S., Chaumette, F.: Visual servo control, part i: Basic approaches.  IEEE Robotics and Automation Magazine 13(4), 82–90 (2006)
  * (10) Kohonen, T.: Self-organized formation of topologically correct feature maps.  Biological cybernetics 43(1), 59–69 (1982)
  * (11) Nair, A., Chen, D., Agrawal, P., Isola, P., Abbeel, P., Malik, J., Levine, S.: Combining self-supervised learning and imitation for vision-based rope manipulation.  In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2146–2153. IEEE (2017)
  * (12) Navarro-Alarcon, D., Liu, Y.H.: Fourier-based shape servoing: a new feedback method to actively deform soft objects into desired 2-d image contours.  IEEE Transactions on Robotics 34(1), 272–279 (2017)
  * (13) Navarro-Alarcon, D., Liu, Y.h., Romero, J.G., Li, P.: On the visual deformation servoing of compliant objects: Uncalibrated control methods and experiments.  The International Journal of Robotics Research 33(11), 1462–1480 (2014)
  * (14) Navarro-Alarcon, D., Yip, H.M., Wang, Z., Liu, Y.H., Zhong, F., Zhang, T., Li, P.: Automatic 3-d manipulation of soft objects by robotic arms with an adaptive deformation model.  IEEE Transactions on Robotics 32(2), 429–441 (2016)
  * (15) Nocedal, J.: Updating quasi-newton matrices with limited storage.  Mathematics of computation 35(151), 773–782 (1980)
  * (16) Ouyang, B., Mo, H., Chen, H., Liu, Y., Sun, D.: Robust model-predictive deformation control of a soft object by using a flexible continuum robot.  In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 613–618. IEEE (2018)
  * (17) Park, D.C.: Centroid neural network for unsupervised competitive learning.  IEEE Transactions on Neural Networks 11(2), 520–528 (2000)
  * (18) Polic, M., Krajacic, I., Lepora, N., Orsag, M.: Convolutional autoencoder for feature extraction in tactile sensing.  IEEE Robotics and Automation Letters 4(4), 3671–3678 (2019)
  * (19) Qi, J., Ma, W., Navarro-Alarcon, D., Gao, H., Ma, G.: Adaptive shape servoing of elastic rods using parameterized regression features and auto-tuning motion controls.  arXiv preprint arXiv:2008.06896 (2020)
  * (20) Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (fpfh) for 3d registration.  In: 2009 IEEE International Conference on Robotics and Automation, pp. 3212–3217 (2009).  DOI 10.1109/ROBOT.2009.5152473
  * (21) Rusu, R.B., Marton, Z.C., Blodow, N., Beetz, M.: Persistent point feature histograms for 3d point clouds.  In: Proc 10th Int Conf Intel Autonomous Syst (IAS-10), Baden-Baden, Germany, pp. 119–128 (2008)
  * (22) Siciliano, B.: Kinematic control of redundant robot manipulators: A tutorial.  Journal of intelligent and robotic systems 3(3), 201–212 (1990)
  * (23) Sun, P., Hu, Z., Pan, J.: A general robotic framework for automated cloth assembly.  In: 2019 IEEE 4th International Conference on Advanced Robotics and Mechatronics (ICARM), pp. 47–52. IEEE (2019)
  * (24) Tang, T., Wang, C., Tomizuka, M.: A framework for manipulating deformable linear objects by coherent point drift.  IEEE Robotics and Automation Letters 3(4), 3426–3433 (2018)
  * (25) Tokumoto, S., Hirai, S.: Deformation control of rheological food dough using a forming process model.  In: Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), vol. 2, pp. 1457–1464. IEEE (2002)
  * (26) Valencia, A.J., Nadon, F., Payeur, P.: Toward real-time 3d shape tracking of deformable objects for robotic manipulation and shape control.  In: 2019 IEEE SENSORS, pp. 1–4. IEEE (2019)
  * (27) Vettigli, G.: Minisom: minimalistic and numpy-based implementation of the self organizing map.  GitHub.[Online]. Available: https://github.com/JustGlowing/minisom/
  * (28) Wakamatsu, H., Hirai, S., Iwata, K.: Modeling of linear objects considering bend, twist, and extensional deformations.  In: Proceedings of 1995 IEEE International Conference on Robotics and Automation, vol. 1, pp. 433–438. IEEE (1995)
  * (29) Yan, M., Zhu, Y., Jin, N., Bohg, J.: Self-supervised learning of state estimation for manipulating deformable linear objects.  IEEE Robotics and Automation Letters 5(2), 2372–2379 (2020)
  * (30) Yeo, N., Lee, K., Venkatesh, Y., Ong, S.H.: Colour image segmentation using the self-organizing map and adaptive resonance theory.  Image and Vision Computing 23(12), 1060–1079 (2005)
  * (31) Zhou, P., Zhu, J., Huo, S., Navarro-Alarcon, D.: Lasesom: A latent representation framework for semantic soft object manipulation.  arXiv preprint arXiv:2012.05412 (2020)
  * (32) Zhu, J., Navarro-Alarcon, D., Passama, R., Cherubini, A.: Vision-based manipulation of deformable and rigid objects using subspace projections of 2d contours.  Rob. Auton. Syst. (2020)