# Learning Human-Inspired Force Strategies for Robotic Assembly

Stefan Scherzinger1, Arne Roennau1 and Rüdiger Dillmann1 1 All authors are
with FZI Research Center for Information Technology, Haid-und-Neu-Str. 10-14,
76131 Karlsruhe, Germany {scherzinger, roennau<EMAIL_ADDRESS>
###### Abstract

The programming of robotic assembly tasks is a key component in manufacturing
and automation. Force-sensitive assembly, however, often requires reactive
strategies to handle slight changes in positioning and unforeseen part
jamming. Learning such strategies from human performance is a promising
approach, but faces two common challenges: the handling of low part clearances
which is difficult to capture from demonstrations and learning intuitive
strategies offline without access to the real hardware. We address these two
challenges by learning probabilistic force strategies from data that are
easily acquired offline in a robot-less simulation from human demonstrations
with a joystick. We combine a Long Short-Term Memory (LSTM) and a Mixture
Density Network (MDN) to model human-inspired behavior in such a way that the
learned strategies transfer easily onto real hardware. The experiments show a
UR10e robot that completes a plastic assembly with clearances of less than 100
micrometers whose strategies were solely demonstrated in simulation.

##  I Introduction

Many industrial assembly processes require mounting together individual
components to build complex products. When plug-inserting two individual
parts, this is referred to as the peg-in-hole problem and is often used to
benchmark new methods in robotics research. In industry, offline programming
[1] is an established process of deriving robot programs before fine-tuning
and deploying them on a plant. This process of fine motion planning [2] is
particularly challenging for force-sensitive assembly: Stiff contacts and
rigid fixtures can build up high reaction forces between the robot and the
environment, and usually require careful calibration of the robots’ motion to
process parameters such as friction. On-site, expert knowledge is often needed
due to the complexity of the programming.

In general, the inclusion of force-torque sensor signals seems crucial for
assembly success [3], [4], [5] and is still used today in form of analytical
solutions [6], [7]. Advances in lightweight robots enable a task-optimized
impedance design and achieve robust assembly skills that are comparable to
human performance [8], [9]. Using the force-torque signals for control
policies can also be learned by the robots from trial-and-error [10], which
has been shown for insertion tasks with tight clearances [11], [12]. On the
contrary, humans are particularly good at handling these tasks manually by
using sophisticated senses with a high bandwidth of information. Although
being intuitive for us where to press, turn and jolt to get two parts mated
when using our hands, these strategies are fuzzy and difficult to describe
quantitatively for robot programs. Capturing and translating these skills into
suitable algorithms is an exciting challenge. In this paper, we consider
offline programming for assembly, where motion is highly constrained and the
parts have low clearances and tight fits. The principle idea is to find
technical representations for these human-inspired skills that can be
transferred to real hardware once programmed.

To work toward this goal, we propose probabilistic force strategies, a data-
driven approach to mimic human behavior for low-clearance assembly skills that
we build onto force control on industrial manipulators. Our proposed deep
neural networks learn from data that is obtained easily offline through
showing the task in simulation on simplified primitives with a joystick.
Building on the experience from our earlier approach [13], we redesign the
previous neural network architecture and present a new concept for modeling
the quintessence of human behavior especially for low-clearance assembly.

##  II Problem statement and Related Work

The peg-in-hole problem is well-known in the assembly literature. Shapes and
sizes vary but the parts usually share a single insertion direction. In
addition to low tolerances and tight fits, we extend this to various insertion
directions with higher geometric complexity in contact. Low-clearance assembly
means dealing with force closure and form closure adaptively during execution,
caused by inevitable model inaccuracies. The additional challenge in offline
programming, however, is programming these skills without real hardware.

We follow the core idea to learn these skills from human demonstrations, which
is known under Programming by Demonstration (PbD) [14], [15], Imitation
Learning (IL) [16] in physiological neurosciences, and Behavioral Cloning (BC)
[17] from machine learning in general. The following sections discuss related
works.

###  II-A Probabilistic skill learning from demonstration

Gaussian Mixture Regression (GMR) is one of the frequently used methods for
trajectory learning in motion and force-based manipulation [18], [19], [20].
Tang et al. [6] use this approach to describe a probabilistic mapping from
contact wrenches to twist commands for admittance control. Although using
probabilistic modeling for approximating the datasets, the regression from
these distributions is usually deterministic. Kronander et al. [21] use
sampling from the conditional probability densities to introduce randomness in
comparison to Least-Squares Estimates (LSE) and thus overcome deadlocks during
assembly. Note that this is mostly valid only if the samples need not be
coherent in time.

Al-Yacoub et al. [22] use GMR during peg-in-hole assembly tasks for clustering
force-torque signals and getting insights into human-applied contact
transitions and skill characteristics.

Dong et al. [23] combine a Hidden Markov Model (HMM) with Locally Weighted
Regression (LWR) to learn discrete contact states of peg-in-hole assembly in a
haptic simulation environment. During reproduction, they use LWR for angle
corrections based on forces-torque signals in these states. Vergara et al.
[24] use Probabilistic Principal Component Analysis (PPCA) for modeling
kinesthetic teachings. Combining this with the constraint specifications of an
optimization framework, they obtain partially transferable skills to new
target positions, in which imitation learning covers wider workspace motions
and constraint-based optimal control considers the last centimeters.

###  II-B DMPs with force profiles

Dynamic Movement Primitives (DMP) [25] are an established method for encoding
human-recorded motion profiles into parametric models. The inclusion of force
profiles makes them suitable for contact-rich tasks, such as assembly [26].
Methods for teaching include e.g. kinesthetic guiding [27] and teleoperation
[28], [29]. The latter has advantages for obtaining unbiased force profiles
for not having to move the robot in its morphology. The Force profiles are
often not embedded into the transformation system of the DMP, but instead are
parameterized and used directly as reference setpoints in end-effector force
controllers. They can be tracked in parallel to the motion trajectories, e.g.
with a proportional-integral (PI) controller [26]. A compliance matrix can map
the force-torque reference setpoints to motion setpoints for velocity-
controlled robots. Nemec et al. [30] and succeeding works [26], [28], [31],
[29] apply this mechanism to peg-in-hole assembly tasks. Abu-Dakka et al. [26]
and Savarimuthu et al. [29] further use the DMP’s phase modulation to slow
down the execution speed when losing track of the target force profiles in
contact.

###  II-C Our approach

Tackling force-based assembly from within offline programming requires
simulation environments. Compared to using real hardware, simulation offers
the advantage of supporting demonstrations with ease but usually poses high
demands on realism to successfully transfer the skills to real hardware. Onda
et al. [32], used a robotic manipulator as the teach device to steer objects
in a frictionless simulation. Their concept did not include force feedback and
purely relied on the visual channel but was sufficient to extract contact
states for deriving simple position/force commands.

Applying more recent methods to this approach, we propose sequential-
probabilistic models for encoding these complex, human strategies for low-
clearance assembly, using the combination of a Mixture Density Network (MDN)
[33] with Long Short-Term Memory (LSTM) [34]. For execution on real hardware,
we connect this new skill model to an admittance control scheme for industrial
manipulators.

##  III Human-Inspired Force Strategies

Klingbeil et al. investigated human behavior in assembly tasks [35] and tested
the hypothesis that humans ground assembly strategies in mental
representations of contact states. Their results showed that some users passed
a few common states during teleoperation but the intermediate behavior was
mostly random among the participants. Physiological studies on control and
planning through the central nervous system further showed that humans use
mental forward models [36] to predict the response to their motor commands
[37].

Concluding from these results, we propose to drop the high-level construct of
contact states and model human-inspired assembly skills directly from the
cyclic interaction between teleoperated force-torque commands and observed
relative motion.

###  III-A Learning features and simulation

Figure 1:  From simulated teleoperation to robot-controlled execution. We use
a 3Dconnexion™ 3D joystick to steer the assembly objects and solve difficult
jamming situations. An LSTM-MDN model learns reactive strategies from recorded
demonstrations and controls the robotic manipulator during the real assembly.

We apply this insight to human-inspired assembly skills as follows: A user
teleoperates an assembly in simulation with a joystick. In our example, the
assembly consists of two plastic components, that are to be joined together
with dexterity. A low clearance between the parts and complicated geometry for
insertion make this a challenging benchmark, which we use to describe the
concepts of this paper. Fig. 1 illustrates the overall concept. In the
simulator, we record the human-initiated vector $\bm{f}^{h}\in\mathbb{R}^{6}$
that we apply with our fingers via the joystick to steer one of the objects.
To obtain this signal, we interpret the joystick’s six-axes displacement as a
combined vector of forces and torques and scale that to suitable ranges for
the real application. We build the simulation environment on our previous work
[13], in which we describe further requirements, such as velocity-dependent
damping while steering the assembly objects. Increased friction and stiction
between the assembly components during demonstrations shall lead to robust
strategies for the transfer to the real setup. We also record the combined
position and orientation vector $\bm{x}\in\mathbb{R}^{7}$, and the goal-
relative velocity $\dot{\bm{x}}\in\mathbb{R}^{6}$ during assembly. In contrast
to our previous simulator, we describe both variables with respect to the
estimated goal pose.

We believe that the interaction between the recorded motor commands
$\bm{f}^{h}$ and the corresponding outcome $\bm{x},\bm{\dot{x}}$ holds concise
information for learning human-inspired strategies, and the goal of this paper
is to present an improved neural network controller that mimics these
strategies for low-clearance assembly.

To keep equations short, we introduce two abbreviations:

$\displaystyle\hat{\bm{x}}_{t}$
$\displaystyle=\left[\bm{x}_{t}\dot{\bm{x}}_{t}{\bm{f}_{t}^{h}}\right]^{T}$
(1) $\displaystyle\hat{\bm{y}}_{t}$ $\displaystyle=\bm{f}_{t+1}^{h}~{},$ (2)

in which $\hat{\bm{x}}_{t}$ is the concatenated state of all variables at the
time step $t$ and $\hat{\bm{y}}_{t}$ is the force-torque vector at the
succeeding time step.

###  III-B Data analysis

Analyzing the recorded data gives us insights into how to design the required
learning algorithm. Fig. 2(a) above shows the z-component $f_{z}$ of the motor
commands $\bm{f}^{h}$ for the exemplary assembly. This component is the
vertical pushing force and is one of the characteristic variables when joining
both parts at the end of the assembly. We use the Euclidean goal distance as
an indicator of assembly progress. For tasks with low clearances, this
distance is mostly unambiguous in geometry and provides a one-dimensional
reference. The plot shows that it will not be possible to ground the force-
based strategies solely in position for this assembly, because the mean loses
the characteristics of the individual executions. Fig. 2(a) below shows the
respective histogram of the z-component $f_{z}$. The darker the spots, the
more frequently do these values appear in the recorded data. This also
indicates in which regions it took longer to complete the assembly. Having
vertical, dark areas reveals an inverse learning problem: For instance, at
around $5\text{\,}\mathrm{m}\mathrm{m}$ goal distance, any $f_{z}$ between
$0\text{\,}\mathrm{N}$ and $-25\text{\,}\mathrm{N}$ seems valid, because these
data appear frequently in this region in the data set. In conclusion, we need
a probabilistic model to capture the non-deterministic components of $f_{z}$.

Fig. 2(b) above shows $f_{z}$ over time for five individual recordings. The
curves share patterns in specific time windows that slightly vary in scale and
offset. For instance, there are characteristic pushing profiles at the end of
the assembly that start at around $8\text{\,}\mathrm{s}$ for the fastest
demonstration and at around $13\text{\,}\mathrm{s}$ for the slowest
demonstration. While a probabilistic model could reproduce the distributions
from Fig. 2(a) below, it could not generate predictions coherent in time as
they appear in our recordings. In conclusion, we additionally need a sort of
memory to model sequential patterns in the human-recorded strategies.

(a)

(b)

Figure 2:  Analysis of recorded demonstrations in the simulator. (a) above
Band plot of the $f_{z}$ component during 100 demonstrations. We used
statistical data binning to cluster all demonstrations into vertical bins,
from which we computed the min, max, and mean values, respectively. An
additional, single demonstration visualizes strategies during assembly. (a)
below The color gradient shows the qualitative occurrence of discretized
values for forces $f_{z}$ in a $180\times 60$ grid. (b) above Time sequences
for 5 performances reveal repeating patterns in $f_{z}$. (b) below The
vertical position $z$ shows the assembly’s progress toward zero, which
indicates the end of the assembly.

##  IV Force Strategies - a Temporal-probabilistic Approach

The previous section motivated two key components in modeling force strategies
for our benchmark: probabilistics to cover the multivalued character of the
distributions, and memory to model their time dependencies. In this section,
we describe how we include both into a temporal probabilistic model for
learning force strategies for low-clearance assembly.

###  IV-A Capturing patterns in sequences

We build our model on the Long Short-Term Memory (LSTM) [38], which has been
used in many domains and which excels at learning long-term dependencies in
data. In contrast to feed forward networks, LSTMs maintain an internal state,
the cell state $c_{t}$, and update information via gated operations. This
allows them to effectively encode long sequences of data. In our work, we use
the LSTM primarily to encode time windows of past assembly progress into a
concise representation and will later combine this with a probabilistic output
layer to predict meaningful next force strategies with high confidence.

We combine $m$ of these cells into an LSTM layer, allowing us to describe the
forward pass with the following matrix-based equations [34], [38]:

$\displaystyle\bm{f}_{t}$
$\displaystyle=\text{sigmoid}(\bm{W}_{f}\hat{\bm{x}}_{t}+\bm{b}_{f}+\bm{U}_{f}\bm{h}_{t-1})$
(3) $\displaystyle\bm{i}_{t}$
$\displaystyle=\text{sigmoid}(\bm{W}_{i}\hat{\bm{x}}_{t}+\bm{b}_{i}+\bm{U}_{i}\bm{h}_{t-1})$
$\displaystyle\bm{o}_{t}$
$\displaystyle=\text{sigmoid}(\bm{W}_{o}\hat{\bm{x}}_{t}+\bm{b}_{o}+\bm{U}_{o}\bm{h}_{t-1})$
$\displaystyle\bm{c}_{t}$
$\displaystyle=\bm{f}_{t}\circ\bm{c}_{t-1}+\bm{i}_{t}\circ\text{tanh}(\bm{W}_{c}\hat{\bm{x}}_{t}+\bm{b}_{c}+\bm{U}_{c}\bm{h}_{t-1})$
$\displaystyle\bm{h}_{t}$
$\displaystyle=\bm{o}_{t}\circ\text{tanh}(\bm{c}_{t}).$

The vector $\hat{\bm{x}}_{t}\in\mathbb{R}^{c}$ is the input to the LSTM layer
and $\bm{h}_{t}\in\mathbb{R}^{m}$ is the output.
$\bm{f}_{t},\bm{i}_{t},\bm{o}_{t}\in\mathbb{R}^{m}$ are the activation vectors
for the forget gates, the input gates, and the output gates respectively. Each
has a non-recurrent weight matrix $\bm{W}\in\mathbb{R}^{m\times c}$ and a
recurrent weight matrix $\bm{U}\in\mathbb{R}^{m\times m}$ for the previous
hidden state $\bm{h}_{t-1}\in\mathbb{R}^{m}$. We further use $\circ$ to denote
the element-wise product. The layer-wide input $\bm{h}_{t-1}$ generates
recurrent connections between each cell in an LSTM layer. We use

$\text{sigmoid}(x)=\frac{1}{1+e^{-x}}$ (4)

as activation for each element in a vector and squash inputs and outputs with
the hyperbolic tangent. The weight matrices $\bm{W},\bm{b}$, and $\bm{U}$
include all learnable parameters. We use a single-layered LSTM and adjust the
number of learnable parameters with the number of cells $m$ in that layer.

Note that Eq. (3) yields $N+1$ consecutive hidden states and cell states for
an input sequence of consecutive points
$\hat{\bm{x}}_{t-N},\dots,\hat{\bm{x}}_{t}$. We are only interested in the
last pair $\bm{h}_{t},\bm{c}_{t}$, (which we refer to as the encoding) for
which the LSTM has seen the complete input sequence:

$\text{LSTM}(\hat{\bm{x}}_{t-N},\dots,\hat{\bm{x}}_{t}):=(\bm{h}_{t},\bm{c}_{t})~{}.$
(5)

The next section shows how we use this encoding to estimate probabilities for
the next force-torque vectors.

###  IV-B Modeling the ambiguity in human behavior

The recorded force strategies had seemingly random trial and error, resulting
in distributions for specific regions. Sampling from these distributions alone
will not be successful, since those samples will not form coherent strategies
over time. Using the encoding from the LSTM, however, we can model more
specific distributions to sample from and thus consider past assembly
progress. We use the Mixture Density Network (MDN) [33] to model these
characteristics. LSTMs and MDNs have been combined for modeling highly complex
human performance, such as generating human handwriting from digital text [39]
or creating full-body, freestyle dance motion [40].

MDNs model conditional probability densities of the target data, conditioned
on the input data [33]. In our notation, we describe this with
$p(\hat{\bm{y}}_{t}\mid\bm{h}_{t},\bm{c}_{t})$, in which the hidden state
$\bm{h}_{t}$ and cell state $\bm{c}_{t}$ are the encoding of the input
sequence according to Eq. (5).

For combining the LSTM encoding and the MDN output, we feed the encoding into
a fully connected layer with linear activation

$\bm{z}=\bm{W}_{z}\left[\bm{h}^{T},\bm{c}^{T}\right]^{T}.$ (6)

The weight matrix $\bm{W}_{z}\in\mathbb{R}^{k(6+2)\times 2m}$ introduces
additional learning parameters. The next step is to split $\bm{z}$ into a
tuple $(\bm{z}^{\alpha},\bm{z}^{\mu},\bm{z}^{\sigma})$ and use the individual
components to parameterize a Gaussian mixture model of $k$ Gaussian
distributions $\mathcal{N}_{6}(\bm{\mu},\sigma^{2})$ that are each
$6$-dimensional. The six dimensions derive from our intention to model the
$6$-dimensional force-torque vector.

First, we compute normalized mixing coefficients for the Gaussians from
$\bm{z}^{\alpha}$ with a softmax according to

$\alpha_{i}=\frac{\text{exp}(z_{i}^{\alpha})}{\sum_{j=1}^{k}\text{exp}(z_{j}^{\alpha})}~{},\qquad
i=1\dots k~{},\qquad\alpha_{i}\in\mathbb{R}~{}.$ (7)

The centers of the Gaussians are computed from $\bm{z}^{\mu}$ with

$\bm{\mu_{i}}=z_{ij}^{\mu}~{},\qquad i=1\dots k~{},j=1\dots
6~{},\qquad\bm{\mu}_{i}\in\mathbb{R}^{6}$ (8)

and the scale parameters from $\bm{z}^{\sigma}$ with

$\sigma_{i}=\text{exp}(z_{i}^{\sigma}),\qquad i=1\dots
k~{},\qquad\sigma_{i}\in\mathbb{R}~{}.$ (9)

Using these parameters, we can compute the conditional density for the $i$th
Gaussian with [33]

$\phi_{i}(\hat{\bm{y}}_{t}\mid\bm{\mu}_{i},\sigma_{i})=\frac{1}{(2\pi)^{6/2}\sigma_{i}^{6}}\text{exp}\left(-\frac{\lVert\hat{\bm{y}}_{t}-\bm{\mu}_{i}\rVert^{2}}{2\sigma_{i}^{2}}\right).$
(10)

Finally, the conditional probability density is composed of a linear
combination of $k$ Gaussian functions $\phi_{i}$ [33]

$p(\hat{\bm{y}}_{t}\mid\bm{h}_{t},\bm{c}_{t})=\sum_{i=1}^{k}\alpha_{i}\phi_{i}(\hat{\bm{y}}_{t}\mid\bm{h}_{t},\bm{c}_{t})~{}.$
(11)

This probability describes how likely $\hat{\bm{y}}_{t}$ follows the observed
sequence $\hat{\bm{x}}_{t-n},n=N\dots 0$, which we feed in form of the LSTM’s
encoding $(\bm{h}_{t},\bm{c}_{t})$ into the MDN. The computed mixture
parameters thus become functions of the encoded sequence for each time step.
During training, this mechanism allows our model to solve ambiguity in the
recordings by parameterizing fine-grained probability density functions along
the assembly progress.

###  IV-C Data composition and training

Figure 3:  (a) Composition of training samples from recorded demonstrations in
simulation. (b) The unrolled LSTM-MDN model with inputs and outputs during the
forward pass. The model is trained with BPTT, using the negative log-
likelihood of the conditional probability density.

The LSTM-MDN model is a recursive prediction model from state-action pairs to
strategies. Its desired output is motor control commands $\bm{f}^{h}$ that
change the assembly object’s state $\bm{x},\dot{\bm{x}}$, which in turn
provokes a next motor command. We use supervised learning for training by
splitting the recorded demonstrations in the simulator into input sequences
and respective labels. Fig. 3(a) illustrates this approach. We obtain training
samples from the data set by selecting individual demonstrations $\xi_{i}$ out
of all recorded assembly demonstrations. A pivotal time step $t$ selects the
starting point. We then compose the sample with the input sequence
$\hat{\bm{x}}_{t-N},\dots,\hat{\bm{x}}_{t}$ and the respective label
$\hat{\bm{y}}_{t}$, which is the force-torque component of the next
$\hat{\bm{x}}_{t+1}$.

The randomly extracted samples have an equal length of $N+2$, so that the
LSTM-MDN model has access to $N$ time steps before predicting the
corresponding probability density function. During training, we sample batches
for parallel processing and cover the entirety of the training data
stochastically.

We train our model with backpropagation through time (BPTT) by unrolling the
model $N$ time steps. Fig. 3(b) shows the unrolled model as a normal
feedforward network. The LSTM layer’s recurrent connections become normal
feedforward connections and propagate the cell state and hidden state. All
LSTM instances from Fig. 3(b) share a common set of learnable weights from Eq.
(3) and refer to the same layer, which is illustrated at specific time steps
together with the respective input from the training sequence. The fully
connected output layer parameterizes the Gaussian mixture model, from which we
compute the conditional probability for the given label. We finally optimize
all learnable weights by minimizing the negative log-likelihood

$\mathcal{L}=-\text{ln}\left(p(\hat{\bm{y}}_{t}\mid\hat{\bm{x}}_{t-N},\dots,\hat{\bm{x}}_{t})\right)$
(12)

of this conditional probability density.

##  V Force Control and Assembly Skills

Figure 4:  Control scheme of the neural network controller for assembly. The
LSTM-MDN receives state feedback from the real robot and generates force-
torque references for our simulation-based force control algorithm.

During deployment, the LSTM-MDN model predicts 6-dimensional Gaussian
distributions, from which we sample force-torque strategies for robot control
in each time step. We use the forward dynamics-based control approach of our
previous work [41] to realize Cartesian force control on the robotic
manipulator. Its implementation is available open-
source111https://github.com/fzi-forschungszentrum-
informatik/cartesian_controllers. The algorithm applies the force-torque
vector at the end-effector of a simulated robot and computes the corresponding
joint acceleration of the system with simplified forward dynamics. The
resulting motion is integrated and sent to the joint position servos of the
real industrial robot. Fig. 4 shows the control scheme.

The hat symbol $\widehat{(.)}$ denotes simulated values that are computed on a
simulated robot model with equal kinematics. First, we compute an error vector
from the reference force-torque profile $\bm{f}^{h}$ as sampled from the LSTM-
MDN model and the measurements $\bm{f}$ from a Cartesian force-torque sensor
in contact with the environment

$\widehat{\bm{f}}=\bm{K}_{p}(\bm{f}^{h}-\bm{f})~{}.$ (13)

$\bm{K}_{p}$ is a positive, diagonal gain matrix. We then simulate the robot’s
response with

$\ddot{\widehat{\bm{q}}}=\widehat{\bm{H}}^{-1}\widehat{\bm{f}}~{},$ (14)

in which $\widehat{\bm{H}}$ is a linearized joint space inertia matrix that is
computed in each control cycle. We refer to our work [41] for more details on
this aspect. We then obtain the joint positions $\widehat{\bm{q}}$ with the
Euler forward method according to

$\displaystyle\widehat{\bm{q}}_{t}=\widehat{\bm{q}}_{t-1}+\dot{\widehat{\bm{q}}}_{t-1}\Delta\widehat{t}$
(15)
$\displaystyle\dot{\widehat{\bm{q}}}_{t}=\dot{\widehat{\bm{q}}}_{t-1}+\ddot{\widehat{\bm{q}}}_{t}\Delta\widehat{t}~{}.$
(16)

An additional joint damping $\dot{\widehat{\bm{q}}}_{t}\leftarrow
0.9~{}\dot{\widehat{\bm{q}}}_{t}$ in each simulation cycle avoids
oscillations. The simulated time window $\Delta\widehat{t}$ can be chosen
independently of the robot’s real control rate and allows for tweaking the
controller’s response to force-torque inputs. The simulated joint positions
$\widehat{\bm{q}}$ are then forwarded to the joint position servos of the real
manipulator. Finally, we use a forward kinematics routine to compute the end-
effector pose $\bm{x}$ and velocity $\dot{\bm{x}}$, and feed them back into
the LSTM-MDN model for the next prediction. Note how model inference and force
control run at different rates.

##  VI Experiments and Results

###  VI-A Dataset and simulation

(a)

(b)

Figure 5:  (a) Histogram for the assembly benchmark consisting of 1155
demonstrations, equalling 7 hours and 10 minutes of demonstrated strategies in
the simulator. The dashed lines $Q_{1}$ to $Q_{3}$ separate the datasets into
equally populated quarters. (b) Learning curves of the different models,
trained with $k=4$, $N=25$, learning rate $5\cdot 10^{-4}$, and a minibatch
size of $128$.

Training data usually determines the quality of data-driven models. We thus
investigated how the quality of demonstrations influences the learning of
assembly strategies. All demonstrations were recorded at
$100\text{\,}\mathrm{H}\mathrm{z}$ from random starting poses until finishing
the assembly in simulation. Since they were all successful, we use their
duration as a measure of quality, implying that the shorter an assembly
demonstration took, the higher the quality of the recorded strategies. We
separated the training set with quartiles $Q_{1}$ \- $Q_{3}$ as depicted in
Fig. 5(a) and used five datasets to train on demonstrations with different
duration:

  * •

$0\rightarrow Q_{1}$ : up to $14.93\text{\,}\mathrm{s}$

  * •

$Q_{1}\rightarrow Q_{2}$ : between $14.93\text{\,}\mathrm{s}$ and
$18.74\text{\,}\mathrm{s}$

  * •

$Q_{2}\rightarrow Q_{3}$: between $18.74\text{\,}\mathrm{s}$ and
$24.32\text{\,}\mathrm{s}$

  * •

$Q_{3}\rightarrow\infty$ : longer than $24.32\text{\,}\mathrm{s}$

  * •

$0\rightarrow\infty$ : all demonstrations

Fig. 5(b) shows the batch loss during training for the different models. They
perform similarly in predicting the labels of the evaluation set with a slight
degradation of the model trained only on the last quarter.

Continuing the experiment, we used the simulator to evaluate the different
models on the assembly task. The models’ predicted force-torque strategies
steered the assembly object similar to as was done during demonstration with
the joystick. Fig. 6 shows a successful execution. The assembly has mainly two
challenging regions: The first is the insertion at $3.6\text{\,}\mathrm{s}$
where parts jam and tilt frequently, and the second is at
$6.6\text{\,}\mathrm{s}$ where pressing in requires dexterity due to low-
clearances. We measured success with the Euclidean distance to the goal pose
$\|\bm{x}\|$ and cut off at $12\text{\,}\mathrm{s}$ simulated time
($60\text{\,}\mathrm{s}$ realtime), after which the execution was considered a
failure. Fig. 6(b) shows the results. The model trained on the complete
dataset performs best on the benchmark. The model trained on the second
quarter of the dataset has a similar performance, which is interesting due to
using only $25$% training data. This could be due to hitting a sweet spot
between data quality and amount of recorded time. The model trained on the
last quartile shows the lowest success rate, corresponding to the degraded
learning curve from Fig. 5.

In conclusion, the quality of demonstrations in the simulator has an impact on
the model to learn successful force-torque strategies and more effective
(shorter) demonstrations can partially replace bigger datasets.

(a)

(b)

Figure 6:  (a) A successful run in simulation at $20$% realtime. (b) Assembly
performance in simulation with the success rate including outliers. We tested
30 trials for each model. Successful executions reached a goal distance of
$0.5\text{\,}\mathrm{m}\mathrm{m}$ in 12s.

###  VI-B Real hardware

The goal of this experiment was to test if the strategies are portable to
real-world systems through our controller. We prepared this transfer in our
concept by allowing partly unrealistic mesh interpenetration in the simulator
during demonstrations and increased friction and stiction between the assembly
components.

Figure 7:  Setup for the strategy transfer with the UR10e robot. Left: The
assembly is clamped in a vice and glued to a surface with adhesive tape.
Right: The blue object is glued to the jaws of a non-actuated gripper, whose
elasticity adds uncertainty to the system.

Fig. 7 shows the setup. We used a passive gripper for this experiment to
attach the blue assembly object to the robot’s end-effector. We calibrated the
setup by teleoperating the robot in force control to the goal pose of the
assembly and recorded the final configuration. During execution, the model
generated force-torque reference commands at $5\text{\,}\mathrm{H}\mathrm{z}$
that were executed with the force controller on the UR10e robot at
$500\text{\,}\mathrm{H}\mathrm{z}$. Fig. 8 shows the strategies and assembly
progress for one such execution.

Figure 8:  Commanded force-torque strategies for a successful assembly on the
real setup.

During the first insertion, the parts quickly jam and require strategies to
advance. Note how the model makes several forward push attempts with the
swings of the $f_{x}$ component. Since this is not immediately successful, the
model tries different re-orientations while searching for part clearance.
After approximately $95\text{\,}\mathrm{s}$, the strategies are successful and
finally plug both parts together. The execution is shown in full length in the
accompanying video.

##  VII Discussion

During the executions on the real setup, configurations that significantly
differed from the dataset decreased the model’s performance. Real-world
assemblies might have several local optima along the goal where parts can be
put in unseen and wrong configurations, leading to possible deadlocks. If
these spots are known or can be anticipated, they should be included randomly
as challenges in the simulator to increase the model’s robustness.

A further limitation is the speed of execution. Admittance controlled
industrial robots that close force control around position-controlled joints,
are not stable during interaction with stiff environments at human speed. It
might thus be worthwhile to deploy this model together with an impedance-
controlled robot.

When programming an assembly with our approach, some experience with the real
parts is required to set suitable ranges for the force-torque signals and to
apply meaningful strategies in the simulator. It is also helpful to touch the
real assembly parts to better imagine how they combine in the simulator. The
strength of our approach is, however, that the predicted strategies can be
post-scaled and sanity-checked before sending them to the controller. For
instance, the force-torque strategies’ amplitude could be decreased by $50$%
on the first run.

##  VIII Conclusions

In this work, we proposed a method to learn human-inspired assembly strategies
from simulated teleoperation. We addressed two challenges that are common in
Imitation Learning for force-sensitive assembly tasks: Handling low clearances
with tight-fitting parts that require much trial and error, and learning these
strategies offline without access to the real hardware. Basing our ideas on
data analysis of recorded demonstrations, we combined an LSTM-based encoder
with an MDN to model probabilistic force strategies that follow patterns over
time. We evaluated this approach experimentally in simulation and on real
hardware with an exemplary benchmark. The results showed that human-inspired
strategies that are learned from simulated data with our method are concise
enough to be successfully transferred to a real robot.

In future research, we plan to reduce the sample intensity that is currently
required in the simulator. A promising method could be including Offline RL
[42] on the demonstration data to exceed human performance without further
interaction with the simulator.

## References

  * [1] Zengxi Pan et al.  “Recent progress on programming methods for industrial robots”  In _Robotics and Computer-Integrated Manufacturing_ 28.2 Elsevier, 2012, pp. 87–94  URL: https://www.sciencedirect.com/science/article/abs/pii/S0736584511001001
  * [2] S. Gottschlich, C. Ramos and D. Lyons  “Assembly and task planning: a taxonomy”  In _IEEE Robotics Automation Magazine_ 1.3, 1994, pp. 4–12  DOI: 10.1109/100.326723
  * [3] W.S. Newman et al.  “Force-responsive robotic assembly of transmission components”  In _Robotics and Automation, 1999. Proceedings. 1999 IEEE International Conference on_ 3, 1999, pp. 2096–2102 vol.3  DOI: 10.1109/ROBOT.1999.770416
  * [4] In-Wook Kim, Dong-Jin Lim and Kab-Il Kim  “Active peg-in-hole of chamferless parts using force/moment sensor”  In _Proceedings 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No.99CH36289)_ 2, 1999, pp. 948–953 vol.2  DOI: 10.1109/IROS.1999.812802
  * [5] Wyatt S Newman, Yonghong Zhao and Y-H Pao  “Interpretation of force and moment signals for compliant peg-in-hole assembly”  In _Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 01CH37164)_ 1, 2001, pp. 571–576  IEEE
  * [6] Te Tang et al.  “Autonomous alignment of peg and hole by force/torque measurement for robotic assembly”  In _2016 IEEE international conference on automation science and engineering (CASE)_ , 2016, pp. 162–167  IEEE
  * [7] A. Wahrburg et al.  “Combined pose-wrench and state machine representation for modeling Robotic Assembly Skills”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2015, pp. 852–857  DOI: 10.1109/IROS.2015.7353471
  * [8] A. Stemmer, A. Albu-Schaffer and G. Hirzinger  “An Analytical Method for the Planning of Robust Assembly Tasks of Complex Shaped Planar Parts”  In _Robotics and Automation, 2007 IEEE International Conference on_ , 2007, pp. 317–323  DOI: 10.1109/ROBOT.2007.363806
  * [9] L. Johannsmeier, M. Gerchow and S. Haddadin  “A Framework for Robot Manipulation: Skill Formalism, Meta Learning and Adaptive Control”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 5844–5850  DOI: 10.1109/ICRA.2019.8793542
  * [10] Sergey Levine, Chelsea Finn, Trevor Darrell and Pieter Abbeel  “End-to-end training of deep visuomotor policies”  In _arXiv preprint arXiv:1504.00702_ , 2015  URL: http://www.jmlr.org/papers/volume17/15-522/source/15-522.pdf
  * [11] Tadanobu Inoue et al.  “Deep reinforcement learning for high precision assembly tasks”  In _arXiv preprint arXiv:1708.04033_ , 2017  URL: https://arxiv.org/abs/1708.04033
  * [12] Gerrit Schoettler et al.  “Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards”  In _arXiv preprint arXiv:1906.05841_ , 2019  URL: https://arxiv.org/abs/1906.05841v2
  * [13] S. Scherzinger, A. Roennau and R. Dillmann  “Contact Skill Imitation Learning for Robot-Independent Assembly Programming”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 4309–4316  DOI: 10.1109/IROS40897.2019.8967523
  * [14] Rüdiger Dillmann, M Kaiser and Ales Ude  “Acquisition of elementary robot skills from human demonstration”  In _International Symposium on Intelligent Robotics Systems_ , 1995, pp. 185–192  Citeseer
  * [15] Aude Billard, Sylvain Calinon, Rüdiger Dillmann and Stefan Schaal  “Robot programming by demonstration”  In _Springer handbook of robotics_ Springer, 2008, pp. 1371–1394
  * [16] Stefan Schaal  “Is imitation learning the route to humanoid robots?”  In _Trends in cognitive sciences_ 3.6 Elsevier, 1999, pp. 233–242  URL: https://www.sciencedirect.com/science/article/abs/pii/S1364661399013273
  * [17] Michael Bain and Claude Sammut  “A Framework for Behavioural Cloning.”  In _Machine Intelligence 15_ , 1995, pp. 103–129
  * [18] Micha Hersch, Florent Guenter, Sylvain Calinon and Aude Billard  “Dynamical system modulation for robot learning via kinesthetic demonstrations”  In _IEEE Transactions on Robotics_ 24.6 IEEE, 2008, pp. 1463–1467  URL: https://ieeexplore.ieee.org/abstract/document/4674620
  * [19] S. Calinon, F. D’halluin, D.. Caldwell and A.. Billard  “Handling of multiple constraints and motion alternatives in a robot programming by demonstration framework”  In _2009 9th IEEE-RAS International Conference on Humanoid Robots_ , 2009, pp. 582–588  URL: https://ieeexplore.ieee.org/abstract/document/5379592
  * [20] L. Rozo et al.  “Learning Physical Collaborative Robot Behaviors From Human Demonstrations”  In _IEEE Transactions on Robotics_ 32.3, 2016, pp. 513–527  URL: https://ieeexplore.ieee.org/abstract/document/7450630
  * [21] Klas Kronander, Etienne Burdet and Aude Billard  “Task transfer via collaborative manipulation for insertion assembly”  In _Workshop on human-robot interaction for industrial manufacturing, robotics, science and systems_ , 2014  Citeseer
  * [22] Ali Al-Yacoub et al.  “Symbolic-based recognition of contact states for learning assembly skills”  In _Frontiers in Robotics and AI_ 6 Frontiers, 2019, pp. 99  URL: https://www.frontiersin.org/articles/10.3389/frobt.2019.00099/full
  * [23] Shen Dong and Fazel Naghdy  “Application of hidden Markov model to acquisition of manipulation skills from haptic rendered virtual environment”  In _Robotics and Computer-Integrated Manufacturing_ 23.3 Elsevier, 2007, pp. 351–360  URL: https://www.sciencedirect.com/science/article/abs/pii/S0736584506000251
  * [24] C.. Vergara P., J. De Schutter and E. Aertbeliën  “Combining Imitation Learning With Constraint-Based Task Specification and Control”  In _IEEE Robotics and Automation Letters_ 4.2, 2019, pp. 1892–1899  DOI: 10.1109/LRA.2019.2898035
  * [25] Auke Jan Ijspeert et al.  “Dynamical movement primitives: learning attractor models for motor behaviors”  In _Neural computation_ 25.2 MIT Press, 2013, pp. 328–373
  * [26] Fares J Abu-Dakka et al.  “Adaptation of manipulation skills in physical contact with the environment to reference force profiles”  In _Autonomous Robots_ 39.2 Springer, 2015, pp. 199–217  URL: https://link.springer.com/article/10.1007/s10514-015-9435-2
  * [27] Aljaž Kramberger et al.  “Transfer of contact skills to new environmental conditions”, 2016, pp. 668–675  IEEE  URL: https://ieeexplore.ieee.org/abstract/document/7803346
  * [28] Norbert Krüger et al.  “Technologies for the Fast Set-Up of Automated Assembly Processes”  In _KI-Künstliche Intelligenz_ 28.4 Springer, 2014, pp. 305–313
  * [29] T.. Savarimuthu et al.  “Teaching a Robot the Semantics of Assembly Tasks”  In _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ PP.99, 2017, pp. 1–23  DOI: 10.1109/TSMC.2016.2635479
  * [30] B. Nemec et al.  “Transfer of assembly operations to new workpiece poses by adaptation to the desired force profile”  In _16th International Conference on Advanced Robotics (ICAR)_ , 2013, pp. 1–7  DOI: 10.1109/ICAR.2013.6766568
  * [31] Aljaz Kramberger et al.  “Generalization of orientation trajectories and force-torque profiles for robotic assembly”  In _Robotics and Autonomous Systems_ 98, 2017, pp. 333–346  DOI: https://doi.org/10.1016/j.robot.2017.09.019
  * [32] H. Onda, H. Hirukawa and K. Takase  “Assembly motion teaching system using position/force simulator - extracting a sequence of contact state transition”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 1995, pp. 9–16  DOI: 10.1109/IROS.1995.525768
  * [33] Christopher M Bishop  “Mixture density networks”, 1994
  * [34] Sepp Hochreiter and Jürgen Schmidhuber  “Long short-term memory”  In _Neural computation_ 9.8 MIT Press, 1997, pp. 1735–1780  URL: https://www.mitpressjournals.org/doi/abs/10.1162/neco.1997.9.8.1735
  * [35] Ellen Klingbeil, Samir Menon and Oussama Khatib  “Experimental Analysis of Human Control Strategies in Contact Manipulation Tasks”  In _2016 International Symposium on Experimental Robotics_ Cham: Springer International Publishing, 2017, pp. 275–286  DOI: 10.1007/978-3-319-50115-4_25
  * [36] Michael I Jordan and Robert A Jacobs  “Hierarchies of adaptive experts”  In _Advances in neural information processing systems_ , 1992, pp. 985–992
  * [37] R.C. Miall and D.M. Wolpert  “Forward Models for Physiological Motor Control” Four Major Hypotheses in Neuroscience  In _Neural Networks_ 9.8, 1996, pp. 1265–1279  DOI: http://dx.doi.org/10.1016/S0893-6080(96)00035-4
  * [38] Felix A. Gers, Jürgen Schmidhuber and Fred A. Cummins  “Learning to Forget: Continual Prediction with LSTM”  In _Neural Computation_ 12, 2000, pp. 2451–2471  URL: https://www.semanticscholar.org/paper/Learning-to-Forget%3A-Continual-Prediction-with-LSTM-Gers-Schmidhuber/11540131eae85b2e11d53df7f1360eeb6476e7f4
  * [39] Alex Graves  “Generating sequences with recurrent neural networks”  In _arXiv preprint arXiv:1308.0850_ , 2013  URL: https://arxiv.org/abs/1308.0850
  * [40] Luka Crnkovic-Friis and Louise Crnkovic-Friis  “Generative choreography using deep learning”  In _arXiv preprint arXiv:1605.06921_ , 2016  URL: https://arxiv.org/abs/1605.06921
  * [41] Stefan Scherzinger, Arne Roennau and Rüdiger Dillmann  “Virtual Forward Dynamics Models for Cartesian Robot Control”, 2020  arXiv: https://arxiv.org/abs/2009.11888
  * [42] Ilya Kostrikov, Ashvin Nair and Sergey Levine  “Offline Reinforcement Learning with Implicit Q-Learning”  arXiv, 2021  DOI: 10.48550/ARXIV.2110.06169