# DexSkills: Skill Segmentation Using Haptic Data for Learning Autonomous
Long-Horizon Robotic Manipulation Tasks

Xiaofeng Mao1,† and Gabriele Giudici2,†,
Claudio Coppola3, Kaspar Althoefer2, Ildar Farkhatdinov2, Zhibin Li4, Lorenzo
Jamone2 Xiaofeng Mao and Gabriele Giudici contributed equally to this work and
share first authorship.Work not related to Claudio Coppola’s position in
Amazon.1University of Edinburgh, email<EMAIL_ADDRESS>authors are
with ARQ (the Centre for Advanced Robotics @ Queen Mary), School of
Engineering and Materials Science, Queen Mary University of London, London,
E14NS, UK (emails:
{g.giudici,k.althoefer,i.farkhatdinov,l.jamone}@qmul.ac.uk).3Amazon
Transportation Services, email<EMAIL_ADDRESS>College
London, email<EMAIL_ADDRESS>study is funded by the UKRI EPSRC grants
EP/R02572X/1 (NCNR), EP/V035304/1 (q-Arena) and Queen Mary University of
London Ph.D. scholarship to G. Giudici.

###### Abstract

Effective execution of long-horizon tasks with dexterous robotic hands remains
a significant challenge in real-world problems. While learning from human
demonstrations have shown encouraging results, they require extensive data
collection for training. Hence, decomposing long-horizon tasks into reusable
primitive skills is a more efficient approach. To achieve so, we developed
DexSkills, a novel supervised learning framework that addresses long-horizon
dexterous manipulation tasks using primitive skills. DexSkills is trained to
recognize and replicate a select set of skills using human demonstration data,
which can then segment a demonstrated long-horizon dexterous manipulation task
into a sequence of primitive skills to achieve one-shot execution by the robot
directly. Significantly, DexSkills operates solely on proprioceptive and
tactile data, i.e., haptic data. Our real-world robotic experiments show that
DexSkills can accurately segment skills, thereby enabling autonomous robot
execution of a diverse range of tasks.

##  I INTRODUCTION

Humans show remarkable dexterity and adaptability in performing manipulation
tasks across various environments, this is attributed to the inherent
capabilities of the human hand. However, enabling robotic dexterous
manipulation with multi-fingered hands remains challenging due to the high-
dimensional action space and occlusion during manipulation. Recent research
efforts have focused on addressing this challenge through either model-based
control methods [1, 2, 3] or learning-based approaches [4, 5, 6]. However,
these methods rely on complex mathematical models, require customized rewards
designed by humans, demand extensive training time, and necessitate large
amounts of demonstration data [7]. Consequently, the training processes become
inefficient and labor-intensive, hindering robots from achieving dexterous
manipulation.

Multi-modal information is crucial in enabling robot dexterous manipulation.
In routine tasks like pick-and-place operations, visual cues provide
information on the position and shape of the object, aiding the robot during
its execution. However, the availability of accurate visual information may be
compromised due to factors such as poor lighting conditions or occlusions. In
addition, certain aspects of objects and tasks, especially in contact-rich
manipulation tasks, are better encoded by tactile and force sensing [8, 9].
Tactile feedback offers valuable insights into the contact dynamics between
the robotic fingers and the manipulated object, enhancing the potential for
accurate and stable manipulation. Hence, numerous studies are currently
investigating the viability of leveraging tactile information as an
alternative or complementary source [10, 11, 12, 13, 14].

Figure 1:  Overview of the proposed long-horizon task segmentation approach.
Individual skills are segmented and classified at each temporal window of the
demonstration. The demonstrations are collected via the teleoperation system
(left) developed in [15].

Learning from human demonstrations makes it possible to learn from real-world
human demonstrations, avoiding the need for intricate dynamic models, well
defined reward functions, and reliable simulated environments [16]. On the
other hand, these approaches often demand a considerable amount of data and
human effort [17]. Moreover, for long-horizon tasks, these methods are prone
to failure due to the accumulation of compounding errors and necessitate model
retraining for each new demonstrated task, even if many task phases are in
common between tasks [18, 19].

Therefore, we propose DexSkills, a system that learns primitive skills from
human demonstrations, and uses such knowledge to segment a human-demonstrated
long-horizon task in a sequence of skills that the robot can execute
autonomously. This requires less data collection efforts with respect to
learning each possible long-horizon task from human demonstration. Notably,
DexSkills relies only on proprioceptive and tactile data (i.e. haptic data).
In particular, the contributions of this work are:

  1. 1.

We introduce a novel supervised representation learning framework where the
latent features from haptic data are jointly trained via an auto-regressive
autoencoder and a label decoder. Our qualitative result proves that by
capturing the latent dynamics of primitive skills, the performance of
segmenting unseen long-horizon tasks into skill sequences is significantly
improved;

  2. 2.

We propose a set of 20 core primitive skills specifically designed to address
robot dexterous manipulation tasks. We trained the autonomous robot controller
for each skill using Multi-Layer Perceptron (MLP) and demonstrated that by
recombining these trained core primitive skills, the robot is able to compose
and achieve long-horizon dexterous tasks effectively and successfully;

  3. 3.

To facilitate future research, we open-sourced our
code111https://github.com/ARQ-CRISP/DexSkills and provided a labeled dataset
comprising 20 primitive skills and 20 long-horizon tasks, which were obtained
from human teleoperated demonstrations.

DexSkills has developed a primary set of reusable primitive skills trained
from haptic human demonstrations, which can be reused and re-combined to
represent various compositions of long-horizon tasks. DexSkills achieves
remarkable segmentation accuracy of $91\%$ for unseen tasks and enables
autonomous robot execution of diverse tasks using only proprioceptive and
tactile data.

##  II Related Works

Imitation learning based robot dexterous manipulation: Robot dexterous
manipulation has recently gained increasing attention [20]. Imitation learning
(IL) is a well-established method that accelerates the skill learning process
in robots by leveraging human experience [21]. In [22], human demonstrations
are captured within a virtual environment, employing learning-based methods
for feedback control in executing non-prehensile manipulations The research
[23] proposes to train an inverse dynamics model to predict actions for state-
only demonstration [23]. In [24], an RGB camera is used to observe a human
operator teleoperate a robot and train the policy with the imitation learning
method.

Long-horizon imitation learning: For long-horizon manipulation tasks,
traditional IL methods might not be sufficient and could fail due to the
accumulation of compounding errors. Breaking down the long-horizon
manipulation task into multiple sub-tasks is a widely adopted strategy for
addressing this challenge. By decomposing the IL for manipulation task into
visual servoing and behaviour replay, [25] has effectively facilitated one-
shot imitation learning for real-world daily tasks. Additionally, the strategy
of automatic waypoint extraction to segment the entire trajectory
significantly reduces the BC horizon, effectively mitigating the problem of
error accumulation [19]. Hierarchical imitation learning serves as a prominent
strategy to solve long-horizon manipulation tasks. By training a high-level
policy for skill selection and a low-level policy for precise motor control,
robots are capable of handling a wide range of dexterous manipulation tasks,
including activities like cable routing [18], drink pouring [26] and bi-manual
cooperation task [27, 28].

Dexterous manipulation with tactile sensing: Recently, some works have
investigated using tactile sensing to mitigate occlusion during manipulation
tasks with robot dexterous hand. By overlaying the binary force sensor to the
one side of the whole robot hand, the work [11] has successfully facilitated
the object in-hand rotation task. With the same robot hand setup, the study
[12] demonstrated the capability of employing tactile sensors to search for
and locate target objects, subsequently manipulating these objects to perform
daily tasks. Similarly, [29] use the tactile sensor Uskin [30] and proposes to
pre-train a tactile encoder to extract features from high-dimensional tactile
information, which enables the robot to perform tasks such as book opening,
bowl and cup unstacking. The work [13] investigates using sparse tactile
feedback to localize, identify and grasp novel objects without any visual
feedback.

Primitive Skills: Daily life manipulation tasks often comprise a variety of
primitive skills. Identifying the sequence in which these skills combine for
any long-horizon task is crucial for facilitating their reuse, thereby
broadening the scope of tasks that can be accomplished. Retrieval skills from
long sequence demonstration can be achieved by using supervised [31, 32] and
unsupervised method [33, 34, 35, 36, 37]. For the reuse of the primitive
skills, [38] proposes to train a universal agent with 12 unique skills that
are capable of multi-task manipulation by using semantic augmentation and
action representation. The research by [39] proposes a hybrid hierarchical
learning framework named ROMAN, which combines behavioural cloning (BC),
reinforcement learning (RL), and IL. A central manipulation network is used in
this work to produce the appropriate sequential actions for various sub-skills
networks, aiming to solve complex long-horizon manipulation tasks. The work
[40] proposes to chain multiple dexterous policies for achieving long-horizon
dexterous manipulation tasks by defining a feasibility function. Different
from previous works, we introduce primitive skills specifically designed for
dexterous manipulation tasks, aimed at achieving unseen long-horizon tasks by
predicting the sequence of primitive skills combination. Inspired by the work
[31], we introduce a specialized set of features designed for telerobot
dexterous manipulation, aiming to substantially enhance classifier performance
through the integration of auto-regressive autoencoder (AE). Moreover, we have
developed and trained a distinct skill policy for each primitive, employing
the predicted skill sequence to successfully execute long-horizon manipulation
tasks.

##  III METHODS

In this study, our objective is to explore methods for accomplishing imitation
learning for long-horizon dexterous manipulation tasks. The framework for
learning long-horizon dexterous manipulation tasks is illustrated in Fig 1. We
start by proposing primitive skills for tasks involving the use of a robotic
arm and dexterous hand. Our research focuses on learning these primitive
skills with haptic data, segmenting the skill sequence from unseen long-
horizon dexterous manipulation tasks, and executing the task by sequentially
performing the identified skill segments.

###  III-A Learning Features

For the robotic setup, which includes a robotic arm, a dexterous hand, and
tactile sensors, we propose a set of features designed to differentiate
primitive skills. These features are summarized as follows:

#### End Effector (EE) Information

For the features related to the EE, we utilize both position and velocity
information to accurately capture the movements. Additionally, to provide a
clearer representation of the movement of EE direction, we incorporate the EE
direction as a feature. This direction is defined with a value into the set
$[-1,0,1]$. Specifically, when the velocity exceeds $0.02m/s$, the EE
direction is designated as 1 to indicate forward movement. Conversely, if the
velocity is less than $-0.02m/s$, the direction is set to -1, indicating
backward movement. In scenarios where the velocity of the EE falls within the
threshold, the direction is set to 0, denoting a stationary state. These
features yield a detailed understanding of the movement of the EE throughout
the execution of tasks.

  * •

EE Pose: $\begin{bmatrix}x,y,z,\alpha,\beta,\gamma\end{bmatrix}$

  * •

EE Velocity:
$\begin{bmatrix}\dot{x},\dot{y},\dot{z},\dot{\alpha},\dot{\beta},\dot{\gamma}\end{bmatrix}$

  * •

EE Direction: $[-1,0,1]$

#### Allegro Hand (AH) Information

The features of the Allegro Hand (AH) include the AH joint state, fingertip
positions, velocities, as well as position and velocity covariance. To
effectively capture the movement correlations between each finger, the $Log$
of the upper triangle of the covariance matrix is utilized. This approach is
chosen for its efficiency in representing finger movements during manipulation
tasks.

  * •

AH Joint State: $[16\times 1]$

  * •

Fingerfitp Position: $[x,y,z]$

  * •

Fingertips Velocity: $[\dot{x},\dot{y},\dot{z}]$

  * •

Fingertip Position Covariance: [$triu(Log(C_{p}))$]

  * •

Fingertip Velocity Covariance: [$triu(Log(C_{v}))$]

#### Tactile Information

A fingertip with a custom-made magnetic sensor, presented in [15], is attached
to each finger of the AH. To quantify the intensity of force applied on each
fingertip, we selected the norm of the tactile force as the feature. This
allows us to accurately capture and analyze the dynamic interactions between
the robot fingertips and manipulated objects. Additionally, a feature
indicating the contact status is selected to reflect the interaction scenario
between the AH and the object. This feature essentially distinguishes whether
there is direct contact between the fingertip and the object. Incorporating
this feature also aids in reducing noise, thereby enhancing the reliability
and clarity of the sensory data used for task analysis and execution.

  * •

Tactile Norm $(Fx,Fy,Fz)\times 4$

  * •

Contact Status: $[(0,1)\times 4]$

Figure 2:  The leader agent generates motor control commands for the end
effector pose and finger joints of the hand. The follower robot executes
corresponding actions based on these commands. During teleoperation, the
follower robot provides haptic feedback. When operating the robot
autonomously, we control the robot using a distinct MLP trained on the
proprioceptive and tactile data (i.e. haptic data) of each separate skill.
Figure 3:  The architecture of our Neural Network for supervised
representation learning incorporates an auto-regressive autoencoder and a
label decoder. This network processes time-series feature data as input, with
the encoder transforming these features into a latent space. The temporal
decoder reconstructs the features along with their predictions, whereas the
label decoder extracts labels from the latent vectors. The label decoder is
jointly trained with the autoencoder generating latent features that improve
the segmentation performance.

###  III-B Primitive Skills Learning

Long-horizon dexterous manipulation tasks often demand extensive
demonstrations and face challenges related to accumulating errors and limited
generalization capabilities. When learning long-horizon dexterous manipulation
skills, humans adeptly analyze and sequence primitive skills. Motivated by
this human approach, our study introduces a methodology for separately
acquiring primitive skills and subsequently predicting the integration of
these skills for unseen tasks. This strategy is aimed at equipping robots with
the capability to perform long-horizon dexterous manipulation tasks. The
specific primitive skills we proposed for robotic manipulation using dexterous
hands are detailed in Table I.

The process of collecting primitive skills datasets and training the
autonomous skill policy is shown in Fig. 2. The single manipulation policy
operates with the assumption that the robot already knows the location of the
object. For each primitive skill, an MLP neural network is utilized to learn
the state-action pair from human demonstrations using the BC method. Given
state information $s$, the goal of the BC is to train a policy
$\pi(\theta_{p})$ that can learn the behaviour of the human demonstrator by
predicting the action $a$:

$\theta_{p}=\arg\max_{\theta_{p}}\underset{\mathbf{s},\mathbf{a}\sim\mathcal{D}}{\mathbb{E}}\left[L(\mathbf{a}\mid\mathbf{s})\right]$
(1)

where the observation $f$ is the input to the network, which includes the
robot EE state, AH joint state, and filtered tactile information as well as
the contact indicator. $L(\cdot)$ represents the loss function. In this work,
the mean squared error (MSE) is used to calculate the loss between the
predicted action and the recorded human action during the demonstration. The
network output $a=[\Delta EE,\Delta Fingertip]$ captures the movement of both
EE and AH fingertip, reflecting the actions executed by humans during the
teleoperation of the robot system. This approach enables the policy
$\pi(\theta_{p})$ to directly learn from human input, controlling the robot
through the same mapping to bypass the complexities of joint-space control and
enhance safety.

###  III-C Supervised Representation Learning

The proposed framework is shown in Fig. 3. This segmentation framework
consists of two parts. Firstly, a temporal auto-regressive AE is utilized to
encode temporal features into a latent space and then reconstruct the features
for the subsequent time step. Secondly, a label decoder extracts the task
label from the encoded latent representation. The AE receives an input
sequence represented by $[\mathbf{f}_{t-2},\mathbf{f}_{t-1},\mathbf{f}_{t}]$,
while the output generated by the encoder is
$[\mathbf{f}_{t-1},\mathbf{f}_{t},\mathbf{f}_{t+1}]$, where $f$ denotes the
features outlined in Section III-A. The interval between each timestep in this
process is set to 0.1 seconds. Furthermore, the representation of skill labels
employs a one-hot encoding scheme.

The choice of an auto-regressive AE is strategic for encapsulating the
temporal dynamics of robot behaviour, crucial for capturing the intricate
nature of robot actions where the sequence of actions is significant. This
setup is designed to capture the nuances of human behaviour through temporal
data analysis. Additionally, the AE facilitates a deeper comprehension of the
input data, which in turn, augments the accuracy of task label predictions by
understanding the underlying patterns and characteristics of human behavior
encoded in the temporal information. During the training, we train the AE
together with the label decoder. The loss function is defined as:

$\text{Loss}=\text{MSE}(f_{\text{true}},f_{\text{pred}})+\text{CrossEntropy}(l_{\text{true}},l_{\text{pred}})$
(2)

where the loss $\text{MSE}(f_{\text{true}},f_{\text{pred}})$ calculates the
MSE between the true temporal features $f_{\text{true}}$ and the predicted
temporal features $f_{\text{pred}}$. The loss function
$\text{CrossEntropy}(l_{\text{true}},l_{\text{pred}})$ is specifically
selected to calculate the discrepancy between the actual probabilities
$l_{\text{true}}$ and the predicted probabilities $l_{\text{pred}}$ for each
class. The combination of these two loss functions encourages the model to
learn rich and informative information that helps in both accurately
predicting temporal features and correctly identifying skill labels.

###  III-D Unknown long-horizon manipulation skills learning

Our objective is to facilitate one-shot imitation learning for long-horizon
dexterous manipulation tasks that were previously unseen, utilizing a set of
predefined primitive skills. Segmenting these tasks into simpler skills makes
it more manageable for the algorithm to understand and master the task,
reducing the overall complexity. For each demonstration, robot data are
gathered, and features are extracted from this collected data. The trained
feature encoder $\mathcal{E}$ and the label decoder $\mathcal{L}$ are used to
predict the sequence of primitive skill for executing dexterous long-horizon
manipulation tasks:

$\mathbf{P_{t}}=\mathcal{L}(\mathcal{E}(\mathbf{f}_{t-2},\mathbf{f}_{t-1},\mathbf{f}_{t};\theta_{\mathcal{E}});\theta_{\mathcal{L}}),$
(3)

where $\mathbf{P_{t}}$ is the predicted probability distribution of primitive
skill at timestep $t$, $\theta_{\mathcal{E}}$ and $\theta_{\mathcal{L}}$ are
the parameters of the feature encoder and label decoder respectively. To
enhance the accuracy of these predictions and reduce errors, a median filter
is applied for smoothing. Upon establishing a sequence for the primitive
skills required for these dexterous long-horizon tasks, the trained primitive
skills are executed in sequence to complete the tasks.

##  IV Experimental Setting

###  IV-A Teleoperation Setup

Depicted in Fig. 1, the teleoperation setup used is similar to the one
proposed by the authors in a previous work [15]. In summary, the setup
consists of two leader devices: Virtuose 6D and HGlove, used to respectively
control the robotic arm UR5 and Allegro dexterous Hand mounted with custom-
made magnetic sensors on the follower side. The mapping and control structure
is the same as in the previous study, as referenced and summarized in Fig. 2.
In this study, the wrist rotation of the robotic arm is enabled in order to
achieve a wider range of skills.

For the data collection phase, all the information used to control the robot
and to extract the training features proposed in Sec.III-A was recorded and
made available in a dataset on the project’s repository††footnotemark: .

###  IV-B The Primitive Skills

In this work, amongst the various possible robotics manipulation skills, 20
primitive skills have been identified and are listed with a unique colour
associated in Table I. A combination of more than three skills defines a Long-
Horizon task. Although all skills are shown independently, some of them
require a pre-action in the real world to be performed. The task of planning
long-horizon tasks is left to the demonstrator. The objective of this work is
to provide a paradigm for segmentation and learning of skills where the
feasibility of a task is defined by the demonstration sequences. For this
reason, the assumption is made that each task is demonstrable and repeatable
in a real-world scenario.

TABLE I: List of primitive skills Group | Skills
---|---
Pre | Skill
No Touch |  white- |  blue!251. Reach
white- |  yellow!252. Setup Position
Touch |  red!253. PreTouch |  orange!504. Touch
white- |  yellow!755. Flip
red!25PreTouch |  cyan!256. Wipe Forth
red!25PreTouch |  magenta!507. Wipe Back
violet!258. PreGrasp |  purple!509. Grasp
white- |  teal!5010. Lift with Grasp
white- |  brown!5011. Transport Forward
white- |  lime!7512. Place
olive!5013. PreRotate |  orange!7514. Rotate
white- |  cyan!7515. Shake Up
white- |  magenta!7516. Shake Down
white- |  green!7517. Twist
white- |  lightgray!5018. Vertical Place
white- |  darkgray!5019. Pour
white- |  teal!7520. Release
TABLE II: Long-Horizon tasks: primitive skill recombinations. Tasks are denoted by alphabetical letters ranging from A to T. Objects utilized for the demonstrations are indicated within parentheses: (s) sponge, (t) tomato passata package, and (b) bottle containing liquid.  Task | Skills
---|---
|  whiteI |  whiteII |  whiteIII |  whiteIV |  whiteV |  whiteVI |  whiteVII |  whiteVIII |  whiteIX |  whiteX
A (s) |  whiteblue!251 |  whiteyellow!755 |  whitered!253 |  whiteorange!504 |  whitemagenta!507 |  whitecyan!256 |  whiteviolet!258 |  whitepurple!509 |  whiteteal!5010 |  whiteteal!7520
B (t) |  whiteorange!504 |  whitemagenta!507 |  whiteviolet!258 |  whitepurple!509 |  whiteteal!5010 |  whitebrown!5011 |  whitelime!7512 |  whiteyellow!252 |  white |  white
C (b) |  whiteolive!5013 |  whiteorange!7514 |  whiteteal!5010 |  whitecyan!7515 |  whitemagenta!7516 |  whitegreen!7517 |  whitelightgray!5018 |  white |  white |  white
D (s) |  whitecyan!256 |  whitemagenta!507 |  whitecyan!256 |  whitemagenta!507 |  whitecyan!256 |  whitemagenta!507 |  white |  white |  white |  white
E (b) |  whiteyellow!755 |  whiteviolet!258 |  whitepurple!509 |  whiteteal!5010 |  whitecyan!7515 |  whitedarkgray!5019 |  white |  white |  white |  white
F (b) |  whiteviolet!258 |  whitepurple!509 |  whiteteal!5010 |  whitegreen!7517 |  white |  white |  white |  white |  white |  white
G (b) |  whiteblue!251 |  whiteyellow!755 |  whiteviolet!258 |  whitepurple!509 |  white |  white |  white |  white |  white |  white
H (t) |  whitecyan!7515 |  whitemagenta!7516 |  whitecyan!7515 |  whitelime!7512 |  white |  white |  white |  white |  white |  white
I (s) |  whitemagenta!7516 |  whitecyan!7515 |  whitemagenta!7516 |  whiteteal!7520 |  white |  white |  white |  white |  white |  white
J (b) |  whitepurple!509 |  whiteteal!5010 |  whitegreen!7517 |  whiteteal!7520 |  white |  white |  white |  white |  white |  white
K (t) |  whiteorange!504 |  whiteviolet!258 |  whitepurple!509 |  white |  white |  white |  white |  white |  white |  white
L (s) |  whiteolive!5013 |  whiteorange!7514 |  whitegreen!7517 |  white |  white |  white |  white |  white |  white |  white
M (s) |  whitepurple!509 |  whiteteal!7520 |  whiteyellow!252 |  white |  white |  white |  white |  white |  white |  white
N (s) |  whitegreen!7517 |  whiteteal!5010 |  whitemagenta!7516 |  white |  white |  white |  white |  white |  white |  white
O (b) |  whiteteal!5010 |  whitegreen!7517 |  whitedarkgray!5019 |  white |  white |  white |  white |  white |  white |  white
P (t) |  whitedarkgray!5019 |  whitegreen!7517 |  whitelightgray!5018 |  white |  white |  white |  white |  white |  white |  white
Q (s) |  whiteyellow!755 |  whiteviolet!258 |  whiteyellow!252 |  white |  white |  white |  white |  white |  white |  white
R (b) |  whiteblue!251 |  whiteolive!5013 |  whiteyellow!252 |  white |  white |  white |  white |  white |  white |  white
S (s) |  whitelightgray!5018 |  whiteteal!5010 |  whiteteal!7520 |  white |  white |  white |  white |  white |  white |  white
T (b) |  whiteteal!5010 |  whitegreen!7517 |  whitelightgray!5018 |  white |  white |  white |  white |  white |  white |  white

###  IV-C Long-Horizon Tasks

As the number of skills increases, the combinations grow exponentially.
Although it is possible to have a sequence with many different skills that can
turn into a challenging demonstration, even a basic task repeated multiple
times may involve just three skills, . For instance, a sequence comprising
”Wipe Forth” and ”Wipe Back” combined with other skills like ”Pick” and
”Place” or ”Twist” may resemble a dish-washing operation. Similarly, an
experiment involving ”Shake Down,” ”Shake Up,” and ”Pour” might evoke the
actions of mixing and serving a cocktail.

For this study, we have identified 20 sparse long-horizon tasks of varying
complexity, as outlined in Table II, which define the experimental set for the
classifier. In this table, each long-horizon task is represented by a numbered
skill sequence and with the same colours according to Table I. To illustrate
the independence of skills from each other in skill recombination, it has been
occasionally chosen to initiate from pre-established verified conditions, as
observed in tasks B, D, J, K, and M. We will evaluate the performance of our
framework on these combinations in Section VI-A.

In the experiments section, we will replicate autonomously long-horizon tasks
defined by sequences A and B. As detailed in the section VI-B, this will be
achieved from learning from the single skills demonstrations used for
classifier training. Thus, no long-horizon sequences will be used in any of
the training phases.

##  V Data Collection

This section delineates the data collection methodology used to evaluate the
proposed approach, while Section VI offers a discussion of the achieved
results. For all the experiments, it is assumed that the position of the
object is known.

The data collection is organized in three phases:

  1. A.

The training data for the 20 primitive skills were gathered through human
teleoperation of the robot, with the majority of skills documented across 10
demonstrations involving a single object. Specifically, for the skills
”Touch,” ”Wipe Back,” and ”Grasp,” data were collected using both a soft
sponge (16x8.5x4.5cm, 20g) and cardboard packages (14x7.5x4 cm, 15g), with 20
demonstrations performed for each skill. Additionally, for the skill ”Lift
with Grasp,” an extra 10 demonstrations were conducted using a bottle
(22x7x7cm, 220g). This collected data was then used to train the model for the
autonomous primitive skill classifier, as shown in Figure 3.

  2. B.

The first phase of experiments is made to evaluate the performance of the
classifier. The demonstrations of the 20 long-horizon tasks presented in Table
II were collected. Each demonstration of a long-horizon task was presented and
manually segmented with labels to make a comparison with the predictions
obtained by the classifier. This process is depicted in Fig. 1.

  3. C.

The second set of experiments consists of replacing the Leader agent with
distinct MLP controllers shown in Fig. 2, each tailored to a specific skill,
trained from the same demonstrations used for the classifier. As in the
methods section III-D, on receiving a skill label, the robot reproduces the
single learned skill and switches to the next one on receipt of a new skill
label.

Figure 4: Confusion matrix ($\%$) of the segmentation system on the Long-
horizon demonstrations (detailed in Section IV-C).

##  VI Experimental Results

In this section, we present the evaluation of our framework through a series
of experiments. We explore the impact of integrating an auto-regressive AE on
the accuracy of skill label prediction. Additionally, we investigate the
benefits of jointly training the auto-regressive AE with the label decoder for
overall performance improvement. Our experiments also validate the efficiency
of the proposed haptic features in accurately differentiating between
primitive skills. Finally, we demonstrate the ability of our framework to
efficiently select and integrate skill primitives for the execution of
complex, long-horizon manipulation tasks.

###  VI-A Classifier Evaluation

We train the classifier using data collected from primitive skills and
validate its performance on unseen long-horizon manipulation tasks. To
validate the effectiveness of our framework, we choose parameters including
Accuracy, Precision, Recall, F1-score, and Average IoU as our key performance
indicators.

Figure 5:  T-SNE visualization of the classifier latent features. Each point
in the graph corresponds to a primitive skill instance, differentiated by
various colours to distinguish among the primitive skills.

####  VI-A1 Framework Validation

We validate the performance of our proposed framework for long-horizon
dexterous manipulation task segmentation by comparing it against three
baselines. First, we examine a setup that utilizes a pre-trained AE for
feature encoding, which is then followed by training a label decoder on these
encoded features. Then, our second baseline explores the combination of a
feature recovery AE with a label decoder, thereby excluding the temporal
prediction from the AE. Finally, we verified the effectiveness of training the
feature encoder and the label decoder together, excluding the temporal feature
decoder. The outcomes are presented in Table III, where ”LD” denote the label
encoder. Our framework achieved an accuracy of 91%, outperforming all baseline
comparisons. This indicates that jointly training the temporal AE with the
label decoder is beneficial for capturing the latent dynamics of robot
behaviour while preserving essential information critical for differentiating
between primitive skills. This capability significantly contributes to
improving the performance of the classifier by ensuring a deeper comprehension
of the temporal features and movements associated with various primitive
skills.

TABLE III: Comparative Performance Evaluation | DexSkills | Pre-trained AE+ LD | Feature recovery AE + LD | Feature Encoder + LD | Raw haptic data | Calculated feature
---|---|---|---|---|---|---
Accuracy | 0.91 | 0.83 | 0.73 | 0.83 | 0.35 | 0.76
Precision | 0.89 | 0.81 | 0.64 | 0.80 | 0.40 | 0.77
Recall | 0.85 | 0.76 | 0.66 | 0.76 | 0.37 | 0.69
F1-Score | 0.87 | 0.76 | 0.63 | 0.75 | 0.34 | 0.69
Average IoU | 0.79 | 0.64 | 0.56 | 0.63 | 0.23 | 0.55

####  VI-A2 Feature importance verification

To verify the importance of the proposed feature for skill classification, we
evaluated the performance of our proposed framework in two distinct scenarios:
firstly, the framework was trained on raw haptic data, including the end-
effector state, filtered tactile information, filtered contact indicators, and
the AH joint state, which are the data used in training the policy for
primitive skills; secondly, by training only with the proposed features while
excluding the raw haptic data. The results are shown in Table III. When
trained with raw haptic information, the classifier achieves an accuracy of
only 35%. In contrast, utilizing the additional features proposed for skill
segmentation improves the accuracy to approximately 76%. This substantial
improvement indicates the effectiveness of our proposed features for skill
segmentation, highlighting their value in enhancing the performance of the
classifier.

####  VI-A3 Confusion matrix for each skill

The confusion matrix provides a comprehensive overview of how well a
classification model is performing across all categories. The result of the
confusion matrix on 20 unseen long-horizon tasks is shown in Fig. 4. The
prediction outcomes for the majority of the skills are satisfactory. However,
the skills of flip and release tend to be confused with other skills,
primarily due to their short duration, which makes their features challenging
to capture accurately. Additionally, it shall be noted that manual labelling
of the ground truth in long-horizon tasks can introduce certain errors
inevitably, which can become especially obvious during the transition phases
between skills.

(a) Long-horizon Task A using a soft sponge (16x8.5x4.5cm, 20g).

(b) Long-horizon Task B using a cardboard package (14x7.5x4 cm, 15g).

Figure 6: Autonomous skills reproduction. Full videos are available on the
project repository

###  VI-B Autonomous Long-horizon Tasks Execution

In this work, we verified our framework on two challenge tasks, which are
identified as tasks A and B in Table II. Each task required the coordination
of over 8 distinct skills, with skill transitions manually controlled by a
human operator. Both tasks were completed by using the predicted primitive
skills combination sequence. This demonstrates the ability of our proposed
framework to reuse primitive skills for achieving unseen long-horizon tasks
effectively. The snapshots of these experiments are presented in Figures 6(a)
and 6(b). Additionally, videos detailing the experimental process are
available on the project GitHub page††footnotemark: .

###  VI-C T-SNE Analysis

We employed t-Distributed Stochastic Neighbor Embedding (t-SNE) analysis on
the latent features extracted before the final layer of the label decoder.
This technique projects high-dimensional vectors onto a two-dimensional plane,
achieving the visualization of high-dimensional data relationships. The
application of t-SNE to our unseen long-horizon task data is depicted in Fig.
5, where a distinct colour represents each skill.

Overall, the t-SNE plot demonstrates that most skills are well-clustered and
exhibit clear distinctions from one another. However, for skills that exhibit
similar behaviours or tend to occur sequentially such as ”PreGrasp”, ”Grasp”,
and ”Lift with Grasp”, there is a noticeable overlap in their representation.
This overlapping tendency is also observed among skills with similar motions,
like ”Wipe Back” and ”Wipe Forth”.

##  VII Conclusion

In this study, we have presented DexSkills a learning framework that addresses
the challenge of executing real-world long-horizon tasks with dexterous
robotic hands by decomposing them into reusable primitive skills, trained from
human demonstrations. This framework, adaptable for various demonstrations
such as bilateral or kinesthetic teleoperation, enables the segmentation of
long-horizon demonstrations into individual tasks, using only proprioceptive
and tactile data, facilitating their integration into the repertoire of known
skills and enabling autonomous robot execution of diverse tasks with high
classification accuracy. In this study, we evaluate the segmentation
performance of unseen multiple long-horizon tasks starting from a limited set
of primitive skills. Our framework demonstrated strong performance in skill
segmentation, achieving an accuracy rate of $91\%$. Comparative analysis
reveals that our framework surpasses alternative approaches by effectively
capturing the latent dynamics of primitive skills within the feature set.
Furthermore, through the same demonstrations employed for the Classifier, a
protocol is presented for learning the autonomous control over each individual
task. Although the accuracy was not studied, it is established how each of the
demonstrated tasks can be replicated autonomously by the robot by providing a
label sequence. Future work should include the development of more complex
models for autonomous reproduction and should accurately assess the
reproducibility of autonomous execution of long-horizon tasks.

## References

  * [1] K. Yao and A. Billard, “Exploiting kinematic redundancy for robotic grasping of multiple objects,” _IEEE Transactions on Robotics_ , 2023.
  * [2] X. Gao, K. Yao, F. Khadivar, and A. Billard, “Real-time motion planning for in-hand manipulation with a multi-fingered hand,” _arXiv preprint arXiv:2309.06955_ , 2023.
  * [3] F. Khadivar and A. Billard, “Adaptive fingers coordination for robust grasp and in-hand manipulation under disturbances and unknown dynamics,” _IEEE Transactions on Robotics_ , 2023.
  * [4] A. Petrenko, A. Allshire, G. State, A. Handa, and V. Makoviychuk, “Dexpbt: Scaling up dexterous manipulation for hand-arm systems with population based training,” _arXiv preprint arXiv:2305.12127_ , 2023.
  * [5] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam _et al._ , “Dextreme: Transfer of agile in-hand manipulation from simulation to reality,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 5977–5984.
  * [6] S. Zhaole, J. Zhu, and R. B. Fisher, “Dexdlo: Learning goal-conditioned dexterous policy for dynamic manipulation of deformable linear objects,” _arXiv preprint arXiv:2312.15204_ , 2023.
  * [7] C. Yu and P. Wang, “Dexterous manipulation for multi-fingered robotic hands with reinforcement learning: a review,” _Frontiers in Neurorobotics_ , vol. 16, p. 861825, 2022.
  * [8] S. Luo, J. Bimbo, R. Dahiya, and H. Liu, “Robotic tactile perception of object properties: A review,” _Mechatronics_ , vol. 48, pp. 54–67, 2017.
  * [9] Q. Li, O. Kroemer, Z. Su, F. F. Veiga, M. Kaboli, and H. J. Ritter, “A review of tactile information: Perception and action through touch,” _IEEE Transactions on Robotics_ , vol. 36, no. 6, pp. 1619–1634, 2020.
  * [10] J. Jiang, G. Cao, J. Deng, T.-T. Do, and S. Luo, “Robotic perception of transparent objects: A review,” _IEEE Transactions on Artificial Intelligence_ , 2023.
  * [11] Z.-H. Yin, B. Huang, Y. Qin, Q. Chen, and X. Wang, “Rotating without seeing: Towards in-hand dexterity through touch,” _arXiv preprint arXiv:2303.10880_ , 2023.
  * [12] K.-W. Lee, Y. Qin, X. Wang, and S.-C. Lim, “Dextouch: Learning to seek and manipulate objects with tactile dexterity,” _arXiv preprint arXiv:2401.12496_ , 2024.
  * [13] S. Pai, T. Chen, M. Tippur, E. Adelson, A. Gupta, and P. Agrawal, “Tactofind: A tactile only system for object retrieval,” _arXiv preprint arXiv:2303.13482_ , 2023.
  * [14] W. Hu, B. Huang, W. W. Lee, S. Yang, Y. Zheng, and Z. Li, “Dexterous in-hand manipulation of slender cylindrical objects through deep reinforcement learning with tactile sensing,” _arXiv preprint arXiv:2304.05141_ , 2023.
  * [15] G. Giudici, B. Omarali, A. A. Bonzini, K. Althoefer, I. Farkhatdinov, and L. Jamone, “Feeling good: Validation of bilateral tactile telemanipulation for a dexterous robot,” in _Annual Conference Towards Autonomous Robotic Systems_. Springer, 2023, pp. 443–454.
  * [16] Í. Elguea-Aguinaco, A. Serrano-Muñoz, D. Chrysostomou, I. Inziarte-Hidalgo, S. Bøgh, and N. Arana-Arexolaleiba, “A review on reinforcement learning for contact-rich robotic manipulation tasks,” _Robotics and Computer-Integrated Manufacturing_ , vol. 81, p. 102517, 2023.
  * [17] A. Correia and L. A. Alexandre, “A survey of demonstration learning,” _arXiv preprint arXiv:2303.11191_ , 2023.
  * [18] J. Luo, C. Xu, X. Geng, G. Feng, K. Fang, L. Tan, S. Schaal, and S. Levine, “Multi-stage cable routing through hierarchical imitation learning,” _IEEE Transactions on Robotics_ , 2024.
  * [19] L. X. Shi, A. Sharma, T. Z. Zhao, and C. Finn, “Waypoint-based imitation learning for robotic manipulation,” _arXiv preprint arXiv:2307.14326_ , 2023.
  * [20] A. Billard and D. Kragic, “Trends and challenges in robot manipulation,” _Science_ , vol. 364, no. 6446, p. eaat8414, 2019.
  * [21] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard, “Recent advances in robot learning from demonstration,” _Annual review of control, robotics, and autonomous systems_ , vol. 3, pp. 297–330, 2020.
  * [22] V. Kumar, A. Gupta, E. Todorov, and S. Levine, “Learning dexterous manipulation policies from experience and imitation,” _arXiv preprint arXiv:1611.05095_ , 2016.
  * [23] I. Radosavovic, X. Wang, L. Pinto, and J. Malik, “State-only imitation learning for dexterous manipulation,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 7865–7871.
  * [24] S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto, “Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation,” in _2023 ieee international conference on robotics and automation (icra)_. IEEE, 2023, pp. 5954–5961.
  * [25] E. Valassakis, G. Papagiannis, N. Di Palo, and E. Johns, “Demonstrate once, imitate immediately (dome): Learning visual servoing for one-shot imitation learning,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 8614–8621.
  * [26] D. Zhang, Q. Li, Y. Zheng, L. Wei, D. Zhang, and Z. Zhang, “Explainable hierarchical imitation learning for robotic drink pouring,” _IEEE Transactions on Automation Science and Engineering_ , vol. 19, no. 4, pp. 3871–3887, 2021.
  * [27] F. Xie, A. Chowdhury, M. De Paolis Kaluza, L. Zhao, L. Wong, and R. Yu, “Deep imitation learning for bimanual robotic manipulation,” _Advances in neural information processing systems_ , vol. 33, pp. 2327–2337, 2020.
  * [28] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” _arXiv preprint arXiv:2304.13705_ , 2023.
  * [29] I. Guzey, B. Evans, S. Chintala, and L. Pinto, “Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play,” _arXiv preprint arXiv:2303.12076_ , 2023.
  * [30] T. P. Tomo, M. Regoli, A. Schmitz, L. Natale, H. Kristanto, S. Somlor, L. Jamone, G. Metta, and S. Sugano, “A new silicone structure for uskin—a soft, distributed, digital 3-axis skin sensor and its integration on the humanoid robot icub,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 2584–2591, 2018.
  * [31] C. Coppola and L. Jamone, “Master of puppets: Multi-modal robot activity segmentation from teleoperated demonstrations,” in _2022 IEEE International Conference on Development and Learning (ICDL)_. IEEE, 2022, pp. 88–94.
  * [32] S. Pirk, K. Hausman, A. Toshev, and M. Khansari, “Modeling long-horizon tasks as sequential interaction landscapes,” _arXiv preprint arXiv:2006.04843_ , 2020.
  * [33] S. Krishnan, A. Garg, S. Patil, C. Lea, G. Hager, P. Abbeel, and K. Goldberg, “Transition state clustering: Unsupervised surgical trajectory segmentation for robot learning,” _The International journal of robotics research_ , vol. 36, no. 13-14, pp. 1595–1618, 2017.
  * [34] D. Meli and P. Fiorini, “Unsupervised identification of surgical robotic actions from small non-homogeneous datasets,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 8205–8212, 2021.
  * [35] Y. Zhu, P. Stone, and Y. Zhu, “Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4126–4133, 2022.
  * [36] W. Wan, Y. Zhu, R. Shah, and Y. Zhu, “Lotus: Continual imitation learning for robot manipulation through unsupervised skill discovery,” _arXiv preprint arXiv:2311.02058_ , 2023.
  * [37] S. Park, K. Lee, Y. Lee, and P. Abbeel, “Controllability-aware unsupervised skill discovery,” _arXiv preprint arXiv:2302.05103_ , 2023.
  * [38] H. Bharadhwaj, J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar, “Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking,” _arXiv preprint arXiv:2309.01918_ , 2023.
  * [39] E. Triantafyllidis, F. Acero, Z. Liu, and Z. Li, “Hybrid hierarchical learning for solving complex sequential tasks using the robotic manipulation network roman,” _Nature Machine Intelligence_ , vol. 5, no. 9, pp. 991–1005, 2023.
  * [40] Y. Chen, C. Wang, L. Fei-Fei, and C. K. Liu, “Sequential dexterity: Chaining dexterous policies for long-horizon manipulation,” _arXiv preprint arXiv:2309.00987_ , 2023.