# Interpretable Robotic Manipulation from Language

Boyuan Zheng  Jianlong Zhou  Fang Chen  University of Technology Sydney

###### Abstract

Humans naturally employ linguistic instructions to convey knowledge, a process
that proves significantly more complex for machines, especially within the
context of multitask robotic manipulation environments. Natural language,
moreover, serves as the primary medium through which humans acquire new
knowledge, presenting a potentially intuitive bridge for translating concepts
understandable by humans into formats that can be learned by machines. In
pursuit of facilitating this integration, we introduce an explainable behavior
cloning agent, named Ex-PERACT, specifically designed for manipulation tasks.
This agent is distinguished by its hierarchical structure, which incorporates
natural language to enhance the learning process. At the top level, the model
is tasked with learning a discrete skill code, while at the bottom level, the
policy network translates the problem into a voxelized grid and maps the
discretized actions to voxel grids. We evaluate our method across eight
challenging manipulation tasks utilizing the RLBench benchmark, demonstrating
that Ex-PERACT not only achieves competitive policy performance but also
effectively bridges the gap between human instructions and machine execution
in complex environments. Code, data, and supplementary materials are publicly
available at https://anonymous.4open.science/r/ExPERACT.

##  1 Introduction

The exploration of sequential decision-making within the realm of artificial
intelligence (AI) is a pivotal research focus, bearing substantial
implications for applications such as autonomous driving, gaming strategy, and
humanoid robotics [23, 22, 36]. Central to this pursuit is imitation learning
(IL), which entails learning from demonstrations and emerges as a promising
primary approach to effectively address the challenges inherent in sequential
decision-making. To enhance the capabilities of intelligent agents, research
communities have integrated various forms of auxiliary information, including
human preferences, task-specific parameters, and the expertise level of
experts [3, 19, 1]. Notably, advancements in large language models have
established natural language as a preferred method among researchers, owing to
its intuitiveness and provision of a more universally applicable means for
instructing agents. Diverse methodologies proposed in studies such as [30, 25,
39] are aimed at training single-task agents, utilizing the conventional
state-action pair as input and supplementing it with natural language as
auxiliary information.

As machine learning techniques progress, there has been a gradual enhancement
in the performance of intelligent agents across single tasks, instilling
anticipation among researchers for adept multitask domain performance. These
intelligent agents are designed to acquire knowledge from offline
demonstrations spanning various tasks and appropriately interact with the
environment during testing time. The integration of natural language holds
promise in strengthening the interconnections between tasks, thereby fostering
an enhanced ability to reuse acquired skills across similar tasks. Studies
such as PERACT [29] and BC-Z [14] exemplify how the incorporation of natural
language into multitask models significantly influences task success rates,
underscoring its potential impact in advancing the field.

However, training a language-conditioned multitask model poses significant
challenges. Firstly, language instructions often fail to fully elucidate tasks
due to inherent limitations in common-sense reasoning. For instance,
instructions like “Open drawer" may overlook crucial steps such as targeting
and gripping the drawer handle, thereby hindering the summarization and
reusability of implicit sub-tasks for comparable activities. Moreover, the
inherently abstract nature of language provides limited information about
subtask divisions and lacks ground truth for potential skills, rendering the
learning process unsupervised. Furthermore, task instructions may vary among
individuals, introducing complexities into the training of multitask models
due to the diversity in expression. Additionally, while natural language
provides a means for humans to probe the model and enhance interpretability,
this aspect has garnered limited attention in multitask model research.

To address the aforementioned challenges and delve deeper into explainability,
we propose a novel method named Ex-PerAct. This approach employs a
hierarchical structure and integrates multitask imitation learning to acquire
reusable skills from language-conditioned offline demonstrations. Ex-PerAct
comprises two transformer-based models in sequence. The top-level model
condenses skills from diverse tasks into discrete skill codes, providing a
comprehensive summary of segmented demonstration snippets. This model acts as
a vital bridge connecting human-understandable natural language to digital
skill vectors, thereby facilitating the reuse of condensed skills in analogous
tasks. To cluster these skill vectors in an unsupervised manner, we employ
Vector Quantization (VQ) [31]. As for the bottom-level model, we leverage the
state-of-the-art PerAct [29], a multitask Behavior Cloning agent renowned for
its competitive performance across a wide array of manipulation tasks. PerAct
represents observations and actions as 3D voxels rather than 2D image pixels,
contributing to its superior capabilities. We conduct experiments on eight
manipulation tasks in the challenging benchmark RLBench [12] with multiple
baselines and ablations. The results demonstrate that Ex-PERACT achieves
better policy performance in almost all tasks while providing an interpretable
way for humans to probe the relationship between language instructions and the
agent’s decision-making process.

We highlight the primary contributions of this paper as follows:

  * •

We present Ex-PerAct, a hierarchical imitation learning method that
effectively integrates diverse modalities, including 3D voxels and language
instructions, thereby achieving competitive performance across tasks.

  * •

Our approach showcases the capacity of Ex-PerAct to extract reusable skills
across tasks, offering significant advantages in the realm of multitask
robotic manipulation.

  * •

Ex-PerAct forges a crucial link between human-understandable natural language
and machine-usable vectors, augmenting interpretability across behavioral
patterns, and language instructions.

##  2 Related Work

###  2.1 Language-conditioned imitation learning in robotic manipulation

The integration of natural language into the training of intelligent agents
has emerged as a prominent trend, with roots extending beyond recent years.
Chen and Mooney introduced a framework facilitating the learning of a semantic
parser, aligning instructions with the world state in navigation tasks [4].
The rise of transformer-based models has ushered in a proliferation of
language-conditioned datasets in contemporary research. Pioneering
investigations such as [14, 16, 20, 29, 8] seamlessly incorporate this
modality with imitation learning, tackling more intricate continuous
manipulation tasks compared to earlier studies.

Noteworthy among recent advancements is the work by Nair et al., who annotated
an offline dataset with crowd-sourced natural language labels. They introduced
LOReL, a method that recovers a language-conditioned reward function
represented by a simple classifier, specifically tailored for integration into
multi-task reinforcement learning [21]. This annotated dataset has since
played a pivotal role in numerous subsequent investigations focusing on
language-conditioned robotic manipulation, exemplified by [7]. Concurrently,
Shridhar et al. introduced CLIPORT, a method leveraging natural language to
achieve commendable performance levels in both simulated and real-world
domains [28].

Despite the breadth of literature in this domain, research into the potential
enhancement of explainability through the incorporation of natural language
remains relatively limited. LISA, as proposed by Garg et al., employs a
hierarchical imitation learning framework that learns interpretable skill
abstractions from a language-conditioned dataset using state-of-the-art
transformer-based models, elucidating the correlation between natural language
and acquired skills [7]. However, LISA introduces additional parameters such
as horizon, and the model’s performance is heavily reliant on these
parameters. Moreover, its evaluation in multitasks related to manipulation is
notably constrained, and its performance in established benchmarks like
RLBench is also limited.

###  2.2 Hierarchical imitation learning

The hierarchical structure has emerged as a beneficial paradigm in numerous
studies on imitation learning, offering various advantages derived from its
ability to harness benefits from different models. One notable advantage is
exemplified by Le et al., who introduced a hierarchical framework encompassing
both imitation learning (IL) and reinforcement learning (RL). Their study
showcased that employing different combinations of IL and RL at distinct
levels could notably mitigate expert effort and exploration costs, rendering
their proposed framework more label-efficient compared to standard IL [17].

Furthermore, hierarchical frameworks excel in facilitating the learning of
separate sub-task policies for complex tasks, leading to performance
enhancements. Hierarchical option frameworks represent a prevalent approach
for learning sub-task policies, categorized into those trained with ground
truth option labels [37, 18] and those utilizing alternative forms of
supervision [27, 32, 7]. For instance, FIST, proposed by Hakhamaneshi et al.,
falls under the former category, employing an inverse dynamic model to extract
generalizable skills offline and achieving competitive performance in long-
horizon tasks under few-shot settings [9]. Conversely, in the latter category,
Zhang and Paschalidis explored the Expectation-Maximization approach to
hierarchical IL from a theoretical standpoint, devising the first convergence
guarantee algorithm that solely observes primitive state-action pairs [38].

In addition to performance improvements, hierarchical IL also enhances
explainability, as will be further elucidated in the subsequent subsection.
Broadly, the high-level options serve as a bridge between human-understandable
knowledge and the model’s performance, acting as a bottleneck to facilitate
clearer comprehension. Our work similarly follows this principle but
emphasizes explainability regarding language over the input demonstrations.

###  2.3 Explainable imitation learning

The exploration of the explainability of IL methods has gained traction in
recent years, with a notable shift towards enhancing interpretability for non-
technical audiences. Early research, proposed in [19], aimed to learn
interpretable and meaningful representations to infer the latent structure of
input demonstrations, albeit without delving deeply into improving
explainability for non-technical users. Subsequently, in 2020, Pan et al.
introduced xGAIL, claiming it as the first explainable GAIL framework. As
explainable AI gains prevalence, an increasing number of studies are emerging
that combine eXplainable AI (XAI) with IL methodologies.

Leech proposed an intrinsically interpretable IL method that leverages a
hierarchical learning framework and integrates IL with logical automata. This
approach represents problems as compact finite state automata with human-
interpretable logic states [18]. Similarly, Bewley et al. (2020) introduced an
explainable IL method employing interpretable models, wherein they model the
behavior policy of a trained black-box agent using a decision tree generated
from analyzing its input-output statistics [2].

In addition to intrinsically interpretable approaches, post-hoc explanation
methodologies are prevalent in the research community [34, 7, 15]. Zhang et
al. (2021) utilized a hierarchical framework to achieve post-hoc explanation,
decomposing complex tasks and elucidating the model’s decision-making process
and causes of failure [37]. Wang et al. proposed subgoal-conditioned
hierarchical IL to emulate doctors’ behavior, providing more explainable post-
hoc recommendations in the dynamic treatment recommendation domain [32].
However, these methods seldom investigate natural language as the primary
communication medium between individuals. With the prevalence of transformer-
based models and their capacity to address sequential data such as language,
natural language is gradually being incorporated into the training process.
This integration should serve as a bottleneck to enhance explainability, as it
enables a more intuitive interaction between humans and machines.

##  3 Approach

Our approach, Ex-PERACT, introduces an explainable hierarchical agent for
6-DOF manipulation tasks. It adopts a behavior cloning approach, training on a
language-conditioned offline dataset. The core concept involves extracting
discrete skill codes to bridge natural language instructions with observation
at a higher level, while simultaneously learning a policy network from the
extracted skill code, language embeddings, and observation-action pair at a
lower level. The skill codes is learned unsupervisedly and enhance
explainability with respect to the natural language and multitask learning.
Additionally, the design incorporates 3D voxel reconstruction inherited from
PERACT, which significantly improves learning efficiency compared to
conventional 2D image-action mapping.

We begin by presenting the preliminary knowledge and our problem formulation
in Section 3.1. Subsequently, Section 3.2 delves into the framework of Ex-
PERACT, providing a comprehensive introduction to the models. Finally, Section
3.3 elucidates further implementation details.

###  3.1 Problem setting

We conceptualize our problem as multitask, where each task $T_{i}$ can be
regarded as an individual Markov Decision Process (MDP), characterized by the
property that the next state $s_{t+1}$ is solely determined by the current
state $s_{t}$ at any time $t$. We assume access to an offline dataset $D$
generated by an optimal policy, comprising $m$ expert demonstration
trajectories from a wide range of tasks, denoted as
$D={\tau_{1},\tau_{2},...,\tau_{m}}$. Each trajectory $\tau$ comprises a
sequence of state-action pairs accompanied by a natural language instruction,
structured as
$\tau_{i}={l^{i},{(s_{1}^{i},a_{1}^{i}),(s_{2}^{i},a_{2}^{i}),...,(s_{n}^{i},a_{n}^{i})}}$.
Each task $T_{i}$ can be further decomposed into multiple snippets of various
lengths, each representing simpler sub-goals. For instance, a task such as
“Take the steak off the grill" may decompose into sub-goals like “Moving the
gripper to target the steak", “Picking up the steak", and “Placing the steak
at the destination". However, such explicit instructions are uncommon in daily
life, often leading individuals to overlook common sense steps. Our method,
Ex-PERACT, aims to categorize these implicit steps at the top-level model from
the offline dataset, and to simplify the problem, we assume the each snippet
only encodes one sub-goal.

In line with previous research, we adopt the approach introduced by James et
al. for trajectory decomposition [13]. A simple heuristic is designed to
extract keyframes from demonstration: a timestep is assigned as a keyframe if
the joint velocity is close to zero or if the gripper’s status changes (from
open to close, and vice versa). The actions of these extracted keyframes are
also utilized as macro-actions during training, enhancing robustness and
mitigating the effects of randomness and noise inherent in original continuous
actions.

We employ PERACT’s voxelization method to reconstruct 3D representations from
RGB-D image observations [29]. This approach partitions a $1.0m^{3}$ space
into $100^{3}$ voxel grids (See Figure 1). These discrete representations of
observation and action facilitate reformulating the problem as a "next best
action" classification problem [13], wherein the $(x,y,z)$ location of the
voxel grid corresponds to the coordination of the gripper’s center, while the
robotic arm’s rotation is discretized similarly over three axes with a
5-degree resolution.

Despite decomposing trajectories into snippets of various lengths, the sub-
goals or skills represented by these snippets remain unknown. Due to this lack
of prior knowledge, we train the top-level model in an unsupervised manner,
utilizing vector quantization to summarize skills into discrete skill codes.

Figure 1: Illustration of Voxelization for the “Open Drawer" Task, adapted
from [29]. RGB-D image inputs undergo transformation into (100,100,100) voxel
grids, representing a cubic meter space.

###  3.2 Ex-PERACT

Figure 2: A diagram illustrating the Ex-PERACT framework. Ex-PERACT, a
hierarchical model, aims to learn reusable skills to improve explainability
and predict actions based on given observations. Language instructions are
encoded using a pre-trained CLIP model, while observations are preprocessed
into embeddings or voxels. The top-level model takes language embeddings and
observation embeddings as input, producing discretized skill code estimates.
Meanwhile, the bottom-level model receives voxel inputs, language embeddings,
and discretized skill code estimates, generating action predictions from three
perspectives.

Ex-PERACT is a two-level hierarchical framework designed to process language
instructions ($\mathcal{L}$) and observations ($\mathcal{O}$), yielding
discretized actions comprising three aspects of behavior: coordination (also
known as translation), quaternion rotation, and gripper state
($\mathcal{A}={\mathcal{A}{\text{coord}},\mathcal{A}{\text{rotation}},\mathcal{A}_{\text{gripper}}}$)
(see Figure 2). Both levels of the model employ transformer-based
architectures to handle sequential input. At the top level, a transformer
model learns discrete skill codes from a single language instruction and a
sequence of observations, represented as
$f(\mathcal{O},\mathcal{L})\mapsto\mathcal{C}$. Meanwhile, the bottom level
transformer learns a policy in a command-conditional behavior cloning manner,
denoted as $\pi(\mathcal{O},\mathcal{C},\mathcal{L})\mapsto\mathcal{A}$, akin
to previous work [5]. Drawing inspiration from the promising results of CLIP
in manipulation tasks reported in prior literature [33, 28, 8], Ex-PERACT also
incorporates CLIP [24]. CLIP is renowned for its generalization capabilities
and effectiveness in multimodal domains. The tokenizer and pre-trained model
of CLIP are employed to tokenize and encode human-understandable language into
language embeddings with dimensions of (77, 512). Additionally, we fine-tune
the language embeddings using a linear layer to further enhance their
effectiveness.

The input observation consists of a set of observations, encompassing not only
the observation at the current timestep but also those from both past and
future keyframes. Each observation comprises RGB-D inputs from four cameras
(front, left shoulder, right shoulder, and wrist). As the heuristically
identified keyframes divide each demonstration into multiple snippets, at
timestep $t$, history observations are randomly sampled within the same
snippet. The skill code at timestep $t$ is then obtained via majority vote
from these history observations, ensuring consistency and robustness. Future
keyframe observations are extracted from the remaining keyframes from timestep
$t$. All these observations are chronologically ordered and fed into the top-
level model as a sequence. Initially, we fuse and encode the RGB-D input of
each observation using a Convolutional Neural Network (CNN). The encoded
observation embeddings are then combined with positional embeddings obtained
from the timestep indexes of the observations and concatenated with the
language embeddings from the pre-trained CLIP model. This language-observation
embeddings are inputted into a simple causal transformer with a single self-
attention layer followed by a linear layer, generating continuous skill code
estimates for each observation in the input, i.e., $\hat{c}_{t}=f(o_{t},l)$,
where $o_{t}$ represents the individual observation at timestep $t$, and $l$
denotes the language instruction of the task. Due to the lack of prior
knowledge regarding the skill codes, we utilize vector quantization
$vq(\cdot)$ to cluster and discretize these learned continuous skill code
estimates, yielding $\widetilde{c}=vq(\hat{c})$.

We train Ex-PERACT end-to-end using the command-conditional behavior cloning
paradigm. The standard command-conditional behavior cloning, as introduced in
[5], can be represented by the following equation:

$\text{minimize}\sum_{i}\mathcal{L}(\pi(o_{i},c_{i}),a_{i}).$ (1)

Since we lack prior knowledge of the skill codes, and they are learned
unsupervisedly in the top-level model, we introduce vector quantization loss
in Equation 1. Vector quantization loss is calculated as the mean squared
error between the discretized skill code estimate and the continuous skill
code. Additionally, considering the action decomposition into three
categories, we independently calculate the cross-entropy loss for each type of
action. Consequently, the objective of Ex-PERACT can be formulated as follows:

$\begin{split}minimize\sum_{i}&\mathcal{L}_{CE}(\pi_{coord}(o_{i},c_{i},l),a_{i}^{\text{coord}})+\\\
&\mathcal{L}_{CE}(\pi_{rot}(o_{i},c_{i},l),a_{i}^{\text{rot}})+\\\
&\mathcal{L}_{CE}(\pi_{grip}(o_{i},c_{i},l),a_{i}^{\text{grip}})+\\\
&\mathcal{L}_{MSE}(f(o_{i},l),\widetilde{c}_{i}),\end{split}$ (2)

where the last term can be directly obtained from the top-level model, and the
first three terms are calculated after each iteration.

To obtain various types of actions, we employ another transformer-based model
to predict coordinates, rotation, and gripper state separately. At timestep
$t$, the top-level model takes the observation set and language instruction as
input, while at the bottom level, the same observation set and language
embeddings together with obtained skill code estimates are used as input. The
observations are reconstructed into voxels $v$ with a shape of $(100,100,100)$
using a similar approach as described in both [13] and [29]. These $100^{3}$
voxel grids are then split and flattened into sequences with a length of
$(100/5)^{3}=8000$ using a 3D convolutional layer with kernel and stride size
set to five. The flattened voxel sequence is subsequently concatenated with
the extracted skill code estimate and language embeddings, and then fed into a
six-layer self-attention model. Considering the potential memory limitations
when processing long sequences on commodity GPUs, we adopt the Perceiver
Transformer $pt(\cdot)$ with 512 latents of dimension 512 [10]. The Perceiver
Transformer allows for projection from a long sequence to a much shorter
latent space and then projects the final output latent back to the original
input size. The features of each voxel grid are obtained from the output of
the Perceiver Transformer after decoding, which are then utilized to predict
coordinates using a 3D CNN or rotation and gripper status using a Multilayer
Perceptron (MLP). Specifically:

$\begin{split}\hat{a}_{t}^{\text{coord}}=&\pi_{coord}(pt(o_{t},c_{t},l))\\\
\hat{a}_{t}^{\text{rot}}=&\pi_{rot}(pt(o_{t},c_{t},l))\\\
\hat{a}_{t}^{\text{grip}}=&\pi_{grip}(pt(o_{t},c_{t},l)).\end{split}$ (3)

Table 1: Multitask performance comparison across state-of-the-art methods. Values represent mean success rate (%) obtained from 25 evaluation episodes per task with the best ones bolded. method | open drawer | meat off grill | slide block | turn tap | close jar | stack blocks | screw bulb | push buttons
---|---|---|---|---|---|---|---|---
BC (CNN) | 4 | 0 | 4 | 20 | 0 | 0 | 0 | 4
BC (ViT) | 16 | 0 | 8 | 24 | 0 | 0 | 0 | 16
LISA | 8 | 0 | 4 | 28 | 0 | 0 | 0 | 20
C2FARM-BC | 28 | 40 | 12 | 60 | 28 | 4 | 12 | 88
Ex-PERACT (Obs only) | 20 | 40 | 8 | 36 | 16 | 0 | 0 | 60
Ex-PERACT (Obs + skill code) | 28 | 56 | 16 | 48 | 16 | 4 | 20 | 72
| Ex-PERACT
---
(Obs + Lang, which is PERACT)
64 | 68 | 32 | 60 | 28 | 8 | 28 | 56
Ex-PERACT (single Lang) | 76 | 80 | 44 | 60 | 32 | 0 | 36 | 92
Ex-PERACT (multi Lang) | 72 | 80 | 44 | 88 | 40 | 8 | 24 | 88

###  3.3 Implementation

The Ex-PERACT framework is hierarchical in nature, and we employ the single
LAMB optimizer [35] to update parameters from both the top and bottom-level
models. To facilitate end-to-end training within a framework containing non-
differentiable components like VQ, we utilize a technique known as the
straight-through estimator. This method directly propagates gradients from the
decoder to the encoder. Given that actions are discretized, cross-entropy
losses for three types of actions are computed by comparing the action
estimates encoded with ground truth one-hot encoding. Specifically, for
rotation, the loss is calculated under the 3D Euler angle coordinate setting,
with the final rotation action reconstructed from the Euler angles into a
quaternion. Demonstration inputs are obtained with the assistance of a motion
planner in RLBench, with each demonstration sourced from a different variance
number. Since observation inputs for specific timesteps consist of observation
sets, we set the sample frequency and batch size to 10 and 1, respectively, to
ensure efficient disk and memory usage on commodity devices. Observations
reveal significant variations in demonstration length across tasks, ranging
from approximately 100 to over 400 timesteps. Additionally, the number of
identified keyframes exhibits considerable deviation, ranging from 2 to around
20. To maintain a similar sample frequency for each sample, we set the
sampling weights as the sum of the total number of observations for each task
and the size of the observation set. In the bottom-level model, we implement
data augmentation prior to voxelization.

##  4 Result

In this section, we present our experimental setup and findings. We assess Ex-
PERACT’s performance using RLBench manipulation tasks, comparing its multitask
capabilities against those of other state-of-the-art methods. Additionally, we
analyze the explainability provided by the hierarchical framework.

###  4.1 Experiment setup

Ex-PERACT is evaluated within the RLBench simulation platform, utilizing a GPU
NVIDIA Quadro RTX 5000 for computations. RLBench operates on CoppeliaSim/V-REP
[26] and PyRep [11] interfaces, featuring a Franka Emika Panda robot arm with
six degrees of freedom (DOF). This platform provides crucial data such as
joint velocities, angles, and forces, alongside four $128\times 128$ RGB-D
images captured from different camera perspectives. Diverse manipulation tasks
are designed, ensuring variability in task execution and accompanying language
summaries. To manage memory usage effectively, all samples are stored on disk.
We select eight tasks spanning from opening a drawer to screwing in a light
bulb, with ten demonstrations collected for each task. For comparison, we
evaluate several state-of-the-art imitation learning (IL) methods as
baselines. Baseline agents include BC with image encoder CNN and Vision
Transformer (ViT) [6], LISA [7], C2FARM-BC [13], and PERACT [29]. BC and LISA
are image-to-action agents mapping observed RGB-D images to eight-dimensional
actions. In contrast, C2FARM-BC, PERACT, and Ex-PERACT are voxel-to-action
agents, reconstructing the 3D space and discretizing the problem as a
classification task. While C2FARM-BC adopts a two-level voxelization allowing
zoom-in operations for higher resolution ($0.47cm^{3}$), PERACT and Ex-PERACT
employ a voxel resolution of $1cm^{3}$. Each agent interacts with the test
environment via a motion planner, and the mean success rate is computed from
25 episodes per task ($8\times 25=200$ total episodes evaluated). Success is
defined as reaching the goal state within 25 steps, while episodes
encountering issues such as exceeding the maximum steps, encountering an
invalid path, or having the target outside the workspace are deemed failures.

###  4.2 Policy performance

Table 1 presents the average success rates of state-of-the-art methods and our
method Ex-PERACT. The first three methods are image-based BC agents, while the
remaining methods employ a 3D representation. The “single lang" Ex-PERACT
denotes that the language instructions are in a consistent format for a
specific task, whereas the “multi-lang" Ex-PERACT indicates variable
instruction formats across batches. For example, in the task “open drawer", a
consistent instruction would be “open the top/middle/bottom drawer"; under the
“multi-lang" condition, the instructions could vary randomly, including “open
the top/middle/bottom drawer", “grip the top/middle/bottom handle and pull the
top/middle/bottom drawer open", or “slide the top/middle/bottom drawer open".
All methods receive the same set of 80 demonstrations ($8\times 10=80$) as
input. Analysis of Table 1 reveals that image-based methods demonstrate poor
performance across all tasks, whereas those utilizing 3D representation
consistently outperform them. With only 10 demonstrations available per task,
image-based agents face significant challenges across all eight tasks. This
finding underscores the formidable task of learning hand-eye coordination from
scratch, given its inherent demands for time and data. Furthermore, this
comparison supports our contention that 3D voxelization offers enhanced data
efficiency. Among methods employing 3D representation, our approach, Ex-
PERACT, which integrates various language inputs, consistently demonstrates
superior policy performance. It surpasses PERACT by an average improvement of
29.07% (the third row from the bottom) and achieves a 63.23% better average
performance compared to C2FARM.

We observe that C2FARM, PERACT, and Ex-PERACT perform better in tasks like
“open drawer", “meat off grill", and “turn tap", which have relatively fewer
variations, namely 3 (top/middle/bottom), 2 (steak/drumstick), and 2
(left/right), respectively. This pattern indicates that the performance of BC
agents is highly dependent on the number of variations in a task. Moreover,
all evaluated methods show limited success in the task “stack blocks" (with a
maximum of 60 variations). In addition to the influence of highest variation
number, we think the principal challenge lies in the degree of freedom of the
manipulable objects. The more states these objects can reach, the more
challenging it is for BC agents to learn, as minor errors can be magnified
during the manipulation of multiple objects. For example, although Ex-PERACT
can successfully stack the first block, any minor misalignment in the location
or rotation of the first blocks degrades the performance of subsequent
actions. The impact of such errors becomes even amplified as more blocks are
stacked. This is likely a major contributing factor to the difficulties
encountered in the “stack blocks" task, considering the promising performance
in task “push buttons" while “push buttons also possesses a maximum 50
variation number.

We conducted an ablation study to investigate the impact of skill code
availability, language, and their combination, as well as language diversity,
on performance. The results of this analysis are presented in the fifth-to-
last row of Table 1. Comparing Ex-PERACT with observation only to Ex-PERACT
with both skill code and observation, we conclude that the inclusion of skill
code provides useful information to enhance agent policy performance. In
comparing Ex-PERACT with skill code to Ex-PERACT with language (PERACT), we
observed that Ex-PERACT with language significantly outperforms Ex-PERACT with
skill code. This finding contradicts the result in [7], where non-hierarchical
Ex-PERACT with language performed worse than hierarchical Ex-PERACT with skill
code. We believe the reason for this discrepancy is that the limited number of
skills is inadequate to represent tasks with large varieties. This assertion
is supported by the performance of Ex-PERACT with skill code, particularly in
low variety tasks such as “meat off grill" and “turn tap", where the agent
performs relatively better than in tasks with greater variety. Additionally,
comparing the performance between single-language Ex-PERACT and multi-language
Ex-PERACT, the results indicate that the inclusion of diverse language
instructions does not detract from the performance of Ex-PERACT.

###  4.3 Explainable skill representation

To evaluate the explainability of Ex-PERACT, we aggregated the input language
instructions alongside the corresponding output skill codes into a dictionary.
In this dictionary, each key represents a skill code generated by the Ex-
PERACT system, while the corresponding value comprises a list of words
extracted from the language instructions. Notably, common stop words such as
“from", “it", and “to" were excluded from the list. The rationale behind this
methodology is to leverage frequently occurring words to elucidate the learned
skill codes. Following this compilation, we explored various visualization
techniques to represent the human-understandable verbal meanings of the skill
codes. Traditional methods, such as heat maps, are less intuitive due to the
heterogeneous formats of language instructions and the potential inundation of
words, resulting in a dense and overwhelming visualization. Consequently, we
opted for a word cloud to enhance accessibility for less technically inclined
audiences.

(a) Word cloud for skill code 04.

(b) Word cloud for skill code 10.

(c) Word cloud for skill code 18.

(d) Word cloud for skill code 19.

Figure 3: Word Clouds for different skill codes, with the size of each word
indicating its frequency within the corresponding skill code.

Figure 3 showcases several word cloud examples corresponding to different
skill codes. Upon inspection, it becomes evident that each skill code
emphasizes distinct aspects of the evaluated tasks. For instance, Skill Code
04 (refer to Figure 3(a)) appears to hold greater significance in tasks such
as “open drawer", “meat off grill", and "slide block", with a notable focus on
colors. Conversely, Skill Code 10 (refer to Figure 3(b)) is centered around
tasks like "stack blocks" and “screw bulb", placing emphasis on numerical
values to indicate the quantity of objects being manipulated. Furthermore,
Skill Codes 18 and 19, both associated with the action of “screw", but exhibit
unique characteristics (refer to Figures 3(c) and 3(d)). Skill Code 18
demonstrates a closer relationship to the task “slide block", featuring words
like “tap" and “jar" that denote screw-like actions, albeit without explicit
mentions of “screwing". In contrast, Skill Code 19 appears to highlight the
relative positions of manipulable objects.

By examining the common tasks associated with each skill code, we can derive
abstract insights into the underlying sub-actions represented by the skills.
For example, Skill Code 04 could suggest a common sub-action of “grabbing an
object" across tasks like “open drawer", “meat off grill", and “slide block".
Similarly, Skill Code 10 implies a common sub-action of “placing an object on
top of another" in tasks such as “stack blocks" and “screw bulb". However,
it’s important to note that these interpretations are based on human
induction, as there may be a lack of alignment between the language
instructions and the delineation of key points.Future endeavors could focus on
refining this process by decomposing the language instructions and correlating
language fragments with demonstration snippets to generate more convincing
explanations.

##  5 Conclusion and Future Work

This paper introduces Ex-PERACT, an explainable hierarchical multi-task
behavior cloning agent. At the top level, Ex-PERACT learns discrete skill
codes via clustering, while the bottom level utilizes skill codes, language,
and observations to acquire a proficient behavioral policy by representing the
problem as discrete 3D voxel grids. Our experiments on 6-DOF manipulation
tasks illustrate that Ex-PERACT demonstrates promising performance compared to
other baseline methods, concurrently providing an human-understandable
visualization of skills learned by the agent.

However, certain limitations of Ex-PERACT persist and warrant exploration as
potential future research directions. Given Ex-PERACT’s ability to accommodate
various forms of language instructions, developing a model to decompose
various language instructions to cooperate with trajectory key points could be
one interesting direction to further enhance explainability. Additionally, Ex-
PERACT faces challenges in long-term tasks and tasks characterized by high
variability. Subsequent research endeavors may prioritize enhancing its
performance, addressing not only the breadth of tasks but also considerations
of task length and variance.

## References

  * Beliaev et al. [2022] M. Beliaev, A. Shih, S. Ermon, D. Sadigh, and R. Pedarsani.  Imitation learning by estimating expertise of demonstrators.  In _International Conference on Machine Learning_ , pages 1732–1748. PMLR, 2022.
  * Bewley et al. [2020] T. Bewley, J. Lawry, and A. Richards.  Modelling agent policies with interpretable imitation learning.  In _International Workshop on the Foundations of Trustworthy AI Integrating Learning, Optimization and Reasoning_ , pages 180–186. Springer, 2020.
  * Brown et al. [2020] D. Brown, R. Coleman, R. Srinivasan, and S. Niekum.  Safe imitation learning via fast bayesian reward inference from preferences.  In _International Conference on Machine Learning_ , pages 1165–1177. PMLR, 2020.
  * Chen and Mooney [2011] D. Chen and R. Mooney.  Learning to interpret natural language navigation instructions from observations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 25, pages 859–865, 2011.
  * Codevilla et al. [2018] F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy.  End-to-end driving via conditional imitation learning.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pages 4693–4700. IEEE, 2018.
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Garg et al. [2022] D. Garg, S. Vaidyanath, K. Kim, J. Song, and S. Ermon.  Lisa: Learning interpretable skill abstractions from language.  _Advances in Neural Information Processing Systems_ , 35:21711–21724, 2022.
  * Guhur et al. [2023] P.-L. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid.  Instruction-driven history-aware policies for robotic manipulations.  In _Conference on Robot Learning_ , pages 175–187. PMLR, 2023.
  * Hakhamaneshi et al. [2022] K. Hakhamaneshi, R. Zhao, A. Zhan, P. Abbeel, and M. Laskin.  Hierarchical few-shot imitation with skill transition models.  In _International Conference on Learning Representations_ , 2022.
  * Jaegle et al. [2021] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, et al.  Perceiver io: A general architecture for structured inputs & outputs.  _arXiv preprint arXiv:2107.14795_ , 2021.
  * James et al. [2019] S. James, M. Freese, and A. J. Davison.  Pyrep: Bringing v-rep to deep robot learning.  _arXiv preprint arXiv:1906.11176_ , 2019.
  * James et al. [2020] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison.  Rlbench: The robot learning benchmark & learning environment.  _IEEE Robotics and Automation Letters_ , 5(2):3019–3026, 2020.
  * James et al. [2022] S. James, K. Wada, T. Laidlow, and A. J. Davison.  Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13739–13748, 2022.
  * Jang et al. [2022] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn.  Bc-z: Zero-shot task generalization with robotic imitation learning.  In _Conference on Robot Learning_ , pages 991–1002. PMLR, 2022.
  * Jiang et al. [2023] Y. Jiang, W. Yu, D. Song, W. Cheng, and H. Chen.  Interpretable skill learning for dynamic treatment regimes through imitation.  In _2023 57th Annual Conference on Information Sciences and Systems (CISS)_ , pages 1–6. IEEE, 2023.
  * Karamcheti et al. [2023] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang.  Language-driven representation learning for robotics.  _arXiv preprint arXiv:2302.12766_ , 2023.
  * Le et al. [2018] H. Le, N. Jiang, A. Agarwal, M. Dudík, Y. Yue, and H. Daumé III.  Hierarchical imitation and reinforcement learning.  In _International conference on machine learning_ , pages 2917–2926. PMLR, 2018.
  * Leech [2019] T. Leech.  _Explainable machine learning for task planning in robotics_.  PhD thesis, Massachusetts Institute of Technology, 2019.
  * Li et al. [2017] Y. Li, J. Song, and S. Ermon.  Infogail: Interpretable imitation learning from visual demonstrations.  _Advances in neural information processing systems_ , 30, 2017.
  * Lynch and Sermanet [2020] C. Lynch and P. Sermanet.  Language conditioned imitation learning over unstructured data.  _arXiv preprint arXiv:2005.07648_ , 2020.
  * Nair et al. [2022] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al.  Learning language-conditioned robot behavior from offline data and crowd-sourced annotation.  In _Conference on Robot Learning_ , pages 1303–1315. PMLR, 2022.
  * Oh et al. [2018] J. Oh, Y. Guo, S. Singh, and H. Lee.  Self-imitation learning.  In _International conference on machine learning_ , pages 3878–3887. PMLR, 2018.
  * Pan et al. [2017] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. Theodorou, and B. Boots.  Agile autonomous driving using end-to-end deep imitation learning.  _arXiv preprint arXiv:1709.07174_ , 2017.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Roh et al. [2020] J. Roh, C. Paxton, A. Pronobis, A. Farhadi, and D. Fox.  Conditional driving from natural language instructions.  In _Conference on Robot Learning_ , pages 540–551. PMLR, 2020.
  * Rohmer et al. [2013] E. Rohmer, S. P. Singh, and M. Freese.  V-rep: A versatile and scalable robot simulation framework.  In _2013 IEEE/RSJ international conference on intelligent robots and systems_ , pages 1321–1326. IEEE, 2013.
  * Sharma et al. [2018] A. Sharma, M. Sharma, N. Rhinehart, and K. M. Kitani.  Directed-info gail: Learning hierarchical policies from unsegmented demonstrations using directed information.  _arXiv preprint arXiv:1810.01266_ , 2018.
  * Shridhar et al. [2022] M. Shridhar, L. Manuelli, and D. Fox.  Cliport: What and where pathways for robotic manipulation.  In _Conference on Robot Learning_ , pages 894–906. PMLR, 2022.
  * Shridhar et al. [2023] M. Shridhar, L. Manuelli, and D. Fox.  Perceiver-actor: A multi-task transformer for robotic manipulation.  In _Conference on Robot Learning_ , pages 785–799. PMLR, 2023.
  * Stepputtis et al. [2020] S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor.  Language-conditioned imitation learning for robot manipulation tasks.  _Advances in Neural Information Processing Systems_ , 33:13139–13150, 2020.
  * Van Den Oord et al. [2017] A. Van Den Oord, O. Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2022] L. Wang, R. Tang, X. He, and X. He.  Hierarchical imitation learning via subgoal representation learning for dynamic treatment recommendation.  In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ , pages 1081–1089, 2022.
  * Xiao et al. [2022] T. Xiao, H. Chan, P. Sermanet, A. Wahid, A. Brohan, K. Hausman, S. Levine, and J. Tompson.  Robotic skill acquisition via instruction augmentation with vision-language models.  _arXiv preprint arXiv:2211.11736_ , 2022.
  * Xie et al. [2022] Y. Xie, S. Vosoughi, and S. Hassanpour.  Towards interpretable deep reinforcement learning models via inverse reinforcement learning.  In _2022 26th International Conference on Pattern Recognition (ICPR)_ , pages 5067–5074. IEEE, 2022.
  * You et al. [2019] Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsieh.  Large batch optimization for deep learning: Training bert in 76 minutes.  _arXiv preprint arXiv:1904.00962_ , 2019.
  * Yu et al. [2018] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and S. Levine.  One-shot imitation from observing humans via domain-adaptive meta-learning.  _arXiv preprint arXiv:1802.01557_ , 2018.
  * Zhang et al. [2021] D. Zhang, Q. Li, Y. Zheng, L. Wei, D. Zhang, and Z. Zhang.  Explainable hierarchical imitation learning for robotic drink pouring.  _IEEE Transactions on Automation Science and Engineering_ , 19(4):3871–3887, 2021.
  * Zhang and Paschalidis [2021] Z. Zhang and I. Paschalidis.  Provable hierarchical imitation learning via em.  In _International Conference on Artificial Intelligence and Statistics_ , pages 883–891. PMLR, 2021.
  * Zhou and Small [2021] L. Zhou and K. Small.  Inverse reinforcement learning with natural language goals.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 11116–11124, 2021.