# SPRINT: Scalable Policy Pre-Training via Language Instruction Relabeling

Jesse Zhang 1, Karl Pertsch1, Jiahui Zhang1, Joseph J. Lim2
1University of Southern California, 2KAIST Correspondence to<EMAIL_ADDRESS>
###### Abstract

Pre-training robot policies with a rich set of skills can substantially
accelerate the learning of downstream tasks. Prior works have defined pre-
training tasks via natural language instructions, but doing so requires
tedious human annotation of hundreds of thousands of instructions. Thus, we
propose SPRINT, a scalable offline policy pre-training approach which
substantially reduces the human effort needed for pre-training a diverse set
of skills. Our method uses two core ideas to automatically expand a base set
of pre-training tasks: instruction relabeling via large language models and
cross-trajectory skill chaining through offline reinforcement learning. As a
result, SPRINT pre-training equips robots with a much richer repertoire of
skills. Experimental results in a household simulator and on a real robot
kitchen manipulation task show that SPRINT leads to substantially faster
learning of new long-horizon tasks than previous pre-training approaches.
Website at https://clvrai.com/sprint.

###  1 Introduction

When humans learn a new task, e.g., how to cook a new dish, we rely on a large
repertoire of previously learned skills, like “chopping vegetables" or
“boiling pasta", that make learning more efficient. Similarly, much work in
robot learning aims to equip robots with a set of useful skills (Sutton et
al., 1999; Schaal, 2006; Hausman et al., 2018; Lynch et al., 2020; Pertsch et
al., 2020; Haarnoja et al., 2023) for improving learning efficiency. A common
approach to acquiring a rich skill set is to pre-train policies on a wide
range of tasks. Recent works have employed _language instructions_ as a way
for humans to manually define such tasks for policy training, typically via
hindsight annotation of large, pre-collected robot experience datasets (Mees
et al., 2022; Lynch and Sermanet, 2021; Lynch et al., 2022; Brohan et al.,
2022). While the resulting policies show impressive capabilities,
generalization to new tasks requires a _large_ set of pre-trained skills and
thus many pre-training tasks. As a result, prior works resorted to annotating
robot trajectory datasets with _hundreds of thousands_ of human instruction
labels (Lynch et al., 2022), limiting their application outside industrial
contexts. Can we instead devise a pre-training approach that similarly equips
robots with a wide repertoire of skills but _minimizes_ the need for human
task annotations?

We introduce SPRINT (Scalable Pre-training via Relabeling Language
INsTructions), a scalable pre-training approach that equips robots with a
large set of skills while substantially reducing human labeling effort (see
Figure 1). SPRINT uses extensive _automated_ relabeling to expand an initial
set of pre-training tasks. Given a dataset of robot trajectories with initial
language instruction annotations, we leverage two core ideas to grow the
number of tasks. First, we leverage the rich knowledge captured in large
language models (LLMs) to iteratively combine consecutive language
instructions into more complex tasks, e.g., “place mug in coffee machine” and
“press brew button” into “make coffee”. Second, we propose a language-
conditioned offline RL objective that “stitches” multiple trajectory segments
from the training data to form new tasks, a process we call “skill chaining”
since it allows the policy to learn longer-horizon skills. Through the
combination of both techniques, SPRINT creates a much richer pre-training task
set. We demonstrate that SPRINT-pre-trained robots can leverage their
resulting larger skill repertoire to more efficiently learn downstream tasks.

Figure 1: We propose SPRINT, a scalable approach for pre-training robot
policies with a rich repertoire of skills while minimizing human annotation
effort. Given a dataset of robot trajectories with an initial set of task
instructions for offline pre-training, SPRINT expands the pre-training task
set without additional human effort via language-model-based instruction
relabeling and cross-trajectory skill chaining. SPRINT-pre-trained policies
enable efficient finetuning on unseen target tasks.

In summary, our contributions are threefold: (1) we propose SPRINT, a scalable
pre-training approach for robot policies that minimizes human task annotation
effort via LLM-based aggregation and cross-trajectory skill chaining, (2) we
introduce ALFRED-RL, an RL benchmark for the popular ALFRED household task
simulator (Shridhar et al., 2020), to test our pre-trained agents on a rich
set of long-horizon, semantically meaningful tasks, (3) we demonstrate that
policies pre-trained with SPRINT learn downstream tasks more efficiently than
prior pre-training approaches, both on challenging ALFRED tasks and in a real
robot kitchen manipulation setup.

###  2 Related Work

Language in RL. There is a large body of work at the intersection of natural
language processing and behavior learning for robotics, and the field has been
further accelerated by the recent successes in training large, general-purpose
language models. Language has been used to structure agents’ representations
(Andreas et al., 2017a; Nair et al., 2022), learn reward functions (Fan et
al., 2022), guide task learning via recipes (Branavan et al., 2009; Andreas et
al., 2017b) and perform long-horizon planning (Huang et al., 2022a; Ahn et
al., 2022; Huang et al., 2022b; Singh et al., 2023). Another line of work has
used language to define a wide range of tasks for pre-training policies,
resulting in impressive generalization capabilities (Lynch and Sermanet, 2021;
Lynch et al., 2022; Brohan et al., 2022). Yet, these works require collecting
hundreds of thousands of costly human language instructions. Our approach
SPRINT builds on this line of work but introduces two novel objectives for
automatic relabeling of training task instructions, thereby substantially
reducing the amount of human labeling required for successful pre-training.
Prior works have also investigated automated language instruction generation
(Colas et al., 2020; Cideron et al., 2020; Li et al., 2022), but they focus on
online learning and make assumptions that are hard to scale, e.g., hand-
defined grammars (Colas et al., 2020) or privileged state information (Li et
al., 2022; Cideron et al., 2020). In contrast, we perform _offline_ training
and use large language models for _scalable_ task generation.

Pre-training Policies for RL. Developing policy pre-training approaches for
faster downstream learning has been investigated for many years (Ijspeert et
al., 2002; Theodorou et al., 2010; Hester et al., 2018). Recent advances in
offline reinforcement learning (Levine et al., 2020) enabled approaches that
can pre-train agents offline and effectively finetune them on online tasks
(Peng et al., 2019; Singh et al., 2020; Nair et al., 2020; Kostrikov et al.,
2022). However, these approaches require target-task reward annotations on the
pre-training data, and the resulting policies are only pre-trained to solve
the target task. Meta-RL approaches, on the other hand, pre-train on a range
of tasks and thus allow fast adaptation to _unseen_ downstream tasks (Duan et
al., 2016; Finn et al., 2017; Rakelly et al., 2019; Nam et al., 2022), yet
require the tedious manual definition of pre-training tasks by experts. To
avoid manual task design, other works have explored unsupervised pre-training
approaches based on behavior diversification (Achiam et al., 2018; Eysenbach
et al., 2019; Sharma et al., 2019), extraction of behavior priors from offline
agent experience (Pertsch et al., 2020; Ajay et al., 2020; Singh et al., 2021)
or goal state reaching (Mendonca et al., 2021; Chebotar et al., 2021). Yet,
such unsupervised pre-training approaches learn skill repertoires without
semantic meaning, which, as we demonstrate in Section 4, lead to worse
downstream task transfer than SPRINT’s skill repertoire learned via language.

Pre-trained Models for Data Augmentation. Obtaining robot (pre-)training data
at scale is costly. Thus, recent works have explored using world knowledge
captured in large pre-trained models for enriching robot learning datasets,
e.g., by increasing the visual diversity of trajectories (Yu et al., 2023;
Chen et al., 2023; Mandi et al., 2023) or annotating unlabeled data (Xiao et
al., 2023). Our approach similarly leverages pre-trained (language) models for
automated data augmentation. By investigating an orthogonal augmentation
direction, aggregation and chaining of natural language instructions, SPRINT
is complementary to these methods.

Figure 2: SPRINT overview. We assume access to a dataset of agent experience
with language instructions for the performed skills (1). Collecting such
instructions with human hindsight annotation is a flexible yet costly approach
for defining pre-training tasks. Thus, SPRINT introduces two approaches for
automatically growing the set of pre-training tasks without additional human
effort: (2) by aggregating language instructions with an LLM and adding the
relabeled trajectories back into the pre-training dataset (Section 3.2), (3)
by performing cross-trajectory chaining of skills to enable pre-training of
skills that are unseen in the offline agent experience (Section 3.3).

###  3 SPRINT: Scalable Policy Pre-Training with Language Instructions

In this paper, we propose SPRINT (Scalable Pre-training via Relabeling
Language INsTructions), an approach for pre-training robot policies that
equips them with a rich repertoire of skills to enable efficient finetuning on
unseen tasks. Following prior work on agent pre-training, SPRINT assumes
access to a large offline dataset $\mathcal{D}$ of agent experience (Gupta et
al., 2019; Lynch et al., 2020; Pertsch et al., 2020; Chebotar et al., 2021;
Ebert et al., 2022; Pertsch et al., 2021), collected e.g., from prior RL runs
or via teleoperation. We further assume that the data is annotated with an
initial set of natural language task instructions, e.g., “put a mug in the
coffee machine” or “push the brew button”, that can be collected _in
hindsight_ via platforms like Amazon Mechanical Turk (Lynch and Sermanet,
2021; Shridhar et al., 2020). Given a sequence $\tau$ of states and actions
from the dataset $\mathcal{D}$, annotators can label sub-trajectories
$\tau_{1}=[s_{0},a_{0},s_{1},\dots],\tau_{2}=\dots$ with free-form language
descriptions $z_{1},z_{2},\dots$ of the skills executed in the respective sub-
trajectories (see Figure 2, left), resulting in a _language-annotated_ dataset
$\mathcal{D}^{L}$.

Approach Overview. SPRINT equips policies with a diverse repertoire of skills
via language-instruction-conditioned offline RL: given a natural language task
description $z$, the policy $\pi(a|s,z)$ is rewarded for successfully
executing the instruction (Section 3.1). Intuitively, the richer the set of
task instructions during pre-training, the more skills the policy will learn
and the more downstream tasks it can finetune on efficiently. Thus, SPRINT
introduces two approaches for increasing the scale and diversity of the pre-
training task instructions without requiring additional costly human inputs.
Firstly, SPRINT leverages pre-trained language models to aggregate consecutive
instructions into new tasks (Figure 2, middle, Section 3.2). Secondly, SPRINT
introduces an objective for cross-trajectory skill-chaining via offline RL
that generates novel instruction chains (Figure 2, right, Section 3.3). SPRINT
pre-trains policies on the combined set of tasks and thereby equips them with
a richer skill repertoire. In our experiments (Section 4) we demonstrate that
this leads to more effective learning of new tasks. See appendix, Alg. 1 for
the pseudocode of our method.

####  3.1 Instruction-Conditioned Offline RL

To pre-train our policy $\pi$ with the natural language instruction dataset
$\mathcal{D}^{L}$, we take inspiration from goal-conditioned RL (Kaelbling,
1993; Schaul et al., 2015; Chebotar et al., 2021): instead of rewarding the
policy for reaching goal states, we condition our policy $\pi(a|s,z)$ on
_language instructions_ $z$ from $\mathcal{D}^{L}$ and provide a sparse reward
$R(s,a,z)$ to the agent for reaching the end-state $s_{T_{z}}$ of the sub-
trajectory. Formally, we define the reward as:

$R(s,a,z)=\begin{cases}1,&\text{for }s=s_{T_{z}}\\\
0,&\text{otherwise.}\end{cases}$ (1)

We train our policy $\pi(a|s,z)$ to maximize this reward with offline RL
(Levine et al., 2020) using an instruction-conditioned critic $Q(s,a,z)$.
Specifically, we use Implicit Q-Learning (Kostrikov et al., 2022) as it is
easy to tune.

Summarize the following steps.
1: Pick up the tomato slice. 2: Heat it up in the microwave. Summary:
Microwave a tomato slice.
1: [SKILL 1] 2: [SKILL 2] … Summary: Figure 3: A shortened example of the LLM
prompt. See the full prompt in appendix, Section A.

####  3.2 Language-Model-Based Instruction Aggregation

Large language models (LLMs), trained on massive corpora of internet text
data, have been shown to be effective at performing a variety of tasks – from
question answering to program synthesis – when prompted with relevant text
(Devlin et al., 2018; Brown et al., 2020; Wang and Komatsuzaki, 2021; Rae et
al., 2021; Hoffmann et al., 2022; Zhang et al., 2022; Chowdhery et al., 2022).
Here we use LLMs to aggregate, i.e., paraphrase, the existing language
instructions in $\mathcal{D}^{L}$ (see Figure 2, middle). Given a trajectory
that contains multiple sub-trajectories, we can aggregate adjacent sub-
trajectories into a longer trajectory and relabel its natural language
annotation with a summary of the individual instructions generated by the LLM,
thereby generating a new higher-level pre-training task that encompasses
instructions from multiple sub-trajectories.111Other relabeling operations,
such as splitting an instruction into lower-level instructions, can also be
performed by the LLM. However, such operations require grounding the LLM in
the agent’s observations to determine sub-trajectory split points. We leave
investigating this to future work. We use a simple summarization prompt to
instruct the language model (see Figure 3). Specifically, we aggregate with
LLAMA-13B (Touvron et al., 2023), an open-source 13 billion parameter LLM (see
Section D.1 for qualitative examples). Like in Section 3.1, the reward for
this new aggregated sub-trajectory is 1 at the last transition and 0
otherwise. For example, we prompt the LLM to summarize the two skills
($z_{1}:$ “Put a mug in the coffee machine,” $z_{2}:$ “Push the brew button”),
resulting in a new annotation $\hat{z}_{1:2}$ describing both skills (e.g.,
“Make coffee”). We then add the new trajectory back to our dataset
$\mathcal{D}^{L}$. Using this technique, we generate new language annotations
for all ${N\choose 2}$ tuples of consecutive sub-trajectories in our dataset.
In practice, this increases the number of task instructions by 2.5x in ALFRED
and 2x in our robot manipulation dataset (see Section 4).

####  3.3 Cross-Trajectory Chaining

Agents trained with offline RL can combine behaviors from multiple
trajectories via value propagation, i.e., perform “stitching” (Levine et al.,
2020). For example, if trajectory (A) shows cleaning the mug in the sink while
trajectory (B) starts with placing the mug in the coffee machine, offline RL
algorithms are able to learn to clean the mug in the sink and then place it in
the coffee machine (see Figure 2, right) and thus learn long-horizon behaviors
that are unseen in the training data. In our case of _instruction-conditioned_
offline RL, enabling such stitching requires special care. Due to the
different language instruction conditionings for the critic $Q(s,a,z_{A})$ and
$Q(s,a,z_{B})$, values do not naturally propagate from trajectory (B) back to
trajectory (A). Instead, we must actively add “chaining examples.” to our
training dataset (Chebotar et al., 2021). To build such chaining examples, we
first sample two sub-trajectories: instead of restricting sampling to
consecutive trajectory segments as in Section 3.2, we can now sample
$\tau_{z_{A}}$ and $\tau_{z_{B}}$ from _different_ trajectories (see Figure
4). Next, we create an aggregate instruction $\hat{z}$ which indicates that
the agent first finishes skill (A) and then finishes skill (B), e.g., “clean
the coffee mug (A) and place it in the coffee machine (B)”.222Note that we
could generate $\hat{z}$ using the same LLM summarization as in Section 3.2.
Yet we found the resulting summaries to often be confusing since randomly
paired instructions _from different trajectories_ can rarely be summarized
meaningfully. We got the best empirical results by simply concatenating the
sampled instructions with the word “and”.

Figure 4: SPRINT chains skills from $\tau_{1}$ and $\tau_{2}$ into a new
trajectory. This is added to the buffer as two separate trajectories with
updated language instructions and appropriate reward values.

Finally, we _cannot_ use the reward function from Section 3.2 that simply sets
the reward to 1 at the end of every subtrajectory, since the last state
$s_{T_{A}}$ of the first trajectory does not solve the combined instruction
$\hat{z}$. Yet, only setting a reward of 1 on the final state of the second
trajectory $s_{T_{B}}$ would not accurately propagate value back into
trajectory (A), since the two trajectories are _separate_ , potentially
requiring many additional transitions to be labeled with the same composite
instruction $\hat{z}$ to “bridge” values back from the initial state of
$\tau_{z_{B}}$ to the final state of $\tau_{z_{A}}$ (see Figure 4). Instead of
relying on this propagation to happen by chance, we can _anchor_ the final
state of $\tau_{z_{A}}$ with a reward that is proportional to the likelihood
of finishing the overall task $\hat{z}$. How can we compute such a reward?

Q functions trained with TD learning (Sutton and Barto, 2018) for the sparse
reward from Equation 1 intuitively represent a value that is proportional to
the probability of reaching the goal state $s_{T_{z}}$ at time $T$ (Eysenbach
et al., 2022; Chebotar et al., 2021):

$\displaystyle Q^{\pi}(s_{t},a_{t},z)$
$\displaystyle=\mathbb{E}[\sum_{t^{\prime}=t}\gamma^{t^{\prime}}R(s_{t^{\prime}},a_{t^{\prime}},z)]$
$\displaystyle=\mathbb{E}\left[\gamma^{T-t}\mathbbm{1}\left[s_{T}=g_{z}\right]\right]\propto
P^{\pi}(s_{T}=g_{z}|s_{t},a_{t}).$ (2)

where $\gamma\in(0,1)$ denotes the discount factor. When anchoring the final
state of the first trajectory, we want to use a reward that is proportional to
the likelihood of finishing _the remainder_ of the combined task $\hat{z}$,
i.e., proportional to the likelihood of finishing the second subtask $z_{B}$
from $s_{T_{A}}$. Following Eq. 2, we can directly use
$Q(s_{T_{A}},a_{T_{A}},z_{B})$ to represent that probability:

$R(s,a,\hat{z})=\begin{cases}1,&\text{for }s=s_{T_{B}}\\\
Q(s,a,z_{B}),&\text{for }s=s_{T_{A}}\\\ 0,&\text{otherwise.}\end{cases}$ (3)

Note that unlike in Section 3.2, we treat both relabeled
$\tau_{z_{A}},\tau_{z_{B}}$ as separate trajectories as we do not know the
states and actions required to transition from the last state of (A) to states
in (B). For a discussion how chaining preserves the structure of the original
MDP, see Appendix Section B.6.1. Finally, since $Q$ changes during training,
we compute the rewards in Eq. 3 online while training.

###  4 Experiments

In our experiments, we investigate how well an agent pre-trained with SPRINT
performs on challenging unseen tasks. Specifically, we answer the following
questions: (1) Does SPRINT enable more efficient finetuning on unseen target
tasks than previous pre-training approaches? (2) Can SPRINT agents execute
unseen language instructions zero-shot? (3) Does _language_ pre-training lead
to better generalization to unseen environments than unsupervised pre-
training?

####  4.1 Experimental Setup

Figure 5: Left: ALFRED provides a rich set of long-horizon, meaningful tasks
and a dataset of 6.6k language-annotated demos (figure drawn w/ permission
from Shridhar et al. (2020)). We introduce the ALFRED-RL Benchmark which tests
finetuning of RL agents on unseen tasks and scenes. Right: Our Jaco robot arm
with RGB image-based control.

We evaluate our approach on two image-based control environments (see Figure
5): ALFRED-RL, a new RL benchmark we develop in the popular ALFRED household
task simulator (Shridhar et al., 2020), and a real robot kitchen manipulation
task with a Jaco 2 robot arm.

ALFRED-RL Benchmark. Our goal is to compare different pre-training approaches
on a diverse set of semantically meaningful, long-horizon tasks. Yet, existing
multi-task RL environments typically evaluate only on short-horizon or
semantically meaningless tasks (Yu et al., 2019; Mees et al., 2022). Thus, we
introduce a new RL benchmark based on the ALFRED household task simulator
(Shridhar et al., 2020). While ALFRED abstracts away low-level agent control
into discrete actions like “pick up” or “turn left,” its 100+ rich indoor
scenes with many interactable objects allow to evaluate an agent’s
capabilities for solving long-horizon household tasks from a rich task
distribution. Although the original benchmark focuses on imitation learning,
we extend it to support RL training of agents through an OpenAI gym interface
with $300\times 300$ egocentric RGB observations and an action space
consisting of 12 discrete action choices and 82 interactable object types
(Pashevich et al., 2021). We create three evaluation task sets that test
progressively more challenging axes of generalization: _EVAL INSTRUCT_ uses
unseen human-generated instructions on familiar scenes, _EVAL LENGTH_ uses
tasks that are longer than any observed in pre-training, testing “stitching”
capabilities, and _EVAL SCENE_ uses tasks in unseen floorplans. For more
details about ALFRED-RL and evaluation set construction, see Appendix, Section
C.1.2.

Real-World Robot Kitchen Manipulation. To evaluate pre-training approaches on
end-to-end _low-level_ robot control, we design a set of stylized kitchen
manipulation tasks with a Kinova Jaco 2 robot arm. The policy’s inputs are RGB
images from a wrist-mounted and a third-person camera and it produces
continuous end-effector displacement actions and a discrete gripper
open/stay/close action at a control frequency of 10Hz. We collect a dataset of
329 long-horizon trajectories via human teleoperation, each consisting of
multiple language-annotated sub-trajectories like “pick up the apple fruit,”,
“place the black bowl in the dish rack,” etc. For evaluation, we construct
three long-horizon tasks, sequencing 2 to 8 “primitive skills” like the ones
mentioned above, in environment configurations that are unseen in the pre-
training data. We collect 25 demonstrations for each of the three tasks to
evaluate offline fine-tuning performance of different pre-trained policies.

Comparisons. We compare SPRINT against common policy pre-training approaches:
behavioral cloning and offline goal-conditioned RL: Language-conditioned BC
(L-BC) (Jang et al., 2021; Lynch and Sermanet, 2021): Behavior cloning (BC)
conditioned on the individual language instructions, Episodic Transformers
(ET) (Pashevich et al., 2021): BC conditioned on sequences of language
instructions – ET is the best-performing end-to-end learned policy on the
ALFRED leaderboard that _does not_ use privileged domain knowledge like hand-
engineered policies or voxel maps, Actionable Models (AM) (Chebotar et al.,
2021): Goal-conditioned offline RL with randomly sampled goal observations
from the same training data as SPRINT.

Implementation Details. All methods use the same transformer-based
architecture and hyperparameters where possible and have access to the same
training data $\mathcal{D}^{L}$. For more implementation details, see appendix
Section B. All results are means and standard deviations over 3 seeds.

Figure 6: Evaluation results on the ALFRED-RL benchmark. Left: Zero shot
performance on the EVALINSTRUCT and EVALLENGTH task sets. SPRINT (green) is
able to complete substantially more subtasks than prior pre-training
approaches. Middle: Breakdown of zero shot performance by task length. SPRINT
performs well on challenging, long-horizon tasks. Numerical results in
appendix Table 4. Right: Finetuning performance in unseen floor plans of the
EVALSCENE task set. SPRINT learns tasks in new floor plans more effectively by
reaching higher peak performance.

####  4.2 SPRINT Solves Long-Horizon Tasks Zero-Shot

We first test the effectiveness of SPRINT’s pre-training by analyzing zero-
shot performance across 100 unseen tasks in the EVALINSTRUCT evaluation set.
We report results in Figure 6 (left). Our approach, SPRINT, achieves 2-8x
higher zero-shot task performance than prior pre-training approaches AM and
L-BC. Even though ET too trains to condition on long-horizon instruction
sequences like SPRINT, ours still outperforms it overall by 2x. To better
understand the differences between the methods, we report the breakdown of
returns by length of the evaluation task in Figure 6 (middle). We find that
L-BC, ET, and SPRINT achieve similar performance on length 1 tasks. However,
on long-horizon tasks, SPRINT achieves much higher returns than all baselines
since it can leverage the language-model to automatically generate longer-
horizon pre-training tasks. In contrast, standard L-BC approaches train only
on the human-provided, shorter-horizon annotations and thus cannot zero-shot
perform long-horizon tasks. Similar to our approach, AM trains to reach long-
horizon goals during pre-training but the results in Figure 6 (left) show that
its pre-training with goal-state conditioning is _less_ effective than our
language-conditioned pre-training. These results also hold for the EVALLENGTH
task set, which tests generalization to task horizons beyond the ones seen
during training. On these most challenging tasks, SPRINT outperforms the best
baseline by 2.5x (see appendix Figure 9 for an example trajectory, Section D.2
for qualitative comparisons).

####  4.3 SPRINT Agents Finetune Effectively in Unseen Environments

ALFRED-RL. We test SPRINT’s finetuning performance to unseen tasks on the most
challenging EVALSCENE task set in unseen household floor plans with 50k
environment interactions. This corresponds to a realistic scenario in which an
agent is placed in a new household environment and needs to leverage skills
learned during pre-training to solve new tasks with minimal environment
interaction. To implement finetuning for SPRINT and AM, we condition the
policy on a language instruction or goal image from the target task
respectively and then run IQL with online data collection. For L-BC and ET, we
first pre-train a language-conditioned critic with IQL on the pre-training
dataset and then finetune both the policy and critic with online IQL.

We report finetuning results in Figure 6 (right), with qualitative examples in
Appendix, Section D.2. SPRINT quickly achieves higher downstream task return
than the best prior work. Specifically, L-BC converges to lower peak
performance than SPRINT. ET performs the worst across all comparisons, perhaps
because transferring from instruction sequences to high-level task
descriptions is challenging. Meanwhile, AM performs similarly to L-BC,
possibly because unseen goal states are more difficult to learn from. In
contrast, SPRINT’s pre-training with language conditioning allows for
effective transfer even to unseen environments since the semantics of the
tasks transfer well: the language description “place cup in coffee machine”
transfers to many environments while the goal image for the same task might
look very different. Thus, pre-training with language instructions can enable
better transfer for learning tasks in new environments than pre-training to
reach goal states.

Real Robot. We also measure finetuning performance on an unseen environment on
our real robot setup. We evaluate on three tasks consisting of 2, 4, and 8
subgoals, respectively, e.g., Serve milk in the bowl and butter and baked
bread in the plate (see Section C.2 for task details). We collect 25
demonstrations per task for offline finetuning. We compare SPRINT against
L-BC, a version of L-BC trained on full sequences of concatenated language
instructions (L-BC Composite), and a method that is trained only on the
downstream task demonstrations (No pre-train).

Table 1: Success rates and number of subgoals completed after fine-tuning on
the tabletop arrangement displayed on the left with unseen object
combinations, averaged over 5 trials.

| Length 2 | Length 4 | Length 8
---|---|---|---
Method | Success | # Tasks | Success | # Tasks | Success | # Tasks
SPRINT (ours) | 100% | 2.0 | 60% | 3.4 | 40% | 6.2
L-BC Composite | 100% | 2.0 | 40% | 2.8 | 20% | 5.2
L-BC | 100% | 2.0 | 40% | 0.4 | 0% | 2.0
No pre-train | 0% | 1.0 | 0% | 0.0 | 0% | 0.0

Results in Table 1 demonstrate that No Pre-train performs poorly, indicating
that pre-training is necessary in this difficult real-world setup. Among the
other pre-training methods, SPRINT achieves the best success rates and
completes the most subgoals on all tasks. Compared to L-BC Composite, SPRINT
achieves higher returns and success rates on the more challenging Length 4 and
8 tasks. See Figure 7 for an example of a length 8 task evaluation and
appendix Section D.2 for more visualizations.

Figure 7: Successful rollout of a SPRINT agent offline finetuned on 25 demos
of the challenging task above with object combinations not in the pre-training
data. SPRINT solves all 8 tasks in sequence.

####  4.4 Ablation Studies

Table 2: Ablations. SPRINT achieves the highest return with both objectives.

Ablation | EVALINSTRUCT | EVALLENGTH
---|---|---
SPRINT (ours) | 1.92 $\pm$ 0.01 | 4.31 $\pm$ 0.51
SPRINT w/o Chain | 1.71 $\pm$ 0.14 | 4.10 $\pm$ 0.15
SPRINT w/o LLM-agg | 0.37 $\pm$ 0.01 | 0.19 $\pm$ 0.11

We verify the effectiveness of the components of our approach, with the
following ablations: SPRINT w/o chain removes cross-trajectory chaining
(Section 3.3), instead trains only on within-trajectory human-provided and
LLM-aggregated tasks; SPRINT w/o LLM-agg additionally removes the LLM
aggregation (Section 3.2), thus trains offline RL agent only on the human-
provided task annotations. We report zero-shot evaluation results on ALFRED in
Table 2. The results show that each component of our approach improves zero-
shot evaluation performance. We observe a particularly large performance loss
when removing the LLM aggregation of pre-training tasks. This underlines the
importance of leveraging the world knowledge in LLMs for automatically
generating long-horizon training tasks.

###  5 Discussion and Limitations

We presented SPRINT, an approach for scalable agent pre-training that
automatically generates training tasks for offline RL via LLM relabeling and
cross-trajectory skill chaining. We demonstrated that SPRINT pre-training
leads to higher zero-shot and finetuning performance on diverse household
tasks in the ALFRED simulator and on real-robot kitchen manipulation tasks.

Limitations. Currently, SPRINT can _only_ leverage language-annotated data.
Investigating approaches that can _combine_ language-annotated and unannotated
data for effective pre-training is an interesting future direction.
Furthermore, the cross-trajectory chaining objective of SPRINT randomly
concatenates sentences together, regardless of whether their combination is
semantically meaningful. Future work can investigate filtering mechanisms for
chaining to only include meaningful tasks. Finally, the development of real
robot benchmarks with rich sets of long-horizon tasks is an open challenge.

### Acknowledgements

We would like to thank Shivin Dass, Sidhant Kaushik, Laura Smith, Siddarth
Verma, and Jullian Yapeter for assisting with task instruction labeling. We
thank Taewook Nam for assisting with the actionable models implementation we
based some of our code on, Xiang Ren for detailed feedback on earlier versions
of the paper, and Yevgen Chebotar for helping us tune the actionable models
baseline. Finally, we thank all of the CLVR lab members at KAIST and USC for
their constructive feedback.

This work was supported by Institute of Information & Communications
Technology Planning & Evaluation (IITP) grants (No.2019-0-00075, Artificial
Intelligence Graduate School Program, KAIST; No.2022-0-00077, AI Technology
Development for Commonsense Extraction, Reasoning, and Inference from
Heterogeneous Data) and National Research Foundation of Korea (NRF) grant
(NRF-2021H1D3A2A03103683), funded by the Korean government (MSIT).

### References

  * Sutton et al. [1999] Richard S. Sutton, Doina Precup, and Satinder Singh.  Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning.  _Artificial Intelligence_ , 112:181–211, 1999.
  * Schaal [2006] Stefan Schaal.  _Dynamic Movement Primitives - A Framework for Motor Control in Humans and Humanoid Robotics_.  Springer Tokyo, 2006.
  * Hausman et al. [2018] Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller.  Learning an embedding space for transferable robot skills.  In _ICLR_ , 2018.
  * Lynch et al. [2020] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet.  Learning latent plans from play.  In _CoRL_ , 2020.
  * Pertsch et al. [2020] Karl Pertsch, Youngwoon Lee, and Joseph J. Lim.  Accelerating reinforcement learning with learned skill priors.  In _CoRL_ , 2020.
  * Haarnoja et al. [2023] Tuomas Haarnoja, Ben Moran, Guy Lever, Sandy H. Huang, Dhruva Tirumala, Markus Wulfmeier, Jan Humplik, Saran Tunyasuvunakool, Noah Y. Siegel, Roland Hafner, Michael Bloesch, Kristian Hartikainen, Arunkumar Byravan, Leonard Hasenclever, Yuval Tassa, Fereshteh Sadeghi, Nathan Batchelor, Federico Casarini, Stefano Saliceti, Charles Game, Neil Sreendra, Kushal Patel, Marlon Gwira, Andrea Huber, Nicole Hurley, Francesco Nori, Raia Hadsell, and Nicolas Heess.  Learning agile soccer skills for a bipedal robot with deep reinforcement learning, 2023.
  * Mees et al. [2022] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard.  Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks.  _IEEE Robotics and Automation Letters (RA-L)_ , 7(3):7327–7334, 2022.
  * Lynch and Sermanet [2021] Corey Lynch and Pierre Sermanet.  Language conditioned imitation learning over unstructured data.  _Robotics: Science and Systems_ , 2021.  URL https://arxiv.org/abs/2005.07648.
  * Lynch et al. [2022] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence.  Interactive language: Talking to robots in real time.  _arXiv preprint arXiv:2210.06407_ , 2022.
  * Brohan et al. [2022] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al.  Rt-1: Robotics transformer for real-world control at scale.  _arXiv preprint arXiv:2212.06817_ , 2022.
  * Shridhar et al. [2020] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox.  ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.  URL https://arxiv.org/abs/1912.01734.
  * Andreas et al. [2017a] Jacob Andreas, Dan Klein, and Sergey Levine.  Learning with latent language.  In _North American Chapter of the Association for Computational Linguistics_ , 2017a.
  * Nair et al. [2022] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta.  R3m: A universal visual representation for robot manipulation.  In _CoRL_ , 2022.
  * Fan et al. [2022] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar.  Minedojo: Building open-ended embodied agents with internet-scale knowledge.  _arXiv preprint arXiv: Arxiv-2206.08853_ , 2022.
  * Branavan et al. [2009] Satchuthananthavale RK Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay.  Reinforcement learning for mapping instructions to actions.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 82–90, 2009.
  * Andreas et al. [2017b] Jacob Andreas, Dan Klein, and Sergey Levine.  Modular multitask reinforcement learning with policy sketches.  In _International Conference on Machine Learning_ , 2017b.
  * Huang et al. [2022a] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  _arXiv preprint arXiv:2201.07207_ , 2022a.
  * Ahn et al. [2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng.  Do as i can and not as i say: Grounding language in robotic affordances.  In _arXiv preprint arXiv:2204.01691_ , 2022.
  * Huang et al. [2022b] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter.  Inner monologue: Embodied reasoning through planning with language models.  In _arXiv preprint arXiv:2207.05608_ , 2022b.
  * Singh et al. [2023] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg.  Progprompt: Generating situated robot task plans using large language models.  In _ICRA_ , 2023.
  * Colas et al. [2020] Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Frier, Peter Dominey, and Pierre-Yves Oudeyer.  Language as a cognitive tool to imagine goals in curiosity driven exploration.  _Advances in Neural Information Processing Systems_ , 2020.
  * Cideron et al. [2020] Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin.  Higher: Improving instruction following with hindsight generation for experience replay.  In _2020 IEEE Symposium Series on Computational Intelligence (SSCI)_ , pages 225–232. IEEE, 2020.
  * Li et al. [2022] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu.  Pre-trained language models for interactive decision-making.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=FWMQYjFso-a.
  * Ijspeert et al. [2002] Auke Jan Ijspeert, Jun Nakanishi, and Stefan Schaal.  Movement imitation with nonlinear dynamical systems in humanoid robots.  In _Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292)_ , volume 2, pages 1398–1403. IEEE, 2002.
  * Theodorou et al. [2010] Evangelos Theodorou, Jonas Buchli, and Stefan Schaal.  Reinforcement learning of motor skills in high dimensions: A path integral approach.  In _2010 IEEE International Conference on Robotics and Automation_ , pages 2397–2403. IEEE, 2010.
  * Hester et al. [2018] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al.  Deep q-learning from demonstrations.  In _Association for the Advancement of Artificial Intelligence_ , 2018\.
  * Levine et al. [2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Peng et al. [2019] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019.  URL https://arxiv.org/abs/1910.00177.
  * Singh et al. [2020] Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine.  Cog: Connecting new skills to past experience with offline reinforcement learning.  _Conference on Robot Learning_ , november 2020.
  * Nair et al. [2020] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine.  Accelerating online reinforcement learning with offline datasets.  _arXiv preprint arXiv:2006.09359_ , 2020.
  * Kostrikov et al. [2022] Ilya Kostrikov, Ashvin Nair, and Sergey Levine.  Offline reinforcement learning with implicit q-learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=68n2s9ZJWF8.
  * Duan et al. [2016] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel.  RL 2: Fast reinforcement learning via slow reinforcement learning.  _arXiv preprint arXiv:1611.02779_ , 2016.
  * Finn et al. [2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  _ICML_ , 2017.
  * Rakelly et al. [2019] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen.  Efficient off-policy meta-reinforcement learning via probabilistic context variables.  In _ICML_ , 2019.
  * Nam et al. [2022] Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph J. Lim.  Skill-based meta-reinforcement learning.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Achiam et al. [2018] Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel.  Variational option discovery algorithms.  _arXiv_ , 2018.
  * Eysenbach et al. [2019] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine.  Diversity is all you need: Learning skills without a reward function.  In _ICLR_ , 2019.
  * Sharma et al. [2019] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman.  Dynamics-aware unsupervised discovery of skills.  _arXiv_ , abs/1907.01657, 2019.
  * Ajay et al. [2020] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum.  Opal: Offline primitive discovery for accelerating offline reinforcement learning.  _arXiv preprint arXiv:2010.13611_ , 2020.
  * Singh et al. [2021] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine.  Parrot: Data-driven behavioral priors for reinforcement learning.  _ICLR_ , 2021.
  * Mendonca et al. [2021] Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak.  Discovering and achieving goals via world models, 2021.
  * Chebotar et al. [2021] Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan C Julian, Chelsea Finn, and Sergey Levine.  Actionable models: Unsupervised offline reinforcement learning of robotic skills.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 1518–1528. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/chebotar21a.html.
  * Yu et al. [2023] Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Brian Ichter, Karol Hausman, and Fei Xia.  Scaling robot learning with semantically imagined experience.  In _arXiv preprint arXiv:2302.11550_ , 2023.
  * Chen et al. [2023] Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar.  Genaug: Retargeting behaviors to unseen situations via generative augmentation.  In _RSS_ , 2023.
  * Mandi et al. [2023] Zhao Mandi, Homanga Bharadhwaj, Vincent Moens, Shuran Song, Aravind Rajeswaran, and Vikash Kumar.  Cacti: A framework for scalable multi-task multi-scene visual imitation learning, 2023.
  * Xiao et al. [2023] Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson.  Robotic skill acquistion via instruction augmentation with vision-language models.  In _Proceedings of Robotics: Science and Systems_ , 2023.
  * Gupta et al. [2019] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman.  Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning.  _CoRL_ , 2019.
  * Ebert et al. [2022] Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine.  Bridge data: Boosting generalization of robotic skills with cross-domain datasets.  In _RSS_ , 2022.
  * Pertsch et al. [2021] Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J. Lim.  Demonstration-guided reinforcement learning with learned skills.  In _CoRL_ , 2021.
  * Kaelbling [1993] Leslie Pack Kaelbling.  Learning to achieve goals.  In _IN PROC. OF IJCAI-93_ , pages 1094–1098. Morgan Kaufmann, 1993\.
  * Schaul et al. [2015] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver.  Universal value function approximators.  In Francis Bach and David Blei, editors, _Proceedings of the 32nd International Conference on Machine Learning_ , volume 37 of _Proceedings of Machine Learning Research_ , pages 1312–1320, Lille, France, 07–09 Jul 2015. PMLR.  URL https://proceedings.mlr.press/v37/schaul15.html.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.  URL https://arxiv.org/abs/1810.04805.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.  URL https://arxiv.org/abs/2005.14165.
  * Wang and Komatsuzaki [2021] Ben Wang and Aran Komatsuzaki.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
  * Rae et al. [2021] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis & insights from training gopher, 2021.  URL https://arxiv.org/abs/2112.11446.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models, 2022.  URL https://arxiv.org/abs/2203.15556.
  * Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  Opt: Open pre-trained transformer language models, 2022.  URL https://arxiv.org/abs/2205.01068.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways, 2022.  URL https://arxiv.org/abs/2204.02311.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * Sutton and Barto [2018] Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Eysenbach et al. [2022] Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine.  Contrastive learning as goal-conditioned reinforcement learning, 2022\.  URL https://arxiv.org/abs/2206.07568.
  * Yu et al. [2019] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, 2019.
  * Pashevich et al. [2021] Alexander Pashevich, Cordelia Schmid, and Chen Sun.  Episodic Transformer for Vision-and-Language Navigation.  In _ICCV_ , 2021.
  * Jang et al. [2021] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn.  BC-z: Zero-shot task generalization with robotic imitation learning.  In _5th Annual Conference on Robot Learning_ , 2021.  URL https://openreview.net/forum?id=8kbp23tSGYv.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016\.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255, 2009.  doi: 10.1109/CVPR.2009.5206848.
  * Snell et al. [2023] Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine.  Offline rl for natural language generation with implicit language q learning, 2023.
  * Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2019\.  URL https://arxiv.org/abs/1908.10084.
  * Zhao et al. [2023] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn.  Learning fine-grained bimanual manipulation with low-cost hardware.  _arXiv preprint arXiv:2304.13705_ , 2023.
  * Perez et al. [2018] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville.  FiLM: Visual Reasoning with a General Conditioning Layer.  In _Association for the Advancement of Artificial Intelligence_ , 2018\.
  * Dass et al. [2023] Shivin Dass, Jullian Yapeter, Jesse Zhang, Jiahui Zhang, Karl Pertsch, Stefanos Nikolaidis, and Joseph J. Lim.  Clvr jaco play dataset, 2023.  URL https://github.com/clvrai/clvr_jaco_play_dataset.

##

Algorithm 1 SPRINT Algorithm

1:Dataset $\mathcal{D}^{L}$ w/ language instruction labels, LLM

2:AggregateSkills($\mathcal{D}^{L}$, LLM)$\triangleright$ Automated LLM skill
relabeling (Sec. 3.2)

3:while not converged do

4: $\tau_{z}\leftarrow\mathcal{D}^{L}$: Sample an annotated skill
(sub-)trajectory

5: Train offline RL on $\tau_{z}$

6: $\tau_{\text{agg}_{1}},\tau_{\text{agg}_{2}}\leftarrow$
CrossAggregateSkills($\mathcal{D}^{L}$, LLM) $\triangleright$ Cross-traj.
aggregation (Sec. 3.3)

7: Train offline RL on $\tau_{\text{agg}_{1}},\tau_{\text{agg}_{2}}$

8:end while

9:

10:procedure AggregateSkills($\mathcal{D}^{L}$, LLM)

11: for composite trajectory $\tau_{\bar{z}}$ in $\mathcal{D}^{L}$ do

12: Separate $\tau_{\bar{z}}$ into language annotated sub-trajectories
$\left[\tau_{z_{1}},...,\tau_{z_{N}}\right]$

13: for all adjacent sub-trajectories
$\left[\tau_{z_{i}}...\tau_{z_{j}}\right]$ do

14: Assign name from LLM: $\text{LLM}(z_{i}...z_{j})=\hat{z}_{i:j}$

15: $\tau_{\hat{z}_{i:j}}\leftarrow$ Concat
$\left[\tau_{z_{i}},...,\tau_{z_{j}}\right]$ and relabel with $\hat{z}_{i:j}$

16: $T\leftarrow$ length of $\tau_{\hat{z}_{i:j}}$

17: $R(s_{T},a_{T},\hat{z}_{i:j})=1$ $\triangleright$ Label last reward with 1
(Eq. 1).

18:
$\mathcal{D}^{L}=\mathcal{D}^{L}\,\cup\,\left\\{\tau_{\hat{z}_{i:j}}\right\\}$

19: end for

20: end for

21:end procedure

22:

23:procedure CrossAggregateSkills($\mathcal{D}^{L}$, LLM)

24: Sample random (sub)trajectories
$\tau_{z_{1}},\tau_{z_{2}}\sim\mathcal{D}^{L}$ with lengths $T_{1},T_{2}$

25: Assign new name : $\hat{z}=$ “$\\{z_{1}\\}.\\{z_{2}\\}$”

26: $\tau_{\text{agg}_{1}}\leftarrow[\left(s_{0},a_{0},0,\hat{z}\right),...,$
$\left(s_{T_{1}},a_{T_{1}},Q^{\pi}(s_{T},a_{T}|z_{2}),\hat{z}\right)]$
$\triangleright$ Relabel reward w/ Eq. 3

27: $\tau_{\text{agg}_{2}}\leftarrow[\left(s_{0},a_{0},0,\hat{z}\right),...,$
$\left(s_{T_{2}},a_{T_{2}},1,\hat{z}\right)]$ $\triangleright$ Relabel reward
w/ Eq. 3.

28: return $\tau_{\text{agg}_{1}},\tau_{\text{agg}_{2}}$

29:end procedure

Instructions: give a high-level description for the following steps describing
common household tasks.
Task Steps: 1\. Pick up the keys on the center table. 2\. Put the keys in the
box. 3\. Pick up the box with keys. 4\. Put the box with keys on the sofa
close to the newspaper.
Summary: Put the box with keys on the sofa.
Task Steps: 1\. Pick up the knife from in front of the tomato. 2\. Cut the
lettuce on the counter. 3\. Set the knife down on the counter in front of the
toaster. 4\. Pick up a slice of the lettuce from the counter. 5\. Put the
lettuce slice in the refrigerator. take the lettuce slice out of the
refrigerator. 6\. Set the lettuce slice on the counter in front of the
toaster.
Summary: Put a cooled slice of lettuce on the counter.
Task Steps: 1\. Pick up the book on the table, in front of the chair. 2\.
Place the book on the left cushion of the couch.
Summary: Put a book on the couch.
Task Steps: 1\. Pick up the fork from the table. 2\. Put the fork in the sink
and fill the sink with water, then empty the water from the sink and remove
the fork. 3\. Put the fork in the drawer.
Summary: Put the cleaned fork in a drawer.
Task Steps: 1\. Take the box of tissues from the makeup vanity. 2\. Put the
tissues on the barred rack. 3\. Take the box of tissues from the top of the
toilet. 4\. Put the tissues on the barred rack.
Summary: Put the box of tissues on the barred rack.
Task Steps: 1\. Pick up the glass from the sink. 2\. Heat the glass in the
microwave. 3\. Put the glass on the wooden rack.
Summary: Put a heated glass on the wooden rack.
Task Steps: 1\. Pick up the box from the far side of the bed. 2\. Hold the box
and turn on the lamp.
Summary: Look at the box under the lamp light.
Task Steps: 1: [SKILL 1]. 2: [SKILL 2]. 3: [SKILL 3]. … N: [SKILL N].
Summary: Figure 8: The full prompt that we use for summarization. Following
the suggestions of Ahn et al. [2022] for prompt design, we explicitly number
each step. The LLM completion task begins after “Summary:”. For brevity, we
omit the new line characters between all numbered steps. Figure 9: Example
successful task execution of our pre-trained SPRINT agent for the challenging
“Warm up a piece of apple” task. Successful execution requires solving 8
subtasks in sequence and a total of 50 steps. This sequence of subtasks was
never observed in the training data. SPRINT uses cross-trajectory stitching
and LLM aggregation to learn unseen tasks.

###  Appendix A Large Language Model Prompt

We list the full large language model summarization prompt in Figure 8. The
examples in the prompt are fixed for all summarization queries. These examples
are selected from the ALFRED validation dataset (which is not otherwise used
in our work) at random: We spell out the primitive skill annotations in the
“Task Steps:” part of each prompt example. Then, the “Summary” for each of
these is the high-level, human-written annotation for that trajectory from
ALFRED. We repeatedly sampled these trajectories until each example mentioned
a different object to prevent biasing the LLM towards certain types of
objects.

We note that the “Look at the box under the lamp light” example is important
to make the LLM give reasonable summaries for similar tasks in ALFRED where
the agent picks something up and turns on a light. This is because most of the
human labels for turning on the lamp do not mention the object in the previous
step, making it difficult for the LLM to realize that the task has to do with
looking at the held object under a lamp.

###  Appendix B Baselines and Implementation

We implement IQL [Kostrikov et al., 2022] as the base offline RL algorithm for
all goal-conditioned offline RL pretraining baselines and ablations due to its
strong offline and finetuning performance on a variety of dense and sparse
reward environments. At a high level, IQL trains on in-distribution
$(s,a,s^{\prime},r,a^{\prime})$ tuples from the dataset rather than sampling a
policy for $a^{\prime}$ to ensure that the Q and value functions represent
accurate estimated returns constrained to actions in the dataset. The value
function is trained with an expectile regression loss controlled by a
hyperparameter $\tau$, where $\tau=0.5$ results in standard mean squared error
loss and $\tau\rightarrow 1$ approximates the max operator, resulting in a
more optimistic value function that can better “stitch” together trajectories
to obtain distant reward in sparse reward settings. The IQL policy is trained
to maximize the following objective:

$e^{\beta\left(Q(s,a)-V(s)\right)}\log\pi(a|s),$

which performs advantage-weighted regression [Peng et al., 2019] with an
inverse temperature term $\beta$. In practice, the exponential advantage term
is limited to a maximum value to avoid numerical overflow issues. We detail
shared training and implementation details below, with method-specific
information and hyperparameters in the following subsections.

####  B.1 ALFRED Details

Observation space. The state space of the ALFRED environment consists of
$300\times 300$ RGB images. Following the baseline method in ALFRED [Shridhar
et al., 2020], we preprocess these images by sending them through a frozen
ResNet-18 encoder [He et al., 2016] pretrained on ImageNet [Deng et al.,
2009]. This results in a $512\times 7\times 7$ feature map that we use as the
observation input to all networks.

Action space. The agent chooses from 12 discrete low-level actions. There are
5 navigation actions: MoveAhead, RotateRight, RotateLeft, LookUp, and LookDown
and 7 interaction actions: Pickup, Put, Open, Close, ToggleOn, ToggleOff, and
Slice. For interaction actions the agent additionally selects one of 82 object
types to interact with, as defined by Pashevich et al. [2021]. In total, the
action space consists of $5+7*82=579$ discrete action choices. Note that this
action space definition is different from the action space in Shridhar et al.
[2020], which used a pixel-wise mask output to determine the object to
interact with. In contrast to Shridhar et al. [2020] we aim to train agents
with _reinforcement learning_ instead of imitation learning and found the
discrete action parametrization more amenable to RL training than dense mask
outputs. For all methods, due to the large discrete action space, we perform
some basic action masking to prevent agents from taking actions that are not
possible. For example, we do not allow the agent to Close objects that aren’t
closeable nor can they ToggleOn objects that can’t be turned on.

Policy and critic networks. For all baselines and SPRINT base models are
implemented on the transformer architecture proposed in Episodic Transformers
[Pashevich et al., 2021]. For offline RL methods (AM, SPRINT) we follow the
advice of [Snell et al., 2023] and parameterize both Q functions and the Value
function of IQL as separate output heads of one transformer backbone that is
used for all critic networks. We train both policies and critic transformer
networks with an observation history of up to 16 previous observations, each
one being processed by a convolutional network before being flattened into a
768-dim feature. Our discrete policy has two output heads of size 12 and 82
for the action and interaction object outputs respectively. Critic networks
are conditioned on both the observation and the discrete action output of the
policy. In networks with language input, words are individually tokenized and
the entire language instruction is fed to the policy and critic networks and
embedded into a sequence of learned 768-dim embeddings, one for each token. We
perform cross-attention between all network inputs: language embeddings,
previous observation embeddings, and the previous action where applicable. The
output of this cross-attention mechanism is then transformed by linear layers
into the final output for the network.

Pre-training hyperparameters. A hyperparameter search was performed first on
the language-conditioned BC-baseline to optimize for training accuracy. These
hyperparameters were carried over to the IQL implementation, and another
search for IQL-specific hyperpameters were performed on a baseline IQL policy
conditioned on language instructions. With these parameters fixed, we
performed one more hyperparameter search specific to Actionable Models but for
the final implementation of SPRINT we re-used the same hyperparameters and
only selected SPRINT-specific parameters heuristically. Hyperparameters for
each method are detailed in separate tables. Shared hyperparameters for all
methods (where applicable) are listed below:

Param | Value
---|---
Batch Size | 1024
# Training Batches | 140k
Learning Rate | 1e-4
Optimizer | AdamW
Dropout Rate | 0.1
Weight Decay | 0.1
Discount $\gamma$ | 0.97
Q Update Polyak Averaging Coefficient | 0.005
Policy and Q Update Period | 1/train iter
Nonlinearity | ReLU
IQL Advantage Clipping | [0, 100]
IQL Advantage Inverse Temperature $\beta$ | 5
IQL Quantile $\tau$ | 0.8
Maximum Transformer Context Length | 16

Finetuning details and hyperparameters. We fine-tune by running IQL on online-
collected data without any of the chaining or aggregation steps. For all
models, we finetune by sampling old pre-training data and newly collected data
at a ratio of 70%/30%. Without this mixed batch training, we found the
transformer-based networks to overfit to the new data, something we did not
see when experimenting with standard MLPs. The newly collected data is also
sampled from two separate buffers at equal proportions, one which contains
trajectories that received at least 1 reward (i.e., completed one sub-task)
and one that contains trajectories that received none. This is another
transformer-specific adaptation we had to make for the models to train stably
with IQL on online-collected data.

Each method is finetuned on every task in the EVALSCENE task set individually;
that is, we pre-train once and then finetune policies for each task in the
task set. We then average returns over all tasks, then report metrics averaged
over all random seeds. For each task, we define a maximum rollout time horizon
of 2 timesteps per environment action required by an expert ALFRED task
planner.

When not specified, finetuning parameters are identical to pre-training
parameters. Finetuning hyperparameters are specified below:

Param | Value
---|---
# Initial Rollouts | 50
Training to Env Step Ratio | 20
$\epsilon$ in $\epsilon$-greedy action sampling | 0
Policy action sampling | True
# Parallel Rollout Samplers | 10

####  B.2 Real Robot Implementation Details

The real-world environment uses a Kinova Jaco 2 robot arm. Below we detail the
implementation and training details specific to the real robot environment.

###### Observation space.

The view observations consist of 224×224×3 cropped RGB images, which are
captured from a Logitech Pro Webcam C920 for the third-person view and an
Intel RealSense D435 for the wrist-view. We leverage a pretrained R3M [Nair et
al., 2022] model to encode each view observation. Additionally, the state
representation includes the robot’s end-effector position, velocity, and
gripper state. Notably, the end-effector position and velocity are two
continuous vectors, while the gripper state is represented as a one-hot
vector, indicating OPEN, CLOSE, or NOT MOVE. To form the observation for the
policy, we concatenate the embedded RGB input with state information.

To condition on language inputs, we use a pre-trained sentence embedder to
embed the entire language annotation into a vector of size 384 (as our network
backbone is an RNN instead of a transformer). This embedding is done with the
all-MiniLM-L12-v2 pre-trained embedding model from the SentenceTransformers
package [Reimers and Gurevych, 2019].

The total state input dimension is: 2048 (third-person R3M) + 2048 (wrist R3M)
+ 15 (Jaco state input) + 384 (language embedding) = 4495.

###### Action space.

The robot action space comprises the changes in the end effector position
between each time stamp, along with the gripper opening/closing commands.
These actions are transmitted to the robot at a frequency of 10 Hz and
interpreted as desired joint poses using PyBullet’s inverse kinematics module.

###### Network architecture and training.

Similar to [Zhao et al., 2023], we use the Action Chunking method to train an
autoregressive policy. Specifically, our policy employs an LSTM model to
predict the next 15 actions, given the initial observation as input, i.e.,
$\pi$($a_{t:t+15}$$\mid$$s_{t}$). Our Q and Value networks are also recurrent,
predicting per-timestep rewards for each action in the sequence. Just like the
policy, they also only see the observation before the action sequence starts.

Because of the fact that the gripper action is discrete and heavily imbalanced
in class distribution, we weigh the gripper action loss inversely
proportionally to the number of examples in each class.

###### Pre-training details and hyperparameters.

We performed a heuristic hyperparameter search by first tuning the language-
conditioned BC baseline to be as effective as possible on zero-shot
evaluations of training tasks, then performed a small heuristic hyperparameter
for the SPRINT. Shared hyperparameters are detailed below:

Param | Value
---|---
Batch Size | 128
# Training Batches | 50k
Learning Rate | 5e-4
Optimizer | AdamW
Weight Decay | 0.1
Discount $\gamma$ | 0.99
Q Update Polyak Averaging Coefficient | 0.005
Policy and Q Update Period | 1/train iter
Nonlinearity | LeakyReLU(0.2)
IQL Advantage Clipping | [0, 100]
IQL Advantage Inverse Temperature $\beta$ | 5
IQL Quantile $\tau$ | 0.8
Action Chunking Length | 15

###### Fine-tuning details and hyperparameters.

We collect 25 demonstrations for each downstream task and perform individual
fine-tuning of the models for each task. In the case of the pre-trained models
(SPRINT, L-BC composite, and L-BC primitive), we conduct 500 epochs of fine-
tuning. As for the model without pre-training, we train 2000 epochs only on
the downstream task demonstrations. The fine-tuning/training hyperparameters
are identical to those for pre-training.

####  B.3 Language-conditioned Behavior Cloning

Our language-conditioned behavior cloning (L-BC) comparison method is inspired
by and replicates BC-Zero [Jang et al., 2021] and LangLfP [Lynch and Sermanet,
2021]. BC-Zero performs language imitation learning [Perez et al., 2018], and
both BC-Zero and LangLfP have an additional image/video-language alignment
objective. In BC-Zero, their video alignment objective aligns language
embeddings with videos of humans performing tasks related to those the BC-Zero
robot agent trains on. LangLfP’s image-language alignment objective allows
their policy to accept both image and natural language goals as input due to
only having a subset of their data labeled with hindsight language labels. As
we don’t have human videos of these tasks and our entire dataset is labeled
with language labels, we do not add a video or image alignment objective.

Hyperparameters for the L-BC baseline are identical to the shared parameters
above for both environments, where applicable.

###### ALFRED:

We implement L-BC by using the same architecture as described in the shared
details section above with just a single transformer policy network that
trains to maximize the log-likelihood of actions in the dataset. As our entire
dataset consists of expert trajectories, this baseline ideally learns optimal
actions for the instructions.

###### Real Robot:

L-BC is implemented with the action-chunked LSTM policy network to maximize
log-likelihood of actions in the dataset as described in the real robot
implementation details section above. Again the dataset consists of human
expert trajectories so L-BC should learn optimal actions for the given
instructions.

####  B.4 Episodic Transformers

Episodic Transformers (ET) [Pashevich et al., 2021] trains a transformer
architecture on full sequences of ALFRED instructions with a behavior cloning
objective. This is currently state of the art in the “Seen Path-Length
Weighted Success Rate” evaluation metric on the ALFRED leaderboard. We adopted
the ET implementation from the official code repository.

For fair comparison, we make a few modifications to make it as close as
possible to SPRINT and the baselines: 1) we train it on the same dataset as
all baselines, so we do not generate new synthetic training data like the
original implementation Pashevich et al. [2021] since it assumes access to an
expert planner, 2) we encode visual frames with a Resnet-18 instead of
Resnet-50 backbone, the same we use for all other models, 3) we remove the
high-level goal specification from the input text tokens as we do not assume
access to those, and 4) we train the model for longer to match the number of
training steps for all methods.

####  B.5 Actionable Models (AM)

Actionable Models [Chebotar et al., 2021] pre-trains a goal-conditioned Q
function conditioned on randomly sampled image goals and also performs a goal-
chaining procedure very similar to our skill chaining procedure. We implement
AM by modifying the base IQL policy and critic networks to take in image goals
instead of natural language embeddings as goals. These goals are provided in
the same way as the observations as a sequence of 5 frames (the last 5 frames
in the trajectory) processed by a frozen ResNet-18.

To allow for fair comparison between our approach and AM, we implement AM with
the same powerful offline RL algorithm, IQL [Kostrikov et al., 2022], used in
our method. IQL ensures that the policy does not choose out-of-distribution
actions by using advantage-weighted regression on in-distribution actions for
policy extraction. With this, we found the conservative auxiliary loss AM adds
to push down Q-values for out-of-distribution actions to be unnecessary and
even hurtful to its overall performance, so we omit this additional loss term.

We also pre-train AM on the same long-horizon trajectories as those generated
by SPRINT during LLM-based skill aggregation. This ensures a fair comparison
in terms of the types and lengths of tasks seen during pre-training.

Finally, after consulting the authors of AM, we tried varying maximum
trajectory lengths when sampling random goals. We found that allowing random
goals to be sampled from anywhere within a trajectory resulted in the best
zero-shot evaluation performance for AM, so our numbers are reported with this
implementation detail.

####  B.6 SPRINT

The implementation details of SPRINT follow from the general discussions at
the top of this section. The key differences are in (1) language model skill
aggregation and (2) cross-trajectory skill chaining, detailed below.

###### LLM Skill Aggregation.

We perform LLM skill aggregation fully offline by iterating through every
trajectory and aggregating sequences of adjacent primitive skill sub-
trajectories. Assuming a trajectory with $N$ primitive skills, we select all
${N\choose 2}$ pairs of start and end skills and aggregate all instructions
from start to end with the LLM. With 73k original language-annotated sub-
trajectories in ALFRED, this procedure allows us to generate an additional
110k aggregated trajectories. We then add these trajectories to the original
dataset and train on the entire set.

On our real-world robot dataset, we start with $\sim$6k language-annotated
sub-trajectories and perform LLM skill aggregation on all pairs of
trajectories directly next to each other (restricting to a maximum of 2 skills
being aggregated at any time). We restrict aggregation in this manner because
each trajectory contains many sub-trajectories of play-like data where many of
the sub-trajectories are not related to each other. Aggregation doubles the
size of our dataset to almost $\sim$13k trajectories.

###### Cross-trajectory skill chaining.

We perform cross-trajectory skill chaining in-batch. Instead of sampling a
second trajectory to perform chaining on, we simply permute the batch indicies
to generate a set of randomly sampled second trajectories. Then, we perform a
second loss function update, in addition to the original update on the sampled
trajectories, with equal loss weighting, to apply the skill-chaining update.
We apply the chaining procedures from Eq. 3 in-batch. Empirically, we found
that cross-trajectory skill chaining works slightly better with the on-policy
Value function obtained through IQL, therefore we use state values at the
chaining targets instead of state-action Q-values.

SPRINT-specific hyperparameters follow:

Param | Value
---|---
LLM | LLAMA-13B [Touvron et al., 2023]
LLM Token Filtering Top-p | 0.9
LLM Token Sampling Temperature | 0.6

#####  B.6.1 Cross-trajectory chaining preserves the MDP.

When performing cross-trajectory chaining using Eq. 3, special care must be
taken to preserve the dynamics of the original Markov Decision Process (MDP).
When chaining together two trajectories $\tau_{A}$ and $\tau_{B}$, we
concatenate the two sentences of each trajectory together and relabel their
rewards with Eq. 3. The new language annotation used to chain together these
trajectories is the concatenation of the two sentences, implying that the
agent finishes skill (A) and then skill (B). However, we cannot concatenate
the two trajectories together into one longer trajectory, as doing so would
imply that the agent can instantaneously jump from the last state of skill (A)
to the first state of skill (B), which may not be possible. Therefore, we
instead treat the relabeled trajectories as separate trajectories with the
same language annotation (lines 36 and 37 of Algorithm 1).

However, this introduces two possible complications: 1) Language annotations
differing in structure from those in the original dataset, and 2) Possible
instruction ambiguity. We detail how these complications are resolved in
SPRINT below:

  1. 1.

Language annotations differing in structure. Language annotations produced by
the chaining procedure will result in annotations that implicitly skip certain
steps. For example, when chaining skill (A), “make the bed,” and skill (B),
“make a cup of coffee,” the resulting chained annotation will be “Make the
bed. Make a cup of coffee.” However to perform skill (B) the agent needs to
first move to the kitchen from the bedroom to make the cup of coffee, which is
skipped in this annotation. LLM-based skill aggregation (Section 3.2) helps
bridge this gap by summarizing long-horizon sequences while skipping certain
implied steps. For example, one real LLM summary summarized the sequence: “1:
Pick up the plaid pillow that is on the left end of the couch. 2: Place the
pillow on the ottoman” into the instruction “Place a plaid pillow on the
ottoman,” which skipped the step of picking up the pillow as it is implied
that you must do so before placing the pillow down. Using the LLM augments our
original dataset such that, in ALFRED, we have 2.5x the original data after
performing offline skill aggregation, and in the real robot manipulation
environment we have 2x the amount of original data. Therefore after performing
LLM aggregation, there are many examples of similar instructions to those used
for chained trajectories that imply certain steps without mentioning them
explicitly.

  2. 2.

Instruction ambiguity. When chaining trajectories, there will be some
ambiguity introduced as we do not have intermediate instructions for going
from the last state of A to the initial state of B (obtaining these
instructions requires additional human effort). This ambiguity is only present
in the states of trajectory A, as when training on trajectory B, the agent can
easily infer that the instructions for trajectory A are finished and the just
follow the instructions relevant for trajectory B. We believe that the effects
of the ambiguity on pre-training performance depends greatly on the given
dataset. In complex and diverse environments, hindsight-labeled annotations
should contain details specific to certain scenes, resolving this ambiguity.
In ALFRED, the annotations usually contain information about the specific
objects that the agent must interact with or locations that the agent must go
to. For example, annotations for rinsing mugs typically are of the form “clean
the MUG in the sink,” or annotations for picking up a candle will often say
something like “pick up the YELLOW CANDLE on the COUNTER,” highlighting
specific details regarding what the agent is supposed to do to complete the
trajectory.

###  Appendix C Dataset, Environment, and Task Details

####  C.1 ALFRED

(a) Skills per trajectory in the original ALFRED dataset.

(b) Skills per trajectory in the merged dataset.

(c) Actions per skill in the merged dataset.

Figure 10: Left: distribution of the number of skills in each trajectory in
the original ALFRED dataset. Middle: distribution of skills per trajectory in
the “merged” dataset with merged navigation skills. Right: distribution of
number of actions per skill in the “merged” dataset.

#####  C.1.1 Dataset Details

For training and evaluation we leverage the ALFRED benchmark and dataset
[Shridhar et al., 2020]. The ALFRED training dataset contains $\sim$6.6k
trajectories collected by an optimal planner following a set of 7 high-level
tasks with randomly sampled objects (e.g., pick up an object and heat it).
Each trajectory has at least three crowd-sourced sets of language instruction
annotations. Each trajectory consists of a sequence of 3-19 individually
annotated skills (see Figure 10, left). This results in a total of 141k
language-annotated skill trajectories.

However, nearly half of the language instructions in the ALFRED dataset are
navigation skill instructions like “turn left, then look up and walk to the
counter on the right”. To get a more balanced skill annotation dataset, we
merge all navigation skills with the skill that immediately follows them,
using only the annotation of the next skill. After this processing step, the
resulting dataset contains 73k language-annotated primitive skill
trajectories. After we merge the navigation skills, the average number of
skills in each trajectory is 3.5 skills per trajectory (Figure 10, middle),
and the average number of actions in each skill is 14.3 (Figure 10, right).

#####  C.1.2 Evaluation Tasks

Table 3: Evaluation Task Specifics. Note that the “number of env actions per task” corresponds to the number of environment actions the ALFRED expert planner required to complete that task. | EVALINSTRUCT | EVALLENGTH | EVALSCENE
---|---|---|---
Number of Tasks | 100 | 20 | 10
Task Lengths (# primitive skills) | [1, 2, 3, 4, 5, 6, 7] | [7, 8] | [1, 2, 3, 4, 5]
Min Number of Env Actions per Task | 1 | 34 | 2
Avg Number of Env Actions per Task | 39.1 | 60.9 | 46.6
Max Number of Env Actions per Task | 113 | 104 | 124
Figure 11: Data collection jupyter notebook page. Note that there is a “Skip”
button so that human annotators can skip an instruction sequence if they do
not feel it is semantically meaningful or easy to summarize.

###### Overview.

We evaluate agents through zero-shot policy evaluation and finetuning on three
sets of evaluation tasks in the ALFRED environment: (1) EVALINSTRUCT to
measure the ability of pre-trained agents to execute semantically meaningful
instructions at varied levels of abstraction, (2) EVALLENGTH to measure the
ability of agents to chain behaviors across multiple trajectories to solve
long tasks, and (3) EVALSCENE to evaluate generalization performance when
finetuning to unseen household floor plans. We did not use the official ALFRED
benchmark test sets to construct EVALSCENE since we require a task
demonstration to compute how many subtasks the agent solved; these
demonstrations are not given for the test set tasks. However, the tasks we
evaluate on generally are designed to be representative of the tasks in the
ALFRED test set: they test the agent on unseen instruction-scene combinations
and consist of varied-length, compositional tasks. Like the ALFRED test set,
our evaluation consists of long-horizon tasks that require sequential
execution of multiple subtasks.

###### Collecting evaluation task data.

The ALFRED dataset provides high-level language annotations for each of the
trajectories in the dataset. We could use these annotations as unseen task-
instructions to evaluate our agents. However, we found that the different
skills are not equally distributed across trajectories of different skill
lengths, e.g., most 2-skill trajectories perform pick-and-place tasks while
tasks involving heating skills only appear in length 7+ trajectories. To allow
evaluation with a less biased skill distribution, we create the EVALINSTRUCT
task set by randomly choosing a trajectory from the ALFRED dataset _and then
randomly sampling a subsequence of skills of a certain length_ from this
trajectory. To obtain a high-level language instruction that summarizes this
new subsequence, we crowd-source labels from human annotators. For labeling,
each annotator is presented with a remotely hosted Jupyter notebook interface
(see Figure 11). Whenever we by chance sample a full ALFRED trajectory for
annotation, we directly used the existing high-level annotation from the
ALFRED dataset. We annotate 80 trajectories with human annotators and combine
them with 20 randomly sampled single-skill trajectories, resulting in a total
of 100 evaluation tasks (see Figure 12 for example instructions). This results
in 20 tasks of length 1 skills, 20 tasks of length 2 skills, 20 tasks of
length 3 skills, 20 tasks of length 4 skills, and 20 tasks of lengths 5+ (5-7)
skills.

For EVALLENGTH, we randomly sampled 20 full trajectories from the ALFRED
dataset that had sequences of 7 or 8 skills (10 of length 7, 10 of length 8)
and removed these trajectories from the training dataset before performing
LLM-based skill aggregation. This ensures AM and SPRINT must perform skill
chaining to solve these tasks by ensuring that there were valid sequences of
skills to chain together to be able to solve these removed tasks. For example,
assume a (shortened for clarity) sampled skill sequence is “pick up apple,”
then “put apple in microwave”, then “slice the apple.” Then, either Actionable
Models or SPRINT can chain together the sub-trajectory associated with “pick
up apple” then “put apple in microwave” with the “slice the apple” sub-
trajectory to solve this task. These trajectories all had annotations from
ALFRED annotators, so we used those annotations directly (see Figure 14 for
example instructions).

Finally, for EVALSCENE, we collected a set of 10 full-length trajectories from
the ALFRED “valid-unseen” dataset consisting of validation tasks in unseen
floor plans. We collected 2 of each length from 1 through 5 for a total of 10
tasks by sampling random full-length trajectories from this dataset, with the
exception of length 1 tasks (we just sample random skills to create length 1
tasks). As these are full trajectories, they already have human annotations
from ALFRED, which we directly use as the task description (see Figure 13 for
example instructions).

We list additional details about the tasks in each evaluation set in Table 3.

Finally, we display 5 randomly sampled tasks, along with their human
annotations, from each of our task sets in Figures 12, 13, and 14.

Skills to Summarize: 1: Grab the knife on the counter. 2: Place the knife in
the sink then turn the faucet on so water fills the sink. Turn the faucet off
and pick up the knife again. 3: Place the knife on the table to the left of
the wooden bowl.
Annotator Summary: Wash the knife from the counter, put in on the table.
Skills to Summarize: 1: Pick up the blue book closest to your and the phone
from the bed. 2: Turn on the lamp to take a look at the book in the light.
Annotator Summary: Examine the book by the light of a lamp.
Skills to Summarize: 1: Pick up yellow candle on counter. 2: Open cabinet, put
candle in cabinet, close cabinet 3: Pick up yellow candle from toilet.
Annotator Summary: Move the candle from the sink to the cabinet under the
sink, close it and and then pick the candle from the top of the toilet in
front of you.
Skills to Summarize: 1: Pick the pot on the left side up from the stove. 2:
Set the bowl and knife on the table next to the tomato.
Annotator Summary: Put the bowl with the knife in it next to the tomato.
Skills to Summarize: 1: Pick up the pen that’s in front of you that’s under
the mug. 2: Put the pencil in the mug that was above it. 3: Pick up the mug
with the pencil in it.
Annotator Summary: Put the pen into the mug and pick up the mug.
Figure 12: Randomly sampled, human language instruction annotations from the
EVALINSTRUCT task set. Skills to Summarize: 1: Pick up the lettuce on the
counter. 2: Chill the lettuce in the fridge. 3: Put the chilled lettuce on the
counter, in front of the bread.
Annotator Summary: Put chilled lettuce on the counter.
Skills to Summarize: 1: Pick up an egg from off of the kitchen counter. 2:
Open the fridge, put the egg in to chill for a few seconds and then take it
back out. 3: Place the cold egg in the sink.
Annotator Summary: Chill an egg and put it in the sink.
Skills to Summarize: 1: Pick up the butter knife off of the right side of the
kitchen island. 2: Put the knife handle down in the frying pan that is on the
front left burner of the stove. 3: Pick up the frying pan with the knife in it
off of the stove. 4: Put the frying pan with the knife in it into the sink
basin to the right of the potato.
Annotator Summary: Put a frying pan with a knife in it into the sink.
Skills to Summarize: 1: Take the pencil from the desk. 2: Put the pencil on
the desk.
Annotator Summary: Take the pencil from the desk, put it on the other side of
the desk.
Skills to Summarize: 1: Pick up the left pillow on the chair. 2: Put the
pillow on the sofa right of the newspaper. 3: Pick up the pillow on the chair.
4: Put the pillow on the sofa left of the newspaper.
Annotator Summary: Place two pillows on a sofa.
Figure 13: Randomly sampled, human language instruction annotations from the
EVALSCENE task set. Skills to Summarize: 1: Pick up the knife in front of the
lettuce. 2: Slice the apple in the sink with the knife. 3: Place the knife
into the sink. 4: Pick up the sliced apple from the sink. 5: Place the apple
slice into the pot on the stove. 6: Pick up the pot from the stove. 7: Pick up
the pot from the stove.
Annotator Summary: Slice an apple for the pot on the stove and put the pot on
the counter to the right of the door.
Skills to Summarize: 1: Take the apple from the counter in front of you. 2:
Place the apple in the sink in front of you. 3: Take the knife by the sink in
front of you. 4: Cut the apple in the sink in front of you. 5: Place the knife
in the sink in front of you. 6: Take an apple slice from the sink in front of
you. 7: Heat the apple in the microwave, take it out and close the microwave.
8: Place the apple slice in the sink in front of you.
Annotator Summary: Place a warm apple slice in the sink.
Skills to Summarize: 1: Pick up the loaf of bread. 2: Put the bread on the
counter above the spatula. 3: Pick up the knife that’s above and to the right
of the loaf of bread. 4: Cut the top half of the loaf of bread into slices. 5:
Put the knife on the edge of the counter in front of you horizontally. 6: Pick
up a slice of bread from the middle of the loaf. 7: Cook the bread in the
microwave then take it out and close the microwave door. 8: Throw the cooked
slice of bread away.
Annotator Summary: Put a microwaved slice of bread in the oven.
Skills to Summarize: 1: Pick the knife up from off of the table. 2: Open the
microwave, slice the potato, and close the microwave. 3: Open the microwave,
place the knife inside of it, and close the microwave. 4: Open the microwave,
pick up the potato slice inside, close the microwave. 5: Place the potato
slice in the pan on the stove. 6: Pick up the pan from the stove. 7: Open the
refrigerator, place the pan inside, and close the refrigerator.
Annotator Summary: Move the pan from the stove top to inside the black
refrigerator.
Skills to Summarize: 1: Pick up the red tomato on the counter to the right of
the stove. 2: Put the tomato onto the island below the butter knife. 3: Pick
up the butter knife off of the kitchen island. 4: Slice up the tomato on the
kitchen island. 5: Place the butter knife onto the island to the right of the
sliced tomato. 6: Pick up a tomato slice off of the kitchen island. 7: Open
the fridge and put the tomato slice on the bottom shelf, then close the door,
after a couple seconds open the fridge and remove the tomato slice then close
the door. 8: Open the microwave door and place the tomato slice inside the
microwave in front of the egg.
Annotator Summary: Put a chilled tomato slice into the microwave.
Figure 14: Randomly sampled, human language instruction annotations from the
EVALLENGTH task set.

###### Online finetuning environment setup.

During online-finetuning we initialize the agent in the same house floor plan
as the trajectory the task was extracted from to ensure executability. During
finetuning, we give each episode a time horizon of 2x the number of
environment actions needed by the expert planner to solve the task. We give
sparse sub-task rewards for each skill solved by the agent during the episode.
Therefore for length 1 tasks, the agent can only be rewarded once before the
episode ends, while for length 5 tasks, the episode terminates on the fifth
reward signal.

####  C.2 Real Robot

Here we detail the dataset and evaluation tasks used for the real world
tabletop environment experiments.

###### Dataset Details.

Our dataset is collected through human operation of a Playstation 4 Dualshock
gamepad that directly controls the Jaco robot arm’s end effector position and
gripper open/close state. Part of our data comes from data collected from
prior work on the same arm setup [Dass et al., 2023], in addition to
additional trajectories collected for this project.

In total, we collected 329 long-horizon trajectories, resulting in $\sim$6k
individual “primitive” skills consisting of pick and place tasks such as “pick
up the black bowl” or “put the apple in the sink.” These trajectories involve
unique scene arrangements of different toy objects such as an apple, orange,
black bowl, white plate, oven, sink, dish rack, etc. The total dataset size is
455,473 individual state-action pairs.

###### Evaluation Tasks.

We formulate 3 unseen evaluation tasks requiring the completion of 2, 4, and 8
subtasks. These tasks are set in an environment configuration that has not
been seen before in the training data, i.e., the object combination is not
present in the training data. For each of the tasks, we collect 25
demonstrations to finetune pre-trained policies for evaluation.

The three tasks are defined below:

  1. 1.

Bake bread in the oven (length 2): The robot must (1) pick up the bread, (2)
place it in the oven.

  2. 2.

Serve heated milk in the bowl (length 4): The robot must (1) pick up the milk
carton, (2) place it in the black bowl bowl, (3) pick up the bowl with the
milk in it, (4) place the bowl in the oven.

  3. 3.

Serve milk in the bowl and butter and baked bread in the plate (length 8): The
robot must: (1) pick up the milk carton, (2) put it in the black bowl, (3)
pick up the butter stick, (4) put it in the plate, (5) pick up the bread, (6)
bake the bread in the oven, (7) pick up the bread from the oven, (8) place the
bread in the plate.

###  Appendix D Extended Experiments, Results, and Analysis

Table 4: EVALINSTRUCT and EVALLENGTH eval dataset per-length and overall skill completion rates. See Section 4 for experiment setup. |  | AM | ET | L-BC | SPRINT
---|---|---|---|---|---
EVALINSTRUCT | Number of Completed Subtasks Overall | 0.82 $\pm$ 0.07 | 1.15 $\pm$ 0.14 | 0.39 $\pm$ 0.02 | 1.92 $\pm$ 0.01
Length 1 Progress | 0.47 $\pm$ 0.06 | 0.75 $\pm$ 0.07 | 0.89 $\pm$ 0.07 | 0.87 $\pm$ 0.05
Length 2 Progress | 0.75 $\pm$ 0.10 | 1.20 $\pm$ 0.18 | 0.66 $\pm$ 0.05 | 1.48 $\pm$ 0.08
Length 3 Progress | 0.96 $\pm$ 0.28 | 1.61 $\pm$ 0.23 | 0.27 $\pm$ 0.08 | 2.21 $\pm$ 0.03
Length 4 Progress | 0.56 $\pm$ 0.17 | 1.45 $\pm$ 0.25 | 0.05 $\pm$ 0.05 | 2.36 $\pm$ 0.16
Length 5 Progress | 1.59 $\pm$ 0.53 | 0.76 $\pm$ 0.14 | 0.07 $\pm$ 0.05 | 3.04 $\pm$ 0.24
Length 6 Progress | 1.20 $\pm$ 0.40 | 0.86 $\pm$ 0.94 | 0.05 $\pm$ 0.08 | 2.87 $\pm$ 0.20
Length 7 Progress | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00
EVALLENGTH | Number of Completed Subtasks Overall | 1.71 $\pm$ 0.43 | 1.76 $\pm$ 0.14 | 0.07 $\pm$ 0.00 | 4.31 $\pm$ 0.51
Length 7 Progress | 0.80 $\pm$ 0.16 | 0.78 $\pm$ 0.45 | 0.06 $\pm$ 0.05 | 3.38 $\pm$ 0.43
Length 8 Progress | 2.62 $\pm$ 0.71 | 2.74 $\pm$ 0.55 | 0.26 $\pm$ 0.18 | 5.25 $\pm$ 0.64

Here, we present additional results complementary to the experiments in the
main paper in Section 4. We present and analyze LLM annotation examples in
Section D.1.

####  D.1 LLM Summary Examples

Skills to summarize: 1\. Pick the clock up from the desk. 2\. Turn the lamp
on.
OPT-350m Summary: Look at the clock.
OPT-1.3b Summary: Look at the clock from the far side of the bed under the
lamp light.
LLaMA-13b Summary: Look at the clock from the desk under the lamp light.

Skills to summarize: 1\. Take the plate from the cabinet to the left of the
stove. 2\. Warm the plate in the microwave, above the stove. 3\. Put the plate
in the cabinet.
OPT-350m Summary: Place the plate in the microwave.
OPT-1.3b Summary: Put the plate on the left side of the table.
LLaMA-13b Summary: Put a warm plate in the cabinet above the stove.

Skills to summarize: 1\. Open the microwave, take out the potato and shut the
door. 2\. Put the potato in the sink, turn the water on and off again. Take
the potato out of the sink. 3\. Place the potato on the counter, to the right
of the sink.
OPT-350m Summary: Put the potato on the counter, to the left of the sink.
OPT-1.3b Summary: Open the microwave, take out the potato, and close the door.
LLaMA-13b Summary: Put a potato on the counter, next to the sink.

Figure 15: Comparison of OPT-350m, OPT-1.3b [Zhang et al., 2022], and
LLaMA-13b summaries on 3 randomly sampled sequences from our dataset. In
general, as the model size increases, the summary becomes better. LLaMA-13b
summaries as a whole tend to be more descriptive and accurate. The smaller
models tend to regurgitate information from the original skills incorrectly,
such as OPT-350m summarizing the third example as putting a potato “to the
left of the sink” when the original skill stated “to the right of the sink.”

We randomly sample 3 LLAMA-13b task summaries produced while performing skill
aggregation in ALFRED (explained in Section 3.2) using the prompt in Figure 8
and display them in Figure 15 along with summaries from OPT-350m and OPT-1.3B
[Zhang et al., 2022], 350M and 1.3B parameter open-source models for
comparison. After analyzing many more examples, we see that LLaMA-13b
generally provides fitting high-level summaries for most sequences by skipping
over implied sub-tasks (although it sometimes also skips over important sub-
tasks, likely due to the prompt). The other smaller models, which are also
pre-trained on smaller corpora, tend to produce worse summaries and make up
details more often.

####  D.2 Qualitative Comparison Results

Here we display and analyze some qualitative task execution examples from
ALFRED and our real robot environment.

#####  D.2.1 ALFRED

###### Zero-shot evaluation.

We compare SPRINT, AM, and L-BC zero-shot evaluation results on long
EVALLENGTH tasks in Figure 16. In general, SPRINT is able to make
substantially more progress on EVALLENGTH tasks as it leverages the large
language model to generate longer-horizon, semantically meaningful pre-
training tasks and performs cross-trajectory chaining to learn to chain its
existing dataset tasks. In the visualized examples, SPRINT is able to
understand and successfully execute many of the sub-tasks implied but not
directly stated by the natural language task instruction. L-BC makes very
little progress on these tasks, not even understanding what the first sub-task
to complete should be as the task annotation is out of distribution from what
it saw while training. Finally, AM is able to make some progress on some of
these tasks due to its long-horizon goal pre-training objective. However, this
is less effective than our language-conditioned pre-training in such zero-shot
evaluations.

###### Finetuning.

We finetune SPRINT, AM, and L-BC on EVALSCENE tasks, in household floorplans
that were never seen while training, and visualize qualitative policy rollout
examples after finetuning in Figure 17. In general, SPRINT is able to finetune
to longer-horizon tasks while AM and L-BC both struggle with making progress
on longer-horizon tasks despite receiving rewards for every completed sub-
task. SPRINT’s ability to complete more sub-tasks on many of the longer-
horizon tasks is demonstrated in Figure 17(a), while a case in which both
SPRINT and AM make partial progress throughout finetuning is demonstrated in
Figure 17(b). We believe that AM has more trouble finetuning on these tasks
than SPRINT because the task specification for AM (goal images) is out of
distribution; pre-training on language tasks with SPRINT allows agents to more
easily learn longer-horizon behaviors as the task specifications may still be
in-distribution of the pre-training tasks that LLM skill-aggregation and skill
chaining produce.

(a) SPRINT successfully solves this task, while AM fails to slice the potato
and repetitively iterates between putting the potato in the fridge and
microwave. L-BC fails even to pick up the potato, as the task annotation does
not directly describe picking up a potato.

(b) SPRINT nearly solves this task, while AM picks up an egg instead of a
potato. L-BC picks up random objects not related to the annotation.

(c) SPRINT completes the entire task. AM picks up the tomato but fails to put
it down onto the counter and slice it. L-BC aimlessly wanders and picks up
random objects.

Figure 16: Visualizations of zero-shot policy rollouts on three tasks in the
EVALLENGTH task set.

(a) SPRINT picks up and places one of the pillows on the sofa, and picks up
the second but does not manage to place the second on the sofa, thus
completing 3/4 subtasks. AM and L-BC both learn to pick up a pillow but never
learned to place it in the correct spot.

(b) SPRINT and AM both learn to pick up a pencil from the desk, although
neither manage to put the pencil down in the correct place “on the other side
of the desk.” Meanwhile, L-BC never picks up the pencil.

Figure 17: Visualizations of policy rollouts on two tasks in the EVALSCENE
task set, after finetuning each method. These floor plans were originally
unseen to all agents until finetuning.

#####  D.2.2 Real Robot

We visualize evaluation rollouts after finetuning on our most difficult,
length 8 task, “Serve milk in the bowl and butter and baked bread in the
plate,” in Figure 18. We display an example comparison between SPRINT and L-BC
composite, the best-performing L-BC baseline in which the fine-tuned SPRINT
model successfully follows and accomplishes the skills in the demonstrated
long-horizon sequences. The L-BC composite agent finishes the first four
skills before encountering confusion about the subsequent skill: pick up the
long bread. This comparison reveals that the L-BC composite model exhibits
proficiency in completing some skills but overall does struggle with long-
horizon tasks. Empirically in our evaluations, we saw that this baseline
exhibits greater variance than SPRINT among its evaluation runs, sometimes
only executing 2 skills and other times finishing 8.

Figure 18: SPRINT picks up the correct objects successfully and places in the
right place accurately, with the same order shown in the demonstration. L-BC
composite model does the right thing on the milk diary and butter diary but is
not able to finish any skills with the long bread.