# SemTra: A Semantic Skill Translator for Cross-Domain Zero-Shot Policy
Adaptation

Sangwoo Shin, Minjong Yoo, Jeongwoo Lee, Honguk Woo Honguk Woo is the
corresponding author.

###### Abstract

This work explores the zero-shot adaptation capability of semantic skills,
semantically interpretable experts’ behavior patterns, in cross-domain
settings, where a user input in interleaved multi-modal snippets can prompt a
new long-horizon task for different domains. In these cross-domain settings,
we present a semantic skill translator framework SemTra which utilizes a set
of multi-modal models to extract skills from the snippets, and leverages the
reasoning capabilities of a pretrained language model to adapt these extracted
skills to the target domain. The framework employs a two-level hierarchy for
adaptation: task adaptation and skill adaptation. During task adaptation, seq-
to-seq translation by the language model transforms the extracted skills into
a semantic skill sequence, which is tailored to fit the cross-domain contexts.
Skill adaptation focuses on optimizing each semantic skill for the target
domain context, through parametric instantiations that are facilitated by
language prompting and contrastive learning-based context inferences. This
hierarchical adaptation empowers the framework to not only infer a complex
task specification in one-shot from the interleaved multi-modal snippets, but
also adapt it to new domains with zero-shot learning abilities. We evaluate
our framework with Meta-World, Franka Kitchen, RLBench, and CARLA
environments. The results clarify the framework’s superiority in performing
long-horizon tasks and adapting to different domains, showing its broad
applicability in practical use cases, such as cognitive robots interpreting
abstract instructions and autonomous vehicles operating under varied
configurations.

##  1 Introduction

The promise of zero-shot policy deployment across different domains stems from
the capability to immediately adapt its pre-trained knowledge to unfamiliar
environments without the need for extensive data collection or fine-tuning.
Such capability could revolutionize fields where safety is crucial, and where
a single failure can lead to significant consequences and substantial costs,
such as autonomous driving and robotics. However, achieving robust zero-shot
adaptation in these fields remains challenging due to the intricacies of given
tasks and the dynamic nature of their deployment environments (Chalaki et al.
2019; Harrison et al. 2017).

In the realm of reinforcement learning (RL) and imitation learning, policy
adaptation has seen some exploration, with researchers leveraging various
forms of task specification to deduce the given task domain. For instance,
several studies utilized video demonstrations or expert trajectories from
similar domains (Zhao et al. 2022; Kim et al. 2020), while others pivoted
towards language instructions provided by users (Goyal, Mooney, and Niekum
2023; Ichter et al. 2022), or video-language interleaved demonstrations in
multi-modal user interfaces (Jiang et al. 2023). Furthermore, in (Xu et al.
2022), expert trajectories from the target domain were directly employed for
task inference.

In this work, we investigate the problem of cross-domain zero-shot policy
adaptation, with a focus on an inclusive and generalized approach to process
user inputs relevant to long-horizon tasks across diverse domains. To be
specific, we consider situations where the policy is prompted with a task-
relevant input, presented as multi-modal interleaved snippets encompassing
video, sensor data, and textual elements. In light of this consideration, we
introduce a novel framework SemTra, grounded in the notion of semantic skill
translation that spans multi-domains. The framework is designed with a two-
tiered hierarchical adaptation process, first at the task level, followed by
the skill level, to enhance the potential for zero-shot policy adaptation to
target domains.

During task adaptation, our framework harnesses the reasoning strengths of
Pretrained Language Models (PLMs) to interpret the task specification (Ichter
et al. 2022; Huang et al. 2022). This specification, encapsulated within a
sequence of multi-modal interleaved snippets, is transformed into a sequence
of semantically interpretable skill representations (semantic skills). For
skill adaptation, the framework employs a parametric approach to instantiate
skills, integrating domain contexts into the semantic skills. These contexts
may not be solely encapsulated in the task specification or input snippets, as
they can also be dynamically captured during the evaluation time through
interactions with the deployment environment. Subsequently, these domain
contexts form domain-randomized meta-knowledge, enabling the framework to
achieve context-parameterized skill instantiation.

The contributions of our work are summarized as follows. (i) We present the
novel framework SemTra, addressing the challenging yet practical issues of
cross-domain zero-shot policy adaptation for long-horizon tasks. (ii) We
devise a robust, hierarchical adaptation algorithm that leverages PLM
prompting for the task-level adaptation and disentangles semantic skills and
domain contexts to enable the skill-level optimization through parametric
instantiations across different domains. (iii) We intensively evaluate our
framework in over 200 cross-domain scenarios with Meta-World, Franka Kitchen,
RLBench, and CARLA simulation environments, demonstrating its broad
applicability in practical use cases, such as cognitive robots and autonomous
driving.

##  2 Problem Formulation

We consider a multi-modal task prompt $\mathcal{P}:=(x_{1},\cdots,x_{N})$
comprising a sequence of snippets, where each snippet $x_{i}$ is given in one
of several modalities such as video, sensor data, and text. We presume that
$\mathcal{P}$ presents a source task specification, and some snippets may
contain pertinent information or guidance beneficial for cross-domain
adaptation. Given a single task prompt $\mathcal{P}$, our work is to establish
a framework to find the optimal model $\phi^{*}$. The model maps $\mathcal{P}$
to a policy $\pi_{\mathcal{P}}$, maximizing the expected evaluation
performance:

$\phi^{*}=\mathop{\textnormal{argmax}}_{\phi}\left[\underset{(D,\mathcal{P})\sim
p_{\textnormal{tgt}}(D,\mathcal{P})}{\mathbb{E}}\left[\textnormal{E}_{D}(\phi(\mathcal{P}))\right]\right].$
(1)

Here, $E_{D}(\phi(\mathcal{P}))$ is an evaluation function for the policy
$\phi(\mathcal{P})$ in a target domain $D$.

Figure 1:  Cross-domain zero-shot adaptation for a multi-modal task prompt:
our framework is given a task prompt filled with partial demonstrations and
instructed contextual cues in multi-modal snippets. The framework conducts a
two-phase adaptation, initially translating the snippets to semantic skills at
the task level, and subsequently optimizing them into actions for the target
domain at the skill level.

As illustrated in Figure 1, the task prompt is comprised of multiple
modalities; e.g., the Video1 snippet offers a visual demonstration of an
expert performing “turning on the kitchen lights”, the Sensor1 snippet
captures the trajectory of “turning on the microwave.”, the Video2 snippet
visually represents the process of “boiling the water in the kettle”, and the
Text1 snippet “do fast” contains contextual clues about the task execution in
the environment, such as a fast-food restaurant. These snippets collectively
formulate the source task specification and also serve as a guide for
adaptation in the target domain. The evaluation function
$E_{D}(\phi(\mathcal{P}))$ in (1) assesses the task performance, particularly
concerning cooking and serving maneuvers in the target domain $D$ with some
stringent time constraints in this case.

Figure 2:  Two-phase policy adaptation in SemTra. (1) Task adaptation: the
multi-modal skill encoders $\Phi_{E}$ produce a skill-level language
instruction from the task prompt. In the figure, we specifically describe the
training of a video skill encoder, contrastively learned through a pretrained
VLM $(\Psi_{V},\Psi_{L})$. The skill-level instruction is then translated into
a semantic skill sequence through the semantic skill sequence generator
$\Phi_{G}$ based on a PLM. The skill boundary detector $\Phi_{B}$ infers the
boundary of semantic skills upon a current state in the target domain. (2)
Skill adaptation: the context encoder $\Phi^{(g)}_{C}$ identifies instructed
domain contexts using both the skill-level instruction and the semantic skill
sequence, generating an executable skill sequence. The online context encoder
$\Phi^{(o)}_{C}$ captures environment hidden contexts at runtime. The behavior
decoder $\pi$ generates actions optimized for the target domain, based on the
executable skills and environment hidden contexts, along with the current
state. 1

##  3 Approach

###  3.1 Overall Framework

To tackle the challenge of cross-domain zero-shot adaptation with a multi-
modal task prompt, we develop a semantic skill translator framework SemTra. As
illustrated in Figure 1, SemTra decomposes the cross-domain adaptation
hierarchically into two-phase linguistic adaptation procedures. In the task
adaptation phase, the framework captures the task discrepancy between the
source and target domains by extracting a domain-invariant task solution using
semantic skills. To do so, a pretrained vision-language model (VLM) is
leveraged to establish the domain-agnostically learned semantic skills from
the video snippets. The knowledge of PLM is also used to align the semantic
skill sequence with the subtasks in the target domain. In the skill adaptation
phase, the semantic skills are instantiated into an actual action sequence
optimized for the target domain.

###  3.2 Task Adaptation

For a task prompt $\mathcal{P}=(x_{1},\cdots x_{N})$, our framework interprets
it to produce a sequence of semantic skill embeddings $z_{t}$. These
embeddings are represented in a language space and correspond to successive
timesteps $t$ during task execution in the target environment.

$\Phi_{D}\circ\Phi_{E}:(s_{t},\mathcal{P}=(x_{1},\cdots,x_{N}))\mapsto z_{t}$
(2)

This adaptation involves multi-modal skill encoders $\Phi_{E}$ and a semantic
skill decoder $\Phi_{D}$, where each encoder takes the input specification as
outlined by the task prompt, and the decoder maps the encoded specification
into the semantic skills to be executed in the target environment. To train
the encoder and decoder models, we use a dataset
$\mathcal{D}=\\{(\mathcal{P}_{i},\tau_{i})\\}_{i}$, where $\mathcal{P}_{i}$ is
a task prompt and $\tau$ is an expert trajectory in the target environment.
Specifically, we use $\tau$ = $\\{(s_{t},a_{t},v_{t},l_{t})\\}_{t}$ with state
$s_{t}$, action $a_{t}$, visual observation $v_{t}$, and semantic annotation
$l_{t}$, where $l_{t}$ is a description of a skill or behavior at timestep
$t$, e.g., “push a button”, following (Shin et al. 2023; Ichter et al. 2022;
Pertsch et al. 2022; Shridhar, Manuelli, and Fox 2022).

#### Multi-modal skill encoders.

The encoders $\Phi_{E}$ convert a task prompt into a skill-level language
instruction $\eta$ as

$\Phi_{E}:\mathcal{P}=(x_{1},\cdots,x_{N})\mapsto\eta:=(l^{(src)}_{i_{1}},\cdots,l^{(src)}_{i_{I}})$
(3)

where each $l^{(src)}_{i_{j}}$ is a language token. In this work, we consider
three different modalities in snippets (i.e., video, sensor data, and text),
although another modality can be incorporated through its respective encoder
implementation.

To train the video skill encoder $\Psi_{V}$, which maps a video snippet
$x=(v_{0:T})$ to skill-level language tokens, we leverage the video-to-text
retrieval and text prompt learning with the pretrained VLM, V-CLIP (Xu et al.
2021), in a similar way of (Shin et al. 2023). Specifically, consider the
V-CLIP $(\Psi_{V},\Psi_{L})$ model of a video encoder $\Psi_{V}$ and a text
encoder $\Psi_{L}$. A text prompt $\theta_{p}$, a small set of learnable
parameters in $\Psi_{L}$ is learned using a contrastive learning loss (van den
Oord, Li, and Vinyals 2018) on a batch
$\\{(v_{t-H:t},l_{t})\\}\sim\mathcal{D}$. This optimization increases the
similarity of video embedding $\Psi_{V}(v_{t-H:t})$ and its language embedding
counterpart $\Psi_{L}(l_{t};\theta_{p})$. Once $\theta_{p}$ is trained, we use
the retrieved semantic skill from the video snippet, i.e.,
$\Psi_{V}(v_{t-H:t}):=$

$\textnormal{argmax}_{l\in\mathcal{L}_{vl}}\textnormal{sim}(\Psi_{V}(v_{t-H:t}),\Psi_{L}(l;\theta_{p}))$
(4)

where $\textnormal{sim}(\cdot,\cdot)$ is a similarity function and
$\mathcal{L}_{vl}=\bigcup_{l\in\tau\in\mathcal{D}}\\{l\\}$ denotes a set of
textually represented semantically interpretable skills. For sensor data
snippets, we use a supervised classifier which predicts the semantic
annotation $l_{t}$ from a state-action sub-trajectory
$(s_{t-H^{\prime}:t},a_{t-H^{\prime}:t})$. For text snippets, we use an
identity function.

#### Semantic skill decoder.

Given a skill-level instruction $\eta$ in (3) and state $s_{t}$, the semantic
skill decoder $\Phi_{D}$ yields a semantic skill $z_{t}$ for each timestep
$t$, i.e.,

$\Phi_{D}:(s_{t},\eta)\mapsto z_{t}$ (5)

by using two models, the semantic skill sequence generator $\Phi_{G}$ and the
skill boundary detector $\Phi_{B}$. The former $\Phi_{G}$ generates a sequence
of target semantic skill embeddings through seq-to-seq translation, leveraging
the knowledge of PLMs.

$\Phi_{G}:\eta\mapsto(z_{j_{1}},\cdots,z_{j_{J}})$ (6)

At each timestep $t$, the latter $\Phi_{B}$ determines whether to continue
executing the current skill $z_{t}$ or to transition to the next skill in the
sequence until the task is completed. It evaluates the probability that
ongoing semantic skill is terminated:

$\Phi_{B}:(s_{t},z_{t},s_{t_{0}})\mapsto p\in[0,1]$ (7)

with state $s_{t}$, semantic skill $z_{t}$, and its initial state $s_{t_{0}}$.

For training $\Phi_{G}$, we exploit the skill encoders $\Phi_{E}$ that can
extract a target semantic sequence $(z_{j_{1}},\cdots,z_{j_{J}})$ from the
video trajectory of $\tau^{(tgt)}$ through (4). Subsequently, $\Phi_{G}$ is
supervised using a cross-entropy-based loss on these target semantic
sequences. Regarding $\Phi_{B}$, it is trained using a binary cross-entropy
loss, i.e., $\textnormal{BCE}(\Phi_{B}(s_{t},z_{t},s_{t_{0}}),1_{z_{t}\neq
z_{t+1}})$. This task adaptation procedure is presented in the middle of
Figure A.13.

Algorithm 1 Two-phase adaptation of SemTra

1: Task prompt $\mathcal{P}$, target trajectory
$\tau^{(tgt)}=\\{(s_{t},v_{t},a_{t},l_{t})\\}$

2: Domain contexts
$\\{\bar{z}^{\prime}_{j_{1}},\cdots,\bar{z}^{\prime}_{j_{J}}\\}$ of
$\tau^{(tgt)}$

3: Multi-modal skill enc. $\Phi_{E}$, skill sequence generator $\Phi_{G}$

4: Skill boundary detector $\Phi_{B}$, context encoder $\Phi^{(g)}_{C}$

5: Online context encoder $\Phi^{(o)}_{C}$, behavior decoder $\pi$ /* Task
adaptation in Section 3.2 */

6: $\eta\leftarrow\Phi_{E}(\mathcal{P})$ using (3),
$\xi\leftarrow\Phi_{G}(\eta)$ using (6)

7: $\xi^{\prime}\leftarrow\\{\Psi_{V}(v_{t-H:t}):t\\}$ with deduplication
using (4)

8:
$\Phi_{G}\leftarrow\Phi_{G}-\nabla_{\Phi_{G}}(\textnormal{CrossEntropy}(\xi,\xi^{\prime}))$

9: $(z_{0},\cdots,z_{T})\leftarrow\\{\Psi_{V}(v_{t-H:t}):t\\}$ using (4)

10:
$\Phi_{B}\leftarrow\Phi_{B}-\nabla_{\Phi_{B}}(\textnormal{BCE}(\Phi_{B}(s_{t},z_{t},s_{t_{0}}),1_{z_{t}\neq
z_{t+1}}))$ /* Skill adaptation in Section 3.3 */

11:
$\\{\bar{z}_{j_{1}},\cdots,\bar{z}_{j_{J}}\\}\leftarrow\Phi^{(g)}_{C}(\eta,\xi)$
using (9)

12:
$\Phi^{(g)}_{C}\leftarrow\Phi^{(g)}_{C}-\nabla_{\Phi^{(g)}_{C}}(\textnormal{CrossEntropy}(\bar{z}_{j_{i}},\bar{z}^{\prime}_{j_{i}}))$

13: $h_{t}\leftarrow\Phi^{(o)}_{C}(s_{t-H^{\prime}:t},a_{t-H^{\prime}:t-1})$
using (11)

14:
$\pi\leftarrow\pi-\nabla_{\pi}(\textnormal{MSE}(\pi(s_{t},\bar{z}_{t},h_{t}),a_{t})$

###  3.3 Skill Adaptation

In conjunction with the task adaptation, which generates a semantic skill
sequence, the framework also encompasses a skill adaptation phase, where each
skill is adapted individually to suit the target domain. It takes as input the
semantic skill sequence $\xi:=(z_{j_{1}},\cdots,z_{j_{J}})$ in (6), paired
with a skill-level language instruction $\eta$ in (3), and the current state
$s_{t}$. It continuously generates an action $a_{t}$ for each timestep during
the execution of $\xi$ in the target domain, employing a context encoder
$\Phi^{(g)}_{C}$ and a behavior decoder $\pi$.

$\pi\circ\Phi^{(g)}_{C}:(s_{t},\eta,\xi)\mapsto a_{t}$ (8)

#### Domain context encoder.

The context encoder $\Phi^{(g)}_{C}$ transforms a semantic skill sequence into
a series of executable skills appropriate for the target domain, by taking
into account the cross-domain contexts encapsulated in a given task prompt.
This involves utilizing the knowledge of PLMs to capture domain-specific
features from the coupled skill-level instruction $\eta$ and its associated
semantic skill sequence $\xi$.

$\Phi^{(g)}_{C}:(\eta,\xi)\mapsto(\bar{z}_{j_{1}},\cdots,\bar{z}_{j_{J}}),\
\text{where}\ \bar{z}:=(z,f_{z},d_{z})$ (9)

Here, each executable skill $\bar{z}$ is parameterized by a semantic skill
$z\in\xi$, a domain factor $f_{z}$, and a magnitude $d_{z}$ associated with
$f_{z}$. The cross-domain context information in $\eta$ is used to infer the
domain factor, which modulates the actual action execution of each semantic
skill according to the specific requirements of the target domain; e.g., the
domain factor for the semantic skill “move to a destination” could be the
surface type or the required speed in the target domain.

#### Behavior decoder.

Given an executable skill sequence $(\bar{z}_{j_{1}},\cdots,\bar{z}_{j_{J}})$
in (9), the behavior decoder $\pi$ is learned to infer an action sequence
optimized for the target domain.

$\pi:(s_{t},\bar{z_{t}})\mapsto a_{t}$ (10)

Furthermore, to adapt to temporal domain shifts that are not fully covered by
the task prompt, an online context encoder $\Phi^{(o)}_{C}$ is utilized to
infer a hidden context $h_{t}$ over timesteps $t$:

$\Phi^{(o)}_{C}:(s_{t-H^{\prime}:t},a_{t-H^{\prime}:t-1})\mapsto h_{t}.$ (11)

Then, this hidden context is concatenated with $\bar{z}_{t}$ in (10). This
$\Phi^{(o)}_{C}$ can be implemented by using reconstruction (Zintgraf et al.
2020; Rakelly et al. 2019) or contrastive learning (Shin et al. 2023; Qiu et
al. 2022). This skill adaptation procedure is on the right side of Figure
A.13. Algorithm 1 outlines the two-phase adaptation in SemTra.

##  4 Evaluation

###  4.1 Experiment Setting

#### Environments.

For cross-domain adaptation evaluations, we use the Franka Kitchen (Fu et al.
2020) (FK) and Meta-World (Yu et al. 2019) (MW) environments. In these
environments, each task $\mathcal{T}$ is composed of $N$ subtasks that must be
completed in the correct order. This allows us to explore the compositionality
of complex tasks and investigate the long-horizon imitation performance (Shin
et al. 2023; Nam et al. 2022; Pertsch, Lee, and Lim 2020; Gupta et al. 2019).

#### Baselines.

For comparison, we test VIMA (Jiang et al. 2023) that incorporates multi-modal
task prompts to a cross-attention-based Transformer for one-shot imitation
learning, and we implement TP-GPT, a multi-modal task prompting approach, that
specifically uses the decoder-only Transformer with self-attention (Shafiullah
et al. 2022; Xu et al. 2022). Furthermore, we implement a video generation-
based task prompting method VCP, similar to (Du et al. 2023; Chane-Sane,
Schmid, and Laptev 2023) that incorporates task prompts to video embeddings-
conditioned policies. We also implement a language generation-based task
prompting method TP-BCZ that employs the state-of-the-art multi-modal zero-
shot imitation framework (Jang et al. 2022).

#### Evaluation metrics.

We employ two performance evaluation metrics in cross-domain zero-shot policy
adaptation: $K$-rate (KR) that evaluates the rate of unseen task domains in
which at least $K$ out of $N$ subtasks are completed, similar to the
evaluation metric used in long-horizon tasks by (Mees et al. 2022); $N$-rate
(NR) that evaluates the average rate of successfully completed subtasks out of
$N$ subtasks. In our evaluation, we set $N=4$ for Franka Kitchen and $N=3$ for
Meta-World.

#### Evaluation datasets.

We use 9 unseen long-horizon evaluation tasks (e.g., each being defined by an
unseen order of subtasks) in Franka Kitchen and 8 unseen long-horizon tasks in
Meta-World. We also use 12 skill-level domain contexts (e.g., instructed
domain contexts for time constraints and time-varying domain contexts for the
non-stationarity of environments). Based on the combination of these unseen
tasks and contexts, we establish a total of 108 and 96 distinct unseen target
domains for Franka Kitchen and Meta-World, respectively. Detailed explanations
for training datasets and examples of task prompts can be found in Appendix B.

###  4.2 Results

#### Cross-domain performance.

We evaluate the cross-domain performance in both $K$-rate and $N$-rate,
wherein a policy is evaluated in a zero-shot manner for different target
domains. In Table 1, SemTra consistently outperforms the baselines,
specifically $36.34\%$ and $66.24\%$ $N$-rate higher than the most competitive
baselines VCP in Franka Kitchen and TP-GPT in Meta-World, respectively. With
respect to the $K$-rate metric, the margin of superiority becomes more
significant for larger $K$, underlining the robust adaptation capabilities of
SemTra, especially for long-horizon tasks.

Env | KR | VIMA | BCZ | GPT | VCP | SemTra
---|---|---|---|---|---|---
FK |  $K$=1 | 63.6 | 55.1 | 51.1 | 62.2 | 100.0
$K$=2 | 31.3 | 19.5 | 32.0 | 32.8 | 80.0
$K$=3 | 13.9 | 3.1 | 14.2 | 16.8 | 61.7
$K$=4 | 5.3 | 0.0 | 7.5 | 7.5 | 23.1
NR | 28.3 | 19.4 | 26.2 | 29.8 | 66.2
MW |  $K$=1 | 18.5 | 17.4 | 35.4 | 22.7 | 86.2
$K$=2 | 4.8 | 6.4 | 3.2 | 4.8 | 82.8
$K$=3 | 0.0 | 0.0 | 0.0 | 0.5 | 72.6
NR | 7.6 | 7.8 | 13.1 | 8.4 | 79.3
Table 1:  Cross-domain performance in $K$-rate and $N$-rate.

#### Single-domain (reference) performance.

To analyze the cross-domain performance, we also use the single-domain
scenarios (i.e., one-shot imitation within a single-domain) as a reference,
where we treat the task prompt as a demonstration specific to the target
domain. As shown in Table 2, SemTra shows $13.89\%$ and $58.33\%$ higher
$N$-rate than the most competitive baseline TP-GPT in Franka Kitchen and Meta-
World, respectively. More importantly, we confirm the robust cross-domain
performance of SemTra (in Table 1), which is comparable to this single-domain
performance; specifically, the degradation is minimal, when transitioning from
single-domain to cross-domain experiments. In contrast, the baselines
experience a relatively large degradation.

Env | KR | VIMA | BCZ | GPT | VCP | SemTra
---|---|---|---|---|---|---
FK |  $K$=1 | 96.2 | 62.9 | 92.5 | 77.7 | 100.0
$K$=2 | 62.9 | 22.2 | 74.0 | 55.5 | 85.1
$K$=3 | 22.2 | 3.7 | 40.7 | 29.6 | 66.6
$K$=4 | 11.1 | 0.0 | 19.6 | 11.1 | 40.7
NR | 48.1 | 22.2 | 59.2 | 43.5 | 73.1
MW |  $K$=1 | 37.5 | 29.1 | 33.3 | 50.0 | 100.0
$K$=2 | 12.5 | 16.6 | 8.3 | 12.5 | 79.1
$K$=3 | 3.7 | 1.5 | 0.0 | 0.0 | 62.5
NR | 16.6 | 15.2 | 22.2 | 20.8 | 80.5
Table 2:  Single-domain performance in $K$-rate and $N$-rate.

In the following, we use this single-domain performance in $N$-rate as a
reference in comparison.

#### Task-level cross-domain performance.

We specifically investigate the cross-domain adaptation performance under
task-level domain changes, the scenarios where the order of subtasks in the
target domain is not the same as that of a given task prompt. For the cross-
domain setting, we consider task prompts involving snippets beneficial for the
adaptation, such as “in reverse order”. With no consideration of skill-level
cross-domain contexts, we test 27 and 24 unseen task prompts for Franka
Kitchen and Meta-World, respectively. Table 3 shows that SemTra exhibits
robust performance for the “Reverse” and “Replace” cases with $N$-rate decline
of $7.08\%$ and $5.97\%$ in Franka Kitchen and Meta-world, respectively,
compared to the single-domain reference performance (“Ref.”). In contrast, TP-
GPT degrades by $9.39\%$ and $2.5\%$ in $N$-rate. This specifies the task-
level adaptation capabilities of SemTra, specifically empowered by the
semantic skill decoder based on a PLM.

Env | Dom | VIMA | BCZ | GPT | VCP | SemTra
---|---|---|---|---|---|---
FK | Ref. | 48.1 | 22.2 | 59.2 | 43.5 | 73.1
Rev | 46.3 | 14.8 | 50.9 | 62.9 | 71.3
Rep | 30.9 | 15.4 | 48.8 | 26.1 | 64.2
MW | Ref. | 16.6 | 15.2 | 22.2 | 20.8 | 80.5
Rev | 5.5 | 9.7 | 8.3 | 8.3 | 76.3
Rep | 8.8 | 8.8 | 13.3 | 5.5 | 80.0
Table 3: Task-level cross-domain performance in $N$-rate.

#### Skill-level cross-domain performance.

Table 4 shows the cross-domain adaptation performance, particularly for skill-
level domain changes, in which each skill or subtask is required to adapt to
the target domain. For each instructed domain context, we compute the average
$N$-rate performance across different non-stationary hidden contexts. For
example, “Fast” in the “Domain” column indicates domain contexts associated
with stringent time limits in the environment, which can be instructed
(specified) by a given task prompt; the snippets beneficial for the skill-
level cross-domain context, such as “do fast”, are given. For each row in the
table, we experiment with 36 and 32 different task prompts for Franka Kitchen
and Meta-World, respectively. SemTra exhibits robust performance compared to
the baselines, showing $39.35\%$ and $65.73\%$ higher in Franka Kitchen and
Meta-World than the most competitive baselines TP-BCZ and TP-GPT,
respectively. This is attributed to our disentanglement scheme, which
effectively separates domain-invariant semantic skills from domain contexts.

Env | Dom | VIMA | BCZ | GPT | VCP | SemTra
---|---|---|---|---|---|---
FK | Ref. | 48.1 | 22.2 | 59.2 | 43.5 | 73.1
Gust | 13.6 | 23.1 | 12.7 | 15.7 | 59.9
Flurry | 7.8 | 19.4 | 4.4 | 16.6 | 61.3
MW | Ref. | 16.6 | 15.2 | 22.2 | 20.8 | 80.5
Slow | 1.0 | 12.5 | 9.3 | 4.1 | 72.1
Fast | 3.4 | 0.0 | 11.4 | 15.2 | 80.2
Table 4: Skill-level cross-domain performance in $N$-rate.

#### Semantic correspondence in multi-modal space.

In Figure 3, we visualize a set of video demonstrations and the trajectories
of the zero-shot policy in SemTra. We use the V-CLIP embedding space. A
numeric index in the legends corresponds to a semantic skill. As observed, the
demonstration trajectories (on the left side of the figure) traverse through
distinct clusters of semantic skills, reflecting the alignment of the skills
with the expert behavior patterns. The policy’s trajectories (in the middle of
the figure) in the target domains share a similar space with the
demonstrations. In addition, the trajectories from different domains (“Dom”s
in legends) for a single semantic skill (on the right side of the figure) tend
to form distinct clusters, indicating the variations in the execution of a
specific skill in different domain contexts.

Figure 3:  Semantic correspondence in V-CLIP space.

###  4.3 Ablation Studies

#### PLMs for skill sequence generation.

To implement the semantic skill sequence generator $\Phi_{G}$ in (6), we test
several PLMs, including relatively lightweight models such as GPT2, GPT2-large
(Radford et al. 2019), and Bloom (Scao et al. 2022). These models are selected
based on the consideration that they can be fine-tuned with the GPU system
resources available in an academic setting. In Figure 4, these PLMs
consistently exhibit higher sample efficiency than the model trained from
scratch. There is also a positive correlation between the size of the PLMs and
the accuracy of skill sequence generation. This specifies that the logical
reasoning abilities of PLMs are instrumental in task adaptation.

We also evaluate the suitability of advanced PLMs such as GPT3 (Brown et al.
2020), GPT3.5, GPT4 (OpenAI 2023), and PaLM (Chowdhery et al. 2022) for the
skill sequence generator, in a zero-shot context, without any fine-tuning. To
this end, we engineer an augmented task prompt to contain a semantic skill
set, which is presented to the PLMs along with the skill-level instruction.
Our observations reveal that GPT3 and GPT3.5 encounter difficulties in
generating the correct sequence, especially when reversing the execution of a
skill sequence, while GPT4 and PaLM achieve relatively better performance.
These results are consistent with our motivation to leverage the advancements
in PLMs in enabling robust translation from intermediate instructions to
semantic skill sequences.

Figure 4:  PLMs for skill sequence generator: (a) PLM fine-tuning case; the
x-axis denotes the gradient update steps, and the y-axis denotes the accuracy
of the skill sequence generation. (b) PLM zero-shot case; only an engineered
prompt is adopted without any model fine-tuning.

#### Conditional components for behavior decoding.

We test several representations for the conditional components to the behavior
decoder in (10) to evaluate our skill adaptation phase. Specifically, we
explore variations in the inputs to the behavior decoder instead of executable
skills $\bar{z_{t}}$ in (10): N-GPT uses a given task prompt in multi-modal
snippets in (2); L-GPT uses a skill-level instruction in (3) that has been
encoded in the language embedding space by the multi-modal encoders; V-GPT is
the same as L-GPT but employs the video embedding space; T-GPT takes the
entire sequence of executable skills in (9) bypassing the sequence
segmentation in (7).

In Table 5, comparing N-GPT and $\\{\textnormal{V, L}\\}$-GPT, we observe that
retrieving semantics from video demonstrations improves performance in
$N$-rate by $16.45\%$. L-GPT outperforms V-GPT, specifying that skill
embeddings in the language space are more efficient than those in the video
space; this pattern was observed in previous work (Jang et al. 2022). T-GPT
outperforms L-GPT by $12.54\%$, indicating the effectiveness of the skill
sequence generation of our framework. Finally, SemTra outperforms T-GPT by
$35.57\%$, demonstrating that a behavior decoder conditioned with a single
skill is suitable. These consistent results clarify the benefits of our skill
adaptation based on disentangled skill semantics and target domain contexts,
enabling parametric skill instantiation.

Env | N-GPT | V-GPT | L-GPT | T-GPT | SemTra
---|---|---|---|---|---
FK | 4.1 | 19.4 | 40.2 | 42.8 | 72.2
MW | 2.7 | 8.1 | 11.8 | 34.3 | 76.1
Table 5: Conditional components for behavior decoding.

#### Sample efficiency in semantic annotations.

As explained in Section 3.2, we leverage a training dataset with semantically
annotated target trajectories. Recognizing the challenges associated with
acquiring a large annotated dataset, we employ the video skill encoder, which
was trained on a small annotated dataset, to pseudo-label on unannotated
demonstrations. We evaluate the impact of the annotated dataset size on the
efficacy of our framework. Table 6 shows the accuracy of the video skill
encoder (in the row of Skill Pred.) and the cross-domain adaptation
performance ($N$-rate) with respect to the number of annotations used.
Remarkably, utilizing only 50 videos per semantic skill (approximately 0.23%
of the entire dataset) is sufficient for our framework to achieve comparable
performance to the cases where annotations are provided at every timestep.

# Ann. | 3 | 5 | 7 | 10 | 50 | Every
---|---|---|---|---|---|---
Skill Pred. | 68.5 | 68.5 | 84.4 | 89.6 | 99.7 | 100.0
$N$-rate | 45.8 | 46.6 | 64.7 | 64.4 | 71.1 | 72.7
Table 6: Sample efficiency in semantic annotations

###  4.4 Use Cases

#### Cognitive robot manipulator.

We verify the applicability of SemTra with cognitive robot manipulation
scenarios using the RLBench robotic manipulation simulator (James et al.
2020), in which a task prompt is presented as abstract user commands in
natural language (contrast with our task prompts containing video
demonstrations.) Specifically, we experiment with three different command
styles, similar to the applications of SayCan (Ichter et al. 2022). A
Summarized sentence command offers an abstract instruction for conveying long-
horizon tasks. An Abstract verbs command encompasses synonymous verbs to
convey desired actions. An Embodiment command narrates the current environment
conditions and the constraints of the robot manipulator. Table 7 demonstrates
the efficacy of SemTra in processing these abstract commands, specifying the
comprehensive task prompting structure as well as the ability to harness the
logical reasoning capabilities of PLMs. Furthermore, the two-phase adaptation
facilitates modular approaches for different levels of policy adaptations
using PLMs. It also supports the flexible integration of future, more advanced
PLMs.

| SayCan | SemTra
---|---|---
GPT-3 | PaLM | GPT-3.5 | GPT-4 | PaLM
Summ. | 76.6 | 100.0 | 53.3 | 100.0 | 100.0
Abst. | 60.0 | 100.0 | 73.3 | 100.0 | 100.0
Emb. | 86.6 | 100.0 | 60.0 | 100.0 | 100.0
Table 7: Performance in cognitive robot manipulations.

#### Autonomous driving.

With the autonomous driving simulator CARLA (Dosovitskiy et al. 2017), we
verify the skill adaptation ability across different vehicle configurations
(agent embodiments) such as sedans and trucks. The driving agent must
calibrate steering, throttle, and brake responses according to the specific
vehicle configuration it manages. Its goal is to reach the destination in the
shortest path as depicted in Figure 5, where skills are semantically defined
as driving straight, turning left, and turning right. We calculate the
normalized return based on the rewards yielded by a rule-based policy. As
shown in Table 8, SemTra outperforms the most competitive baseline VCP by
$1.94\sim 5.00\%$.

Figure 5:  The left graph represents the difference in expert action distribution based on vehicle configurations. The right figure shows a semantic skill sequence to reach the goal.  Vehicle type | VIMA | BCZ | GPT | VCP | SemTra
---|---|---|---|---|---
Sedan | 79.4 | 74.7 | 62.7 | 78.3 | 83.3
Truck | 33.4 | 68.6 | 65.1 | 70.5 | 72.4
Table 8:  Performance in autonomous driving.

##  5 Related Works

#### Cross-domain policy adaptation.

Recently, there has been a surge in cross-domain policy adaptation techniques
exploring task specifications (Goyal, Mooney, and Niekum 2023; Pertsch et al.
2022). A relevant subset of this trend is cross-domain one-shot imitation,
which exploits demonstrations from the source domain to tackle tasks in
different target domains (Zhao et al. 2022; Kim et al. 2020; Liu et al. 2018;
Yu et al. 2018; Gupta et al. 2017). For instance, (Zhao et al. 2022)
introduced a contrastive learning-based video encoder to bridge visual domain
gaps between demonstrations and target environments. (Kim et al. 2020)
employed the generative adversarial learning to handle embodiment and dynamics
differences, while (Goyal, Mooney, and Niekum 2023) delved into task-level
domain disparities, directly transferring source expert trajectories to target
actions using a Transformer-based policy. Our work is in the same vein, but it
distinguishes itself by accommodating a unified multi-modal specification for
task prompting. We also concentrate on long-horizon tasks, incorporating the
knowledge of PLMs into our framework to explore the compositionality and
parametric execution capability of semantic skills.

#### Semantic skills in RL.

In the RL literature, several studies introduced policy learning and
adaptation methods, particularly investigating semantically interpretable
skills (semantic skills). Recently, Saycan (Ichter et al. 2022) employed a PLM
to plan semantic skills from abstract linguistic instructions. Utilizing an
offline RL dataset collected from an environment with domain-specific rewards,
this novel approach learns to estimate the value of each semantic skill at a
specific scene, effectively opting to execute the most valuable skill.
Furthermore, (Pertsch et al. 2022) exploited human demonstrations with
semantic annotations via state distribution matching to expedite RL training.
However, our emphasis lies on their ability to adapt in a zero-shot manner
across domains by developing a novel two-phase adaptation process.

#### Multi-modality in robotic manipulation.

In recent studies on robotic manipulation, several approaches leveraging
large-scale pretrained multi-modal models have been introduced, showing
promise in enhanced policy generalization by adeptly integrating both visual
and language instructions (Guhur et al. 2023; Shridhar, Manuelli, and Fox
2022; Liu et al. 2022). (Jang et al. 2022) proposed a multi-modal imitation
learning framework, which trains the vision encoder by maximizing the
similarity of video embeddings and the PLMs embedding vectors of video
descriptions. (Jiang et al. 2023) proposed a unified robot-manipulation
framework capable of integrating interleaved video-language inputs by encoding
them with pretrained vision and language models. To the best of our knowledge,
SemTra is the first to consider a unified multi-modal task prompt and
incorporate the knowledge of PLMs and parametric semantic skills into cross-
domain policy adaptation procedures.

##  6 Conclusion

We presented the semantic skill translator SemTra, a framework that leverages
the knowledge of PLMs to achieve zero-shot policy adaptation across different
domains. In the framework, we employ a two-tiered adaptation process in which
a sequence of multi-modal interleaved snippets serves as a task prompt,
facilitating task-level semantic skill generation and skill-level refinement
tailored to the target domain contexts. In our future work, we will explore
the versatility of PLMs more under intricate cross-domain conditions that
require dynamic adjustments of a skill sequence based on real-time
observations.

## Acknowledgements

We would like to thank anonymous reviewers for their valuable comments and
suggestions. This work was supported by Institute of Information &
communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government (MSIT) (No. 2022-0-01045, 2022-0-00043, 2019-0-00421,
2020-0-01821) and by the National Research Foundation of Korea (NRF) grant
funded by the MSIT (No. RS-2023-00213118).

## References

  * Ba, Kiros, and Hinton (2016) Ba, L. J.; Kiros, J. R.; and Hinton, G. E. 2016.  Layer Normalization.  _CoRR_ , abs/1607.06450.
  * BigScience Workshop (2022) BigScience Workshop. 2022.  BLOOM (Revision 4ab0472).
  * Brown et al. (2020) Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020.  Language Models are Few-Shot Learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chalaki et al. (2019) Chalaki, B.; Beaver, L.; Remer, B.; Jang, K.; Vinitsky, E.; Bayen, A. M.; and Malikopoulos, A. A. 2019.  Zero-Shot Autonomous Vehicle Policy Transfer: From Simulation to Real-World via Adversarial Learning.
  * Chane-Sane, Schmid, and Laptev (2023) Chane-Sane, E.; Schmid, C.; and Laptev, I. 2023.  Learning Video-Conditioned Policies for Unseen Manipulation Tasks.  _CoRR_ , abs/2305.06289.
  * Chen, Liu, and Chen (2020) Chen, J.; Liu, G.; and Chen, X. 2020.  AnimeGAN: a novel lightweight GAN for photo animation.  In _International symposium on intelligence computation and applications_ , 242–256. Springer.
  * Chen et al. (2021) Chen, L.; Lu, K.; Rajeswaran, A.; Lee, K.; Grover, A.; Laskin, M.; Abbeel, P.; Srinivas, A.; and Mordatch, I. 2021.  Decision Transformer: Reinforcement Learning via Sequence Modeling.  In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , 15084–15097.
  * Chowdhery et al. (2022) Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; Schuh, P.; Shi, K.; Tsvyashchenko, S.; Maynez, J.; Rao, A.; Barnes, P.; Tay, Y.; Shazeer, N.; Prabhakaran, V.; Reif, E.; Du, N.; Hutchinson, B.; Pope, R.; Bradbury, J.; Austin, J.; Isard, M.; Gur-Ari, G.; Yin, P.; Duke, T.; Levskaya, A.; Ghemawat, S.; Dev, S.; Michalewski, H.; Garcia, X.; Misra, V.; Robinson, K.; Fedus, L.; Zhou, D.; Ippolito, D.; Luan, D.; Lim, H.; Zoph, B.; Spiridonov, A.; Sepassi, R.; Dohan, D.; Agrawal, S.; Omernick, M.; Dai, A. M.; Pillai, T. S.; Pellat, M.; Lewkowycz, A.; Moreira, E.; Child, R.; Polozov, O.; Lee, K.; Zhou, Z.; Wang, X.; Saeta, B.; Diaz, M.; Firat, O.; Catasta, M.; Wei, J.; Meier-Hellstern, K.; Eck, D.; Dean, J.; Petrov, S.; and Fiedel, N. 2022.  PaLM: Scaling Language Modeling with Pathways.  _CoRR_ , abs/2204.02311.
  * Devlin et al. (2019) Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , 4171–4186.
  * Dosovitskiy et al. (2021) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_.
  * Dosovitskiy et al. (2017) Dosovitskiy, A.; Ros, G.; Codevilla, F.; López, A. M.; and Koltun, V. 2017.  CARLA: An Open Urban Driving Simulator.  In _1st Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings_ , 1–16.
  * Du et al. (2023) Du, Y.; Yang, M.; Dai, B.; Dai, H.; Nachum, O.; Tenenbaum, J. B.; Schuurmans, D.; and Abbeel, P. 2023.  Learning Universal Policies via Text-Guided Video Generation.  _CoRR_ , abs/2302.00111.
  * Fu et al. (2020) Fu, J.; Kumar, A.; Nachum, O.; Tucker, G.; and Levine, S. 2020.  D4RL: Datasets for Deep Data-Driven Reinforcement Learning.  _CoRR_ , abs/2004.07219.
  * Goyal, Mooney, and Niekum (2023) Goyal, P.; Mooney, R. J.; and Niekum, S. 2023.  Language-guided Task Adaptation for Imitation Learning.  _CoRR_.
  * Guhur et al. (2023) Guhur, P.-L.; Chen, S.; Pinel, R. G.; Tapaswi, M.; Laptev, I.; and Schmid, C. 2023.  Instruction-driven history-aware policies for robotic manipulations.  In _Conference on Robot Learning_ , 175–187.
  * Gupta et al. (2017) Gupta, A.; Devin, C.; Liu, Y.; Abbeel, P.; and Levine, S. 2017.  Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_.
  * Gupta et al. (2019) Gupta, A.; Kumar, V.; Lynch, C.; Levine, S.; and Hausman, K. 2019.  Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning.  In _3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings_ , volume 100 of _Proceedings of Machine Learning Research_ , 1025–1037.
  * Harrison et al. (2017) Harrison, J.; Garg, A.; Ivanovic, B.; Zhu, Y.; Savarese, S.; Fei-Fei, L.; and Pavone, M. 2017.  AdaPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems.  In _Robotics Research, The 18th International Symposium, ISRR 2017, Puerto Varas, Chile, December 11-14, 2017_ , 437–453.
  * HF Canonical Model Maintainers (2022) HF Canonical Model Maintainers. 2022.  gpt2 (Revision 909a290).
  * Huang et al. (2022) Huang, W.; Xia, F.; Xiao, T.; Chan, H.; Liang, J.; Florence, P.; Zeng, A.; Tompson, J.; Mordatch, I.; Chebotar, Y.; Sermanet, P.; Jackson, T.; Brown, N.; Luu, L.; Levine, S.; Hausman, K.; and Ichter, B. 2022.  Inner Monologue: Embodied Reasoning through Planning with Language Models.  In _Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand_ , Proceedings of Machine Learning Research, 1769–1782. PMLR.
  * Ichter et al. (2022) Ichter, B.; Brohan, A.; Chebotar, Y.; Finn, C.; Hausman, K.; Herzog, A.; Ho, D.; Ibarz, J.; Irpan, A.; Jang, E.; Julian, R.; Kalashnikov, D.; Levine, S.; Lu, Y.; Parada, C.; Rao, K.; Sermanet, P.; Toshev, A.; Vanhoucke, V.; Xia, F.; Xiao, T.; Xu, P.; Yan, M.; Brown, N.; Ahn, M.; Cortes, O.; Sievers, N.; Tan, C.; Xu, S.; Reyes, D.; Rettinghouse, J.; Quiambao, J.; Pastor, P.; Luu, L.; Lee, K.; Kuang, Y.; Jesmonth, S.; Joshi, N. J.; Jeffrey, K.; Ruano, R. J.; Hsu, J.; Gopalakrishnan, K.; David, B.; Zeng, A.; and Fu, C. K. 2022.  Do As I Can, Not As I Say: Grounding Language in Robotic Affordances.  In _Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand_ , Proceedings of Machine Learning Research, 287–318.
  * James et al. (2020) James, S.; Ma, Z.; Arrojo, D. R.; and Davison, A. J. 2020.  RLBench: The Robot Learning Benchmark & Learning Environment.  _IEEE Robotics Autom. Lett._ , 3019–3026.
  * Jang et al. (2022) Jang, E.; Irpan, A.; Khansari, M.; Kappler, D.; Ebert, F.; Lynch, C.; Levine, S.; and Finn, C. 2022.  BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning.  _CoRR_.
  * Jiang et al. (2023) Jiang, Y.; Gupta, A.; Zhang, Z.; Wang, G.; Dou, Y.; Chen, Y.; Fei-Fei, L.; Anandkumar, A.; Zhu, Y.; and Fan, L. 2023.  VIMA: General Robot Manipulation with Multimodal Prompts.  arXiv:2210.03094.
  * Kim et al. (2020) Kim, K.; Gu, Y.; Song, J.; Zhao, S.; and Ermon, S. 2020.  Domain Adaptive Imitation Learning.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , Proceedings of Machine Learning Research, 5286–5295.
  * Liu et al. (2022) Liu, H.; Lee, L.; Lee, K.; and Abbeel, P. 2022.  Instruction-Following Agents with Jointly Pre-Trained Vision-Language Models.  _arXiv preprint arXiv:2210.13431_.
  * Liu et al. (2018) Liu, Y.; Gupta, A.; Abbeel, P.; and Levine, S. 2018.  Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation.  In _2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018_ , 1118–1125.
  * Mees et al. (2022) Mees, O.; Hermann, L.; Rosete-Beas, E.; and Burgard, W. 2022.  CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks.  _IEEE Robotics Autom. Lett._ , 7.
  * Nam et al. (2022) Nam, T.; Sun, S.; Pertsch, K.; Hwang, S. J.; and Lim, J. J. 2022.  Skill-based Meta-Reinforcement Learning.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  _CoRR_ , abs/2303.08774.
  * Ott et al. (2019) Ott, M.; Edunov, S.; Baevski, A.; Fan, A.; Gross, S.; Ng, N.; Grangier, D.; and Auli, M. 2019.  fairseq: A Fast, Extensible Toolkit for Sequence Modeling.  arXiv:1904.01038.
  * Pertsch et al. (2022) Pertsch, K.; Desai, R.; Kumar, V.; Meier, F.; Lim, J. J.; Batra, D.; and Rai, A. 2022.  Cross-Domain Transfer via Semantic Skill Imitation.  In _Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand_ , 690–700.
  * Pertsch, Lee, and Lim (2020) Pertsch, K.; Lee, Y.; and Lim, J. J. 2020.  Accelerating Reinforcement Learning with Learned Skill Priors.  In _4th Conference on Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event / Cambridge, MA, USA_ , volume 155 of _Proceedings of Machine Learning Research_ , 188–204.
  * Qiu et al. (2022) Qiu, Y.; Wu, J.; Cao, Z.; and Long, M. 2022.  Out-of-Dynamics Imitation Learning from Multimodal Demonstrations.  In _Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand_ , volume 205 of _Proceedings of Machine Learning Research_ , 1071–1080.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021.  Learning Transferable Visual Models From Natural Language Supervision.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , 8748–8763.
  * Radford et al. (2019) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8): 9.
  * Rakelly et al. (2019) Rakelly, K.; Zhou, A.; Finn, C.; Levine, S.; and Quillen, D. 2019.  Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , 5331–5340.
  * Scao et al. (2022) Scao, T. L.; Fan, A.; Akiki, C.; Pavlick, E.; Ilic, S.; Hesslow, D.; Castagné, R.; Luccioni, A. S.; Yvon, F.; Gallé, M.; Tow, J.; Rush, A. M.; Biderman, S.; Webson, A.; Ammanamanchi, P. S.; Wang, T.; Sagot, B.; Muennighoff, N.; del Moral, A. V.; Ruwase, O.; Bawden, R.; Bekman, S.; McMillan-Major, A.; Beltagy, I.; Nguyen, H.; Saulnier, L.; Tan, S.; Suarez, P. O.; Sanh, V.; Laurençon, H.; Jernite, Y.; Launay, J.; Mitchell, M.; Raffel, C.; Gokaslan, A.; Simhi, A.; Soroa, A.; Aji, A. F.; Alfassy, A.; Rogers, A.; Nitzav, A. K.; Xu, C.; Mou, C.; Emezue, C.; Klamm, C.; Leong, C.; van Strien, D.; Adelani, D. I.; and et al. 2022.  BLOOM: A 176B-Parameter Open-Access Multilingual Language Model.  _CoRR_.
  * Shafiullah et al. (2022) Shafiullah, N. M. M.; Cui, Z. J.; Altanzaya, A.; and Pinto, L. 2022.  Behavior Transformers: Cloning k modes with one stone.  _CoRR_ , abs/2206.11251.
  * Shin et al. (2023) Shin, S.; Lee, D.; Yoo, M.; Kim, W. K.; and Woo, H. 2023.  One-shot Imitation in a Non-Stationary Environment via Multi-Modal Skill.
  * Shridhar, Manuelli, and Fox (2022) Shridhar, M.; Manuelli, L.; and Fox, D. 2022.  Cliport: What and where pathways for robotic manipulation.  In _Conference on Robot Learning_ , 894–906. PMLR.
  * van den Oord, Li, and Vinyals (2018) van den Oord, A.; Li, Y.; and Vinyals, O. 2018.  Representation Learning with Contrastive Predictive Coding.  _CoRR_.
  * van den Oord, Vinyals, and Kavukcuoglu (2017) van den Oord, A.; Vinyals, O.; and Kavukcuoglu, K. 2017.  Neural Discrete Representation Learning.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , 6306–6315.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All you Need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , 5998–6008.
  * Xu et al. (2021) Xu, H.; Ghosh, G.; Huang, P.; Okhonko, D.; Aghajanyan, A.; Metze, F.; Zettlemoyer, L.; and Feichtenhofer, C. 2021.  VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , 6787–6800.
  * Xu et al. (2022) Xu, M.; Shen, Y.; Zhang, S.; Lu, Y.; Zhao, D.; Tenenbaum, J. B.; and Gan, C. 2022.  Prompting Decision Transformer for Few-Shot Policy Generalization.  In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , Proceedings of Machine Learning Research, 24631–24645.
  * Yu et al. (2018) Yu, T.; Finn, C.; Xie, A.; Dasari, S.; Zhang, T.; Abbeel, P.; and Levine, S. 2018.  One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning.  _CoRR_ , abs/1802.01557.
  * Yu et al. (2019) Yu, T.; Quillen, D.; He, Z.; Julian, R.; Hausman, K.; Finn, C.; and Levine, S. 2019.  Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning.  In _3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings_ , 1094–1100.
  * Zhao et al. (2022) Zhao, M.; Liu, F.; Lee, K.; and Abbeel, P. 2022.  Towards More Generalizable One-shot Visual Imitation Learning.  In _2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022_ , 2434–2444. IEEE.
  * Zintgraf et al. (2020) Zintgraf, L. M.; Shiarlis, K.; Igl, M.; Schulze, S.; Gal, Y.; Hofmann, K.; and Whiteson, S. 2020.  VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_.

##  Appendix A Implementation Details

###  A.1 SemTra Implementation

Our model is implemented using Python v3.8.16 and automatic gradient framework
Jax v0.4.4. We train the models on a system with an NVIDIA RTX 4090 GPU.
SemTra consists of six parameterized learnable models: multi-modal skill
encoder $\Phi_{E}$, semantic skill sequence generator $\Phi_{G}$, skill
boundary detector $\Phi_{B}$, context encoder $\Phi^{(g)}_{C}$, online context
encoder $\Phi^{(o)}_{C}$, and behavior decoder $\pi$. We will now provide
detailed explanations of their implementations.

#### Multi-modal skill encoder.

To implement the video skill encoder $\Psi_{V}$, which maps a video snippet
$x=(v_{0:T})$ to skill-level language tokens, we utilize the Video-CLIP
(V-CLIP) (Xu et al. 2021) which consists of a video encoder $\Psi_{V}$ and a
language encoder $\Psi_{L}$. The implementation of this video skill encoder is
based on an open-source project by (Ott et al. 2019). The prompt $\theta_{p}$
is a set of learnable parameters, which is concatenated with the text to form
a comprehensive input vector for the language encoder $\Psi_{L}$ in (4).

For mapping a sensor snippet $x=(s_{t-H^{\prime}:t},a_{t-H^{\prime}:t})$ to a
skill-level language token, we utilize a Transformer-based classifier (Vaswani
et al. 2017) $\Psi_{S}$, which predicts the corresponding semantic annotation
$l_{t}$. The hyperparameter settings for the multi-modal skill encoder
$\Phi_{E}$ are summarized in Table A.9.

Hyperparameter | Value
---|---
Size of $\theta_{p}$ | 16 vector in 768-dim
Batch size for $\theta_{p}$ | 1
Learning rate for $\theta_{p}$ | 1e-4
The number of layer for $\Psi_{S}$ | 4
The number of head for $\Psi_{S}$ | 4
Activation fn for $\Psi_{S}$ | Relu
Dropout for $\Psi_{S}$ | 0.1
Hidden dimension for $\Psi_{S}$ | 256
Batch size for $\Psi_{S}$ | 256
Learning rate for $\Psi_{S}$ | 5e-4

Table A.9: Hyperparameters for the multi-modal skill encoder.

#### Semantic skill sequence generator.

To train the semantic skill sequence generator $\Phi_{G}$, we employ a
relatively small-sized pretrained language model GPT2 (Radford et al. 2019)
with 124.43 million parameters, obtained from the open-source Huggingface
library (HF Canonical Model Maintainers 2022). Other pretrained models used in
Figure 4 of the main manuscript were also obtained from Huggingface’s models
(HF Canonical Model Maintainers 2022; BigScience Workshop 2022). For the model
from scratch $\Phi^{(scr)}_{G}$ in Figure 4, we train an encoder-decoder
Transformer having 7.05 million parameters from scratch, using the Bert
(Devlin et al. 2019) token embedding. The hyperparameter settings for the
semantic skill sequence generator $\Phi_{G}^{(scr)}$ are summarized in Table
A.10.

Hyperparameter | Value
---|---
Batch size for $\Phi^{(GPT)}_{G}$ | 3
Learning rate for $\Phi^{(GPT)}_{G}$ | 1e-5
Batch size for $\Phi^{(scr)}_{G}$ | 3
Learning rate for $\Phi^{(scr)}_{G}$ | 1e-4
The number of layer of the encoder for $\Phi^{(scr)}_{G}$ | 1
The number of head of the encoder for $\Phi^{(scr)}_{G}$ | 8
The number of layer of the decoder for $\Phi^{(scr)}_{G}$ | 1
The number of head of the decoder for $\Phi^{(scr)}_{G}$ | 4
Feed-forward network dim. for $\Phi^{(scr)}_{G}$ | 32
Activation fn for $\Phi^{(scr)}_{G}$ | Relu
Dropout for $\Phi^{(scr)}_{G}$ | 0.15

Table A.10: Hyperparameters for the skill sequence generator.

#### Skill boundary detector.

To train the skill boundary detector $\Phi_{B}$, we utilize pseudo-labeling by
assigning each timestep of the expert trajectory $\tau^{(tgt)}$ with the
predicted semantic skill $z_{t}=\Psi_{V}(v_{t-H:t})$, using the video skill
encoder $\Psi_{V}$. As discussed in Section 4.3 of the main manuscript, if
semantic labels already exist for all timesteps, we directly use them as
$z_{t}$. Since the number of times the semantic skill changes in a target
trajectory is relatively low compared to the total length of timesteps, we
observe that this results in a low proportion of positive labels for the skill
boundary detector to predict. To address this, we extend the positive labels
around the timesteps where the actual semantic skill changes by $\pm J$. The
hyperparameter settings for the skill boundary detector $\Phi_{B}$ are
summarized in Table A.11.

Hyperparameter | Value
---|---
Batch size for $\Phi^{(g)}_{G}$ | 3
Learning rate for $\Phi^{(g)}_{G}$ | 1e-5

Table A.11: Hyperparameters for the skill boundary detector.

#### Context encoder.

To train the context encoder $\Phi^{(g)}_{C}$, we fine-tune a pretrained GPT2
model (HF Canonical Model Maintainers 2022) similar to the semantic skill
sequence generator. The context encoder takes natural language input, which
consists of a skill-level language instruction derived from the task prompt
and a semantic skill sequence connected with conjunctions. The target output
is the executable skill sequence for the target trajectory. Each executable
skill is represented as a tuple in natural language form, consisting of (skill
semantic, domain context, magnitude), e.g., (“Open door”, “Speed”, “Slow”).
The procedure of creating the target trajectory and the corresponding task
prompt for each trajectory is described in detail in Appendix Section B. The
hyperparameter settings for the context encoder $\Phi_{C}$ are summarized in
Table A.12.

Hyperparameter | Value
---|---
Network architecture for $\Phi_{B}$ | 4 FC with 128 units
Batch size for $\Phi_{B}$ | 256
Learning rate for $\Phi_{B}$ | 1e-4
Activation fn for $\Phi_{B}$ | Leaky Relu
Dropout for $\Phi_{B}$ | 0.0
Positive label augmentation $J$ | 4

Table A.12: Hyperparameters for the context encoder

#### Online context encoder.

To train the online context encoder $\Phi^{(o)}_{C}$, we adopt a contrastive
learning-based dynamics encoder, as proposed in (Shin et al. 2023; Qiu et al.
2022). Specifically, this model takes an expert sub-trajectory
$\tau=(s_{t-H^{\prime}:t},a_{t-H^{\prime}:t-1})$ as input and encodes the
dynamics parameter $h_{t}$ of the environment. This is quantized into a
discrete value using vector quantization (van den Oord, Vinyals, and
Kavukcuoglu 2017). For contrastive learning, two different sub-trajectories
are considered as positive pairs if they are sampled from the same trajectory.
The online context encoder $\Phi^{(o)}_{C}$ is jointly trained with the
inverse dynamics decoder $\psi_{dec}$, which takes the output $h_{t}$ of
$\Phi^{(o)}_{C}$ as input and reconstructs the sub-trajectory. The learning
process is controlled by the coefficient $\alpha$ of the contrastive loss. The
hyperparameter settings for the online context encoder $\Phi^{(o)}_{C}$ are
summarized in Table A.13.

Hyperparameter | Value
---|---
Network architecture for $\Phi^{(o)}_{C}$ | LSTM with 32 units
Batch size for $\Phi^{(o)}_{C}$ | 256
Learning rate for $\Phi^{(o)}_{C}$ | 1e-4
Activation function for $\Phi^{(o)}_{C}$ | Relu
Sub-trajectory length $H^{\prime}$ | 3
Coefficient $\alpha$ for Meta-World | 1.0
Coefficient $\alpha$ for Franka Kitchen | 0.1
Dimension of codebook for $\Phi^{(o)}_{C}$ | 16
The number of codebooks for $\Phi^{(o)}_{C}$ | 50
Network architecture for $\psi_{dec}$ | 2 FC with 32 units
Batch size for $\psi_{dec}$ | 256
Learning rate for $\psi_{dec}$ | 1e-4
Activation function for $\psi_{dec}$ | Relu

Table A.13: Hyperparameters for the online context encoder

#### Behavior decoder.

To train the behavior decoder $\pi$, we utilize a multi-layer perceptron (MLP)
with layer normalization (Ba, Kiros, and Hinton 2016). The hyperparameter
settings for the behavior decoder $\pi$ are summarized in Table A.14.

Hyperparameter | Value
---|---
Network architecture of $\pi$ for Meta-World | 7 FC with 128 units
Network architecture of $\pi$ for Franka kitchen | 4 FC with 128 units
Batch size for $\pi$ | 256
Learning rate for $\pi$ | 1e-4
Activation function of $\pi$ for Meta-World | Relu
Activation function of $\pi$ for Franka kitchen | Leaky Relu

Table A.14: Hyperparameters for the behavior decoder

###  A.2 Baseline Implementation

#### VIMA.

VIMA is a framework that accomplishes various robotics tasks, such as one-shot
imitation or rearrangement, through interleaved multi-modal prompting of video
and language. We adopt the same tokenizing technique as presented in the
original paper to address multi-modal snippets. Specifically, the frames of
video snippets are embedded using a pretrained vision Transformer (Dosovitskiy
et al. 2021). Text snippets are embedded using a pretrained language model T5,
with the last 2 layers fine-tuned through behavior cloning loss. As the
original paper does not cover sensor snippets, we additionally train an MLP
$\Psi_{S}$ with the same output dimension as the embeddings of other snippets,
using behavior cloning loss. The embeddings of each snippet are concatenated
with the current state $s_{t}$ and used as input to a cross-attention-based
Transformer $\pi$. The hyperparameter settings for VIMA are summarized in
Table A.15.

Hyperparameter | Value
---|---
Batch size | 256
Learning rate | 1e-4
Network architecture for $\Psi_{S}$ | 1 FC with 768 units
Learning rate for $\pi$ | 1e-4
The number of layers for $\pi$ | 3
The number of heads for $\pi$ | 4
Subsequence length | 20

Table A.15: Hyperparameters for VIMA

#### TP-BCZ.

BCZ (Jang et al. 2022) is the state-of-the-art in multi-modal imitation
learning framework, which employs contrastive learning to increase similarity
in the multi-modal space between videos and their corresponding instructions
or captions. To adapt this idea to the cross-domain context, we implement TP-
BCZ, which trains an additional regression network $\psi_{g}$ which performs
contrastive learning to maximize the similarity between the embeddings of task
prompt $\mathcal{P}$ and the embeddings of language instruction of the
corresponding target trajectory. The embedding method for each snippet in the
task prompt is the same as in SemTra, and we construct the language
instruction of the target trajectory by concatenating the semantic skill
sequence with the conjunction “and”. The policy $\pi$ is conditioned on the
embeddings of task prompt $\mathcal{P}$. The hyperparameter settings for TP-
BCZ are summarized in Table A.16.

Hyperparameter | Value
---|---
Batch size | 256
Network architecture for $\psi_{g}$ | 3 FC with 256 units
Learning rate for $\psi_{g}$ | 1e-4
Network architecture for $\pi$ | 4 FC with 128 units
Learning rate for $\pi$ | 1e-4

Table A.16: Hyperparameters for TP-BCZ

#### TP-GPT.

TP-GPT is implemented based on a Decision Transformer (Chen et al. 2021), a
model tailored for imitation learning. To handle multi-modal snippets, we
utilize pretrained large models, where each frame of the video snippet is
embedded using the image encoder of CLIP (Radford et al. 2021), and the text
snippet is embedded using the language model Bert (Devlin et al. 2019). For
processing the sensor snippet, we employ an MLP $\Psi_{S}$ while ensuring that
it has the same dimension of output as the embeddings of other snippets. The
embeddings of each snippet are concatenated with the current state $s_{t}$ and
input into a self-attention-based Transformer $\pi$. The hyperparameter
settings for TP-GPT are summarized in Table A.17.

Hyperparameter | Value
---|---
Batch size | 256
Learning rate | 1e-4
Network architecture for $\Psi_{S}$ | 1 FC with 768 units
Learning rate for $\pi$ | 1e-4
The number of layers for $\pi$ | 3
The number of heads for $\pi$ | 4
Subsequence length | 20

Table A.17: Hyperparameters for TP-GPT

#### VCP.

We implement VCP (Video-Conditioned Policy) based on video-conditioned
policies (Chane-Sane, Schmid, and Laptev 2023; Du et al. 2023). This model
trains a video embedding generator $\psi_{g}$, which takes the embedding of
the task prompt $\mathcal{P}$ as input and generates the video embedding of
the target trajectory. The generated video embedding, along with the current
timestep state $s_{t}$, is then presented to the policy $\pi$ to predict the
action. For cross-domain evaluation, we explicitly employ snippet embeddings
that hold cross-domain guidance for task prompting in addition to video
generation and state. Similar to TP-BCZ, each snippet of the multi-modal task
prompt $\mathcal{P}$ is embedded using a set of pretrained large models. The
hyperparameter settings for VCP are provided in Table A.18.

Hyperparameter | Value
---|---
Batch size | 256
Network architecture for $\psi_{g}$ | 3 FC with (256, 128, 256)
Learning rate for $\psi_{g}$ | 1e-5
Network architecture for $\pi$ | 4 FC with 128 units
Learning rate for $\pi$ | 1e-4

Table A.18: Hyperparameters for VCP

##  Appendix B Environments and Datasets

We test our framework in various environments, including Franka Kitchen (Fu et
al. 2020), Meta-World (Yu et al. 2019), RLBench (James et al. 2020), and CARLA
(Dosovitskiy et al. 2017), as described in Figure A.10. In this section, we
provide detailed descriptions of each environment and training dataset
collection.

Figure A.6: Franka kitchen

Figure A.7: Meta-World

Figure A.8: RLBench

Figure A.9: CARLA

Figure A.10: Evaluation environments

###  B.1 Task prompts

To generate training datasets, we execute a series of semantic skills across
several different domain contexts. The continuity of each trajectory is
maintained by setting the final state of one semantic skill as the starting
point for the subsequent one. For each trajectory involving both the semantic
skill sequence and the skill-level domain context, we then compute a plausible
task prompt, which might have been given, based on predefined language
templates. It is worth noting that this linkage between the task prompt and
the target trajectory is a straightforward linguistic process that operates
independently of trajectory generation. Here, we provide a typical example of
a task prompt in Listing 1, and specific examples used in each environment are
detailed in Sections B.2 and B.3. Each task prompt includes placeholders
SRC_TASK, SEQ_REQ, CONSTRAINED_SK, and MAGNITUDE.

  * •

SRC_TASK contains information specifying the source task in a multi-modal form
(e.g., sensor data, video demonstration performing “Open microwave” and then
“Light a kitchen”, or text itself).

  * •

SEQ_REQ forms the task-level cross-domain contexts. For example, if “Replace
lighting a kitchen with open a cabinet” is provided in this placeholder
SEQ_REQ, then, combined with the example of SRC_TASK, the agent needs to
perform “Open microwave” and then “Open a cabinet.”

  * •

CONSTRAINED_SK represents a specific skill or “all skills” to which the DEGREE
will be applied.

  * •

MAGNITUDE: Specifies the degree related to the considered domain context
(e.g., “flurry” for the domain context wind, and “fast” for the domain context
speed).

Listing 1: Task prompt examples for Franka Kitchen and Meta-World.

⬇

Ex1. "Do {SRC_TASK}, under {SEQ_REQ}. Furthermore, for {CONSTRAINED_SK} the
speed {MAGNITUDE} is applied."

Ex2. "When undertaking {SRC_TASK}, ensure {SEQ_REQ} and adapt the speed
{MAGNITUDE} for {CONSTRAINED_SK}."

Ex3. "While carrying out {SRC_TASK}, comply with {SEQ_REQ} and speed
modifications {MAGNITUDE} to {CONSTRAINED_SK}."

Ex4. "{SRC_TASK} is performed with {SEQ_REQ} and speed {MAGNITUDE} during
{CONSTRAINED_SK}."

Ex5. "During {SRC_TASK}, {SEQ_REQ} is maintained as the wind {MAGNITUDE}
blows."

Ex6. "While performing {SRC_TASK} and following {SEQ_REQ}, the wind
{MAGNITUDE}."

Ex7. "The wind {MAGNITUDE} blows during {SRC_TASK}, while adhering to
{SEQ_REQ}."

Ex8. "{SEQ_REQ} is upheld while {SRC_TASK} and the wind {MAGNITUDE} blows."

###  B.2 Franka Kitchen

In this environment, a 7-DoF robot arm needs to perform sequential subtasks,
such as “open microwave” and “slide cabinet”. Each trajectory in the dataset
has a length of 200-500 timesteps, and the timestep limit for the target
environment is set to 280. The domain context in this environment is
represented by the wind blowing in the environment, i.e., “breeze”, “gust”,
and “flurry”.

To obtain trajectories executed in such domain contexts, we use the offline
dataset provided by the offline-RL benchmark D4RL (Fu et al. 2020). The first
three axes of actions in the dataset, which represent the positions of the
robot arm’s grippers, are modified by adding 3 different constants $c$,
similar to the approach in (Shin et al. 2023). The value of $c$ is set to 0.05
for “breeze,” 0.2 for “gust,” and 0.4 for “flurry”. This results in different
action distributions according to the instructed cross-domain contexts.
Additionally, 4 different parameters $w\in\\{-0.6,-0.3,0.3,0.6\\}$ are added
to the fourth axis. This generates a total of 6,612 trajectories (by
$551\times 3\times 4$). As the original dataset does not include semantic
annotations, we manually mapped semantic annotations to each timestep by
judging the video between two timesteps where positive rewards occurred (i.e.,
the start and completion points of a subtask).

During the evaluation, we introduce time-varying parameters into the fourth
axis of agents’ actions to simulate a non-stationary environment as explained
in Section C.1. Examples of task prompts used in Franka Kitchen are in Listing
2 and Figure A.11.

Listing 2: Task prompt examples for Franka Kitchen.

⬇

# Single-domain

Ex1. "Do the actions presented in this {SRC_TASK}."

Ex2. "Carry out {SRC_TASK} sequentially."

# Task-level cross-domain

Ex3. "Do the actions presented in this {SRC_TASK}, but replace moving the
kettle with turning on the light."

Ex4. "Carry out {SRC_TASK} in reverse order."

# Skill-level cross-domain

Ex5. "Do the actions presented in this {SRC_TASK}. The flurry wind blows."

Ex6. "Carry out {SRC_TASK} sequentially, as the gust wind blows."

# Task-level and Skill-level cross-domain

Ex7. "Do the actions presented in this {SRC_TASK}. But replace moving the
kettle with turning on the light as the blurry blows."

Ex8. "Carry out {SRC_TASK} in reverse order, as the gust wind blows."

Figure A.11:  Examples of multi-modal task prompts for Franka Kitchen. The
prompt comprises (video) demonstration, text, and sensor data. A numeric index
$n$ in the figure corresponds to a semantic skill (e.g., 0 corresponds to
“open microwave”). Certain snippets contain cross-domain beneficial
information, such as the magnitude of wind blowing in the target environment.

###  B.3 Meta-World

In this environment, a 3-DoF robot arm needs to perform sequential subtasks,
such as “push button” and “close drawer”. To create a multi-stage setting
similar to the Franka Kitchen environment, we utilize a multi-stage Meta-World
environment (Shin et al. 2023). In this environment, we use 16 training tasks,
and for each task, we generate 15 episodes.

The domain context in this environment includes the execution speed of the
skill and the external wind blowing. To control the speed of semantic skill
execution in three degrees (Slow, Normal, and Fast), we use a rule-based
expert policy. Depending on the speeds, each trajectory in the dataset has a
length of 200-900 timesteps, and the timestep limit for the target environment
is set to 1,000. For the domain context of wind, similar to the process used
in the Franka Kitchen environment, we construct three different domain
contexts. Additionally, 4 different parameters $w\in\\{-0.6,-0.3,0.3,0.6\\}$
are added to the third axis.

As a result, this setup comprises 5,760 trajectories (by $16\times 15\times
2\times 3\times 4)$. We compute the number of timesteps required for executing
each semantic skill in each domain context using the collected offline
dataset, as shown in Table A.19. During evaluations, if a task prompt includes
a domain context related to the speed requirement, we determine the success or
failure of the agent by calculating a confidence interval of 70% based on the
previously computed number of timesteps.

Figure A.12: Multi-modal task prompts for Meta-World. The prompt comprises
video, text, and sensor data. Certain snippets contain instructed cross-domain
context, such as the speed requirement.

Furthermore, we introduce time-varying parameters into the third axis of
agents’ actions to simulate a non-stationary environment as explained in
Section C.1. Examples of task prompts used in Franka Kitchen are in Listing 2.
Examples of task prompts used in Meta-World are in Listing 3 and Figure A.12.

Domain Slide puck Close drawer Push button Open door Slow 166.08 162.23 177
374.35 Normal 100.73 84.68 68.2 177 Fast 75.8 56.62 53.1 104.34

Table A.19:  Performance in autonomous driving.  Listing 3: Task prompt
examples for Meta-World.

⬇

# Single-domain

Ex1. "Do {SRC_TASK} in sequential order."

Ex2. "Sequentially follow {SRC_TASK}."

# Task-level cross-domain

Ex3. "Do {SRC_TASK} in reverse order."

Ex4. "Replace opening the door with closing the drawer when following
{SRC_TASK}."

# Skill-level cross-domain

Ex5. "Do {SRC_TASK} in sequential order, but during opening the drawer, the
speed slow is applied."

Ex6. "Sequentially follow {SRC_TASK} and adapt the slow speed for pushing the
puck into the goal."

# Task-level and Skill-level cross-domain

Ex7. "Replace opening the door with closing the drawer when following
{SRC_TASK}, but during opening the drawer, the speed slow is applied."

Ex8. "Replacing opening the door with closing the drawer when following
{SRC_TASK}, and adapt the slow speed for pushing the puck into the goal."

###  B.4 RLBench

This environment simulates the cognitive robot manipulation scenario discussed
in Table 7 of the main manuscript. In this environment, an 8-DoF robot arm
needs to perform sequential subtasks, such as “pick up red block”, and “close
the box”. To compute the affordance score of Saycan (Ichter et al. 2022) in
Table 7, we collect semantically annotated datasets comprising 460
trajectories. We use a rule-based policy, which performs given skill-level
instruction, for the evaluation of both SayCan and our framework.

In these scenarios, we explore three different command styles: A Summarized
sentence command that offers an abstract instruction rather than direct skill-
level instruction for the agent; An Abstract verbs command that encompasses
synonymous verbs to communicate actions; An Embodiment command that provides
the current environment conditions and the constraints of the robots.

For evaluations, we leverage advanced PLMs such as GPT3.5, GPT4, and PaLM in a
zero-shot manner, in a similar way of (Ichter et al. 2022). Instead of fine-
tuning, we engineer an augmented task prompt to contain beneficial information
for semantic skill planning, which is presented to the PLMs with the
respective commands. The examples of abstract commands and augmented
engineered prompts are in Linsting 4.

Listing 4: Engineered augmented prompt and abstract commands examples for
RLBench.

⬇

# Prompt

Ex1. "You are a model trained to sequence tasks. You have a list of seven
skills:

pick up red block

pick up blue block

pick up yellow block

close the box

open the box

place the block on the desk

place the block in the box

Your task is to sequence these skills in a logical order to complete a task.
You can only use the skills in the list.

After all skills have been used, write ‘Finish’. Please list the skills in the
order they should be performed, starting with ‘1. First skill 2. Second
skill’, and so on".

/* Summarized Sentence */

# Command examples

Ex2. "Place the red block into the closed box. Currently, Could you suggest an
appropriate skill sequence to handle this situation?"

Ex3. "Move the blue block into the closed box. Currently, Could you suggest an
appropriate skill sequence to handle this situation?"

Ex4. "Put the yellow block into the closed box. Currently, Could you suggest
an appropriate skill sequence to handle this situation?"

/* Abstract verbs */

# Command examples

Ex5. "Given task is executed in the following skill sequence:

1. open the box

2. move the red block in the box

3. close the box

4. Finish

Currently, Could you suggest an appropriate skill sequence to handle this
situation?"

Ex6. "Given task is executed in the following sequence:

1. move the red block on the top of the blue block to the desk

2. open the box

3. move the blue block in the box

4. close the box

5. Finish

Currently, Could you suggest an appropriate skill sequence to handle this
situation?"

Ex7. "Given task is executed in the following sequence:

1. pick up {block_color_1} block

2. place the block on the desk

3. open the box

4. move the {block_color_2} block in the box

5. close the box

6. Finish

Currently, Could you suggest an appropriate skill sequence to handle this
situation?"

/* Embodiment */

# Command examples

Ex8. "Given task is executed in the following skill sequence:

1. open the box

2. pick up red block

3. place the block in the box

4. close the box

5. Finish

Currently, a blue block is placed on top of the box lid and it cannot be
opened. Could you suggest an appropriate skill sequence to handle this
situation?"

Ex9. " Given task is executed in the following skill sequence:

1. open the box

2. pick up red block

3. place the block in the box

4. close the box

5. Finish

Currently, the box lid is already open. Could you suggest an appropriate skill
sequence to handle this situation?"

Ex10. "Given task is executed in the following skill sequence:

1. open the box

2. pick up {block_color_1} block

3. place the block in the box

4. close the box

5. Finish

Currently, there is a {block_color_2} block is placed on top of the
{block_color_1} block so you cannot pick it up. Could you suggest an
appropriate skill sequence to handle this situation?"

Figure A.13:  Cross-domain context affects the action distribution of
adaptation policy: In the figure, a numeric index corresponds to a specific
semantic skill (for example, 0 corresponds to ”open microwave”). We use t-SNE
to visualize the distribution of actions, which sequentially perform four
semantic skills in a trajectory. Each subfigure consists of three distinct
trajectories, each of which performs the same order of semantic skill
sequences but in different domain contexts. As observed, while the action
sequences show a similar path when the semantics are the same, but
perturbations occur based on the domain context.

###  B.5 CARLA

In this environment, the agent needs to adjust a 3-dimensional action for
steering, throttle, and brake to reach a goal position via the shortest path.
In this setting, we consider different car configurations (agent embodiments),
such as sedan and truck, as distinct domain contexts. As described in Figure 5
of the main manuscript, the actual actions to be executed for skills, “going
straight”, “turning left”, and “turning right”, need to be adjusted according
to the different car configurations.

Regarding the dataset generation, we collect 200 expert trajectories by
implementing a waypoint-based expert policy for each car configuration. The
evaluation task involves the sequential execution of five skills: “going
straight”, “turning right”, “going straight”, “turning left”, and “going
straight”, to reach the goal. The training dataset only contains trajectories
composed of four or fewer skills; no trajectories involve all five skills in
sequence. Consequently, the agent has to adapt to the target domain task by
composing the skills learned during training, and aligning them with the
domain context (car configuration).

Listing 5: Task prompt examples for CARLA.

⬇

# Training task prompts

Ex1. "Go right, and then go straight to drive a truck."

Ex2. "Drive a truck go straight, and then go right."

Ex3. "Go straight, and then go right to drive a sedan."

Ex4. "Drive a sedan with go left, and then go straight."

# Evaluation task prompts

Ex5. "Go straight, then go right, then go straight, then go left, and then go
straight, to drive a sedan."

Ex6. "Drive a truck with go straight, then go right, then go straight, then go
left, and then go straight."

##  Appendix C Experiment Details

Algorithm 2 Evaluation of SemTra

1: Task prompt $\mathcal{P}$

2: Multi-modal skill enc. $\Phi_{E}$, skill sequence generator $\Phi_{G}$

3: Skill boundary detector $\Phi_{B}$, context encoder $\Phi^{(g)}_{C}$

4: Online context encoder $\Phi^{(o)}_{C}$, history buffer $\mathcal{B}$

5: Behavior decoder $\pi$ /* Task interpretation */

6: $\eta\leftarrow\Phi_{E}(\mathcal{P})$ using (3)

7: $\xi=(z_{j_{1}},\cdots,z_{j_{J}})\leftarrow\Phi_{G}(\eta)$ using (6)

8: $(\bar{z}_{j_{1}},\cdots\bar{z}_{j_{J}})\leftarrow\Phi^{(g)}_{C}(\eta,\xi)$
using (9) /* Skill interpretation */

9: $t\leftarrow 0$, $i\leftarrow 0$, done $\leftarrow$ False

10: $s_{t}\leftarrow$ env.reset(), $s_{t_{0}}\leftarrow s_{0}$,
$\mathcal{B}$.append($s_{t}$)

11: while not done do

12: $h_{t}\leftarrow\Phi^{(o)}_{C}(s_{t-H^{\prime}:t},a_{t-H^{\prime}:t-1})$
using (11)

13: $a_{t}\leftarrow\pi(s_{t},\bar{z}_{j_{i}},h_{t})$ using (10)

14: $(s_{t+1},\textnormal{done})\leftarrow$ env.step($a_{t}$)

15: $\mathcal{B}$.append($(s_{t+1},a_{t})$)

16: $m\leftarrow\Phi_{B}(s_{t},z_{j_{i}},s_{t_{0}})$ using (7), $t\leftarrow
t+1$

17: if $m=1$ then

18: $s_{t_{0}}\leftarrow s_{t}$, $i\leftarrow i+1$

19: end if

20: end while

###  C.1 Complete results

To implement a cross-domain scenario featuring temporal domain shift, we
employ a non-stationary environment. In this setup, a time-varying parameter
$w_{t}$ is added to specific axes of actions before an agent interacts with
the environment; this interaction follows the transition probability

$s_{t+1}\sim\mathbb{P}(s_{t+1}|s_{t},a_{t}+w_{t}),$ (A.1)

which the probability changes over time.

We explore four distinct non-stationary settings, each paired with three
distinct instructed domain contexts (i.e., contexts that can be encapsulated
in the task prompt or demonstration). This results in a total of twelve cross-
domain contexts. These contexts are combined with nine unseen tasks (i.e.,
unseen order of subtasks) in Franka Kitchen and eight unseen tasks in Meta-
World, forming 108 and 96 cross-domain environment scenarios, respectively.

Table A.20 presents the performance on a total of 12 cross-domain contexts,
which are combinations of the three types of skill-level domain contexts and
the four types of non-stationary contexts. Due to their absence of history-
aware model architecture, TP-BCZ and VCP are evaluated solely on instructed
domain contexts, as described in Table A.21.

Below are the different types of time-varying parameters.

  * •

Sin:

$w_{t}=\begin{cases}0.6\times\textnormal{sin}(\frac{8\pi\times
t}{T})\,\,\,\textnormal{if}\,\,\,u_{t}\leq 0.5\\\
0.25\times\textnormal{sin}(\frac{8\pi\times
t^{2}}{T})\,\,\,\textnormal{if}\,\,\,u_{t}>0.5\end{cases}$ (A.2)

  * •

Bias(m)

$w_{t}=\begin{cases}-m+\textnormal{Unif}(0,0.1)\,\,\,\textnormal{if}\,\,\,u_{t}\leq
0.25\\\ m+\textnormal{Unif}(0,0.1)\,\,\,\textnormal{if}\,\,\,u_{t}>0.25\\\
\end{cases}$ (A.3)

  * •

Bias(M)

$w_{t}=\begin{cases}-M+\textnormal{Unif}(0,0.1)\,\,\,\textnormal{if}\,\,\,u_{t}\leq
0.25\\\ M+\textnormal{Unif}(0,0.1)\,\,\,\textnormal{if}\,\,\,u_{t}>0.25\\\
\end{cases}$ (A.4)

  * •

Zigzag

$w_{t}=\begin{cases}0.45\,\,\,\textnormal{if}\,\,\,u_{t}\leq 0.5\\\
-0.45\,\,\,\textnormal{if}\,\,\,u_{t}>0.5\\\ \end{cases}$ (A.5)

where $u_{t}\sim\textnormal{Unif}(0,1)$ is a random value sampled at each
timestep $t$.

Env. Domain VIMA TP-GPT SemTra Franka Kitchen breeze, sin 39.82% 42.59% 59.26%
breeze, bias(m) 32.40% 42.59% 67.59% breeze, bias(M) 35.18% 42.59% 64.81%
breeze, zigzag 35.18% 40.74% 50.00% gust, sin 13.89% 15.74% 54.62% gust,
bias(m) 15.74% 14.82% 62.03% gust, bias(M) 14.81% 10.19% 64.81% gust, zigzag
10.19% 10.19% 58.33 % flurry, sin 10.18% 0.00% 66.67% flurry, bias(m) 7.41%
0.00% 66.67% flurry, bias(M) 6.48% 0.00% 66.67% flurry, zigzag 7.41% 1.86%
55.56% Meta- World normal, sin 4.17% 15.28% 73.61% normal, bias(m) 4.17%
11.11% 81.94% normal, bias(M) 1.39% 6.94% 80.56% normal, zigzag 1.39% 12.50%
80.56% slow, sin 1.39% 9.73% 56.94% slow, bias(m) 0.00% 8.33% 58.33% slow,
bias(M) 0.00% 8.33% 62.50% slow, zigzag 2.78% 11.11% 55.56% fast, sin 0.00%
8.33% 86.11% fast, bias(m) 0.00% 8.33% 86.11% fast, bias(M) 0.00% 16.67%
79.17% fast, zigzag 0.00% 11.11% 80.56%

Table A.20: Performances in environments without temporal domain shifts.

Env. Domain VIMA TP-BCZ TP-GPT VCP SemTra Franka Kitchen seq, breeze 48.15%
22.22% 59.26% 43.52% 73.15% rev, breeze 46.30% 14.82% 50.93% 62.97% 71.30%
rep, breeze 30.95% 15.48% 48.81% 26.19% 64.29% seq, gust 38.89% 23.15% 17.59%
15.74% 69.45% rev, gust 38.89% 22.22% 26.86% 20.37% 71.30% rep, gust 23.81%
11.90% 25.00% 27.39% 64.29% seq, flurry 8.33% 19.45% 1.86% 16.67% 60.19% rev,
flurry 10.19% 25.30% 2.78% 18.52% 62.04% rep, flurry 3.57% 16.67% 2.39% 38.10%
58.33% Meta- World seq, stand. 16.67% 15.28% 22.22% 20.93% 80.56% rev, stand.
5.56% 8.33% 22.22% 8.33% 76.39% rep, stand. 8.89% 8.89% 20.00% 2.22% 80.00%
seq, slow 5.56% 12.50% 8.33% 4.17% 94.44% rev, slow. 2.78% 12.50% 5.56% 9.72%
79.17% rep, slow. 4.45% 11.11% 13.33% 0.00% 97.78% seq, fast. 12.50% 0.00%
8.33% 11.11% 73.61% rev, fast. 5.56% 0.00% 6.95% 15.28% 72.22% rep, fast.
6.67% 2.22% 11.11% 4.44% 60.00%

Table A.21: Performances in environments with temporal domain shifts.

###  C.2 Visual cross-domain evaluation

We record real-world video demonstrations of subtasks that construct Franka
Kitchen and Meta-World environments at a resolution of $700\times 700$ in
residential kitchens and offices, respectively. Then, to generate a noisy
human demonstration, we add a Gaussian noise which is sampled from the
distribution of $\mathcal{N}(0,1)$. Finally, to generate an illustrative
(animation) demonstration we use AnimeGAN (Chen, Liu, and Chen 2020), which
transfers the video style from real world to animation. These videos are
videos were then cropped to a size of 224x224 and used as input to V-CLIP (Xu
et al. 2021). Examples of the real-world videos are showcased in Figure A.14.

Figure A.14:  Examples of real-world videos used for visual cross-domain
scenarios. From top to bottom, the video represents “turn on burner”, “open
microwave”, “open door”, and “close drawer”. Each video snippet constitutes a
multi-modal task prompt.

#### Video encoder tuning for human demonstrations.

To analyze the effect of fine-tuning the VLM-based video skill encoder in (4),
we evaluate our framework with a task prompt involving noisy human-level
demonstrations in various scenarios: demonstrations without noise (wo/ noise
in the Demo column of Table A.22), noisy demos, and illustrative (animation)
videos. Noise-free demos are accurately translated to semantic skills due to
the domain-invariant semantic knowledge of the pre-trained VLM (wo/ F.T in the
column name). Fine-tuning the skill encoder (w/ F.T) enhances the skill
prediction when a demonstration is semantically annotated. Since the skill
sequence generator is trained in the language semantic space, it generates
suitable skill sequences, resulting in only a minor performance drop.

Env Domain Skill prediction Adaptation wo/ F.T w/ F.T wo/ F.T w/ F.T Franka
Kitchen wo/ noise 94.44% 97.22% 57.41% 70.37% w/ noise 86.11% 94.44% 67.59%
69.44% Illust. 75.00% 94.44% 37.04% 62.97% Meta- World wo/ noise 100.0% 100.0%
75.00% 75.00% w/ noise 83.33% 95.83% 55.56% 69.44% Illust. 79.17% 91.67%
40.28% 69.50%

Table A.22: Fine-tuning for human demonstrations.