# Video-Language Critic: Transferable Reward
Functions for Language-Conditioned Robotics

Minttu Alakuijala1 Reginald McLean2 Isaac Woungang2 Nariman Farsad2
Samuel Kaski1,3 Pekka Marttinen1 Kai Yuan4

1Department of Computer Science, Aalto University
2Department of Computer Science, Toronto Metropolitan University
3Department of Computer Science, University of Manchester
4Intel Corporation

<EMAIL_ADDRESS>
###### Abstract

Natural language is often the easiest and most convenient modality for humans
to specify tasks for robots. However, learning to ground language to behavior
typically requires impractical amounts of diverse, language-annotated
demonstrations collected on each target robot. In this work, we aim to
separate the problem of _what_ to accomplish from _how_ to accomplish it, as
the former can benefit from substantial amounts of external observation-only
data, and only the latter depends on a specific robot embodiment. To this end,
we propose Video-Language Critic, a reward model that can be trained on
readily available cross-embodiment data using contrastive learning and a
temporal ranking objective, and use it to score behavior traces from a
separate reinforcement learning actor. When trained on Open X-Embodiment data,
our reward model enables 2x more sample-efficient policy training on Meta-
World tasks than a sparse reward only, despite a significant domain gap. Using
in-domain data but in a challenging task generalization setting on Meta-World,
we further demonstrate more sample-efficient training than is possible with
prior language-conditioned reward models that are either trained with binary
classification, use static images, or do not leverage the temporal information
present in video data.111Source code, data and trained models are available at
https://github.com/minttusofia/video_language_critic.

##  1 Introduction

Advances in natural language processing and vision-language representations
have enabled a significant increase in the scalability and generalization
abilities of learned control policies for robotics. Methods involving large
architectures, such as Transformers [42], and internet-scale pretraining have
transferred well to both high-level [17, 43] and low-level [5, 21, 34] robotic
control. Natural language has many desirable features as a modality for
specifying tasks. Unlike structured, hand-designed task sets, natural language
is unrestricted and open-domain. Moreover, prompts can be specified as
precisely or vaguely as appropriate. While goal images, demonstration videos,
or goal states more broadly, have been considered as an alternative open-
domain task definition modality [7, 2, 23], they typically have to specify
irrelevant environment details, such as the background. Furthermore, language
readily supports task definitions with novel combinations of actions, objects
and their attributes (e.g. _pick up the large green ball_) as well as subtask
sequencing (e.g. _place the toy inside the box, then close the box_) in a way
that facilitates the policy’s understanding of unseen tasks.

The majority of prior work has proposed to learn language-conditioned policies
end-to-end, i.e., directly predicting an action in the robot’s action space
given the current state and task description. However, this has several
downsides: first, fitting large models on the full problem requires a
significant amount of high-quality demonstration data from the target robot
domain. Second, the resulting policy depends entirely on the specific robot
instance, observation space, and controller type (e.g., joint or task space
control) and does not easily transfer to other settings. Moreover, much of the
prior work addresses vision-language grounding in robotics purely with
imitation learning [5, 21, 34, 27], without attempting to discriminate between
low-quality and expert demonstrations. As a result, the resulting policies are
inherently limited by the skills of the demonstrators, and no novel solutions
can be discovered through planning or interaction. This line of work overlooks
performance gains that could be obtained by converting the language prompts to
a scalar reward function. Manually defining a well-specified dense reward
function to communicate task success is typically laborious and error-prone,
and must be repeated for each task. To make progress towards a general-purpose
robotic system that can learn human-level skills both in terms of quality
(dexterity, robustness) and capability (variety of skills), we argue these
systems will need to be able to critique their own behavior, by learning
reward functions at scale.

We address this problem by learning a foundation _video-language-conditioned
reward model_ , i.e., a critic that evaluates the progress (in the form of a
video) of a task, given as a human-language instruction, and assigns a reward
based on how close the robot is to completing the task. By leveraging large
cross-task pretraining data, which may come from a variety of robots, our
Video-Language Critic (VLC), can learn to score the alignment between a
textual description and task execution regardless of the specific robot
embodiment. Our experimental evaluation on Meta-World [46] manipulation tasks
shows that VLC can learn useful general-purpose reward functions not only from
in-domain, but also out-of-domain data (Open X-Embodiment [27]) collected from
different robot embodiments. In Section 4, we show that VLC 1) accelerates the
training of a wide range of manipulation tasks and 2) enables zero-shot
learning on unseen tasks, when combined with a sparse task completion signal.

Figure 1: Overview of Vision-Language Critic: Our similarity function,
$S_{\theta}$, is trained using video-caption pairs $(v_{1:T}^{i},c^{i})$. The
visual encoder (ViT, [9]) is applied separately to each video frame $v_{t}$ to
produce a sequence of image features, which are appended to the caption
embedding produced by the text encoder. The temporal aggregator then predicts
a similarity score for each time step $t$ of the video. We use cross-entropy
and sequence ranking objectives to encourage the predicted scores to be high
for matching video-caption pairs and low for mismatching pairs, and to
monotonously increase over a successful execution.

Recent work in defining language-conditioned rewards for robotic manipulation
has used either binary classification [33, 35, 25], contrastive vision-
language alignment [38, 22, 26] or reconstruction [15] objectives. However,
prior methods have not fully leveraged temporal ordering of frames to
encourage increasing scores over a successful episode. We focus on providing a
dense reward function that evaluates the actor’s progress throughout an
episode by posing learning as a contrastive ranking task.

The advantages and contributions of our approach are as follows:

  * •

We learn vision-language manipulation using actor-agnostic videos and
instructions at scale without requiring tedious demonstration collection on a
specific robot. Unlike end-to-end policy learning, our method can learn from
cross-embodiment data without action labels.

  * •

Through maximizing the learned reward, our policies can improve over sub-
optimal demonstrations, by executing the task faster or by finding better
solutions.

  * •

Our method, VLC, enables a 3x sample efficiency gain over a sparse task-
completion reward, or 2x when trained exclusively on cross-embodiment data
with a significant domain gap.

  * •

VLC generalizes to unseen tasks through large-scale pretraining and language
conditioning and leads to faster policy training than prior reward learning
methods.

  * •

VLC is agnostic to the type of policy learning, and can be combined with
model-free or model-based reinforcement learning, affordance-based grasping or
model-predictive control.

##  2 Related Work

#### Vision-language imitation

Many prior works have aimed to connect language instructions and vision-based
observations in robotics [5, 21, 34, 20, 12] and in video games [10], mostly
through large-scale demonstrations [5, 21, 20, 10] or pretraining [34, 12].
However, the majority of approaches have considered imitation-based objectives
only, without ranking existing trajectories using offline reinforcement
learning (RL) or attempting to outperform prior data using model-based or
model-free RL. This has the obvious downside of requiring large amounts of
high-quality data from the target domain with low-quality examples already
filtered out, and limits the performance of the agent to predicting the mean
of the demonstrated behavior instead of improving over it. We instead propose
to learn a state-value function from cross-domain offline behavior, which can
be optimized using either online, offline or model-based policy training.

#### Multi-modal representations

Pretrained vision-language representations [28] have been adapted to a wide
range of downstream tasks [34, 12]. Shridhar et al. [34] propose to augment
pretrained CLIP [28] with Transporter nets [48] to handle fine-grained spatial
awareness required for precise manipulation. Xiao et al. [44] train a CLIP-
like contrastive embedding space from crowd-sourced language annotations for
trajectories from the robot. We draw inspiration from these works, but instead
define an embodiment-agnostic, language-conditioned reward function, which
supports improvement over demonstration data.

#### Video retrieval

Our work is closely related to video retrieval as we seek to move beyond
image-language correspondence and match task descriptions with history-aware
state representations. As the task of learning representations across time is
computationally expensive, many prior works have proposed to start from
pretrained, CLIP-like static image features and aggregate them over time,
while fine-tuning the aggregation function’s weights on video retrieval [19,
3, 18]. Unlike in video retrieval, we aim to not only assign high alignment
scores to full videos, but provide smoothly increasing reward over the whole
video to indicate task progress.

#### Inverse RL

Several works have proposed to infer the reward function of a task using
examples of expert behavior, and to train an RL policy to optimize this reward
[30]. Most relevantly to our setting, a line of prior inverse RL methods
considers the case where the observed behavior is not annotated with actions
and may come from different action and observation spaces altogether,
typically a human demonstrator [7, 2, 23, 33, 35, 25, 31, 32, 47]. Many of
these works use either a goal image [2, 23, 47] or a demonstration video [7,
32] rather than language conditioning, and some are only applicable for data
from a single task at a time [31, 47]. Moreover, handling multi-task reward
learning with an additional task identifier state variable, as done by Chen et
al. [7], requires a predefined grouping into a discrete set of tasks. By
contrast, our use of language to define tasks enables a more subtle and
composable task space.

#### Language-conditioned inverse RL

Although a few prior works have used unrestricted natural language to define
rewards for robotic manipulation tasks using either binary classification [33,
35, 25] or contrastive vision-language alignment [10, 26, 38, 22], these
methods have not fully leveraged the temporal ordering of frames in their
objective to encourage increasing scores over a successful episode. We instead
propose to explicitly learn increasing rewards for partial trajectories making
progress towards solving the task. Moreover, most prior methods use only a
single image or a pair of images, whereas we consider the full episode to
better represent partially observable tasks. Moreover, many prior works only
considered data from the actor’s own observation space [35, 25, 10], whereas
we propose to learn from cross-embodiment data allowing zero-shot transfer to
a robot with different morphology, kinematics, and visual appearance.

Specifically, a few prior works [10, 29, 4] have explored CLIP-based models
for learning vision-language reward functions. However, these works have been
limited to image features of the current time step [29, 4] or a snapshot of
the most recent history [10]. We instead propose to score behavior at the time
series level by comparing the task description and a full video trajectory.
Despite its name, RoboCLIP [38] (proposed to solve a similar problem to our
work) is not based on the CLIP architecture or the pretrained representations
unlike our method, but the S3D architecture [45], and mainly shares the
InfoNCE loss with CLIP.

##  3 Video-Language Critic

We propose to learn language-conditioned robotic manipulation by first
training an embodiment-agnostic reward function on video-caption pairs, and
then using the learned reward model to guide the autonomous training of a
robot-specific policy. To serve as a useful reward signal for downstream
policy learning, the learned function should accurately represent the intended
task, while providing enough signal to the agent to enable efficient learning
[1, 36, 37, 39]. It should exhibit at least two key properties: _accuracy_ and
_temporal smoothness_. Making progress in the specified task should be
rewarded with positive feedback with as little delay as possible, i.e., the
function should smoothly increase over a successful execution. In fact, the
problem of optimal reward shaping is equivalent to learning the value function
for the optimal policy [39], suggesting that an optimal densely shaped reward
should monotonously increase over a successful demonstration (assuming the
reward we ultimately wish to maximize corresponds to sparse goal reaching).
Moreover, the end-of-episode scores for successful trajectories should exceed
those of incomplete or failed executions: classification accuracy between
successes and failures should be high. With these desiderata, we formulate
Video-Language Critic, a language-conditioned reward model trained with cross-
entropy and sequential ranking objectives to encourage progressively
increasing scores over a successful video’s duration.

###  3.1 Contrastive video-language training

Our approach is motivated by the success of contrastive image-language
pretraining and the wide applicability of pretrained CLIP [28] encoders as
foundation models. The problem of comparing observed behavior to a desired
task description is analogous to the setting of CLIP; however, we extend the
contrastive learning approach to scoring videos. Compared to a single image,
using sequences of frames sampled across the full trajectory increases the
generality of our reward function, and could allow it to handle non-Markovian
(i.e., history-dependent) tasks. Such tasks might involve partial
observability, repetitive or circular movements, or be described relative to
an earlier state; even simple object displacement tasks may fall in this
category.

#### Reward model architecture

We define video and text encoder networks similar to CLIP4Clip used for video
retrieval [19], the task of finding videos within an existing dataset that
most closely match a given textual query. The general architecture is shown in
Fig. 1. First, each video frame is processed with an image encoder network
while the video caption is processed with a text encoder, both initialized
with CLIP in order to benefit from its large-scale vision-language
pretraining. Luo et al. [19] tested different aggregation strategies for
reasoning over the resulting sequence of image features. In video retrieval,
averaging image features over time was found to be sufficient, and no
performance benefit could be obtained with an attention-based aggregation.
While video retrieval shares similarities with our setting, task progress
evaluation requires a much more nuanced understanding of temporal dynamics:
for one, reversing the video should typically result in a very different
reward value.

To support this, it is necessary for the temporal aggregation function to
process the input frames as an ordered sequence, which in the case of
Transformer aggregators is achieved using position embeddings. Furthermore, we
find that embedding both modalities independently of each other and comparing
with a cosine similarity, as done by CLIP as well as CLIP4Clip’s _sequence
Transformer_ aggregation, causes the resulting video representation to lose
too much of the information relevant to task completion. Thus, instead of
using cosine similarity, we train a single temporal aggregation Transformer to
directly output a similarity score based on the concatenated textual and image
features. Each encoder’s weights are fine-tuned and the aggregation function
is trained from scratch for the video-text matching task. We refer to the full
architecture, consisting of both the encoder networks and the aggregation
Transformer, as a similarity function $S_{\theta}$, parameterized by $\theta$.

#### Contrastive objective

As training signal, we wish to leverage weak supervision from video-level
captions without known spatial or temporal extent. We use a contrastive
objective function to encourage each caption to better match its corresponding
video than other videos, and vice versa. We therefore train the similarity
prediction network $S_{\theta}$ with symmetric cross-entropy as done by
Radford et al. [28], i.e., with the mean of text-to-video and video-to-text
cross-entropy terms, for video-caption pairs $(v^{i},c^{i}),i=1..N$:

$\displaystyle\mathcal{L}_{xent}=(\mathcal{L}(v^{1:N},c^{1:N})+\mathcal{L}(c^{1:N},v^{1:N}))/2,$
(1)

with the cross-entropy loss from modality $x$ to modality $y$ defined as:

$\displaystyle\mathcal{L}(x^{1:N},y^{1:N})=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(S_{\theta}(x^{i},y^{i}))}{\sum_{j=1}^{N}\exp(S_{\theta}(x^{i},y^{j}))}.$
(2)

#### Sequential ranking objective

Video inputs also contain implicit information about the relative _ranking_ of
states, which is not leveraged in prior reward learning approaches [25, 10,
38, 26, 15, 22]. We propose to learn from this temporal signal by extending
the cross-entropy objective with a sequential ranking term. Each subsequent
state in a successful trajectory should, in general, have higher value for
completing the task than its predecessors, which the reward function should
reflect. Our total loss then becomes:

$\displaystyle\mathcal{L}_{VLC}=\mathcal{L}_{xent}+\frac{\alpha}{N}\sum_{i=1}^{N}\sum_{t=1}^{|v^{i}|-1}|S_{\theta}(v^{i}_{1:t},c^{i})-S_{\theta}(v^{i}_{1:t+1},c^{i})|_{+}$
(3)

where $\alpha$ is a hyperparameter balancing both objective terms and
$|x|_{+}$ denotes $\max(x,0)$.

In order to ensure the reward model learns to discriminate videos based on
task completion rather than simply the presence of relevant objects in the
scene, it may be beneficial to include failure examples featuring similar
environments. Task failures are typically easier to generate than success
examples and, as a result, are fairly inexpensive to collect. When available,
we leave these videos uncaptioned and treat them only as additional negatives
in contrastive learning, and do not include them in the ranking loss.

##  4 Experiments

We demonstrate the accuracy and effectiveness of the learned video-language
rewards with experimental evaluation on simulated robotic manipulation tasks
from the Meta-World benchmark [46]. We evaluate VLC’s ability to inform
successful policy training in three settings of increasing difficulty. First,
we assess the ability of our model to jointly represent several robotic tasks
with a single language-conditioned prediction network in Section 4.1. Second,
we test our models’ ability to generalize to unseen Meta-World tasks with the
help of vision-language pretraining as well as extrapolation from training
tasks in Section 4.2. In Section 4.3, we demonstrate our method’s out-of-
domain transfer ability: VLC is used to learn an embodiment-agnostic reward
function for any language-conditioned manipulation task by observing a variety
of robot actors from Open X-Embodiment [27], a large dataset collected from a
variety of real-world robots in different environments. We further report
comparisons to prior work, both quantitatively and qualitatively, in Section
4.4. Finally, we demonstrate VLC’s effectiveness in planning with a known
dynamical model in Section 4.5.

#### Reward model training details

We perform hyperparameter selection, ablation studies, and finalize all
training details on data from VLMBench [49], a second robotic benchmark. This
avoids Meta-World specific tuning and allows for a fairer comparison with
prior work. Details on this dataset and our model ablations are included in
Appendix A. We observe the same hyperparameters perform well both on VLMBench
videos and in interactive RL policy training on Meta-World, which highlights
the method’s applicability across domains. Further training details are
reported in Appendix B.

#### Policy training

To train control policies, we use Soft Actor-Critic (SAC) [13], an off-policy
RL algorithm. The policy $\pi(a_{t}|s_{t})$ is trained to produce actions
$a_{t}$ conditioned on the current environment state $s_{t}$ in order to
maximize $\mathbb{E}[R_{t}]$, the expectation of discounted cumulative reward
from the current time $t$ onwards:
$R_{t}=\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t}r_{t^{\prime}}$, where
$r_{t^{\prime}}$ is the reward obtained for taking action $a_{t^{\prime}}$ in
state $s_{t^{\prime}}$, $\gamma$ is a discount factor in $[0,1)$ and $T$ is
the maximum episode length. The reward function $r$ in our case is either a
sparse task completion reward, equal to $1$ if the task has been solved and
$0$ otherwise, or a learned dense function consisting of a weighted sum of the
trained reward function and the sparse reward. As the similarity
$S_{\theta}(v_{1:t},c)$ predicted by VLC can take arbitrary values, we apply a
few normalization operations to aid in the stability of RL optimization [41].
As predictions for the starting state can vary across initial states, even for
the same task, we shift the rewards of the episode such that $r_{1}=0$, as the
subsequent behavior should be scored relative to this state. In addition to
this, we apply the standard normalization logic from Gymnasium [40], which
scales reward values such that their exponential moving average has fixed
variance $(1-\gamma)^{2}$.

#### Environments

We evaluate our method in RL policy training on robotic manipulation tasks
from the Meta-World benchmark. As the focus of this work is the problem of
learning a foundation reward model, we keep to a standard single-task policy
training setup and condition on full state information as defined by Meta-
World (see Appendix D for more details on the environment definitions).
Compared to using image-only observations, this reduces training time and
computational cost while allowing us to demonstrate the effectiveness of our
reward function. We consider the subset of tasks that can be reliably solved
using the dense Meta-World rewards, which are manually specified for each
environment based on the full state of the environment. Specifically, we
include tasks that can be solved with $\geq$98% success within a maximum
training length of 800,000 steps. We further split these in half into 12 easy
(learned in $<$240,000 steps) and 13 hard tasks (240k – 800k steps).

###  4.1 Multi-task reward function

To validate VLC’s effectiveness as a multi-task reward function, we first
train our model on video data from all 50 tasks. We collect 40 video
demonstrations per task for a total video dataset of 2000 successful
executions. We further collect 1600 failure examples by replacing the
demonstrator’s actions with random actions with probability 0.7, and refer to
this joint dataset as MW50 (short for Meta-World). We do not make any
modifications to the data generating process to explicitly encourage
exploration, as we want to validate our method in the context of existing
offline data, which typically does not cover the full state space. A key
challenge VLC needs to overcome is to sufficiently generalize from the
successes and failures present in the data to evaluate out-of-distribution
trajectories, as the RL policy may act very differently from the demonstration
data.

The policy training results are shown in Table 1, with learning curves for the
hard task set in Fig. 2. To summarize learning speed with a single number, we
report success rate of the policy evaluated at the training length at which
the manually specified Meta-World reward solves the task to $\geq$98% success.
VLC trained on MW50 enables improved sample efficiency relative to the sparse
reward only, which demonstrates that VLC can sufficiently generalize to
trajectories not seen in demonstration data, and can effectively represent
task progress for multiple tasks at once. However, a few tasks, such as Handle
Press, are learned in so few trials even with sparse reward alone that there
is little room for improvement in reward design, and learning is instead
bottlenecked by the policy training’s sample efficiency. This is why the
biggest gains are obtained for the harder tasks.

Figure 2: Success rate of RL policies on hard Meta-World tasks, over the number of environment steps (mean of 5 random seeds and standard error). Dashed lines denote convergence ($\geq 0.98$). Table 1: Success rates (%) of sparse reward only training vs. VLC trained on either MW50 or Open X, evaluated at the training length at which the Meta-World hand-designed reward solves the task. | sparse only | VLC MW50 | VLC Open X
---|---|---|---
easy (12) | 53 $\pm$ 6 |  65 $\pm$ 5 |  54 $\pm$ 6
hard (13) | 62 $\pm$ 5 |  91 $\pm$ 3 |  72 $\pm$ 5
mean | 58 $\pm$ 4 |  78 $\pm$ 3 |  63 $\pm$ 4

###  4.2 Task generalization to unseen environments

Having verified VLC can effectively represent several tasks using language
conditioning, we further evaluate its ability to use language to generalize to
entirely unseen tasks. The goal is to see whether knowledge of training tasks,
which in part contain similar actions or objects as the held-out tasks (such
as Close Faucet in the training set and Open Faucet in the test set), as well
as pretrained CLIP language features, are sufficient to allow VLC to
generalize to novel tasks. For this purpose, we split the Meta-World tasks
into 40 training and 10 test tasks: every 5th task in alphabetical order is
reserved for testing. This leaves roughly 1600 successful and 1300
unsuccessful videos as training data – we refer to this subset as MW40. Of the
test tasks, 2 are in the easy set, 4 in hard and 4 are unsolved even with the
curated single-task Meta-World rewards, and hence not our primary evaluation
target.

Success rates of RL policies on these 6 held-out tasks are shown in Table 2,
with learning curves in Fig. 3. In most tasks, the addition of VLC rewards
improves sample efficiency of RL training despite the tasks having never been
seen in reward function training, with average success rates 34 percentage
points over the sparse baseline and an average sample efficiency improvement
of over 5x.

###  4.3 Embodiment generalisation to unseen domains

The advantage of our method, and pretraining a reward function in general, is
that no data collection on the target robot and in the target environment is
required. To demonstrate this, we train VLC on cross-embodiment data from Open
X-Embodiment [27]. We use the language-annotated subset, with a total of
698,000 episodes of diverse tasks filmed in various real-world robotic labs.
Although some of this data does feature the Sawyer robot used in Meta-World
simulations, this is only a marginal subset of 0.33% of the language-annotated
videos. Moreover, the domain gap remains significant due to real-world
variations in objects, backgrounds, lighting conditions, task instances and
instruction formats, as well as the embodiment gap between the simulated and
the real robots.

We successfully train policies (see Table 1 and Fig. 2) using Open X trained
models despite a significant domain gap, highlighting the generalizability of
large-scale vision-language training. We obtain an average 2.1x sample
efficiency gain relative to the sparse reward in tasks that are solved by both
rewards (sample efficiency is ill-defined if either does not solve the task),
with particularly large improvements in Handle Pull Side (7x), Reach Wall (7x)
and Slide Plate (6x) and an average 5 percentage point success rate increase
across all 25 tasks despite misrepresenting a few tasks. Note that unlike RT-X
[27], our method does not use action labels, and remains equally applicable on
observation-only data.

Figure 3: Success rate of RL policies on held-out Meta-World tasks, over the number of environment steps. VLC achieves improved sample efficiency compared to any prior approach in 3/6 tasks. Table 2: Success rates (%) of VLC and prior work trained on MW40, evaluated on 6 unseen tasks (2 easy, 4 hard) at the training length at which the Meta-World hand-designed reward solves the task. | LOReL | RoboCLIP | LIV | Voltron | R3M | VLC (ours) | sparse only
---|---|---|---|---|---|---|---
easy (2) | 45 $\pm$ 16 | 47 $\pm$ 26 | 30 $\pm$ 10 | 18 $\pm$ 10 |  22 $\pm$ 11 |  51 $\pm$ 15 | 39 $\pm$ 12
hard (4) | 72 $\pm$ 19 | 57 $\pm$ 12 | 63 $\pm$ 15 | 35 $\pm$ 11 | 24 $\pm$ 14 |  87 $\pm$ 17 | 42 $\pm$ 10
mean | 63 $\pm$ 18 | 53 $\pm$ 13 | 39 $\pm$ 11 | 29 $\pm$ 19 | 16 $\pm$ 10 |  75 $\pm$ 17 | 41 $\pm$ 18

###  4.4 Comparison to prior work

To validate the benefits of VLC, we compare its performance to prior language-
conditioned reward models LOReL [25], RoboCLIP [38], LIV [22], Voltron [15]
and R3M [26], each fine-tuned on MW40. Training and implementation details are
included in Appendix C.

We find VLC’s combination of cross-entropy and the sequential ranking
objective as well as full video conditioning to produce more informative
reward predictions than existing methods, as shown by faster policy training
on average in Table 2 and Fig. 3. Moreover, on qualitative inspection of the
shape of the predicted rewards (Fig. 4), we find VLC’s outputs to better
distinguish successes from failures compared to either LOReL or RoboCLIP, the
two strongest baselines. Thanks to its broad coverage of execution history and
the sequential loss term, VLC produces rewards that more smoothly increase
over time for successful executions than either prior method.

Figure 4: Qualitative evaluation of rewards predicted over time steps of an
example successful (above) and unsuccessful (below) test episode. Predictions
are offset so the episode start has reward 0, as done in policy training,
except for RoboCLIP, which only assigns a reward for the final time step in RL
training, as in the original work (the full curve is shown here for
visualization only). In this figure, rewards are further normalized per task
and method so that the success and failure episode rewards are comparable:
e.g., in Faucet Open, VLC assigns at most 75% of its reward prediction for the
end of the success episode to any step of the failure episode; this is similar
in spirit to the scale normalization used in policy training (which instead
normalizes running statistics). A good reward model gives higher rewards in
the top row than the bottom row. Good correlation with the Meta-World reward
also implies an understanding of the task, but is not strictly required for
successful RL training.

###  4.5 Model-based evaluation

As a pretrained reward function, VLC can also be very useful in the context of
model-based planning: any imagined future state can be scored, allowing the
planner to identify the best action plan. We demonstrate this in a proof-of-
concept experiment, where we do not learn the model but instead assume access
to a known transition model as well as action primitives. The action
primitives include grasping and reaching, parameterized by target positions
(such as the locations of objects detected in the scene), and are defined
using segments from the expert policies available in Meta-World. We evaluate
VLC’s ability to identify the action primitive with the correct execution for
a held-out task.

In each task, we compare one successful trajectory with 5 unsuccessful ones
with randomly sampled target positions. Two action primitives are required to
solve Assembly and Pick Out of Hole: first grasping the required object, then
moving it as specified, such as lifting it out of the hole. The action
primitive is selected with $\arg\max_{a}S_{\theta}(o(s_{t+1}),c)$ where $a$ is
the action primitive that transitions state $s_{t}$ to $s_{t+1}$, and $o$ is a
function describing the observation visible to $S_{\theta}$, in this case a
video.

In Table 3, we report the accuracy of our model in identifying executions
solving each held-out task, averaged over 50 episodes. VLC generalizes well to
these tasks zero-shot, with 80% mean accuracy.

Table 3: Accuracy (%) of VLC-MW40 in identifying a successful execution out of 6 samples. Plate Slide Side |  Stick Push |  Handle Pull Side |  Faucet Open |  Door Open |  Coffee Push |  Button Topdown Wall |  Reach |  Pick Out of Hole (Grasp) |  Pick Out of Hole (Goal) |  Assembly (Grasp) |  Assembly (Goal) |  Mean
---|---|---|---|---|---|---|---|---|---|---|---|---
100 | 20 | 96 | 100 | 40 | 94 | 54 | 82 | 100 | 100 | 97 | 100 | 80

##  5 Limitations

To learn reward functions from cross-embodiment data, a shared input
representation, typically images or video, is required. This limits the
application to tasks whose progress can be evaluated from vision; however,
additional robot-specific variables could be added to the policy’s input
representation. Furthermore, the size of the dataset we use in cross-
embodiment experiments is significant and imposes considerable computational
cost, which is why we chose to focus on Meta-World task generalization when
comparing to prior work, in order to be able to adequately train several
methods. Further work is needed to investigate the performance with smaller
datasets. For tasks that clearly deviate from the Open X training tasks, such
as in their length or specificity, additional data collection might be
beneficial. In our formulation, we focused on tasks whose rewards can be
simplified to goal reaching, but other rewards may be relevant (such as
penalizing large actions). We further assumed that a sparse task completion
reward can be provided, which may not always be the case.

In this work, we focused on simulated robotic manipulation tasks, and
evaluation on a real robot is left for further work. Moreover, extending VLC
to learn from videos of humans performing manipulation tasks, such as from the
Something-Something dataset [11], is a promising avenue for pushing the
generalization capabilities of pretrained reward functions in the future.

##  6 Conclusion

We proposed Video-Language Critic (VLC), an approach to pretrain a foundation
reward model for vision-language manipulation. In particular, we train our
critic using contrastive video-language alignment and a ranking loss
encouraging monotonic increases for successful trajectories. Our model
predicts a language-conditioned state-value function conditioned on only a
history of image observations, and can therefore readily be scaled to leverage
external observation data from other actors. VLC can be used for various
downstream tasks, such as model-free and model-based reinforcement learning.
Further, we experimentally validated its usefulness as a reward function in
out-of-distribution tasks (unseen, held-out environments in the same domain)
and out-of-domain tasks (an unseen environment and embodiment in a different
domain). Our experiments on Meta-World demonstrated improved results compared
to 5 prior methods and a sparse reward baseline, with success rate increases
of 12 and 34 percentage units, respectively, and sample efficiency gains of
65% and 300%, respectively. Unlike methods based on predicting expert actions
in a given embodiment, VLC remains applicable in the absence of action labels.
Therefore, it can be extended to equally learn from videos of humans in future
work.

## Acknowledgments and Disclosure of Funding

This work was supported by the Research Council of Finland (Flagship
programme: Finnish Center for Artificial Intelligence FCAI, and grants 345604,
352986, 358246, 359207), EU Horizon 2020 (European Network of AI Excellence
Centres ELISE, grants 951847, 101016775 and NextGenerationEU) and UKRI Turing
AI World-Leading Researcher Fellowship (EP/W002973/1). We acknowledge the
computational resources provided by the Aalto Science-IT Project from Computer
Science IT, and CSC for awarding this project access to the LUMI
supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland)
and the LUMI consortium through Finland.

## References

  * Ackley and Littman [1991] David Ackley and Michael Littman.  Interactions between Learning and Evolution.  _Artificial Life II_ , 1991.
  * Alakuijala et al. [2023] Minttu Alakuijala, Gabriel Dulac-Arnold, Julien Mairal, Jean Ponce, and Cordelia Schmid.  Learning reward functions for robotic manipulation by observing humans.  _International Conference on Robotics and Automation_ , 2023.
  * Bain et al. [2022] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  A clip-hitchhiker’s guide to long video retrieval.  _arXiv preprint arXiv:2205.08508_ , 2022.
  * Baumli et al. [2023] Kate Baumli, Satinder Baveja, Feryal Behbahani, Harris Chan, Gheorghe Comanici, Sebastian Flennerhag, Maxime Gazeau, Kristian Holsheimer, Dan Horgan, Michael Laskin, et al.  Vision-language models as a source of rewards.  _NeurIPS Workshop on Agent Learning in Open-Endedness_ , 2023.
  * Brohan et al. [2022] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich.  RT-1: Robotics transformer for real-world control at scale.  _arXiv preprint arXiv:2212.06817_ , 2022.
  * Buslaev et al. [2020] Alexander Buslaev, Vladimir I. Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A. Kalinin.  Albumentations: Fast and flexible image augmentations.  _Information_ , 11(2), 2020.
  * Chen et al. [2021] Annie S Chen, Suraj Nair, and Chelsea Finn.  Learning generalizable robotic reward functions from "in-the-wild" human videos.  _Robotics: Science and Systems_ , 2021.
  * Damen et al. [2022] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.  _International Journal of Computer Vision_ , 130:33–55, 2022.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _International Conference on Learning Representations_ , 2021.
  * Fan et al. [2022] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar.  MineDojo: Building open-ended embodied agents with internet-scale knowledge.  _Advances in Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.
  * Goyal et al. [2017] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.  The "something something" video database for learning and evaluating visual common sense.  _International Conference on Computer Vision_ , 2017.
  * Guhur et al. [2023] Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi, Ivan Laptev, and Cordelia Schmid.  Instruction-driven history-aware policies for robotic manipulations.  _Conference on Robot Learning_ , 2023.
  * Haarnoja et al. [2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _International Conference on Machine Learning_ , 2018.
  * Huang et al. [2022] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo.  CleanRL: High-quality single-file implementations of deep reinforcement learning algorithms.  _Journal of Machine Learning Research_ , 23(274):1–18, 2022.
  * Karamcheti et al. [2023] Siddharth Karamcheti, Suraj Nair, Annie S. Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang.  Language-driven representation learning for robotics.  _Robotics: Science and Systems_ , 2023.
  * Kingma and Ba [2015] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _International Conference on Learning Representations_ , 2015.
  * Liang et al. [2022] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng.  Code as policies: Language model programs for embodied control.  _arXiv preprint arXiv:2209.07753_ , 2022.
  * Lu et al. [2024] Haoyu Lu, Mingyu Ding, Yuqi Huo, Guoxing Yang, Zhiwu Lu, Masayoshi Tomizuka, and Wei Zhan.  Uniadapter: Unified parameter-efficient transfer learning for cross-modal modeling.  _International Conference on Learning Representations_ , 2024.
  * Luo et al. [2022] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  CLIP4clip: An empirical study of clip for end to end video clip retrieval and captioning.  _Neurocomputing_ , 508:293–304, 2022.
  * Lynch and Sermanet [2021] Corey Lynch and Pierre Sermanet.  Grounding language in play.  _Robotics: Science and Systems_ , 2021.
  * Lynch et al. [2023] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence.  Interactive language: Talking to robots in real time.  _IEEE Robotics and Automation Letters_ , 2023.
  * Ma et al. [2023a] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman.  LIV: Language-image representations and rewards for robotic control.  _International Conference on Machine Learning_ , 2023a.
  * Ma et al. [2023b] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang.  VIP: Towards universal visual reward and representation via value-implicit pre-training.  _International Conference on Learning Representations_ , 2023b.
  * Miech et al. [2019] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips.  _International Conference on Computer Vision_ , 2019.
  * Nair et al. [2022a] Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al.  Learning language-conditioned robot behavior from offline data and crowd-sourced annotation.  _Conference on Robot Learning_ , 2022a.
  * Nair et al. [2022b] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta.  R3M: A universal visual representation for robot manipulation.  _Conference on Robot Learning_ , 2022b.
  * Open X-Embodiment Collaboration et al. [2023] Open X-Embodiment Collaboration, Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Brian Ichter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Federico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Hao Su, Hao-Shu Fang, Haochen Shi, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jaehyung Kim, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jiajun Wu, Jialin Wu, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jitendra Malik, Jonathan Tompson, Jonathan Yang, Joseph J. Lim, João Silvério, Junhyek Han, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Zhang, Keyvan Majd, Krishan Rana, Krishnan Srinivasan, Lawrence Yunliang Chen, Lerrel Pinto, Liam Tan, Lionel Ott, Lisa Lee, Masayoshi Tomizuka, Maximilian Du, Michael Ahn, Mingtong Zhang, Mingyu Ding, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Pannag R Sanketi, Paul Wohlhart, Peng Xu, Pierre Sermanet, Priya Sundaresan, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martín-Martín, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Sherry Moore, Shikhar Bahl, Shivin Dass, Shuran Song, Sichun Xu, Siddhant Haldar, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Sudeep Dasari, Suneel Belkhale, Takayuki Osa, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xuanlin Li, Yao Lu, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yueh hua Wu, Yujin Tang, Yuke Zhu, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zhuo Xu, and Zichen Jeff Cui.  Open X-Embodiment: Robotic learning datasets and RT-X models.  _arXiv preprint arXiv:2310.08864_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  _International Conference on Machine Learning_ , 2021.
  * Rocamonde et al. [2024] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner.  Vision-language models are zero-shot reward models for reinforcement learning.  _International Conference on Learning Representations_ , 2024.
  * Russell [1998] Stuart Russell.  Learning agents for uncertain environments.  _Conference on Computational learning theory_ , pages 101–103, 1998.
  * Schmeckpeper et al. [2020] Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, and Chelsea Finn.  Reinforcement learning with videos: Combining offline observations with interaction.  _Conference on Robot Learning_ , 2020.
  * Sermanet et al. [2018] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain.  Time-contrastive networks: Self-supervised learning from video.  _International Conference on Robotics and Automation_ , 2018.
  * Shao et al. [2021] Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg.  Concept2robot: Learning manipulation concepts from instructions and human demonstrations.  _The International Journal of Robotics Research_ , 40(12-14):1419–1434, 2021.
  * Shridhar et al. [2022] Mohit Shridhar, Lucas Manuelli, and Dieter Fox.  CLIPort: What and where pathways for robotic manipulation.  _Conference on Robot Learning_ , 2022.
  * Silva et al. [2021] Andrew Silva, Nina Moorman, William Silva, Zulfiqar Zaidi, Nakul Gopalan, and Matthew Gombolay.  LanCon-learn: Learning with language to enable generalization in multi-task manipulation.  _IEEE Robotics and Automation Letters_ , 7(2):1635–1642, 2021.
  * Singh et al. [2009a] Satinder Singh, Richard L Lewis, and Andrew G Barto.  Where do rewards come from?  _Proceedings of the Annual Meeting of the Cognitive Science Society_ , 31, 2009a.
  * Singh et al. [2009b] Satinder Singh, Richard L Lewis, Jonathan Sorg, Andrew G Barto, and Akram Helou.  On separating agent designer goals from agent goals: breaking the preferences–parameters confound.  _Proceedings of the Annual Conference of the Cognitive Science Society_ , 2009b.
  * Sontakke et al. [2023] Sumedh Sontakke, Jesse Zhang, Séb Arnold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti.  RoboCLIP: One demonstration is enough to learn robot policies.  _Advances in Neural Information Processing Systems_ , 36, 2023.
  * Sorg [2011] Jonathan Daniel Sorg.  The optimal reward problem: Designing effective reward for bounded agents.  _PhD Thesis, University of Michigan_ , 2011.
  * Towers et al. [2023] Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis.  Gymnasium, 2023.
  * van Hasselt et al. [2016] Hado P van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver.  Learning values across many orders of magnitude.  _Advances in Neural Information Processing Systems_ , 29, 2016.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Vemprala et al. [2023] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor.  ChatGPT for robotics: Design principles and model abilities.  Technical Report MSR-TR-2023-8, Microsoft, February 2023.
  * Xiao et al. [2023] Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson.  Robotic skill acquistion via instruction augmentation with vision-language models.  _Robotics: Science and Systems_ , 2023.
  * Xie et al. [2018] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  _European Conference on Computer Vision_ , 2018.
  * Yu et al. [2019] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  _Conference on Robot Learning_ , 2019.
  * Zakka et al. [2022] Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi.  XIRL: Cross-embodiment inverse reinforcement learning.  _Conference on Robot Learning_ , 2022.
  * Zeng et al. [2021] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al.  Transporter networks: Rearranging the visual world for robotic manipulation.  _Conference on Robot Learning_ , 2021.
  * Zheng et al. [2022] Kaizhi Zheng, Xiaotong Chen, Odest Jenkins, and Xin Eric Wang.  VLMbench: A compositional benchmark for vision-and-language manipulation.  _Advances in Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.

## Supplementary Material

##  Appendix A Model ablations

###  A.1 VLMBench dataset

We use the VLMBench manipulation task suite to develop and validate our method
without any Meta-World specific tuning. For this purpose, we collect 2700
video demonstrations and 1600 failure cases from variations of the picking
task – covering different object shapes, sizes, colors and relative positions,
as well as distractor objects. The natural language instructions match this
diversity in task variants, such as _Grasp the cylinder_ or _Grasp the cyan
object_ , and require distinguishing relevant objects from distractors with
either absolute (color, shape) or relative (size, position) properties, such
as the the _larger_ or the _front_ object. For more details on the benchmark,
see Zheng et al. [49].

We use these VLMBench videos to validate VLC design decisions, but defining a
single informative metric on the dataset of video-caption pairs
$(v^{i},c^{i})$, $i=1,...,N$, is difficult. Test loss, video retrieval metrics
such as mean recall, or classification metrics such as area under the ROC
curve do not correspond well to the models’ ability to model task progress.
The main difficulty is that part of the caption-to-video matching task can be
solved by simply connecting objects referred to in the caption to objects
present in the scene, without considering temporal information or actual task
success.

To support informative evaluation of our models, we therefore further define a
set of 19 test episodes: in each test case, the same initialization of the
scene is used to generate alternative trajectories that grasp at different
objects in the scene, only one of which solves the correct task. The accuracy
over this set of videos is our main model selection metric of interest, i.e.,
in how many out of 19 instances does the model assign a higher score to the
successful video than any incorrect video from the same initialization. Out of
evaluation metrics available at training time, we find video-to-text cross-
entropy to correlate the most with this test-time accuracy, and so use this
metric on a set of validation trajectories to choose model checkpoints.

###  A.2 Ablation results

We compare two temporal aggregation methods as proposed by Luo et al. [19]:
the sequence transformer and the tight-type transformer. The sequence
transformer aggregates the sequence of image features
$[\text{ViT}(v_{1}),\text{ViT}(v_{2}),...,\text{ViT}(v_{T})]$ into a single
embedding vector $S_{\theta}(v_{1:T})$, which it then compares to the caption
embedding $\text{TextEnc}(c)$ with cosine similarity. The tight-type
transformer, on the other hand, includes the caption embedding as an
additional input to the temporal aggregator $S_{\theta}(v_{1:T},c)$, as shown
in Fig. 1.

We report the results of our ablation study in Table 4. In addition to the
choice of architecture, we observe performance gains from adding image
augmentations from the Albumentations library [6], by sampling frames randomly
from uniform intervals instead of deterministic uniform sampling, the addition
of the sequence ranking term, as well as considering failure examples only as
negatives in the contrastive objective, and report results using these
settings in Section 4.

Table 4: Accuracy on VLMBench test episodes for various model ablations. $\alpha$ is the weight of the sequence ranking loss term. Architecture | $\alpha$ | Data augmentations | Failures | Accuracy (%)
---|---|---|---|---
Sequence transf. | 0 | - | as negatives only | 51.6 $\pm$ 3.1
" | " | image | " | 62.1 $\pm$ 4.2
" | " | frame sampling | " | 44.2 $\pm$ 3.6
" | " | image & frame sampling | " | 60.0 $\pm$ 4.3
" | " | image & frame sampling | yes | 72.6 $\pm$ 7.1
Tight type | " | - | as negatives only | 52.6 $\pm$ 4.7
" | " | image | " | 72.6 $\pm$ 3.1
" | " | frame sampling | " | 55.8 $\pm$ 8.4
" | " | image & frame sampling | " | 80.0 $\pm$ 4.5
" | " | image & frame sampling | yes | 76.8 $\pm$ 4.6
" | 3.3 | image & frame sampling | as negatives only | 82.1 $\pm$ 4.3
" | 10 | " | " | 83.2 $\pm$ 2.0
" | 33 | " | " |  88.4 $\pm$ 5.4
" | 100 | " | " | 85.3 $\pm$ 3.1

##  Appendix B Training details

###  B.1 Reward training

We subsample the videos to 12 time steps. Capping the maximum video length is
a practical choice both in terms of learning ability and computational cost.
We keep the default value of 12 frames in CLIP4Clip, though we change these to
be linearly sampled from across the entire video. Informed by the findings of
our ablation studies in Section A, at training time, we additionally apply
image augmentations and randomize frame sampling. We set $\alpha$, the ranking
loss weight, to 33 based on accuracy on VLMBench test episodes.

Reward training on Meta-World videos took 2 hours for MW50 on a single NVIDIA
A100 GPU, and 1 hour for MW40 on a GeForce RTX 3090 GPU. Training on the
significantly larger Open X-Embodiment dataset took 256 hours (nearly 11 days)
on a single A100. However, we believe this length could be greatly reduced in
future work by improving data loading throughput and running on multiple GPUs.

###  B.2 Policy training

For RL training experiments, we adapt the SAC implementation of CleanRL [14].
Policy evaluation is done every 20,000 timesteps for 50 episodes. Both the
actor and critic networks contain three hidden layers of size 400, and
optimization is done using Adam [16]. Other algorithm hyperparameters were
kept at the implementation’s default values.

We experimentally set the relative weights of the VLC and sparse reward
components to 1 and 50, respectively, the motivation being that the sparse
reward, once obtained, should be able to override the dense intermediate
reward predictions. We chose these values after testing three other settings:
(0.01, 10), (0.1, 10), and (0.1, 20), which also performed quite well. For
sparse reward only experiments, we did not find significant differences in the
scale of the reward, but for some tasks using a weight of 50 seemed to perform
better than 1 or 10, so we report results using this value for consistency
with the VLC experiments.

Training length and hence computational cost varies considerably across tasks.
We terminate training after convergence to $\geq$98% success (averaged over
the 10 most recent evaluations), or after a maximal training length set per
task. The resulting average training length across all 25 tasks was 600k
environment steps for VLC MW50 and 750k for the Open X trained model. The
corresponding GPU hours vary slightly based on exact architecture used, but we
obtained approximately 750k steps in 24 hours on a single NVIDIA V100 or P100
GPU. Our total computational budget was therefore 19.2 GPU hours $\times$ 5
random seeds $\times$ 25 tasks = 2,400 GPU hours for the RL training
experiments with VLC MW50 and 24 $\times$ 5 $\times$ 25 = 3,000 hours with VLC
Open X. For the MW40 task generalization experiments, the average training
length was again 600k for VLC, so the corresponding cost for training on 6
tasks was approximately 580 GPU hours for our method, and slightly more for
baselines that took longer to train.

##  Appendix C Baseline implementations

For the LOReL baseline experiment, we used the proposed binary classification
objective, reversed negatives and 2-frame conditioning (first and last), while
keeping all of our other training details and data augmentations identical to
our method. This is to ensure LOReL’s smaller and older original architecture
did not account for any difference in performance. The original
implementations were used for all other methods, and pretrained checkpoints
were reused when applicable (RoboCLIP was pretrained on HowTo100M [24], LIV on
EPIC-KITCHENS [8] and Voltron on Something-Something [11]).

Although not considered as a source of data in the original works, we also use
our failure videos as additional negatives in the contrastive objective of
LIV, RoboCLIP and R3M. Model selection was performed based on validation loss
on held-out trajectories. Otherwise, the training procedure and hyperparameter
settings were kept unchanged from the original work. In policy training, we
apply identical reward normalization (offset and scale, as described in
Section 4) for all methods, and reuse the multiplier 50 for the sparse reward.

##  Appendix D Meta-World environment definitions

The Meta-World environments are a set of continuous control robotic
manipulation tasks. The state space of each environment is $\mathbb{R}^{39}$,
containing the positions and orientations of relevant objects as well as the
xyz-position of the robot’s end-effector. An action $a$ in $[-1,1]^{4}$
consists of the desired end-effector xyz-translation, and one dimension for
controlling the openness of the gripper. We refer to Yu et al. [46] for more
details on the state information.


P(\textbf{E}^{1}_{+}|\textbf{S}_{2})=\int_{\lambda\in\text{supp}\,\mu_{\textbf{S}_{2}}}\text{d}\lambda\,\mu_{\textbf{S}_{2}}(\lambda)\xi_{\textbf{E}^{1}_{+}}(\lambda)=0$
(124)

implies $\xi_{\textbf{E}^{1}_{+}}(\lambda)=0$ for
$\lambda\in\text{supp}(\mu_{\textbf{S}_{2}})$. The ontic response function
$\xi_{\textbf{E}^{1}_{+}}(\lambda)$ is therefore deterministic on
$\text{supp}\,\mu_{\textbf{S}_{1}}\cup\text{supp}\,\mu_{\textbf{S}_{1}}=\Lambda$.
Analogous arguments using the pairs $\boldsymbol{v}_{1}$ and
$\boldsymbol{v}_{2}$, or $\boldsymbol{v}_{3}$ and $\boldsymbol{v}_{4}$ can be
constructed to show that $\xi_{\textbf{E}^{2}_{+}}(\lambda)$ and
$\xi_{\textbf{E}^{3}_{+}}(\lambda)$ are also deterministic on $\Lambda$. We
therefore have

$\displaystyle\xi_{\textbf{E}_{k}^{+}}(\lambda)\in\\{0,1\\}\quad\forall\lambda\in\Lambda.$
(125)

We now turn to the operational equivalence (39). Applying noncontextuality,
this implies

$\displaystyle\frac{1}{3}(\xi_{\textbf{E}^{1}_{+}}(\lambda)+\xi_{\textbf{E}^{2}_{+}}(\lambda)+\xi_{\textbf{E}^{3}_{+}}(\lambda))=\frac{1}{3}(\xi_{\textbf{E}^{1}_{-}}(\lambda)+\xi_{\textbf{E}^{2}_{-}}(\lambda)+\xi_{\textbf{E}^{3}_{-}}(\lambda))\quad\forall\lambda,$
(126)

or using
$\xi_{\textbf{E}_{k}^{-}}(\lambda)=1-\xi_{\textbf{E}_{k}^{+}}(\lambda)$,

$\displaystyle\xi_{\textbf{E}^{1}_{+}}(\lambda)+\xi_{\textbf{E}^{2}_{+}}(\lambda)+\xi_{\textbf{E}^{3}_{+}}(\lambda)=\frac{3}{2}\quad\forall\lambda.$
(127)

Since we have proven that the functions $\xi_{\textbf{E}_{k}^{+}}$ are
deterministic in (125), there is no way of satisfying these equations for any
$\lambda$. This contradiction therefore proves impossibility of a
noncontextual learning model.

##  Appendix C Example of a model without preparation equivalences

As an example of such a model, consider a contextual multi-task model
$\\{\mathcal{P}_{\theta}^{1}(y|\boldsymbol{x}),\mathcal{P}_{\theta}^{2}(y|\boldsymbol{x}),\mathcal{P}_{\theta}^{3}(y|\boldsymbol{x})\\}$.
Construct a joint model from this as
$\mathcal{P}_{\theta}^{1}(\boldsymbol{y}|\boldsymbol{x})=\mathcal{P}_{\theta}^{1}(y^{(1)}|\boldsymbol{x})\mathcal{P}_{\theta}^{2}(y^{(2)}|\boldsymbol{x})\mathcal{P}_{\theta}^{3}(y^{(3)}|\boldsymbol{x})$,
i.e., the labels of the joint model are sampled independently using the multi-
task model. Let us assume that the multi-task model is contextual in the sense
of Theorem 4.1 for $\eta=1$, so that the six preparations densities $s_{i}$
correspond to the six vectors $\boldsymbol{v}_{i}$. From (31) if follows that
for the preparation density $\frac{1}{2}s_{1}+\frac{1}{2}s_{2}$ all labels can
occur in the joint model except the labels $\boldsymbol{y}=(0,0,1),(1,1,1)$.
Similarly for $\frac{1}{2}s_{3}+\frac{1}{2}s_{4}$ the labels
$\boldsymbol{y}=(1,0,0),(1,1,1)$ never occurs, and for
$\frac{1}{2}s_{5}+\frac{1}{2}s_{6}$ the labels
$\boldsymbol{y}=(0,1,0),(1,1,1)$ never occur. This means that the label
distributions for the different pairs of preparations densities are not the
same, and hence the pairs of preparation densities are not operationally
equivalent.

##  Appendix D Higher dimensional measurements satisfying the zero-sum bias
condition

Here we give an example of a set of three $\pm 1$ valued observables
$\mathcal{O}_{k}$ in even dimension $d$ that satisfy the zero-sum bias
condition $\mathcal{O}_{1}+\mathcal{O}_{2}+\mathcal{O}_{3}=0$ for all even
$d\geq 2$. We write the three observables as $\mathcal{O}_{1}=2P-\mathbb{I}$
and $\mathcal{O}_{2}=2Q-\mathbb{I}$ (and
$\mathcal{O}_{3}=-\mathcal{O}_{1}-\mathcal{O}_{2}$), where $P$ and $Q$ are the
projections onto the positive eigenspaces of the observables $\mathcal{O}_{1}$
and $\mathcal{O}_{2}$, respectively. Since $\mathcal{O}_{3}$ is a $\pm 1$
valued observable it follows that $\mathcal{O}_{3}^{2}=\mathbb{I}$, that is,

$\mathbb{I}=(\mathcal{O}_{1}+\mathcal{O}_{2})^{2}=4(P+Q-\mathbb{I})^{2}=4(P+Q)^{2}+4\mathbb{I}-8(P+Q)=4(P+Q+PQ+QP)+4\mathbb{I}-8(P+Q)$
(128)

and therefore,

$\frac{3}{4}\mathbb{I}=P+Q-PQ-QP=(P-Q)^{2}.$ (129)

The problem is therefore equivalent to finding projections $P$ and $Q$ such
that $(P-Q)^{2}=\frac{3}{4}\mathbb{I}$.

To achieve this for even dimensional Hilbert spaces, we write the Hilbert
space as
$\mathbb{C}^{d}\simeq\mathbb{C}^{\frac{d}{2}}\oplus\mathbb{C}^{\frac{d}{2}}$,
and define $P$ and $Q$ in block forms

$P=\begin{pmatrix}\mathbb{I}&0\\\ 0&0\end{pmatrix},\quad
Q=\frac{1}{4}\begin{pmatrix}\mathbb{I}&\sqrt{3}\mathbb{I}\\\
\sqrt{3}\mathbb{I}&3\mathbb{I}\end{pmatrix}.$ (130)

It is straightforward to verify that $P$ and $Q$ are projections and that
$(P-Q)^{2}=\frac{3}{4}\mathbb{I}$ holds. Therefore, the corresponding
observables satisfy the desired bias.

##  Appendix E Details of the unbiased quantum model

Figure 10: The unbiased quantum model. As for the biased qunatum model, each
unitary $U(\boldsymbol{\theta})$ has 9 parameters given by the three
parameterised qubit rotations. The data encoding unitaries are identical to
the biased quantum model, and so this model has access to the same Fourier
spectrum.

The generic unbiased quantum model that we use for the numerical investigation
of section (8) is described in Figure 10.

##  Appendix F Data generation and training details

Here we outline the specific details pertaining to the numerical investigation
of section 8. To generate each row of the $3\times 3$ strategy matrix
$\boldsymbol{x}$, we sample three uniformly random numbers $r_{1},r_{2},r_{3}$
in $[0,1]$ and set the row entries as $(r_{1},r_{2},r_{3})/\sum_{i}r_{i}$,
which results in a valid probability distribution. This process defines the
distribution $\mathcal{D}(\boldsymbol{x})$ described in the main text. We
repeat this 1500 times to generate all input strategies in the training set.
The corresponding labels are sampled according to the rules of the RPS game
via (66).

Both quantum models are initialised by sampling each of their parameters
uniformly in $[0,2\pi]$. Each parameter of the surrogate model is initialised
by sampling a number uniformly in $[-1,1]$ then multiplying by
$1/\\#\texttt{params}$ where $\\#\texttt{params}=2(2L+1)^{6}$ is the total
number of parameters of the model. To optimise the models we use JAX and the
optax package, in combination with pennylane in the case of the quantum
models. For the gradient descent, we employ full batch gradient descent (no
significant different was observed using mini-batches). For the quantum models
we use the adam update with the optax default settings and an initial learning
rate of $0.001$. For the suroogate model, since the L2 regularised cost
function is convex in the optimisation parameters, we used standard gradient
descent with a learning rate of $0.001$ (L=1) and $0.0001$ (L=2).

Figure 11: The effect of adding L2 regularisation to the cost function for the
surrogate model for $L=1$ for different regularisation strengths (lighter
curves imply stronger regularisation).