# “Task Success” is not Enough: Investigating the Use of Video-Language Models
as Behavior Critics for Catching Undesirable Agent Behaviors

Lin Guan  Yifan Zhou  Denis Liu  Yantian Zha  Heni Ben Amor  Subbarao
Kambhampati

###### Abstract

Large-scale generative models are shown to be useful for sampling meaningful
candidate solutions, yet they often overlook task constraints and user
preferences. Their full power is better harnessed when the models are coupled
with external verifiers and the final solutions are derived iteratively or
progressively according to the verification feedback. In the context of
embodied AI, verification often solely involves assessing whether goal
conditions specified in the instructions have been met. Nonetheless, for these
agents to be seamlessly integrated into daily life, it is crucial to account
for a broader range of constraints and preferences beyond bare task success
(e.g., a robot should grasp bread with care to avoid significant
deformations). However, given the unbounded scope of robot tasks, it is
infeasible to construct scripted verifiers akin to those used for explicit-
knowledge tasks like the game of Go and theorem proving. This begs the
question: when no sound verifier is available, can we use large vision and
language models (VLMs), which are approximately omniscient, as scalable
Behavior Critics to catch undesirable robot behaviors in videos? To answer
this, we first construct a benchmark that contains diverse cases of goal-
reaching yet undesirable robot policies. Then, we comprehensively evaluate VLM
critics to gain a deeper understanding of their strengths and failure modes.
Based on the evaluation, we provide guidelines on how to effectively utilize
VLM critiques and showcase a practical way to integrate the feedback into an
iterative process of policy refinement. The dataset and codebase are released
at: https://guansuns.github.io/pages/vlm-critic.

Machine Learning, ICML

##  1 Introduction

Figure 1: Examples of GPT-4V critic accurately catching undesirable behaviors.
Failure cases can be found in Fig. 3.

Large-scale pre-trained generative models, such as large language models
(LLMs), exhibit impressive capabilities across a broad spectrum of tasks,
including language modeling and code generation. However, despite their
success in numerous applications, it has been observed that they often fail to
produce desired outputs in a single attempt. For example, in daily AI-assisted
writing scenarios, users usually sample multiple outputs from an LLM and
provide corrective feedback in a multi-turn manner until the desired writing
is obtained. Similarly, in AI-based mathematics problem solving, the state-of-
the-art performance (Romera-Paredes et al., 2023; Trinh et al., 2024) is
achieved by repeatedly (re-)prompting LLMs with verification feedback from
program executors and scripted evaluator. Likewise, in several LLMs-driven
task planning frameworks (Valmeekam et al., 2023b; Guan et al., 2023),
candidate plans sampled from LLMs are validated by external model-based
verifiers, e.g., to see if there is any unfulfilled precondition at each step.
Overall, while large-scale generative models are capable of generating
meaningful candidate solutions, they tend to consistently overlook task
constraints and user preferences. To fully harness their power, it has become
common practice to couple them with external critics or verifiers (e.g.,
humans or programs) such that the models can iteratively refine their outputs
according to the critiques or verification feedback.

Motivated by the success of large pre-trained models in other domains,
robotics and embodied AI communities have initiated efforts in building
general-purpose language-conditioned policy models (Padalkar et al., 2023;
Team et al., 2023; Nair et al., 2022). Although the versatility and
reliability of state-of-the-art policy models are not yet on par with models
like LLMs, their capabilities continue to grow as researchers enrich robot-
specific datasets (Zhao et al., 2023a) and devise methodologies that can
leverage internet-scale embodiment-agnostic data (Yang et al., 2023; Du et
al., 2023b). Nevertheless, even if data sparsity is addressed, we envision
that large policy models would still face the same limitations as we observe
in LLMs development — the models may not produce desired policies in one shot.
Several factors can result in such inaccuracies: (a) the intrinsic
stochasticity of large parametric models; (b) the inevitable noise in large-
scale datasets, especially in collective data sources like the internet; and
(c) the adoption of fully goal-driven policy optimization mechanisms – in
other words, the current literature typically only verifies the satisfaction
of goal conditions specified in symbolic or natural-language instructions
(e.g., by ensuring high CLIP-style similarity between image observations and
the language instructions (Ma et al., 2022; Cui et al., 2022; Nair et al.,
2022; Rocamonde et al., 2023)). However, since language instructions (or any
abstracted representation of the world) are always an incomplete specification
of goal conditions and user preferences (Guan et al., 2022a), a fully goal-
driven learning mechanism would encourage “shortcuts” and overlook
undesirability in the agent’s behaviors (Fig. 1). For example, when instructed
to “hand a pair of scissors to a person,” the robot holds the pointy end of
the scissors towards the person. In another example, the robot forcefully
opens a cabinet door, potentially causing damage to the door hinge, even if
the damage is not immediately noticeable.

Therefore, in the context of embodied AI, to form a loop of policy
(re-)generation and verification (Fig. 2), it is necessary to account for both
goal reachability and a broader range of common constraints and preferences.
However, unlike explicit-knowledge tasks (e.g., the game of Go and theorem
proving) wherein it is feasible to list out all the rules and constraints,
given the unbounded scope of robot tasks, it is intractable to exhaustively
articulate the considerations and undesirability. This raises a question: in
the absence of a sound verifier, can we use large vision and language models
(VLMs) like GPT-4V (Achiam et al., 2023), which are considered approximately
omniscient, as scalable Behavior Critics to catch, at least to a significant
extent, the undesirability presented in videos of robot behaviors?
Furthermore, given VLMs may exhibit inaccuracies in any use case,
understanding the pattern of their outputs becomes crucial prior to their
integration into any policy generation system.

To answer these questions, we conduct the first comprehensive study on
leveraging VLMs (i.e., GPT-4V and Gemini Pro in this work) as behavior critics
to freely comment on undesirable behaviors presented in videos. We first
collect real-world videos that encompass diverse robot behaviors which are
goal-reaching but undesirable (Fig. 1). Then we use the videos as a benchmark
to investigate the feasibility of VLM critics, with an emphasis on testing how
many of the undesirable behaviors are covered by the critiques (i.e., recall
rate) and how many of the critiques are valid (i.e., precision rate). In
addition to scalar metrics, we also manually examine each critique and develop
a taxonomy to characterize the failure modes. Our evaluation indicates that
GPT-4V can identify a significant portion of undesirable robot behaviors (with
a recall rate of 69%). However, it also generates critiques that contain a
considerable amount of hallucinated information (resulting in a precision rate
of 62%), mainly due to limited visual-grounding capability.

Based on these findings, we discuss how to effectively utilize VLM critiques.
We demonstrate that, in an ideal case, by providing GPT-4V with grounding
feedback that verifies the occurrences of events mentioned in the critiques,
it can “refine” its outputs and achieve a precision rate of over 98% (with a
minor impact on the recall). Lastly, while this work does not take a strong
stance on how the critiques should be integrated into closed-loop policy-
generation systems, we do present a candidate framework using a real robot in
five household scenarios, wherein a Code-as-Policies agent (Liang et al.,
2023) iteratively refines the policy according to VLM critiques on the
rollouts.

##  2 Related Work

Figure 2: Positioning of this work. This figure also illustrates a loop of
policy (re-)generation and verification.

Coupling generative models with verifiers. Generative models, like LLMs, are
generalists that can serve as powerful heuristics for numerous problems.
However, to effectively explore the solution space and select a working
solution with guaranteed correctness, it is necessary to couple these models
with external sound verifiers. Such generator-verifier paradigms are widely
adopted (Valmeekam et al., 2023b; Guan et al., 2023; Romera-Paredes et al.,
2023; Trinh et al., 2024).

The use of AI critiques and feedback. It may be infeasible to construct formal
verifiers for tasks that are tacit or have a wide scope. In these situations,
people have explored the option of using AI models to criticize the outputs,
either generated by themselves or by other models (Bai et al., 2022; Lee et
al., 2023; Wang et al., 2023; Yuan et al., 2024; Ahn et al., 2024). Different
from existing works that employ AI models to assess how well each output
conforms to a predefined “constitution,” this study focuses on utilizing AI
models to identify overlooked aspects of undesirability (e.g., to complete the
constitution). Overall, the use of AI feedback offers improved scalability but
also comes at the expense of losing correctness guarantees.

LLMs and VLMs for decision making. LLMs and VLMs have been used in various
aspects of decision-making, such as generating heuristic plans (Ahn et al.,
2022; Du et al., 2023a; Zhao et al., 2023b), acquiring domain knowledge and
models (Guan et al., 2023; Wong et al., 2023), constructing reward functions
(Xie et al., 2023; Ma et al., 2023; Yu et al., 2023), automating task design
(Ahn et al., 2024), and explaining execution failures (Liu et al., 2023). More
comprehensive surveys can be found in (Firoozi et al., 2023; Zeng et al.,
2023). Among all the use cases, reward construction and failure explanation
are most relevant to this work. LLMs-based reward construction also elaborates
on desirability and undesirability in a reward function. However, its focus is
on how to create an easily-optimizable dense reward function by adjusting the
relative weights of a pre-defined set of features or by modifying the logic of
a Python reward program. Differently, behavior critics emphasize knowledge
acquisition by proposing new factors that are overlooked in the current
behavior. These factors may be used to expand the feature set and be
incorporated into a refined reward function. In failure explanation frameworks
like REFLECT (Liu et al., 2023) and reflexion (Shinn et al., 2023), the focus
is on identifying the causes of errors that result in termination of plan
execution and prevent the agent from reaching goal states. Instead, our work
argues that there can still be undesirability in goal-reaching behaviors
especially given that human instructions always under-specify common
constraints and human preferences. The positioning of our work is illustrated
in Fig. 2.

Robotic foundation models. Robotic foundation models, such as the RT series
(Zitkovich et al., 2023; Padalkar et al., 2023) and Octo (Team et al., 2023),
are typically waypoint-based policy models that connect semantic task
specifications to low-level motion. Critics, including our behavior critics,
and a continuously-improved policy model can complement each other in the
sense that a desired behavior can be more efficiently derived from the policy
within a guided generation framework.

##  3 Problem Setting

We consider a setting where an agent (e.g., a motion planner) tries to solve a
task specified with a language instruction $i$. The task is represented as a
finite-horizon discounted MDP $\mathcal{M}=\langle
S,A,R,T,S_{0},\gamma\rangle$, where $S$ is the state space comprising features
that encapsulate necessary information about the environment, the agent, and,
optionally, the history; $A$ is the set of actions; $T:S\times A\times
S\rightarrow[0,1]$ is the world dynamic model; $S_{0}$ is the set of possible
initial states; $R$ is the reward function; and $\gamma$ is the discounted
factor. Instruction $i$ corresponds to a symbolic specification of goal
conditions, which characterizes a set of absorbing goal states $S_{g}\subset
S$. Without loss of generality, $R$ can be constructed as $\mathbbm{1}(s\in
S_{g})$ for $s\in S$, which encourages the agent to reach a goal state with
the least time steps if $\gamma<1$. Let $S_{g}^{*}\subset S_{g}$ denote the
set of states that satisfy all the conditions in $i$ as well as common human
preferences $\mathcal{P}$. Due to the potential incompleteness of symbolic
representations, instruction $i$ may allow for undesirable goal-satisfying
states $S_{g}^{-}=S_{g}\setminus S_{g}^{*}$. Considering the unbounded or
tacit nature of $\mathcal{P}$, our primary objective is to investigate how
well VLMs can approximate $\mathcal{P}$ and determine if any state $s$ is in
$S_{g}^{-}$.

##  4 Methodology

We begin by describing the prompt used to obtain behavior critiques from VLMs.
Next, as the core contribution of this paper, we will explain how we collect
and construct our benchmark which consists of video clips demonstrating
suboptimal yet goal-reaching policies in diverse household tasks. Accordingly,
we introduce the metrics and taxonomy for characterizing the strengths and
failure modes of VLM critics. Finally, we conclude this section with a
discussion on practical ways to utilize and integrate VLM critiques.

###  4.1 Obtaining behavior critiques from VLMs

Fig. 9 in Appendix shows the prompt template for instructing VLMs to generate
critiques over undesirable behaviors. It has 4 key components: (a) a brief
description of the task of being a behavior critic; (b) two textual examples
that demonstrate the expected output format for cases where there are and are
not undesirable behaviors in a video; (c) frames from the video to be
criticized; and (d) a short note explaining that the selected frames maintain
the same relative ordering as in the original video and that there may be zero
or multiple undesirable behaviors presented in a video.

Each $\langle\text{frame}_{i}\rangle$ tag in the prompt template corresponds
to an image. In our benchmark, each video showcases a policy for a short-
horizon manipulation task, and we find that 30 frames adequately capture the
essence of each policy. Hence, each input contains a maximum of 30 frames, and
each frame undergoes preprocessing to ensure that its longest side does not
exceed 512 pixels. Also, adjustments may need to be made based on the VLM
being used. For example, Gemini Pro currently only supports 16 frames per
conversation.

Moreover, the examples used to illustrate the output format are fixed across
all test cases. We acknowledge the possibility of marginally improving overall
performance by retrieving more contextually relevant examples from a pool of
pre-recorded undesirable behaviors. However, it is important to remember that
the main reason for using VLMs as behavior critics is the difficulty in
elaborating on the potential undesirability. Therefore, it is crucial to
assess the performance of behavior critics under an “unforeseeable” setting.

In addition to having a behavior critic comment on individual videos
separately, we also experiment with having a critic compare pairs of
trajectories and provide preferences with justification. Pairwise preferences
can complement verbal critiques when the undesirability involves subtle and
tacit components. For example, when selecting a trajectory that opens a door
less “forcefully” from a set of virtual rollouts, it would be easier to use
preference labels. However, we do note that verbal critiques still play a
central role in this study as they provide the most informative signal to
guide an agent toward desirable behaviors. For more details on preference
elicitation, see Appx. A.1.

###  4.2 Benchmarking VLMs behavior critics

Data collection.  In our benchmark, a video of suboptimal policy may feature
one or more negative events. These negative events are based on undesirability
observed in public robot videos (e.g., on homepages of relevant academic
projects) and demonstration datasets (Padalkar et al., 2023) as well as our
own experience of testing robot policies (with both public and proprietary
models). All task setups take place in real-world household environments.

To facilitate and ensure the safety of data collection, the majority of videos
were recorded by reproducing different policies with a human-controlled
grabber tool (Fig. 1). The grabber tool closely resembles a robot gripper in
appearance. Furthermore, we have taken care to crop the videos to eliminate
any visible cues indicating manual operation of the “gripper”. This setup has
also been employed in several visual imitation work (Young et al., 2021; Pari
et al., 2021).

Benchmark structure.  We collected 175 videos in total, forming 114 test
cases. Each test case has all the following components: (a) a negative video
that features the undesirable behaviors; (b) another negative video that
exhibits the same undesirable behaviors; (c) a positive video that
demonstrates a satisfactory policy – the inclusion of positive examples aims
to assess the frequency of VLM critics producing false alarms; (d) a language
instruction that specifies the task – policy shown in a positive video is
supposed to satisfy the goal conditions given in the corresponding
instruction; (e) a list of human-annotated undesirable behaviors that are
presented in the respective negative videos; and (f) a narrative description
of the negative video, covering key actions that perform the task as
anticipated and those that result in undesirable outcomes (see examples in
App. A.4). The inclusion of narration is to examine whether the text backbones
of existing multi-modal models have sufficient knowledge to differentiate
between undesirable and aligned agent behaviors.

Evaluation metrics.  Recall and precision are fundamental metrics in assessing
behavior critics. Recall measures the proportion of the most critical negative
events, as annotated by humans, that the VLM critics successfully identify.
Precision, on the other hand, quantifies the fraction of the critiques that
accurately reflect the negative events within the videos. Note that the
calculation of precision rate involves manual assessment because, by
definition, “precision” should permit the critics to suggest valid critiques
even when they do not associate with any human-annotated negative event.

Furthermore, in order to gain deeper insights that scalar metrics alone cannot
provide, we also conducted a thorough examination and annotation of individual
critiques based on the properties they exhibit. We adhere to a taxonomy that
allows us to delineate the success cases and failure modes. While our taxonomy
may not be exhaustively comprehensive, we find it sufficiently captures
patterns of VLM critiques that we observed in our evaluations. The dimensions
we take into account include:

Figure 3: Failure cases of GPT-4V critic. More examples can be found in Fig. 6
in Appendix.

(a) Critique operationalizability: a critique is expected to identify negative
events in a sense that the event itself, not supplementary information such as
underlying causes, is precisely described. However, an intriguing question
arises as to whether a critique can extend beyond mere description to provide
an operationalizable explanation for the cause. Here, an operationalizable
explanation refers to an explanation that can be directly implemented by an
agent to rectify the suboptimal policy, without requiring extra human-like
reasoning about the cause. An example of an unoperationalizable critique is
“the robot lifted the bag without securing the contents, leading to the
carrots falling onto the table,” which does not pinpoint the root cause, i.e.,
the robot’s failure to securely hold or seal the bag’s opening. Another
example (Fig. 3) is “the robot caused pasta sauce to drip onto the counter
while transferring the spoon from the jar to the bowl,” which does not
identify that the core issue lies in the robot not waiting for the remnants to
drip back to the jar before moving the spoon. Note that in some circumstances,
an operationalizable explanation is inherently part of the description, for
example, “the robot gripper made contact with the plate while trying to grasp
the carrot.”

(b) Visual grounding errors: visual grounding errors are instances where a
critic refers to events that are not presented in the video. This is also
known as hallucination in the context of perception and video understanding.
Note that perception errors can occur throughout the entire critique –
critiquing behavior not depicted in the video – or within fragments of a
critique, for instance, fabricating a behavior to explain the cause of a
negative event.

(c) Causal understanding errors: this type of error pertains to cases where a
critic misconstrues the cause of negative events (if an explanation is
provided as part of the critique). Such errors may arise when the critic
erroneously links two independent events or when the critic attributes a
nonexistent event as the cause. The latter case is also labelled as a visual
grounding error – different error types in our taxonomy are not mutually
exclusive.

(d) Incomplete understanding of the problem: in this type of inaccuracy, the
critic carelessly judges an event as negative while it may not be undesirable
under certain conditions. This is a relatively rare error type. In the case of
GPT-4V, only two critiques fell into this category. One instance criticizes
the robot for “releasing the scissors above the table when handing them to a
person, potentially causing them to fall and injure the person or damage the
table”. However, in the video, the person comfortably and firmly received the
scissors from the robot’s gripper. Therefore, releasing the scissors above the
table did not necessarily exhibit risky behavior, considering the person’s
reaction.

In addition to the aforementioned error types, we also investigated whether
the behavior critic provides incorrect “advice” that leads the agent away from
an optimal policy. An example scenario could be when the critique hinders a
robot from reaching an object, contradicting the task requirement of
delivering the object to a person. Fortunately, GPT-4V never exhibit such
errors in our evaluation, and as a result, this particular category is
deliberately excluded from our taxonomy and future analysis to avoid potential
confusion.

###  4.3 Utilizing behavior critiques

The primary objective of our benchmark is to shed light on the output patterns
of VLM critics and provide guidelines on how to utilize them. As shown in Sec.
5.1, critiques from GPT-4V exhibit the following characteristics: (a) GPT-4V
can identify a significant portion of undesirable events, but it also produces
inaccurate critiques, as reflected in its relatively low precision rate; and
(b) visual grounding errors are common and contribute to the majority of
inaccuracies, meaning that GPT-4V’s critiques often contain hallucinated
information. Therefore, from the perspective of critique-taking agents, while
attempting to address these “hallucinated” critiques may not necessarily
deviate an agent from optimal behaviors, it could be a waste of time.
Additionally, since VLM critics may criticize optimal behaviors based on
hallucinated events, a system involving such critics will fail to provide a
clear notion of convergence, i.e., determining whether a desirable behavior is
achieved. The remainder of this section will discuss how to address these
challenges.

Augmenting VLM critics with grounding feedback. When it comes to tasks like
identifying potential negative events and their causes, currently there may
not be more performant models than state-of-the-art VLMs. However, this does
not hold true for visual grounding. For one, as noted in the OpenAI official
document and several empirical studies (Chen et al., 2024), GPT-4V may not
match the performance of specialized vision models in specific types of visual
grounding tasks (Zhang et al., 2022; Kirillov et al., 2023). Moreover,
accurately grounding certain events may require additional information from
modalities other than images. For example, depth maps can provide more direct
information about spatial relationships between objects, such as collisions,
distances, or relative positions. Similarly, audio and robot sensory readings
can be more informative about the contact that the robot’s gripper exerts on
other objects.

Based on the above analysis, we propose a practical pipeline to leverage VLM
critics by incorporating grounding feedback from a suite of specialized
grounding modules. Firstly, events in the critiques are extracted and sent to
the most suitable specialized model(s) to check their existence in the robot’s
behavior. The process of model selection can be automated with an LLM (Zeng et
al., 2022; Wu et al., 2023). Next, the grounding feedback, which typically
provides one bit of extra information (i.e., whether an event is present), is
returned to the VLM critic to continue the critique-extraction dialogue. As we
will see in Sec. 5.3, when the grounding feedback is perfect, a precision rate
of over 98% can be achieved. In essence, this augmented pipeline shares the
same principles as tool-augmented LLMs (Parisi et al., 2022; Schick et al.,
2023) and retrieval-augmented generation (Guu et al., 2020), wherein LLMs and
VLMs serve as a hub to manipulate information from more accurate sources (Zeng
et al., 2022; Chen et al., 2024). In this work, VLM critics initially suggest
potential negative events by retrieving approximately from its vast internal
knowledge and then refine the outputs with more precise grounding tools (which
are not necessarily generative models).

Integrating critiques into a closed-loop system. The development of general-
purpose policy models that can accept verbal instructions and corrective
feedback is an ongoing area of research. Some works have been able to utilize
verbal feedback in specific problem setups (Sharma et al., 2022; Cui et al.,
2023; Bucker et al., 2023; Zha et al., 2023; Miyaoka et al., 2023; Yow et al.,
2024). For instance, RBAs (Guan et al., 2022b) and AlignDiff (Dong et al.,
2023) learn conditional reward or policy models that can implement relative
concepts like “take bigger stride.” However, there is currently no robot
policy model that demonstrates the same level of generality and flexibility as
models in other domains, such as LLMs.

In this work, we remain neutral on the choice of underlying models to drive
the agent, as our main focus is on verbal critiques. Nevertheless, we present
a viable closed-loop system wherein Code-as-Policies (Liang et al., 2023) is
employed to control the robot. In Code-as-Policies (CaP), an LLM “programs”
the robot’s motion by invoking and adjusting input parameters of a predefined
library of control primitive APIs. With this setup, a closed-loop system can
be constructed by repeating these steps: (a) CaP synthesizes a control program
as policy; (b) the robot executes the program in the environment or, ideally,
a faithful simulator; (c) a video recording of the rollout is collected and
sent to the VLM critic for review; and (d) the process terminates if no
undesirable behavior is detected; otherwise, the CaP agent is informed of the
negative events.

Apart from directly taking verbal critiques, one may also discard the textual
description of the undesirable behaviors and convert the critiques into scalar
training signals like binary preference labels, e.g., by assuming a trajectory
with fewer undesirable events is preferred over others. The preference labels
can then be used within frameworks like preference-based RL (Christiano et
al., 2017; Zhang et al., 2019; Hejna et al., 2023). Here, we will not delve
into this direction as the inability to accept explicit verbal guidance is a
fundamental cause of high sample complexity in preference-based policy
learning, especially in hard-exploration problems like continuous control
(Kambhampati, 2021).

##  5 Empirical Evaluation

This section will elaborate on the evaluation results, focusing particularly
on results obtained with GPT-4V as of January 2024. Alongside, an evaluation
of Gemini has been conducted but is limited because Gemini Pro is currently
restricted to a maximum of 16 images per conversation. For clarity, Gemini’s
results are presented in Appx. A.2. It is important to note that the objective
of this study is to investigate the feasibility of using state-of-the-art VLMs
as behavior critics, rather than establishing comparisons between different
models.

Metrics | Acc. | Recall | Precision
---|---|---|---
Positive | Negative
GPT-4V | 6.14% | 92.98% | 69.42% | 62.21%
GPT-4V-Augmented | 98.24% | 78.07% | 62.80% | 98.02%
Table 1: Evaluation results of GPT-4V critic. GPT-4V-Augmented refers to the
experiment of augmenting GPT-4V with perfect grounding feedback (Sec. 5.3).

###  5.1 Recall and precision

Table 1 presents the recall and precision rates of GPT-4V critic. These
results are based on the direct output from GPT-4V prior to any post-
filtering. In terms of recall, GPT-4V achieves a rate of 69.42%. We note that
given the extensive pool of potential negative events within household tasks,
a recall rate of 69.42% already signifies a decent performance and a high
value in saving human efforts to eliminate undesirability. Nonetheless, this
number also suggests room for improvement, such as fine-tuning the model with
a collective set of undesirable events. Compared to recall, the precision rate
of 62.21% highlights more practical challenges. A significant portion of the
critiques do not accurately cover any undesirable events in the videos,
emphasizing the need for post-filtering (Sec. 4.3 and Sec. 5.3).

Note that the recall-precision evaluation is conducted solely with videos
containing negative events (hereafter referred to as “negative samples”). It
is also important to examine how GPT-4V comments on videos showcasing
preferable behaviors (hereafter referred to as “positive samples”). Table 1
shows GPT-4V’s accuracy in identifying the presence of undesirable events
among both negative and positive samples. The results clearly illustrate that
GPT-4V always provides criticism, irrespective of the presence or absence of
undesirable behaviors. Upon reviewing the “critiques” on positive samples, we
find that almost all of them are invalid as they criticize “hallucinated”
negative events. This result further confirms that GPT-4V suffers from limited
visual grounding capability, and its critiques should not be directly used as
an indication of “optimality”.

Figure 4: Distributions of error types. GPT-4V-Augmented refers to the
experiment of augmenting GPT-4V with perfect grounding feedback (Sec. 5.3).

###  5.2 Insights into the distribution of error types

Sec. 4.2 outlines the taxonomy of errors that is used here to analyze the
output patterns of GPT-4V critic (on negative samples). Fig. 4 displays the
number of errors per category. Notably, within 262 critiques, we find 116
instances of visual-grounding errors, highlighting a major limitation of
GPT-4V in understanding videos. Visual grounding errors also occur
significantly more frequently compared to other error types. However, we would
interpret this as a positive finding since grounding-related errors are more
manageable to address than others (see discussions in Sec. 4.3). Additionally,
we find that only 69.05% of the critiques capturing undesirability can
accurately identify both the negative events and their causes (by providing
operationalizable explanations). Hence, at present, it is more reliable to
consider the critique as a means of detecting undesirability rather than
providing actionable advice.

###  5.3 Augmenting VLM critics with grounding feedback

Most analyses so far suggest the need to augment VLMs with improved grounding
capabilities (more discussions in Sec. 4.3). As a preliminary study, we assess
whether GPT-4V critic can refine its output when perfect grounding feedback is
provided (by human participants for experimental purposes). To achieve this,
we continue the critique-elicitation conversation with feedback messages
mentioning that “A perfect perception model has been used to verify the
existence of events in the critiques.” Each grounding feedback is provided in
the format of “The following event is not detected: $\langle$event
description$\rangle$”.

Results are shown in Table 1 and Fig. 4. With perfect grounding feedback, the
precision rate increases significantly to over 98%, although there is a slight
decrease in recall. Upon manual examination, we find that this is because
GPT-4V tends to simply delete critiques that contain “hallucinated”
information instead of refining them (e.g., by generating a better guess of
causes). Moreover, grounding feedback also substantially reduces the
occurrence of false alarms on positive samples (evidenced by the accuracy of
98.24% in determining the presence of undesirability.).

###  5.4 Additional evaluations

This subsection focuses on two additional experiments. The first experiment
aims to assess the feasibility of complementing verbal critiques with
preference labels (see Sec. 4.1). Our results show that (a) when contrasting a
negative sample with a positive sample, in 95.61% of time, GPT-4V manages to
prefer the positive one; (b) when comparing pairs of negative samples or pairs
of positive samples, GPT-4V always establishes invalid rankings within the
pairs by “hallucinating” that one sample is perfect and one is flawed, as seen
in its justifications. The positive aspect is that GPT-4V can accurately
select near-optimal behaviors over suboptimal ones even though it tends to
establish unnecessary orderings in other cases. More details of this
experiment are explained in Appx. A.1.

In the second experiment, instead of feeding videos into GPT-4V, we provide a
narration of the video (see Sec. 4.2 and Appx. A.2) and ask GPT-4 to identify
which steps contain undesirable behaviors. The goal of this study is to verify
that (a) the text backbone of GPT-4V has good understanding of what
constitutes undesirable and well-aligned behaviors, and (b) the primary
obstacle lies within visual grounding. Out of 114 test cases, GPT-4 accurately
locates and explains the undesirable behavior in 110 cases. We note that
although this may not be a rigorous evaluation as human-written narrations
already provide strong information filtering, it does shed some light on the
current bottleneck of VLMs.

###  5.5 Demonstrating a closed-loop system with CaP

Figure 5: Illustrations of the initial policies and the updated policies in
response to verbal critiques. A more detailed visualization can be found in
Fig. 7 in Appendix.

In this subsection, we showcase a candidate closed-loop system with Code-as-
Policies as the planner. We would reiterate that we do not restrict the
selection of policy models, and the main reason for employing CaP is the
underlying LLM’s strong ability to adapt in response to verbal critiques, even
though CaP may be less expressive compared to other waypoint-based approaches
(Padalkar et al., 2023; Team et al., 2023).

We use a UR5 robot and Robotiq-85 gripper with operational space control.
Accordingly, the CaP agent is provided with APIs that can control the robot to
translate freely on xyz, rotate regarding a specified axis, and adjust the
gripper (see details in Appx. A.3). We consider the following tasks within
table-top setups: (a) scissor handover; (b) lifting an opened bag; (c) bread
grasping; (d) knife placing; and (e) spoon picking. In scissor handover, the
CaP agent occasionally generates policies that point the sharp end of the
scissors at a human. After receiving one critique, the agent manages to rotate
the end-effector by 180 degrees. In lifting an opened bag, the agent
occasionally makes the mistake of gripping the bottom of the bag, leading to
spills. After a few attempts (3 in avg.) to grasp different edges of the bag,
the agent manages to hold the opening. In bread grasping, the agent initially
applies excessive force, resulting in significant deformation of the bread.
After $\sim$5 iterations, the agent manages to reduce the force to a minimal
value. Likewise, in knife placing, the agent starts with a policy that causes
forceful contact between the knife tip and the cutting board. After 2
iterations, the agent successfully lowers its gripper and rotates the gripper
by 90 degrees, ensuring a gentle placement of the knife and preventing contact
between the tip and the cutting board.

There are also instances where CaP fails to reach a preferable behavior. In a
scenario where the robot is tasked with taking a spoon out of a jar, the CaP
agent neglects the step of waiting for the sauce residue to drip back into the
jar before repositioning the spoon. Upon receiving an unoperationalizable
critique “the robot caused pasta sauce to drip onto the counter while
transferring the spoon from the jar to the bowl”, the CaP agent continues to
move the spoon immediately after extracting it, albeit via an alternative
path. This reveals the overall performance may also be limited by the
planner’s capability to infer causes of negative events.

##  6 Conclusion

We conducted a thorough investigation into the feasibility of using VLMs as
behavior critics to identify common undesirability in videos of robot
behaviors. We evaluated GPT-4V on our benchmark that covers diverse cases of
undesirability. Our findings not only demonstrate the feasibility but also
comprehensively characterize the strengths and limitations of GPT-4V critic,
which provide practical guidelines for effectively utilizing the critiques.
Lastly, we also demonstrate how to align the behaviors of a Code-as-Policies
agent with our critic.

Going forward, it would be important to understand the connections between
this work and other studies that show LLMs cannot reliably verify plans
(Valmeekam et al., 2023a; Stechly et al., 2023). Our hypothesis is that, in
our test cases, the behavior critic can identify undesirability with local
information without the need to reason over long horizons and capture
interactions between subgoals. It would be beneficial to further expand the
benchmark in this direction. Additionally, it would be meaningful to test if
we can turn the critic into a stronger “universal” human-preference “enforcer”
by fine-tuning a VLM with undesirability data collected through collaborative
efforts from the community.

## References

  * Achiam et al. (2023) Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Ahn et al. (2022) Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ , 2022.
  * Ahn et al. (2024) Ahn, M., Dwibedi, D., Finn, C., Arenas, M. G., Gopalakrishnan, K., Hausman, K., Ichter, B., Irpan, A., Joshi, N., Julian, R., et al.  Autort: Embodied foundation models for large scale orchestration of robotic agents.  _arXiv preprint arXiv:2401.12963_ , 2024.
  * Bai et al. (2022) Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022.
  * Bucker et al. (2023) Bucker, A., Figueredo, L., Haddadin, S., Kapoor, A., Ma, S., Vemprala, S., and Bonatti, R.  Latte: Language trajectory transformer.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 7287–7294. IEEE, 2023.
  * Chen et al. (2024) Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., Florence, P., Sadigh, D., Guibas, L., and Xia, F.  Spatialvlm: Endowing vision-language models with spatial reasoning capabilities.  _arXiv preprint arXiv:2401.12168_ , 2024.
  * Christiano et al. (2017) Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Cui et al. (2022) Cui, Y., Niekum, S., Gupta, A., Kumar, V., and Rajeswaran, A.  Can foundation models perform zero-shot task specification for robot manipulation?  In _Learning for Dynamics and Control Conference_ , pp. 893–905. PMLR, 2022.
  * Cui et al. (2023) Cui, Y., Karamcheti, S., Palleti, R., Shivakumar, N., Liang, P., and Sadigh, D.  No, to the right: Online language corrections for robotic manipulation via shared autonomy.  In _Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ , pp. 93–101, 2023.
  * Dong et al. (2023) Dong, Z., Yuan, Y., Hao, J., Ni, F., Mu, Y., Zheng, Y., Hu, Y., Lv, T., Fan, C., and Hu, Z.  Aligndiff: Aligning diverse human preferences via behavior-customisable diffusion model.  _arXiv preprint arXiv:2310.02054_ , 2023.
  * Du et al. (2023a) Du, Y., Watkins, O., Wang, Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, J.  Guiding pretraining in reinforcement learning with large language models.  _arXiv preprint arXiv:2302.06692_ , 2023a.
  * Du et al. (2023b) Du, Y., Yang, M., Dai, B., Dai, H., Nachum, O., Tenenbaum, J. B., Schuurmans, D., and Abbeel, P.  Learning universal policies via text-guided video generation.  _arXiv preprint arXiv:2302.00111_ , 2023b.
  * Firoozi et al. (2023) Firoozi, R., Tucker, J., Tian, S., Majumdar, A., Sun, J., Liu, W., Zhu, Y., Song, S., Kapoor, A., Hausman, K., et al.  Foundation models in robotics: Applications, challenges, and the future.  _arXiv preprint arXiv:2312.07843_ , 2023.
  * Guan et al. (2022a) Guan, L., Sreedharan, S., and Kambhampati, S.  Leveraging approximate symbolic models for reinforcement learning via skill diversity.  In _International Conference on Machine Learning_ , pp. 7949–7967. PMLR, 2022a.
  * Guan et al. (2022b) Guan, L., Valmeekam, K., and Kambhampati, S.  Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences.  _arXiv preprint arXiv:2210.15906_ , 2022b.
  * Guan et al. (2023) Guan, L., Valmeekam, K., Sreedharan, S., and Kambhampati, S.  Leveraging pre-trained large language models to construct and utilize world models for model-based task planning.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=zDbsSscmuj.
  * Guu et al. (2020) Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , pp. 3929–3938. PMLR, 2020.
  * Hejna et al. (2023) Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W. B., and Sadigh, D.  Contrastive prefence learning: Learning from human feedback without rl.  _arXiv preprint arXiv:2310.13639_ , 2023.
  * Kambhampati (2021) Kambhampati, S.  Polanyi’s revenge and ai’s new romance with tacit knowledge.  _Communications of the ACM_ , 64(2):31–32, 2021.
  * Kirillov et al. (2023) Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Lee et al. (2023) Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V., and Rastogi, A.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  _arXiv preprint arXiv:2309.00267_ , 2023.
  * Liang et al. (2023) Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A.  Code as policies: Language model programs for embodied control.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 9493–9500. IEEE, 2023.
  * Liu et al. (2023) Liu, Z., Bahety, A., and Song, S.  Reflect: Summarizing robot experiences for failure explanation and correction.  _arXiv preprint arXiv:2306.15724_ , 2023.
  * Ma et al. (2022) Ma, Y. J., Sodhani, S., Jayaraman, D., Bastani, O., Kumar, V., and Zhang, A.  Vip: Towards universal visual reward and representation via value-implicit pre-training.  _arXiv preprint arXiv:2210.00030_ , 2022.
  * Ma et al. (2023) Ma, Y. J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A.  Eureka: Human-level reward design via coding large language models.  _arXiv preprint arXiv:2310.12931_ , 2023.
  * Miyaoka et al. (2023) Miyaoka, Y., Inoue, M., and Nii, T.  Chatmpc: Natural language based mpc personalization.  _arXiv preprint arXiv:2309.05952_ , 2023.
  * Nair et al. (2022) Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A.  R3m: A universal visual representation for robot manipulation.  _arXiv preprint arXiv:2203.12601_ , 2022.
  * Padalkar et al. (2023) Padalkar, A., Pooley, A., Jain, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Singh, A., Brohan, A., et al.  Open x-embodiment: Robotic learning datasets and rt-x models.  _arXiv preprint arXiv:2310.08864_ , 2023.
  * Pari et al. (2021) Pari, J., Shafiullah, N. M., Arunachalam, S. P., and Pinto, L.  The surprising effectiveness of representation learning for visual imitation.  _arXiv preprint arXiv:2112.01511_ , 2021.
  * Parisi et al. (2022) Parisi, A., Zhao, Y., and Fiedel, N.  Talm: Tool augmented language models.  _arXiv preprint arXiv:2205.12255_ , 2022.
  * Rocamonde et al. (2023) Rocamonde, J., Montesinos, V., Nava, E., Perez, E., and Lindner, D.  Vision-language models are zero-shot reward models for reinforcement learning.  _arXiv preprint arXiv:2310.12921_ , 2023.
  * Romera-Paredes et al. (2023) Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J., Wang, P., Fawzi, O., Kohli, P., and Fawzi, A.  Mathematical discoveries from program search with large language models.  _Nature_ , 2023.  doi: 10.1038/s41586-023-06924-6.
  * Schick et al. (2023) Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_ , 2023.
  * Sharma et al. (2022) Sharma, P., Sundaralingam, B., Blukis, V., Paxton, C., Hermans, T., Torralba, A., Andreas, J., and Fox, D.  Correcting robot plans with natural language feedback.  _arXiv preprint arXiv:2204.05186_ , 2022.
  * Shinn et al. (2023) Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S.  Reflexion: Language agents with verbal reinforcement learning.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Stechly et al. (2023) Stechly, K., Marquez, M., and Kambhampati, S.  GPT-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems.  In _NeurIPS 2023 Foundation Models for Decision Making Workshop_ , 2023.  URL https://openreview.net/forum?id=PMtZjDYB68.
  * Team et al. (2023) Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Xu, C., Luo, J., et al.  Octo: An open-source generalist robot policy, 2023.
  * Trinh et al. (2024) Trinh, T. H., Wu, Y., Le, Q. V., He, H., and Luong, T.  Solving olympiad geometry without human demonstrations.  _Nature_ , 625(7995):476–482, 2024.
  * Valmeekam et al. (2023a) Valmeekam, K., Marquez, M., and Kambhampati, S.  Investigating the effectiveness of self-critiquing in LLMs solving planning tasks.  In _NeurIPS 2023 Foundation Models for Decision Making Workshop_ , 2023a.  URL https://openreview.net/forum?id=gGQfkyb0KL.
  * Valmeekam et al. (2023b) Valmeekam, K., Marquez, M., Sreedharan, S., and Kambhampati, S.  On the planning abilities of large language models - a critical investigation.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023b.  URL https://openreview.net/forum?id=X6dEqXIsEW.
  * Wang et al. (2023) Wang, T., Yu, P., Tan, X. E., O’Brien, S., Pasunuru, R., Dwivedi-Yu, J., Golovneva, O., Zettlemoyer, L., Fazel-Zarandi, M., and Celikyilmaz, A.  Shepherd: A critic for language model generation.  _arXiv preprint arXiv:2308.04592_ , 2023.
  * Wong et al. (2023) Wong, L., Mao, J., Sharma, P., Siegel, Z. S., Feng, J., Korneev, N., Tenenbaum, J. B., and Andreas, J.  Learning adaptive planning representations with natural language guidance.  _arXiv preprint arXiv:2312.08566_ , 2023.
  * Wu et al. (2023) Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., and Duan, N.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  _arXiv preprint arXiv:2303.04671_ , 2023.
  * Xie et al. (2023) Xie, T., Zhao, S., Wu, C. H., Liu, Y., Luo, Q., Zhong, V., Yang, Y., and Yu, T.  Text2reward: Automated dense reward function generation for reinforcement learning.  _arXiv preprint arXiv:2309.11489_ , 2023.
  * Yang et al. (2023) Yang, M., Du, Y., Ghasemipour, K., Tompson, J., Schuurmans, D., and Abbeel, P.  Learning interactive real-world simulators.  _arXiv preprint arXiv:2310.06114_ , 2023.
  * Young et al. (2021) Young, S., Gandhi, D., Tulsiani, S., Gupta, A., Abbeel, P., and Pinto, L.  Visual imitation made easy.  In _Conference on Robot Learning_ , pp. 1992–2005. PMLR, 2021.
  * Yow et al. (2024) Yow, J., Garg, N. P., Ramanathan, M., Ang, W. T., et al.  Extract–explainable trajectory corrections from language inputs using textual description of features.  _arXiv preprint arXiv:2401.03701_ , 2024.
  * Yu et al. (2023) Yu, W., Gileadi, N., Fu, C., Kirmani, S., Lee, K.-H., Arenas, M. G., Chiang, H.-T. L., Erez, T., Hasenclever, L., Humplik, J., et al.  Language to rewards for robotic skill synthesis.  _arXiv preprint arXiv:2306.08647_ , 2023.
  * Yuan et al. (2024) Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J.  Self-rewarding language models.  _arXiv preprint arXiv:2401.10020_ , 2024.
  * Zeng et al. (2022) Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al.  Socratic models: Composing zero-shot multimodal reasoning with language.  _arXiv preprint arXiv:2204.00598_ , 2022.
  * Zeng et al. (2023) Zeng, F., Gan, W., Wang, Y., Liu, N., and Yu, P. S.  Large language models for robotics: A survey.  _arXiv preprint arXiv:2311.07226_ , 2023.
  * Zha et al. (2023) Zha, L., Cui, Y., Lin, L.-H., Kwon, M., Arenas, M. G., Zeng, A., Xia, F., and Sadigh, D.  Distilling and retrieving generalizable knowledge for robot manipulation via language corrections.  _arXiv preprint arXiv:2311.10678_ , 2023.
  * Zhang et al. (2022) Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., and Shum, H.-Y.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  _arXiv preprint arXiv:2203.03605_ , 2022.
  * Zhang et al. (2019) Zhang, R., Torabi, F., Guan, L., Ballard, D. H., and Stone, P.  Leveraging human guidance for deep reinforcement learning tasks.  _arXiv preprint arXiv:1909.09906_ , 2019.
  * Zhao et al. (2023a) Zhao, T. Z., Kumar, V., Levine, S., and Finn, C.  Learning fine-grained bimanual manipulation with low-cost hardware.  _arXiv preprint arXiv:2304.13705_ , 2023a.
  * Zhao et al. (2023b) Zhao, Z., Lee, W. S., and Hsu, D.  Large language models as commonsense knowledge for large-scale task planning.  _arXiv preprint arXiv:2305.14078_ , 2023b.
  * Zitkovich et al. (2023) Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  In _Conference on Robot Learning_ , pp. 2165–2183. PMLR, 2023.

##  Appendix A Appendix

Figure 6: Additional failure cases of GPT-4V critic.

Figure 7: Visualization of the tasks and policies.

Figure 8: Preference-elicitation prompt.

Figure 9: Critique-elicitation prompt.

###  A.1 Preference elicitation

Fig. 8 shows the prompt template for instructing VLMs to provide preferences
over trajectory pairs. It follows a similar structure as the prompt for
critique elicitation, with the addition of frames from one extra video.
Similar to critique elicitation (Sec. 4.1), the examples used to illustrate
the output format are fixed across all test cases. There are three examples
corresponding to the following conditions: (a) one of the trajectories is
preferable to the other; (b) neither trajectory is desirable; and (c) both
trajectories are desirable.

As mentioned in Sec. 4.2, for each task, we have one positive sample along
with two negative samples that exhibit the same undesirable behavior(s).
Accordingly, in the evaluation, we test whether GPT-4V can give accurate
preference labels in the following cases: (a) when pairs of positive and
negative samples (denoted as positive-negative) are given; (b) pairs of
negative samples (denoted as negative-negative) are given; and (c) pairs of
positive samples (denoted as positive-positive) are given. Note that, to
create positive-positive pairs, we simply horizontally flip each frame of a
positive sample. Also note that for negative-negative samples, we expect
GPT-4V to not establish any ordering within the pairs unless valid
justifications are provided. For positive-positive samples, given how the
pairs are formed, we expect GPT-4V to state both videos are equally preferred.

| Recall | Precision
---|---|---
GPT-4V (Preference) | 65.29% | 60.65%
Table 2: Recall rate and precision rate of critiques when GPT-4V contrasts positive samples with negative samples | positive-negative | positive-positive | negative-negative
---|---|---|---
GPT-4V (Preference) | 95.61% | 3.7% | 0%
Table 3: Accuracy scores of preference labels generated by GPT-4V Metrics | Acc. | Recall | Precision
---|---|---|---
Positive | Negative
Gemini Pro | 71.92% | 42.98% | 18.18% | 44.9%
Table 4: Evaluation results of Gemini Pro critic.

Figure 10: Distributions of error types within preference justifications.

Table 3 summarizes the evaluation results (i.e., accuracy scores of preference
labels). Results suggest that GPT-4V can select the positive samples from
positive-negative pairs with high accuracy. However, in two other conditions,
GPT-4V shows subpar performance. GPT-4V always establishes invalid rankings
within negative-negative and positive-positive pairs by “hallucinating” that
one sample is perfect and one is flawed, as seen in its justifications. Along
with accuracy scores, we also report the recall and precision rates (Table 2),
and the error types within the justifications (Fig. 10) generated when
contrasting negative samples with positive samples. The results resemble those
obtained when GPT-4V criticizes individual videos, indicating that the ways of
extracting critiques do not significantly alter the output pattern.

###  A.2 Results of Gemini Pro

Table 4 shows the performance of Gemini critic in terms of critique accuracy,
recall, and precision. The evaluation setup is identical to that of GPT-4V
(Sec. 5.1). Compared to GPT-4V, Gemini tends to be “over-optimistic” about the
video, meaning that it more frequently states there is no undesirability
presented. However, this over-optimism also leads to a significantly lower
recall rate, as many undesirable events are not captured. Overall, the results
indicate that there is still a performance gap between the Pro version of
Gemini and GPT-4V, suggesting that GPT-4V currently is a better choice for
behavior critics. We did not evaluate Gemini Pro for preference elicitation
due to the restriction on the number of frames per conversion.

###  A.3 Robot experiment details

We provide the following primitive control APIs to CaP agent:

  * •

get_xy(object), which queries a top-down camera and pre-trained vision-
language models to get the location of an object on the table.

  * •

move_gripper_to(x, y, z), which moves the robot end-effector to the desired x,
y, z in the space.

  * •

control_gripper(force, position), which controls the gripper and close it to
indicated position with specified force.

  * •

rotate(degree), which rotates the end-effector according to the input degrees.

  * •

pause(second), which does nothing and hold the robot for indicated seconds.

We strictly follow the prompt structure outlined in the Code-as-Policies
paper. When it comes to code examples in prompts, we use examples drawn from a
pool of policies for tasks within the same category. For instance, for bread
grasping and spoon picking, examples may come from tasks such as picking up a
Coke can, picking up a bag of chips, and picking up bread packaged in
different ways. Similarly, for scissor handover, the example tasks include
handing over a tomato, handing over a cup of water, and handing over a pair of
scissors from different initial positions. For knife putting down, the
examples are putting-related ones, such as putting down a banana onto a book,
putting down an apple, and putting down a knife held in diverse ways by the
robot initially.

###  A.4 Examples of human-written narrations

Instruction: pick up the ceramic bowl and put in the small box for storing
ceramic household items

  * •

t=0: The robot’s gripper reaches toward the ceramic bowl

  * •

t=1: The robot’s gripper closes its gripper on the rim of the ceramic bowl,
securely grasping it

  * •

t=2: The robot lifts the bowl to a few inches above the top of the container;
the robot twists its gripper such that the bottom of the bowl faces to the
right

  * •

t=3: The robot drops the bowl into the container; it lands on its side,
bounces, and then lands upright in the container

  * •

t=4: The robot’s gripper exits the frame

Instruction: use the spatula to stir-fry the food thoroughly

  * •

t=0: The robot’s gripper grasps a spatula and prepares to stir-fry food in a
pan; the sound of the gas-powered flame fills the otherwise silent room

  * •

t=1: The robot begins to stir-fry the food; a yellow vegetable jumps out of
the pan and lands on the grating

  * •

t=2: The robot continues to stir-fry the food; the vegetables turn and flip
over inside the pan

Instruction: move the turner and place it at the left edge of the table

  * •

t=0: the robot’s gripper approaches the spatula and hovers over its handle

  * •

t=1: the robot closes its gripper and securely grasps the spatula

  * •

t=2: the robot lifts the gripper and bumps into the pot with eggs in it,
moving it a few inches closer to the edge of the table

  * •

t=3: the robot, still gripping the spatula, moves around the pot and places
the spatula on the other side of the pot

  * •

t=4: the robot withdraws its gripper from the frame

Instruction: put the facial cleanser onto the white tray

  * •

t=0: The robot’s gripper approaches a bottle of lotion which sits by itself on
a tray

  * •

t=1: The robot’s gripper pumps lotion; the lotion falls partially onto the
bottle and also onto the tray

  * •

t=2: The robot’s gripper adjusts itself to securely grasp the bottle of lotion

  * •

t=3: The robot lifts the bottle of lotion and places it into the white tray
with all the other lotions

Instruction: pour some orange juice into the guest’s glass

  * •

t=0: the robot’s gripper is firmly grasping a glass of orange juice

  * •

t=1: the robot, while approaching the cup, spills some orange juice onto the
table

  * •

t=2: the robot’s gripper tilts the glass to pour the juice into the cup

  * •

t=3: the robot pours all the orange juice from the glass into the cup

  * •

t=4: the robot’s gripper tilts the glass back to right side up and lifts the
glass out of frame