# Generating Executable Action Plans with Environmentally-Aware Language
Models

Maitrey Gramopadhye1 and Daniel Szafir1 1University of North Carolina at
Chapel Hill, United States

###### Abstract

Large Language Models (LLMs) trained using massive text datasets have recently
shown promise in generating action plans for robotic agents from high level
text queries. However, these models typically do not consider the robot’s
environment, resulting in generated plans that may not actually be executable
due to ambiguities in the planned actions or environmental constraints. In
this paper, we propose an approach to generate environmentally-aware action
plans that can be directly mapped to executable agent actions. Our approach
involves integrating environmental objects and object relations as additional
inputs into LLM action plan generation to provide the system with an awareness
of its surroundings, resulting in plans where each generated action is mapped
to objects present in the scene. We also design a novel scoring function that,
along with generating the action steps and associating them with objects,
helps the system disambiguate among object instances and take into account
their states. We evaluate our approach using the VirtualHome simulator and the
ActivityPrograms knowledge base. Our results show that the action plans
generated from our system outperform prior work in terms of their correctness
and executability by 5.3% and 8.9% respectively.

##  I INTRODUCTION

Recent work in the natural language processing (NLP) and machine learning (ML)
communities has made tremendous breakthroughs in several core aspects of
computational linguistics and language modeling driven by advances in deep
learning, data, hardware, and techniques. These advancements have led to the
release of pretrained large (million and billion+ parameter) language models
(LLMs) that have achieved state-of-the-art performance across a variety of
tasks such as text classification, generation, and summarization, question
answering, and machine translation, that demonstrate some ability to
incorporate meaningful understandings of the real world [1, 2, 3, 4, 5, 6, 7,
8]. Research has shown that LLMs have some ability to generalize across
modalities and domains, such as video retrieval from text, visual question
answering, task planning, etc. [9]. In particular, recent works have explored
using LLMs to convert high-level natural language commands to actionable steps
(e.g. “Bring water” $\rightarrow$ “Grab glass”, “fill glass with water”, “walk
to table”, “put glass on table”) for intelligent agents [10, 11, 12, 13, 14,
15].

While promising, the action steps generated by LLMs in prior work are not
always actually executable by a robot platform. For instance, for a task
“clean the room” a LLM might generate an output “call cleaning agency on
phone”; while being correct, this action plan might not be executable since
the agent might not understand the concept of “call” or have the “phone”
object in it’s environment. This limitation arises because LLMs are trained
solely on large text corpora and have essentially never had any interaction
with a physical environment. As a result, the action steps they generate lack
context on the robot’s surroundings and capabilities.

To address these issues, prior works have explored grounding LLMs by fine-
tuning models using human interactions [15, 16, 17] or training models for
downstream tasks using pretrained LLM representations [18, 19, 20, 21, 22, 23,
24, 25]. However, these methods often require additional annotated data, which
can be expensive or infeasible to obtain, and can lead to loss of generalized
knowledge from the LLM. Instead, recent research has investigated biasing LLM
output without altering their weights by using prompt engineering [10, 11, 26]
or constraining LLM output to a corpus of available action steps defined a
priori that are known to be within a robot’s capabilities [10, 11]. While
these systems effectively perform common sense grounding by extracting
knowledge from an LLM, they employ a one-fits-all approach without considering
the variations possible in the actionable environment. As a result, executing
the action plans generated by these systems either requires approximations to
the agent’s environment or time-consuming and costly pretraining to generate
an affordance score to determine the probability that an action will succeed
or produce a favourable outcome towards task completion, given the current
agent and environment states. Additionally, since prior systems are
environment agnostic, it is not possible to use them to generate executable
action plans for tasks requiring object disambiguation. For example, a task
such as “set the table” requires an agent to execute actions like “find
plate”, “grab plate”, “put plate on table”, etc. repeatedly, but with
different instances of the same object type. Current systems can lead to plans
that involve a robot grabbing and placing the same plate over and over. To
generate correct action plans for such tasks (i.e., plans where the robot
finds and sets multiple different plates), the system needs to be aware of
different object instances.

We propose a new method to address these issues while generating low-level
action plans from high-level task specifications. Our approach is inspired by
Huang et al., 2022 [10]. From an Example Dataset (see §IV) using the
ActivityPrograms knowledge base collected by [27], we sample an example
similar to the query task and environment and use it to design a prompt for a
LLM, details of which are given in Section III-A. We then use the LLM to
autoregressively generate candidates for each action step and objects
associated with that action step. We design multiple scores to rank the
generated candidates, including LLM probability for the action and object, a
matching score for the generated action to the closest action in the corpus of
available actions, a matching score for the object to the corpus of available
objects in the scene, an object disambiguation score, etc. (see §III-B and
III-C). After the top candidate is selected, we append it to the action plan
and repeat this process till the entire action plan is generated.

To evaluate our approach, we use the recently released VirtualHome environment
[27]. We use several metrics (details in §IV-A), including executability,
Longest Common Subsequence (LCS), and final graph correctness to autonomously
test generated action plans on VirtualHome, enabling us to compare our
approach against plans generated by Huang et al., 2022 [10], which represents
a state-of-the-art baseline. Overall, we found that our method increased
action plan executability and final graph correctness by 5.3% and 8.9%
respectively over [10].

In summary, we propose a novel approach using LLMs to generate low-level
action plans from high-level tasks that are directly executable. In
particular, we contribute:

  * •

A method for conditioning LLM-generated action plans on an agent’s environment
without any fine-tuning,

  * •

A method to disambiguate and associate objects in an environment to action
steps,

  * •

An experimental evaluation of our approach on the VirtualHome platform and an
evaluation methodology with metrics that enable future direct comparisons of
robot action plan generation.

##  II RELATED WORK

Our work builds upon recent efforts in robotics to leverage the potential of
LLMs. For instance, researchers are beginning to explore LLMs in the context
of applying commonsense reasoning to natural language instructions [28],
providing robotic agents with zero-shot action plans [10], and supplying high-
level semantic knowledge about robot tasks [11]. Below, we review related
research in task planning, LLMs, and action plan grounding.

###  II-A Task Planning

The problem of task planning involves generating a series of steps to
accomplish a goal in a constrained environment. Historically, this problem has
been widely studied in robotics [29, 30, 31], with most approaches solving it
by optimizing the generated plan, given environment constraints, [32, 33] and
using symbolic planning [29, 31]. Recently, machine learning methods have been
employed to relax the constraints on the environment and allow higher-level
task specifications by leveraging techniques such as reinforcement learning or
graph learning to learn task hierarchy [34, 35, 36, 37, 38, 39, 40, 41, 42,
43, 44, 45, 46, 47, 48, 49]. However, most of these methods require extensive
and costly training from demonstrations or explicitly encoded environmental
knowledge and may not generalize to unseen environments and tasks. The use of
LLMs, which encapsulate generalized world knowledge, may help plan for novel
tasks and new environments.

###  II-B Large Language Models

Large language models (LLMs) are language models, usually inspired by the
transformer architecture [50], tens of gigabytes in size and trained on
enormous amounts of unstructured text data. Recent advances in the field of
natural language processing have shown that LLMs are useful for several
downstream applications including interactive dialogue, essay generation,
creating websites from text descriptions, automatic code completion, etc. [1,
2, 4, 3]. During their pretraining, LLMs can accumulate diverse and extensive
knowledge [51, 52, 53] that enable use in applications beyond NLP, such as
retrieving visual features [54] and solving mathematical problems [55, 56] or
as pretrained models for other modalities [57, 58]. In particular, knowledge
embedded in LLMs can be utilised in robotics to generate actionable plans for
agents from high-level queries. However, in order for a plan to be executable
by a robot, the outputs from the LLMs need to be grounded in the context of
the robot’s environment and capabilities.

###  II-C Grounding Natural Language in Action Plans

There has been considerable work towards grounding natural language in
actionable environments. Prior research has focused on parsing natural
language or analysing it as series of lexical tokens to remove ambiguity and
mapping language commands to admissible actions [59, 60, 61, 62]. However,
these methods usually require extensive, manually coded rules and thus fail to
generalize to novel environments and tasks. More relevant to our approach,
recent work has explored grounding language models using additional
environment elements as inputs to the system [63, 64, 65, 66, 67]. Alternative
techniques include prompting [10, 26] and constraining language model outputs
to admissible actions [11, 12, 13, 14]. To also ground the output of language
models in the environment of the agent, prior works have tried using LLMs as
fixed backbones for applications [18, 20, 21, 22, 23, 24, 25, 68] or fine-tune
or rank model outputs through interactions with the environment [15, 16, 17].
Our work extends such approaches, where we use additional inputs from the
environment (i.e., objects and their properties) to condition the model output
without any fine-tuning of the LLM or pretraining to learn value functions for
ranking LLM outputs.

##  III APPROACH

In this section, we discuss our proposed method to generate directly
executable action plans from high-level tasks (Figure 1 provides a visual
overview). Following Huang et al., 2022 [10], our approach uses two language
models, a planning LM ($LM_{P}$) to generate the action plan and calculate a
score for the similarity of an object with the other objects associated with
the action plan; and a translation LM ($LM_{T}$) to calculate embeddings for
objects and actions.

Figure 1: An overview of our approach. We generate action plans by first
selecting an example that has a similar task and environment to the query. We
use this example to autoregressively prompt the Planning LM to generate an
action plan and map the output to admissible actions and objects using the
Translation LM.

###  III-A LLM Action Plan Prompt Generation

Large language models have the ability to learn from context during inference,
i.e., when autoregressively sampled, LLMs can generate meaningful text to
complete or extend a given textual prompt [1]. Motivated by Huang et al., 2022
[10], we leverage this capability in designing prompts for LLM sampling that
generate action plans. Specifically, we select an example from an Example set
of task and action plans synthesized from the ActivityPrograms dataset (see
§IV) and construct a prompt for the LLM by prepending the example task and
action plan to the current task.

We dynamically select the example during inference to design a prompt similar
to the query. As in Huang et al., 2022 [10], we use the query task to select
the example. However, one of our novel extensions is to also use the
environment associated with the query to construct a prompt, keeping in mind
the objects the agent can currently interact with and their states. For
example, for a query ($Q$) with task “Play video games”, an example ($E_{1}$)
with task “play board games” may be chosen considering just the task
similarity. However, another example ($E_{2}$) with task “Use the computer”
may be more relevant because the action plan for both $Q$ and $E_{2}$ would
have actions involving similar objects, such as “switch on computer”, “type on
keyboard”, “push mouse”, etc., which may not be present in the action plan for
$E_{1}$. Considering the environment in selecting the example may also help in
disambiguating between examples with high task similarity but different
objects. For example, a “clean room” action plan, which uses a rag, and a
“clean floor” plan, which uses a mop, may both have high task similarity to a
“clean the house” query. Considering the objects present in the query
environment (e.g., a rag is present, but not a mop) can help determine the
better example.

We start by selecting $\\{Q^{i}_{e}\\}_{i=1}^{N_{e}}$ examples that have tasks
$\\{T^{i}_{e}\\}_{i=1}^{N_{e}}$ similar to the task $T$ of the query $Q$. We
use the cosine similarity ($C$) of task embeddings to calculate task
similarity given by:

$S_{M}(T,T_{e})=C(LM_{T}(T),LM_{T}(T_{e}))$

We than compare the environments $\\{E^{i}_{e}\\}_{i=1}^{N_{e}}$ of the
selected $N_{e}$ examples with the environment of the query $E$, to calculate
the environment similarity given by:

$S_{G}(E,E_{e})=IoU(nodes(E),nodes(E_{e}))+\\\ IoU(edges(E),edges(E_{e}))$

Finally, from the selected $N_{e}$ examples, we select one example $Q^{*}_{e}$
that maximises the example score given by:

$S_{M}(T^{*}_{e},T)+W_{s}\cdot S_{G}(E^{*}_{e},E)$

where $W_{s}$ is a hyperparameter. With the example task $T^{*}_{e}$, action
plan $A^{*}_{e}$, and query task $T$, we form a prompt
($Pr_{a}=T^{*}_{e}+A^{*}_{e}+T$) for generating the action plan and a prompt
($Pr_{o}$) from the set of objects associated with the actions plan
$A^{*}_{e}$. We use $Pr_{o}$ to calculate the similarity scores between any
new objects and the objects already associated with the action plan.

###  III-B Action Step Generation

As in Huang et al., 2022 [10], we sample the $LM_{P}$ multiple times using
prompt $Pr_{a}$ to get $k$ samples for each action step, and the LLM
generation probability associated with each sample ($P_{a}$). $P_{a}$ gives a
score for how relevant the planner LM thinks the action is to the current
action plan and prompt. However, since the output of the language model is
unconstrained, it can include infeasible actions that the agent cannot
actually accomplish. To make sure the actions generated are executable, we map
each sample to its closest admissible action. We calculate the closest
admissible action ($a^{{}^{\prime}}_{s}$) to a sample action $a_{s}$ as the
admissible action that maximises the action matching score given by:

$P_{aM}=max(S_{M}(a_{s},a_{v});\forall a_{v}\in A_{v})$

Where $S_{M}(a_{s},a_{v})=C(LM_{T}(a_{s}),LM_{T}(a_{v}))$ and $A_{v}$ is the
corpus of all admissible atomic actions. We also include a penalty for actions
that match previous actions to discourage action plan loops.

###  III-C Object Association

Each action step discussed so far is of the format [Action] <Object names>.
For an agent to execute an action step, it needs to associate the object names
in the step with objects in the environment. However, action plans generated
without considering the environment often contain steps with object names that
cannot be directly mapped to objects present in the environment, meaning the
plan is not executable.

Prior research that is agnostic to environment information has often relied on
approximate object associations, where a specific object is chosen randomly
from a set of multiple objects with the same names for association with the
action step. However, such random assignment of objects might not lead to
executable or even correct action plans. As an alternative, object assignment
may be performed in post-processing using hard-coded environment information
[10]. Many simulation platforms, including VirtualHome, used in this work
[27], offer a way to associate objects using hard coded rules. However, it is
unlikely that such information would be readily available in real-world robot
deployments. Such systems are also difficult to evaluate as they rely on human
evaluators to judge action plan correctness and executability; while human
evaluations may provide significant feedback, humans often lack the complete
knowledge of the environment and agent constraints necessary to properly
evaluate an action plan.

Instead, we propose a way to autonomously associate objects in the query
environment with action steps during action plan generation, without the use
of any hard-coded information. As a result, our action plans can be used
directly in the query environment. Additionally, since our action plans
consider the agent’s environment, we are able to generate action plans for
tasks that were previously infeasible. For example, a task “set the table” has
an action plan with steps “find a plate”, “put plate on table”, “grab cup”,
“put cup on table”, etc. repeated multiple times, each time for different
“plate” and “cup” objects. A system that does not have any scene information
may correctly generate an action plan that has these action steps repeated in
a relevant order, but during execution all “cup” objects will be mapped to the
same “cup” and all “plate” objects will be mapped to the same “plate”,
creating an incorrect execution result. Since our method can reason about the
“cup” and “plate” objects in the scene (and their states), we can associate
different objects of the same name with repeated action steps, leading to
correct execution.

Since the admissible actions for our agent all follow the same schema, we
first parse the action step to extract the object (e.g., “put cup on table”
has the object names “cup” and “table”). For each object name ($\hat{o}$), we
then assign an object name from the environment. To do so, we first use a
translation LM $LM_{T}$ to find $o^{\prime}$, the closest available object
name to $\hat{o}$ that is present in the agent’s environment, chosen by
maximizing an object matching score given by:

$P_{oM}=max(S_{M}(\hat{o},o);\forall o\in O)$

where $O$ is the corpus of all object names present in the agent’s current
environment. We also calculate an object relevance score that denotes the
similarity of $o^{\prime}$ with the object names in the action plan of the
example ($A^{*}$) and those already associated with the previous steps of the
generated action plan. We append $o^{\prime}$ to the prompt for objects
$Pr_{o}$ and use it as an input to the planning LM $LM_{P}$ to calculate the
perplexity loss $L_{ppl}$ for $o^{\prime}$, and use it to construct the object
relevance score ($P_{o}$) given by:

$P_{o}=log(\frac{1}{L_{ppl}})$

The object matching and relevance scores enable a consideration of environment
objects for action plan generation, but are associated with object names and
will be the same for all objects in the scene with the same name. To
disambiguate among such objects, we calculate an object disambiguation score
($P_{oD}$) as being inversely related to mean distance of the object ($d_{o}$)
from the objects that the agent interacted with in the last generated step:

$P_{oD}=\frac{exp(\frac{-dist}{10})}{10}$

This score prioritizes objects that are near the agent’s location in the prior
step. In some cases, when the object location is absent in the environment,
object disambiguation is unnecessary and $P_{oD}=0$. For each object, we also
look for repetitions, i.e., instances where the current ($action$, $objects$)
pair is found in the already generated action plan. For each such instance we
incur a negative penalty to encourages the agent to interact with a different
object with the same name each time $action$ is associated with some object.
This negative score builds for every instance of repetition and can also
reduce the overall score for the ($action$, $objects$) pair, deterring our
action plans from loops of multiple $actions$ (e.g., “walk to table”, “walk to
chair” repeated over and over) that prior action plans suffer from.

Finally, we calculate a mean for the 3 object scores ($P_{oM},P_{o},P_{oD}$),
over all objects extracted for the action step.

###  III-D Ranking and Termination

We take a weighted sum of the scores described above to rank all ($action$,
$objects$) pairs:

$W_{a}\cdot P_{a}+W_{aM}\cdot P_{aM}+W_{o}\cdot P_{o}+W_{oM}\cdot
P_{oM}+W_{oD}\cdot P_{oD}$

Where $W_{a},W_{aM},W_{o},W_{oM}$ and $W_{oD}$ are hyperparameter weights for
each score. We then select the highest ranking ($action$, $objects$) pair and
append it to the action plan. We also append $action$ to the prompt for action
generation $Pr_{a}$ and $objects$ to prompt for object relevance score
$Pr_{o}$.

If the score for the highest ranking ($action$, $objects$) pair is below a
cutoff hyperparameter, we terminate the action plan. Following Huang et al.,
2022 [10], we also terminate the action plan if at any point most samples
generated by $LM_{P}$ for an action step are null.

The resulting action plans generated by the language model are in natural
language. To execute and evaluate plans, we used the VirtualHome simulation
platform (see §IV), which required parsing this plan to create a raw action
plan matching the VirtualHome agent schema. This parsing was performed using a
predefined mapping since all action steps follow a fixed pattern.

##  IV EVALUATION

We evaluated our approach in generating environmentally-aware action plans
against plans generated using the method in Huang et al., 2022 [10], treated
as a baseline state-of-the-art LLM action plan generation system that does not
consider environment information. We executed all plans using VirtualHome, a
multi-agent simulation platform. VirtualHome provides diverse and customizable
household environments that support a wide array of possible interactions in
the form of atomic agent actions. An atomic action is specified by [Action]
<Objects> (object_ids), for example: [PutBack] <glass> (2) <sink> (1).

For our experiments, we use the ActivityPrograms Knowledge Base released by
Puig et al., 2018 [27]. This dataset contains 292 unique high-level tasks,
with 1374 unique action plans and 6201 unique environments in total, curated
from task and action plan samples manually annotated by Amazon Mechanical Turk
workers. Each such sample consists of a high-level task, a natural language
description detailing the task, and an action plan consisting of atomic
actions. We tested with the 292 unique high-level tasks, randomly split into 3
sets: an Example set of 167 tasks, a Validation set of 25 tasks and a Test set
of the remaining 100 tasks.

###  IV-A Metrics

We designed three metrics to evaluate action plans: executability, longest
common sequence, and correctness.

Executability measures whether the generated actions follow a logical order
and satisfy the constraints of the environment (e.g., action precondtions are
met by preceding actions, action plan objects are present in the environment,
and object states support planned actions). To measure executability, we
execute the action plan step-by-step on the VirtualHome platform and calculate
the number of steps that execute without the action plan failing. We report
the percentage of steps that execute as the action plan executability score.

Following Puig et al., 2018 [27], we compute Longest Common Subsequence (LCS)
as the length of the longest common subsequence between a generated action
plan (either via our approach or from our baseline) and a “ground truth”
action plan written by human annotators from the ActivityPrograms dataset,
divided by the length of the longer action plan. LCS provides a metric to
understand the number of correct actions being generated and also their short
term order, while penalizing longer action plans that have irrelevant or
repeated actions. However, it does not judge the correctness of the action
plan as a whole because it does not consider the position of the longest
common sequence in the action plan. Also, as LCS only compares the natural
language action plans, it does not offer a way to judge whether the action
plan is able to disambiguate among objects of the same name.

We propose Final Graph Correctness as a new metric to evaluate the final
correctness of the environment graph (a graphical structure that represents
the environment, where nodes represent objects and their states, with edges
encoding object relationships) after execution of the action plans. To measure
correctness, we execute each generated action plan in VirtualHome through the
last executable step and extract the graph for the resulting environment
($E_{o}$). We then compare this graph with the graph of the environment formed
after executing the action plan in the dataset ($E_{G}$). We independently
compute the set of nodes and edges that have changed in the initial
environment ($E_{init}$) after executing the output and human annotator ground
truth action plans respectively: $Nodes_{o}$, $Edges_{o}$, $Nodes_{G}$,
$Edges_{G}$. The final graph correctness is calculated as the sum of
intersection over unions of the two sets of nodes and edges thus obtained:

$Final\ correctness=IoU(Nodes_{o},Nodes_{G})+\\\ IoU(Edges_{o},Edges_{G})$

###  IV-B Experimental Setup

We evaluate over the Test set (100 tasks) and use the Example set (167 tasks)
for prompt engineering. The Validation set (25 tasks) was used for
hyperparameter search and ablation studies. As each task can have multiple
action plans and associated environments, we conduct 5 runs every time we
generate an action plan; in each run we randomly select an action plan and
environment for each task and report the average results over all the runs.

In action plans generated by our baseline, when objects in action steps are
not associated with the objects in the environment, for each object in each
action step, we randomly select an object of the same name and assign it to
the action step. To account for this randomness in object ID assignment, for
every run we compute the results for the baseline 3 times and report the
average results over the sub-runs.

We use open-source resources from Hugging Face Transformers [69] and
SentenceTransformers [70] for our model choices. Our primary results use
GPT2-large as the Planning LM and Sentence-RoBERTa-large as the Translation
LM, while Tables III and IV present results with other models.

Figure 2: Example plans generated by our system. For each action, matched
environment objects with ids are identified in brackets. Note that our system
can handle plans containing actions with multiple objects (e.g., keys and
hanger) and can consider multiple objects of the same name (e.g. chair and
desk).

##  V RESULTS

In this section we present our results, all of which are reported as a
percentage of 100 unless mentioned otherwise.

###  V-A Environment Aware Action Plans

We conduct a total of 5 runs and compute the action plans for each of the
samples in our Test set, using our method and the baseline [10] that doesn’t
use any scene information. Table I reports the mean score of executability,
LCS and final graph correctness, with standard deviation in parentheses.

TABLE I: Evaluation results of generated action plans Method | Executability | LCS | Final Correctness
---|---|---|---
Baseline [10] | 31.57 (1.54) | 10.22 (0.55) | 31.53 (1.59)
Ours | 36.87 (1.88) | 9.03 (0.47) | 40.43 (1.02)

Overall, we found that our method generated action plans with improved
correctness and executability scores. These results are promising as our
method, which implicitly considers object states and properties in our example
selection module, could be readily integrated with a robot perception stack
during real deployments.

We found that LCS scores were lower in our action plans in comparison to the
baseline. In examining generated plans, we found the reason for lower LCS is
due to an increase in null action plans in our method, which receive an LCS
score of 0. In some cases, when none of the samples generated for the first
step of the action plan have an object that can be associated with an object
present in the agent’s environment (e.g., the sample action step is “go to
bedroom” but the agent’s environment does not have a “bedroom” object), the
object matching score ($P_{o}M$) for all samples of the first step is low and
thus our resultant action plan is null, while the baseline still produces an
action plan that has a few steps. While both plans get an executability score
of 0 and we would argue our null action plan is more accurate (i.e., it is
impossible for an agent to perform a task with objects that do not exist, so
generating a null plan appears more reasonable than a plan involving attempts
to interact with non-existent objects), the baseline plan gets a non-zero LCS
score since LCS only compares the natural language action plans.

###  V-B Ablations

####  V-B1 Ablation of Implementation Choices

We show ablation results for the two main contributions of our method: using
environment information to select an example for prompts and including scores
for objects to generate action plans, discussed in §III-A and III-C. For our
ablation studies, we conducted 5 runs over the Validation set and report both
average results and standard deviation (Table II).

TABLE II: Ablations of implementation choices Method | Executability | LCS | Final Correctness
---|---|---|---
Final approach | 40.34 (7.76) | 9.27 (0.70) | 43.20 (2.37)
w/o scene score | 34.5 (4.33) | 9.53 (0.63) | 42.89 (1.53)
w/o object score | 40.60 (9.23) | 9.53 (1.27) | 41.55 (1.64)
baseline scores | 42.22 (8.40) | 9.69 (0.50) | 40.84 (0.84)

####  V-B2 Ablation over Planning LMs

Ablating over different choices for the Planning LM while using stsb-roberta-
large as the model for Translation LM, we found a general trend that
increasing model size for the same types of LMs increased action plan length
and correctness (Table III). As larger models have a greater capacity to hold
generalised knowledge, they are more capable at producing complicated action
plans for high-level tasks. However, we see exceptions to this trend across
different types of models, for example the larger OPT-1.3b model [71] with
1.3B parameters performed worse than the GPT2-large model [3] with 774M
parameters. It is possible that more extensive hyperparameter search per model
could maximise results. In our ablation studies, we used the same
hyperparameter set for all models.

TABLE III: Ablations of Planning LM Planning LM | Final Correctness | Plan Length
---|---|---
distilgpt2 | 29.56 (1.49) | 0.10 (0.02)
GPT2 | 29.89 (1.88) | 0.08 (0.06)
GPT2-large | 43.20 (2.37) | 4.46 (0.34)
OPT-125M | 35.79 (2.80) | 1.56 (0.36)
OPT-1.3B | 31.12 (2.61) | 2.70 (0.44)

####  V-B3 Ablation over Translation LMs

We follow the ablation design of Huang et al., 2022 [10] and ablate over 2
different sized models from Sentense BERT and Sentence RoBERTa [2, 70, 72]
using GPT2-large as the Planning LM (Table IV). We noticed that larger
translation LMs created more accurate, but shorter programs. We speculate that
larger models may create more meaningful embeddings for actions and objects
and thus better guide the Planning LM to correct actions; however they are
harsher towards planning LM outputs and thus cause the action plans to
terminate faster.

TABLE IV: Ablations of Translation LM Translation LM | Final Correctness | Plan Length
---|---|---
stsb-bert-base | 39.53 (2.08) | 5.02 (0.90)
stsb-bert-large | 44.1 (0.86) | 4.85 (0.47)
stsb-roberta-base | 42.42 (2.36) | 5.70 (0.57)
stsb-roberta-large | 43.20 (2.37) | 4.46 (0.34)

##  VI FUTURE WORK & CONCLUSIONS

In this paper, we propose a method to condition large language models on the
information contained in an agent’s environment to generate environmentally-
aware action plans from high-level tasks. We propose multiple scores to rank
the outputs of LLMs and ground them in the agent’s surroundings. We discuss
the performance of our generated action plans for several complex tasks on the
VirtualHome platform. While our results demonstrate improved performance in
terms of plan executability and correctness over a state-of-the-art baseline,
there are several areas for further improvements. For example, our approach
makes implicit use of various object properties and states when selecting
examples for prompt generation, but cannot make use of this information
directly during plan generation. Future research might explore how to further
improve plan executability by addressing this limitation. In addition, future
work is needed to validate our approach in a real-world robot deployment,
beyond the VirtualHome simulator.

## References

  * [1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _NAACL_ , 2019\.
  * [3] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019.
  * [4] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu _et al._ , “Exploring the limits of transfer learning with a unified text-to-text transformer.” _J. Mach. Learn. Res._ , vol. 21, no. 140, pp. 1–67, 2020.
  * [5] B. Z. Li, M. Nye, and J. Andreas, “Implicit representations of meaning in neural language models,” in _ACL_ , 2021.
  * [6] A. Roberts, C. Raffel, and N. M. Shazeer, “How much knowledge can you pack into the parameters of a language model?” _ArXiv_ , vol. abs/2002.08910, 2020\.
  * [7] D. W. Otter, J. R. Medina, and J. K. Kalita, “A survey of the usages of deep learning for natural language processing,” _IEEE transactions on neural networks and learning systems_ , vol. 32, no. 2, pp. 604–624, 2020.
  * [8] A. Torfi, R. A. Shirvani, Y. Keneshloo, N. Tavaf, and E. A. Fox, “Natural language processing advancements by deep learning: A survey,” _arXiv preprint arXiv:2003.01200_ , 2020.
  * [9] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill _et al._ , “On the opportunities and risks of foundation models,” _arXiv preprint arXiv:2108.07258_ , 2021.
  * [10] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” _arXiv preprint arXiv:2201.07207_ , 2022.
  * [11] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog _et al._ , “Do as i can, not as i say: Grounding language in robotic affordances,” _arXiv preprint arXiv:2204.01691_ , 2022.
  * [12] A. Suglia, Q. Gao, J. Thomason, G. Thattai, and G. Sukhatme, “Embodied BERT: A transformer model for embodied, language-guided visual task completion,” _arXiv_ , 2021. [Online]. Available: https://arxiv.org/abs/2108.04927
  * [13] A. Pashevich, C. Schmid, and C. Sun, “Episodic Transformer for Vision-and-Language Navigation,” in _ICCV_ , 2021.
  * [14] P. Sharma, A. Torralba, and J. Andreas, “Skill induction and planning with latent language,” _ArXiv_ , vol. abs/2110.01517, 2022.
  * [15] S. Li, X. Puig, Y. Du, C. J. Wang, E. Akyürek, A. Torralba, J. Andreas, and I. Mordatch, “Pre-trained language models for interactive decision-making,” _ArXiv_ , vol. abs/2202.01771, 2022.
  * [16] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. E. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. J. Lowe, “Training language models to follow instructions with human feedback,” _ArXiv_ , vol. abs/2203.02155, 2022.
  * [17] M. Reid, Y. Yamada, and S. S. Gu, “Can wikipedia help offline reinforcement learning?” 2022.
  * [18] S. Nair, E. Mitchell, K. Chen, B. Ichter, S. Savarese, and C. Finn, “Learning language-conditioned robot behavior from offline data and crowd-sourced annotation,” in _CoRL_ , 2021.
  * [19] C. Lynch and P. Sermanet, “Language conditioned imitation learning over unstructured data,” _Robotics: Science and Systems_ , 2021. [Online]. Available: https://arxiv.org/abs/2005.07648
  * [20] V. Blukis, C. Paxton, D. Fox, A. Garg, and Y. Artzi, “A persistent spatial semantic representation for high-level natural language instruction execution,” in _5th Annual Conference on Robot Learning_ , 2021. [Online]. Available: https://openreview.net/forum?id=NeGDZeyjcKa
  * [21] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,” 2022. [Online]. Available: https://arxiv.org/abs/2203.12601
  * [22] A. Akakzia, C. Colas, P.-Y. Oudeyer, M. CHETOUANI, and O. Sigaud, “Grounding language to autonomously-acquired skills via goal generation,” in _International Conference on Learning Representations_ , 2021. [Online]. Available: https://openreview.net/forum?id=chPj_I5KMHG
  * [23] R. Zellers, A. Holtzman, M. Peters, R. Mottaghi, A. Kembhavi, A. Farhadi, and Y. Choi, “PIGLeT: Language grounding through neuro-symbolic interaction in a 3D world,” in _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Online: Association for Computational Linguistics, Aug. 2021, pp. 2040–2050. [Online]. Available: https://aclanthology.org/2021.acl-long.159
  * [24] F. Hill, S. Mokra, N. Wong, and T. Harley, “Human instruction-following with deep reinforcement learning via transfer-learning from text,” _ArXiv_ , vol. abs/2005.09382, 2020.
  * [25] P. C. Humphreys, D. Raposo, T. Pohlen, G. Thornton, R. Chhaparia, A. Muldal, J. Abramson, P. Georgiev, A. Santoro, and T. Lillicrap, “A data-driven approach for learning to control computers,” in _Proceedings of the 39th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Eds., vol. 162. PMLR, 17–23 Jul 2022, pp. 9466–9482. [Online]. Available: https://proceedings.mlr.press/v162/humphreys22a.html
  * [26] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,” _ArXiv_ , vol. abs/2201.11903, 2022.
  * [27] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba, “Virtualhome: Simulating household activities via programs,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8494–8502.
  * [28] H. Chen, H. Tan, A. Kuntz, M. Bansal, and R. Alterovitz, “Enabling robots to understand incomplete natural language instructions using commonsense reasoning,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 1963–1969.
  * [29] R. E. Fikes and N. J. Nilsson, “Strips: A new approach to the application of theorem proving to problem solving,” _Artificial Intelligence_ , vol. 2, no. 3, pp. 189–208, 1971. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0004370271900105
  * [30] E. D. Sacerdoti, “A structure for plans and behavior,” 1977.
  * [31] D. S. Nau, Y. Cao, A. Lotem, and H. Muñoz-Avila, “Shop: Simple hierarchical ordered planner,” in _IJCAI_ , 1999.
  * [32] M. Toussaint, “Logic-geometric programming: An optimization-based approach to combined task and motion planning,” in _IJCAI_ , 2015.
  * [33] M. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum, “Differentiable physics and stable modes for tool-use and manipulation planning - extended abtract,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_. International Joint Conferences on Artificial Intelligence Organization, 7 2019, pp. 6231–6235. [Online]. Available: https://doi.org/10.24963/ijcai.2019/869
  * [34] D. Xu, S. Nair, Y. Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese, “Neural task programming: Learning to generalize across hierarchical tasks,” _CoRR_ , vol. abs/1710.01813, 2017. [Online]. Available: http://arxiv.org/abs/1710.01813
  * [35] D. Xu, R. Martín-Martín, D.-A. Huang, Y. Zhu, S. Savarese, and L. Fei-Fei, “Regression planning networks,” in _Thirty-third Conference on Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [36] D. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and J. C. Niebles, “Neural task graphs: Generalizing to unseen tasks from a single video demonstration,” _CoRR_ , vol. abs/1807.03480, 2018. [Online]. Available: http://arxiv.org/abs/1807.03480
  * [37] B. Eysenbach, R. R. Salakhutdinov, and S. Levine, “Search on the replay buffer: Bridging planning and reinforcement learning,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https://proceedings.neurips.cc/paper/2019/file/5c48ff18e0a47baaf81d8b8ea51eec92-Paper.pdf
  * [38] N. Savinov, A. Dosovitskiy, and V. Koltun, “Semi-parametric topological memory for navigation,” in _International Conference on Learning Representations_ , 2018. [Online]. Available: https://openreview.net/forum?id=SygwwGbRW
  * [39] B. Ichter, P. Sermanet, and C. Lynch, “Broadly-exploring, local-policy trees for long-horizon task planning,” in _CoRL_ , 2021.
  * [40] C. Matuszek, N. FitzGerald, L. Zettlemoyer, L. Bo, and D. Fox, “A joint model of language and perception for grounded attribute learning,” in _ICML_ , 2012\.
  * [41] T. Silver, R. Chitnis, N. Kumar, W. McClinton, T. Lozano-Perez, L. P. Kaelbling, and J. Tenenbaum, “Inventing relational state and action abstractions for effective and efficient bilevel planning,” in _The Multi-disciplinary Conference on Reinforcement Learning and Decision Making (RLDM)_ , 2022.
  * [42] C. R. Garrett, C. Paxton, T. Lozano-Perez, L. P. Kaelbling, and D. Fox, “Online replanning in belief space for partially observable task and motion problems,” _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 5678–5684, 2020.
  * [43] Y. Zhu, D. Gordon, E. Kolve, D. Fox, L. Fei-Fei, A. K. Gupta, R. Mottaghi, and A. Farhadi, “Visual semantic planning using deep successor representations,” _2017 IEEE International Conference on Computer Vision (ICCV)_ , pp. 483–492, 2017.
  * [44] B. Wu, S. Nair, L. Fei-Fei, and C. Finn, “Example-driven model-based reinforcement learning for solving long-horizon visuomotor tasks,” _ArXiv_ , vol. abs/2109.10312, 2021.
  * [45] S. Nair and C. Finn, “Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation,” in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=H1gzR2VKDH
  * [46] F. Xia, C. Li, R. Martín-Martín, O. Litany, A. Toshev, and S. Savarese, “Relmogen: Leveraging motion generation in reinforcement learning for mobile manipulation,” _arXiv preprint arXiv:2008.07792_ , 2020\.
  * [47] C. Li, F. Xia, R. Martin-Martin, and S. Savarese, “Hrl4in: Hierarchical reinforcement learning for interactive navigation with mobile manipulators,” _arXiv preprint arXiv:1910.11432_ , 2019.
  * [48] A. Pu, A. Katabarwa, B. Hiziroglu, and O. Dai, “[re] language as an abstraction for hierarchical deep reinforcement learning,” 2020, submitted to NeurIPS 2019 Reproducibility Challenge. [Online]. Available: https://openreview.net/forum?id=S1xcL6qfpr
  * [49] D. Shah, A. T. Toshev, S. Levine, and brian ichter, “Value function spaces: Skill-centric state abstractions for long-horizon reasoning,” in _International Conference on Learning Representations_ , 2022. [Online]. Available: https://openreview.net/forum?id=vgqS1vkkCbE
  * [50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017\. [Online]. Available: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [51] J. Davison, J. Feldman, and A. Rush, “Commonsense knowledge mining from pretrained models,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 1173–1178. [Online]. Available: https://aclanthology.org/D19-1109
  * [52] Z. Jiang, F. F. Xu, J. Araki, and G. Neubig, “How can we know what language models know?” _CoRR_ , vol. abs/1911.12543, 2019. [Online]. Available: http://arxiv.org/abs/1911.12543
  * [53] F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, “Language models as knowledge bases?” _ArXiv_ , vol. abs/1909.01066, 2019.
  * [54] G. Ilharco, R. Zellers, A. Farhadi, and H. Hajishirzi, “Probing text models for common ground with visual representations,” _ArXiv_ , vol. abs/2005.00619, 2020.
  * [55] K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” _ArXiv_ , vol. abs/2110.14168, 2021.
  * [56] J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu, “Generate & rank: A multi-task framework for math word problems,” _arXiv preprint arXiv:2109.03034_ , 2021.
  * [57] K. Lu, A. Grover, P. Abbeel, and I. Mordatch, “Pretrained transformers as universal computation engines,” _arXiv preprint arXiv:2103.05247_ , 2021\.
  * [58] M. Tsimpoukelli, J. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill, “Multimodal few-shot learning with frozen language models,” _Proc. Neural Information Processing Systems_ , 2021.
  * [59] Y. Artzi and L. Zettlemoyer, “Weakly supervised learning of semantic parsers for mapping instructions to actions,” _Transactions of the Association for Computational Linguistics_ , vol. 1, pp. 49–62, 2013. [Online]. Available: https://aclanthology.org/Q13-1005
  * [60] D. Misra, K. Tao, P. Liang, and A. Saxena, “Environment-driven lexicon induction for high-level instructions,” in _ACL_ , 2015.
  * [61] D. Misra, J. Sung, K. Lee, and A. Saxena, “Tell me dave: Context-sensitive grounding of natural language to manipulation instructions,” _The International Journal of Robotics Research_ , vol. 35, pp. 281 – 300, 2016.
  * [62] M. Tenorth, D. Nyga, and M. Beetz, “Understanding and Executing Instructions for Everyday Manipulation Tasks from the World Wide Web,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , Anchorage, AK, USA, May 3–8 2010, pp. 1486–1491. [Online]. Available: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5509955
  * [63] C. Sun, A. Myers, C. Vondrick, K. P. Murphy, and C. Schmid, “Videobert: A joint model for video and language representation learning,” _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 7463–7472, 2019\.
  * [64] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,” in _Arxiv_ , 2019\.
  * [65] J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” in _NeurIPS_ , 2019.
  * [66] R. Zellers, X. Lu, J. Hessel, Y. Yu, J. S. Park, J. Cao, A. Farhadi, and Y. Choi, “Merlot: Multimodal neural script knowledge models,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 23 634–23 651. [Online]. Available: https://proceedings.neurips.cc/paper/2021/file/c6d4eb15f1e84a36eff58eca3627c82e-Paper.pdf
  * [67] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” _CoRR_ , vol. abs/2103.00020, 2021. [Online]. Available: https://arxiv.org/abs/2103.00020
  * [68] C. Lynch and P. Sermanet, “Grounding language in play,” _ArXiv_ , vol. abs/2005.07648, 2020.
  * [69] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew, “Huggingface’s transformers: State-of-the-art natural language processing,” _ArXiv_ , vol. abs/1910.03771, 2019.
  * [70] N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” _ArXiv_ , vol. abs/1908.10084, 2019.
  * [71] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,” 2022.
  * [72] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _ArXiv_ , vol. abs/1907.11692, 2019.