checkmark

# HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive
Vision-Language Domains with Memory-Augmented Language Models

###### Abstract

Recent research on instructable agents has used memory-augmented Large
Language Models (LLMs) as task planners, a technique that retrieves language-
program examples relevant to the input instruction and uses them as in-context
examples in the LLM prompt to improve the performance of the LLM in inferring
the correct action and task plans. In this technical report, we extend the
capabilities of HELPER, by expanding its memory with a wider array of examples
and prompts, and by integrating additional APIs for asking questions. This
simple expansion of HELPER into a shared memory enables the agent to work
across the domains of executing plans from dialogue, natural language
instruction following, active question asking, and commonsense room
reorganization. We evaluate the agent on four diverse interactive visual-
language embodied agent benchmarks: ALFRED, TEACh, DialFRED, and the Tidy
Task. HELPER-X achieves few-shot, state-of-the-art performance across these
benchmarks using a single agent, without requiring in-domain training, and
remains competitive with agents that have undergone in-domain training.

Gabriel Sarch Sahil Somani∗ Raghav Kapoor∗
Michael J. Tarr Katerina Fragkiadaki

∗equal contribution

Carnegie Mellon University

helper-agent-llm.github.io

##  1 Introduction

A typical way to adapt LLMs to downstream applications is through prompting
(Brown, 2020; Alayrac, 2022; Liu, 2022b; Hongjin, 2022; Mishra, 2022; Wei,
2021; Song, 2022b), exploiting their strong in-context and few-shot learning
abilities. When the amount of in-context examples and task descriptions
necessary to cover the task constraints increases, inference costs
significantly rise due to additional attention operations. To handle
computational challenges and LLM context length, a growing body of research
explores the concept of “memory-augmented prompting” – a method that involves
retrieving a set of pertinent in-context examples to append to the prompt,
thereby broadening their applicability (Perez, 2021; Schick, 2021; Gao, 2020;
Liu, 2022c; Song, 2023; Sarch, 2023; Lewis, 2020; Mao, 2021). HELPER Sarch
(2023) retrieves a set of language-program examples based on the user’s input
instruction and adds them to the prompt to provide contextualized examples for
GPT-4 task planning.

Figure 1:  TEACh-tailored HELPER Sarch (2023) demonstrates a 6.9% drop in
success when applied to ALFRED, despite sharing the same action space and
environments, due to variations in language inputs and tasks. HELPER-X
consistently performs well in both domains with one model.

Despite GPT-4’s robust generalization, memory-augmented prompting tailored for
one domain does not guarantee high performance in a similar yet distinct
domain. Applying HELPER, prompted with TEACh-specific examples and
descriptions, to ALFRED – a related domain that shares the same action space
and environments but differs in language inputs and task types – results in a
notable 6.9% decrease in accuracy compared to a HELPER model with a
specialized prompt and customized example memory specifically for ALFRED, and
vice versa when doing the same on TEACh (3.2% decrease), as shown in Figure 1
(right).

We report that two simple extensions of HELPER that allows for strong
performance across four domains, by expanding its memory with a wider array of
examples and prompts, and by integrating a additional APIs for asking
questions. Specifically, we introduce two HELPER-X variants: HELPER-XP, which
retrieves domain-specific prompt templates and related in-context examples for
the LLM, and HELPER-XS, which retrieves in-context examples from a shared
memory under a single prompt template.

We evaluate HELPER-X across four domains that include dialogue-based task
completion on TEACh (Padmakumar, 2021), following instructions from natural
language on ALFRED (Shridhar, 2020), engaging in instruction following with
active question asking on DialFRED (Gao, 2022), and organizing rooms using
spatial commonsense reasoning in the Tidy Task (Sarch, 2022). HELPER-X
demonstrates state-of-the-art performance in the few-shot example domain, that
is, without in-domain training. Extending the language-program memory does not
cause interference and does not hinder performance. In fact, HELPER-X matches,
and sometimes even exceeds, the performance of agents prompted with a single-
domain in mind.

##  2 HELPER-X

We extend HELPER (Sarch, 2023) to work across four domains. We propose two
versions to extend the memory-augmented prompting of LLMs in HELPER: 1)
HELPER-XP that retrieves from a memory of domain-tailored prompt templates and
associated domain-specific examples (Section 2.2.1), and 2) HELPER-XS that
expands the memory of HELPER into a shared memory of in-context examples
across domains combined with a domain-agnostic prompt template (Section
2.2.2). Additionally, we extend the capabilities of HELPER for question
asking, by appending a question API with functions defining possible questions
and their arguments to the LLM prompt (Section 2.2.3). We use HELPER (Sarch,
2023) for execution of the generated program using standard perception
modules.

###  2.1 Background

Here, we give an account of HELPER to make the paper self-contained.

HELPER prompts an LLM, namely GPT-4 (gpt, 2023), to generate plans as Python
programs. It assumes that the agent has access to a set of action skills $S$
(e.g., go_to(X), pickup(X), etc.). HELPER adds these skills to the prompt in
the form of a Python API. The LLM is instructed only to call these pre-defined
skill functions in its generated programs. HELPER considers a key-value memory
of language inputs and successful program pairs. It retrieves a set of in-
context examples relevant to the current input language to add to the prompt
to assist program generation. Each key is encoded into a language embedding.
The top-$k$ language-program pairs are retrieved based on their euclidean
distance to the encoding of the language input $I$ encoding.The HELPER prompt
also contains a role description ("You are a helpful assistant with expertise
in…"), a task description ("Your task is to …") and guidelines to follow for
program generation ("You should only use functions in the API…"), that are
commonly tailored to the domain-of-interest.

HELPER (Sarch, 2023), using RGB input, estimates depth maps, object masks, and
agent egomotion at each timestep. This facilitates the creation and upkeep of
a 3D occupancy map and an object memory database, essential for obstacle
navigation and object tracking. Object detection in each frame leads to
instance aggregation based on 3D centroid proximity, with each instance
characterized by dynamic state attributes (e.g., cooked, sliced, dirty). When
an action fails, a Vision-Language Model (CLIP (Radford, 2021)) provides
feedback, prompting the LLM to re-plan. For objects not present in the map,
the LLM suggests areas for HELPER to search (e.g., “near the sink").

Figure 2: Illustration of the shared example memory (HELPER-XS; top) and the
prompt retrieval (HELPER-XP; bottom). The memory is shared across domains in
both versions, allowing language and task inputs from any of the domains.

###  2.2 Unified Memory-Augmented Prompting

We explore two ways to expand HELPER to work across four domains, either
through prompt retrieval (Section 2.2.1) or through a shared example memory
(Section 2.2.2).

####  2.2.1 Prompt Retrieval

Given an input language instruction $I$, the prompt retrieval agent HELPER-XP
retrieves a specialized prompt template $P$ and an associated set of
specialized examples $E$. Each specialized prompt template contains role
descriptions, task instructions and guidelines tailored to each domain,
namely, dialogue-based task completion (based on TEACh (Padmakumar, 2021)),
instruction following from natural language (based on ALFRED (Shridhar,
2020)), instruction following with active question asking (based on DialFRED
(Gao, 2022)), or tidying up rooms (based on Tidy Task (Sarch, 2022)). For
retrieval, a query is generated from the input instruction by encoding the
instruction into an embedding vector using a pre-trained language model
(Greene, 2022). The query retrieves the closest key from memory, where each
key represents the language encodings of each prompt template and example text
, as shown in Figure 2. The top-$k$ in-context examples are further retrieved
from the specialized set of examples associated with the retrieved prompt and
added to the retrieved prompt template, and the resulting prompt is used for
LLM’s program generation.

####  2.2.2 Shared Example Memory

Given an input language instruction $I$, the shared example memory agent
HELPER-XS retrieves a set of in-context examples from a shared memory that
includes in-context examples from all domains considered. These examples are
added to a domain-agnostic prompt template that does not have a specialized
role description, task instructions or guidelines for any single domain. A
query is generated from the input instruction by encoding it into an embedding
vector using a pre-trained language model (Greene, 2022). The keys represent
encodings of each in-context example language in the shared memory. The query
embedding retrieves the top-$k$ nearest neighbors keys and their values. These
are added to the prompt as relevant in-context examples for LLM program
generation, as shown in Figure 2.

####  2.2.3 Question Asking API

A natural limitation of asking questions in a simulator is that only certain
types of questions can be understood and answered. We constrained the set of
possible questions asked by the agent by defining an API of available
questions in the DialFRED (Gao, 2022) benchmark and their arguments that
HELPER-X can call on to gather more information. These include questions in
three categories—Location, Appearance, and Direction—pertaining to the agent’s
next interaction object. Importantly, this API can be continuously expanded by
adding an additional function to the question-asking API.

Implementation details. We follow the network implementation of HELPER (Sarch,
2023). We use GPT-4-0613 (gpt, 2023) for text generation and text-embedding-
ada-002 (Greene, 2022) for text embeddings. We use the SOLQ object detector
(Dong, 2021) and ZoeDepth network (Bhat, 2023) for depth estimation from RGB
input. We use $k=3$ for example retrieval.

##  3 Experiments

We test HELPER-X in the following benchmarks: 1. Inferring and executing
action plans from dialogue (TEACh (Padmakumar, 2021)), 2. Inferring and
executing action plans from instructions (ALFRED (Shridhar, 2020)), 3. Active
question asking for seeking help during instruction execution (DialFRED (Gao,
2022)), and 4. Tidying up rooms (Tidy Task (Sarch, 2022)).

###  3.1 Inferring and Executing Action Plans from Dialogue

Inferring and executing task plans from dialogue involves understanding
dialogue segments and executing related instructions using the provided
information in the dialogue. This task evaluates the agent’s ability in
understanding noisy and free-form conversations between two humans discussing
about a household task.

##### Dataset

We use the TEACh benchmark (Padmakumar, 2021), which consists of over 3,000
dialogues focused on household tasks in the AI2-THOR environment Kolve (2017).
We use the Trajectory from Dialogue (TfD) variant, where an agent, given a
dialogue segment, must infer action sequences to fulfill tasks like making a
coffee or preparing a salad. The training dataset contains 1482 expert
demonstrations with associated dialogues. The evaluation includes 181 ’seen’
and 612 ’unseen’ episodes, with ’seen’ having different object placements and
states than in training, and ’unseen’ also having different object instances
and room environments. The agent receives an egocentric image at each step and
selects actions to execute, such as pickup(X), turn_left(), etc.

Baselines We consider two kinds of baselines: 1. Methods that supervise low-
level or high-level action prediction from language and visual input using
expert demonstrations in the training set (1482 in number) (Pashevich, 2021;
Zheng, 2022; Min, 2021; Zhang, 2022), and 2. Methods that use a small amount
of expert demonstrations for prompting pretrained LLM (Sarch, 2023).
Specifically, HELPER and HELPER-X use 11 domain-specific examples. We
additionally include a comparison to HELPER-ALF, which uses specialized
prompts and examples for ALFRED.

Evaluation Metrics We follow the TEACh evaluation metrics. Task success rate
(SR) is a binary metric of whether all subtasks were successfully completed.
Goal condition success rate (GC) quantifies the proportion of achieved goal
conditions across all sessions. Both SR and GC have path-length weighted
variants weighted by (path length of the expert trajectory) / (path length
taken by the agent).

Results are reported in Table 1. On validation unseen, HELPER-XS and HELPER-XP
demonstrate performance on-par with HELPER, with HELPER-XS even slightly
outperforming HELPER, despite HELPER-X being shared across four domains.
HELPER-X also outperforms the best supervised baselines trained in-domain with
many demonstrations. On validation seen, both HELPER-X variants outperform
HELPER, with the best model HELPER-XP outperforming HELPER by 2.7% in success
rate..

Table 1: Trajectory from Dialogue (TfD) evaluation on the TEACh validation set. Trajectory length weighted metrics are included in ( parentheses ). FS = few shot. Sup. = supervised. G = generalist; shared across benchmarks. GC = goal-condition success.  |  | Unseen | Seen
---|---|---|---
|  | Success | GC | Success | GC
Sup. |  E.T. (Pashevich, 2021) | 0.5 (0.1) | 0.4 (0.6) | 1.0 (0.2) | 1.4 (4.8)
JARVIS (Zheng, 2022) | 1.8 (0.3) | 3.1 (1.6) | 1.7 (0.2) | 5.4 (4.5)
FILM (Min, 2022) | 2.9 (1.0) | 6.1 (2.5) | 5.5 (2.6) | 5.8 (11.6)
DANLI (Zhang, 2022) | 8.0 (3.2) | 6.8 (6.6) | 5.0 (1.9) | 10.5 (10.3)
ECLAIR (Kim, 2023) | 13.2 | – | – | –
FS | HELPER-ALF | 10.5 (1.8) | 10.3 (4.5) | 13.3 (2.8) | 14.2 (7.5)
HELPER (Sarch, 2023) | 13.7 (1.6) | 14.2 (4.6) | 12.2 (1.8) | 18.6 (9.3)
G+FS | HELPER-XP | 13.6 (2.0) | 13.6 (5.6) | 14.9 (3.6) | 20.3 (11.0)
HELPER-XS | 14.5 (2.1) | 14.0 (5.4) | 14.4 (3.5) | 19.9 (11.0)

###  3.2 Following Natural Language Instructions

Natural language instruction following evaluates the agent’s ability to carry
out high-level instructions (“Rinse off a mug and place it in the coffee
maker") and low-level ones (“Walk to the coffee maker on the right") provided
by a human user. Importantly, the language and tasks in this evaluation differ
from the ones in the TEACh benchmark (Section 3.1).

##### Dataset

Te ALFRED (Shridhar, 2020) is a vision-and-language navigation benchmark
designed for embodied agents to execute tasks in domestic settings from RGB
sensory input. It includes seven task types across 207 environments, that
involve 115 object types in 4,703 task instances, varying from simple object
relocation to placing a heated item in a receptacle. The dataset includes
detailed human-authored instructions and high-level goals, based on 21,023
expert demonstrations. It also comprises 820 ’seen’ and 821 ’unseen’
validation episodes. Agents receive egocentric RGB images at each step and
select actions from a predefined set to progress, such as pickup(X),
turn_left(), etc.

Baselines Again, we consider two sets of baselines: those that supervise low-
level or high-level action prediction using the expert demonstrations in the
training set (Pashevich, 2021; Zheng, 2022; Min, 2021; Zhang, 2022; 2021;
Song, 2022a; Blukis, 2022; Bhambri, 2023; Liu, 2022a; Murray, 2022; Inoue,
2022), and those that use a small amount of demonstrations ($<=100$) (few-
shot) (Sarch, 2023; Song, 2023; Brohan, 2023; Liu, 2023). The SayCan (Brohan,
2023), FILM (Min, 2021) (FS), and HLSM (Blukis, 2022) (FS) few shot baselines
are adapted for the few-shot ALFRED setting by the authors of Song (2023),
where the planning modules in FILM and HLSM are re-trained using only 100
demonstrations. We adapt HELPER (Sarch, 2023) as a baseline with specialized
prompts and examples for ALFRED, as well as a comparison to HELPER-TEACh,
which uses specialized prompts and examples for TEACh. HELPER and our HELPER-X
model each use 7 domain-specific examples in their memory.

Evaluation Metrics We follow the ALFRED evaluation metrics: 1\. Task success
rate (SR) and 2\. Goal condition success rate (GC). These are defined the same
as in TEACh (Section 3.1).

Results are reported in Table 2. Our conclusions are similar to Section 3.1.
On validation unseen, HELPER-XS and HELPER-XP demonstrate performance on-par
with HELPER, with HELPER-XP marginally outperforming HELPER by 1.0%, despite.
HELPER-X is also competitive with the best supervised baselines, despite only
requiring a few in-domain demonstrations. On validation seen, we observe both
HELPER-X models marginally outperforming HELPER. We additionally show that
using HELPER-TEACh which has prompts and examples for a different domain
(TEACh) causes a significant 6.9% drop in performance.

Table 2: Evaluation on the ALFRED validation unseen set. Trajectory length weighted metrics are included in ( parentheses ). FS = few shot. Sup. = supervised. G = generalist; shared across benchmarks.  |  | Unseen | Seen
---|---|---|---
|  | Success | GC | Success | GC
Sup. | E.T. (Pashevich, 2021) | 7.3 (3.3) | 20.9 (11.3) | 46.6 (32.3) | 52.9 (42.2)
HiTUT (Zhang, 2021) | 12.4 (6.9) | 23.7 (12.0) | 25.2 (12.2) | 34.9 (18.5)
M-TRACK(Song, 2022a) | 17.29 | 28.98 | 26.70 | 33.21
HLSM (Blukis, 2022) | 18.3 | 31.2 | 29.6 | 38.7
FILM (Min, 2021) | 20.1 | 32.5 | 24.6 | 37.2
MCR-Agent (Bhambri, 2023) | 20.1 (10.8) | – | 34.4 (23.0) | –
LEBP (Liu, 2022a) | 22.36 | 29.58 | 27.63 | 35.76
LGS-RPA (Murray, 2022) | 33.18 | 44.68 | 43.86 | 52.51
EPA (Liu, 2023) | 40.11 | 44.14 | 45.78 | 51.03
Prompter (Inoue, 2022) | 53.3 (19.6) | 63.0 (21.7) | – | –
FS | HLSM (Blukis, 2022) (FS) | 0.00 | 1.86 | 0.1 | 2.8
FILM (Min, 2021) (FS) | 0.00 | 9.65 | 0.0 | 0.0
SayCan (Brohan, 2023) | 9.9 | 22.5 | 12.3 | 24.5
LLM-Planner (Song, 2023) | 15.4 | 23.4 | 16.5 | 30.1
HELPER-TEACh | 27.5 (5.9) | 44.3 (10.1) | 24.5 (6.3) | 38.2 (11.2)
HELPER (Sarch, 2023) | 34.4 (7.6) | 51.5 (11.9) | 27.6 (7.4) | 42.0 (12.7)
G+FS | HELPER-XP | 35.4 (7.9) | 52.9 (12.3) | 28.2 (7.5) | 42.5 (12.9)
HELPER-XS | 34.0 (7.5) | 51.1 (11.9) | 28.0 (7.4) | 42.1 (12.8)

###  3.3 Instruction Following with Asking Questions

Question asking instruction following allows the agent to choose to ask
questions to an oracle to gain additional information to help it complete a
task defined by an initial natural language instruction.

##### Dataset

The DialFRED benchmark (Gao, 2022) enables an agent to query users while
executing language instructions, utilizing user responses for task
improvement. It features a human-annotated dataset with 53K relevant questions
and answers, plus an oracle for responding to agent queries. Agents can ask
questions in three categories—Location, Appearance, and Direction—pertaining
to their next interaction object. The dataset covers 25 task types across 207
environments, 115 object types, and includes ’seen’ and ’unseen’ episodes. The
agent receives egocentric RGB images at each step and selects actions from a
set, like pickup(X), turn_left(), etc. This benchmark’s instructions and tasks
are distinct from TEACh and partially overlap with ALFRED, with significant
modifications and 18 new task types.

Questioning Implementation To ask questions in the DialFRED task, we add to
the question asking API functions to query the oracle in DialFRED (see Section
2.2.3). HELPER-X asks questions when it does not know the location of an
object required for the task at hand. Unlike ALFRED, success in DialFRED
requires interacting with a specific instance of an object class. To account
for this, HELPER-X also asks questions to help disambiguate when it has seen
multiple instances of the same object.

Baselines We compare with the baselines in the DialFRED paper (Gao, 2022),
which includes a sequence-to-sequence architecture for choosing to ask a
question, trained with reinforcement learning, and the Episodic Transformer
architecture (Pashevich, 2021), trained with behavioral cloning. We adapt
HELPER (Sarch, 2023) as a baseline with specialized prompts and examples for
DialFRED, as well as our question asking API. We consider a few shot setting
with each few shot model receiving 7 domain-specific examples.

Evaluation Metrics We follow the conventions of the DialFRED benchmark. We use
the Task success rate (SR) metric. This is defined the same as TEACh (Section
3.1).

Results are reported in Table 4. On validation unseen, we observe HELPER-XS
marginally outperforming HELPER by 0.38 points in success rate, despite
HELPER-X being shared across all domains. While HELPER-X is outperformed by
the best supervised baselines, HELPER-X only requires a few in-domain
demonstrations compared to the thousands of language-action demonstrations and
RL interactions needed to train the baseline models. Most importantly, we see
the addition of question-asking in HELPER-X improves success rate by 2.48
points, highlighting its efficiency in question selection and response
utilization.

Table 3: Evaluation on the DialFRED validation unseen set. FS = few shot. Sup. = supervised. G = generalist; shared across benchmarks. |  | Success |
---|---|---|---
Sup. | Instructions Only (Gao, 2022) | 18.3 |
All QAs (Gao, 2022) | 32.0 |
RL Anytime (Gao, 2022) | 33.6 |
FS | HELPER (Sarch, 2023) | 19.62 |
G+FS | HELPER-XP | 18.96 |
without QA | 16.48 |
HELPER-XS | 19.99 |

Table 4: Evaluation on the Tidy Task test set. Trajectory length weighted metrics are included in ( parentheses ). FS = few shot. S = supervised. G = generalist; shared across benchmarks. CM = Correctly Moved. IM = Incorrectly Moved.  |  | CM $\uparrow$ | IM $\downarrow$ | Energy% $\downarrow$ | Steps
---|---|---|---|---|---
S | TIDEE (Sarch, 2022) | 2.7 | 0.3 | 64.9 | 437.6
FS | Random Receptacle | 2.0 | 0.3 | 95.5 | 329.3
HELPER (Sarch, 2023) | 2.1 | 0.2 | 83.9 | 348.3
G+FS | HELPER-XP | 2.1 | 0.3 | 86.9 | 368.2
HELPER-XS | 2.2 | 0.2 | 83.4 | 333.9

###  3.4 Tidying Up using Spatial Commonsense Reasoning

Tidying up involves figuring out where to place items without explicit
instructions, relying on spatial commonsense to infer a proper location for an
object. This task tests the agent’s ability to use commonsense reasoning
regarding contextual, object-object, and object-room spatial relations.

##### Dataset

We evaluate on the Tidy Task (Sarch, 2022) benchmark, where the agent is
spawned in a disorganized room, and must reposition objects to bring them to
an organized tidy state. The dataset consists of 8000 training, 200
validation, and 100 testing messy configurations in 120 distinct scenes of
bedrooms, living rooms, kitchens and bathrooms. At each time step, the agent
obtains an egocentric RGB and depth image and must choose an action from a
specified set to transition to the next step, such as pickup(X), turn_left(),
etc. In this setup, the models are prompted to tidy up the room, given a set
of objects that are out of place obtained using the visual detector from TIDEE
(Sarch, 2022).

Baselines We compare against TIDEE (Sarch, 2022), which includes a graph
neural network encoding common object arrangements. This is supervised in the
training set of the Tidy Task to predict where a target object should be re-
positioned to in the current scene. We adapt HELPER (Sarch, 2023) as a
baseline with specialized prompts and examples for the Tidy Task. We
additionally include a random receptacle baseline which chooses random
receptacle placement locations for the out of place objects. We consider a few
shot setting with each few shot model receiving 3 domain-specific examples.

Evaluation Metrics We use the following evaluation metrics for the Tidy Task:
Correctly Moved (CM) Average number of correctly moved objects that are out of
place in the scene, and moved by the agent. Higher is better. Incorrectly
Moved (IM) Average number of incorrectly moved objects that are not out of
place, but were moved by the agent. Lower is better. Energy The "cleaniness"
energy, where lower energy represents a higher likelihood of the room object
configuration aligning with the configurations in the organized AI2THOR rooms.
Following ProcThor (Deitke, 2022), for each receptacle object, the probability
that each object type appears on its surface is computed across the AI2THOR
scenes. See the Appendix for more details.

Results are in Table 4. On the Tidy Task, HELPER-XS and HELPER-XP demonstrates
performance on-par with HELPER. HELPER-X does significantly better than if
object locations are randomly placed (Random Receptacle). We find that the
supervised baseline, TIDEE, outperforms HELPER-X, especially in the Energy
metric, revealing that in-domain training on this benchmark is helpful for
learning the common object configurations within the AI2THOR environments.
However, we find that HELPER-X accomplishes the task in significantly fewer
steps compared to TIDEE.

##  4 Conclusion

We introduce HELPER-X, an embodied agent that executes tasks from dialogue or
language instructions, ask questions, and tides up rooms. HELPER-X has two
variants, HELPER-XP and HELPER-XS, enhancing HELPER’s memory capabilities.
HELPER-XP retrieves domain-specific templates and examples for large language
models, while HELPER-XS retrieves only examples for a domain-agnostic prompt
template through a shared memory. Evaluation of HELPER-X in four domains:
TEACh, ALFRED, DialFRED, and the Tidy Task, yields state-of-the-art
performance in the few-shot example setting. Memory and API expansions we
considered maintained or improved performance for the LLM, highlighting the
effectiveness of memory-enhanced LLMs in building versatile, instructable
agents.

##  5 Related Work

###  5.1 Memory-Augmented Prompting of Large Language Models

Recently, external memories have been instrumental in scaling language models
Borgeaud (2021); Khandelwal (2019), overcoming the constraints of limited
context windows in parametric transformers Wu (2022). They also facilitate
knowledge storage in various forms such as entity mentions de Jong (2021),
knowledge graphs Das (2022), and question-answer pairs Chen (2022). Retrieval-
augmented generation (RAG) (Lewis, 2020; Mao, 2021) has been shown to
significantly improve response quality in large language models (LLMs) by
integrating external knowledge sources with the model’s internal
representations. In agent-based domains, memory-augmented prompting has
enhanced task planning in embodied instructional contexts (Song, 2023; Sarch,
2023) and open-world gaming (Wang, 2023b; a; Majumder, 2023). Our model,
HELPER-X, employs memory-augmented prompting across four benchmarks,
demonstrating that memory expansion across related domains can maintain
performance.

###  5.2 Instructable Embodied Agents that Interact with their Environments

Numerous benchmarks assess embodied vision-and-language tasks, with
significant advancements in learning-based embodied AI agents across tasks
like scene rearrangement Gan (2022); Weihs (2021); Batra (2020); Sarch (2022);
Trabucco (2022), object-goal navigation Anderson (2018); Yang (2019); Wortsman
(2019); Chaplot (2020); Gupta (2017); Chang (2020); Gervet (2023); Chang
(2023), point-goal navigation and exploration Anderson (2018); Savva (2019);
Wijmans (2020); Ramakrishnan (2020); Gupta (2017); Chen (2019); Chaplot
(2019); Kumar (2021), embodied question answering Gordon (2018); Das (2018);
Zhu (2023); Datta (2022); Das (2020); Gao (2022), instructional and image
navigation (Ku, 2020; Krantz, 2023), audio-visual navigation (Chen, 2020),
interactive dialogue and natural language instruction following (Yenamandra,
2023; Shridhar, 2020; Padmakumar, 2021; Gao, 2023), and embodied commonsense
reasoning (Kant, 2022; Sarch, 2022; Wu, 2023). Interactive instruction
benchmarks (e.g., ALFRED (Shridhar, 2020) and TEACh (Padmakumar, 2021))
require agents to follow natural language directives and dialogue, identifying
objects in scenes via interaction masks. Variants like DialFRED (Gao, 2022)
allow agent inquiries about objects and locations. Benchmarks such as TIDEE
(Sarch, 2022) and HouseKeep (Kant, 2022) test agents’ ability to tidy rooms
using commonsense, without explicit object placement directives. Unlike most
methods confined to a single domain, our work focuses on creating a multi-
domain agent adept in dialogue-based task planning, natural language
instruction following, asking questions for disambiguation of instructions,
and tidying up scenes. Our method shows competitive performance across the
four domains with a few task-specific demonstrations and without domain-
specific weights, beyond the single image object detector.

Interactive vision-language embodied agent methods train distinct agents for
each language-defined task, using large datasets from expert demonstrations
(Min, 2021; Inoue, 2022; Zhang, 2022; Kim, 2023; Pashevich, 2021). Some
approaches use these demonstrations for end-to-end network training to
directly predict actions from observations (Pashevich, 2021; Gao, 2022; Zhang,
2021). Others employ modular methods, training planners to generate subgoals
handled by specialized perception, manipulation, and obstacle avoidance
modules (Min, 2021; Inoue, 2022; Blukis, 2022; Zheng, 2022; Kim, 2023;
Bhambri, 2023; Liu, 2022a; Murray, 2022; Liu, 2023). However, these methods
often over-specialize to specific datasets and tasks, limited by the training
domain’s language and task structure. In contrast, our method performs
competitively across multiple benchmarks with minimal task-specific
demonstrations and without needing domain-specific networks.

## References

  * gpt (2023) Openai. gpt-4 technical report.  _arXiv preprint arxiv:2303.08774_ , 2023.
  * Alayrac (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Anderson (2018) Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al.  On evaluation of embodied navigation agents.  _arXiv preprint arXiv:1807.06757_ , 2018.
  * Batra (2020) Dhruv Batra, Angel Xuan Chang, S. Chernova, Andrew J. Davison, Jun Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su.  Rearrangement: A challenge for embodied ai.  _ArXiv_ , abs/2011.01975, 2020.
  * Bhambri (2023) Suvaansh Bhambri, Byeonghwi Kim, and Jonghyun Choi.  Multi-level compositional reasoning for interactive instruction following.  _Interaction_ , 3:4, 2023.
  * Bhat (2023) Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller.  Zoedepth: Zero-shot transfer by combining relative and metric depth.  _arXiv preprint arXiv:2302.12288_ , 2023.
  * Blukis (2022) Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi.  A persistent spatial semantic representation for high-level natural language instruction execution.  In _Conference on Robot Learning_ , pp. 706–717. PMLR, 2022.
  * Borgeaud (2021) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre.  Improving language models by retrieving from trillions of tokens.  _CoRR_ , abs/2112.04426, 2021.  URL https://arxiv.org/abs/2112.04426.
  * Brohan (2023) Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al.  Do as i can, not as i say: Grounding language in robotic affordances.  In _Conference on Robot Learning_ , pp. 287–318. PMLR, 2023.
  * Brown (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chang (2020) Matthew Chang, Arjun Gupta, and Saurabh Gupta.  Semantic visual navigation by watching youtube videos.  _Advances in Neural Information Processing Systems_ , 33:4283–4294, 2020.
  * Chang (2023) Matthew Chang, Theophile Gervet, Mukul Khanna, Sriram Yenamandra, Dhruv Shah, So Yeon Min, Kavit Shah, Chris Paxton, Saurabh Gupta, Dhruv Batra, Roozbeh Mottaghi, Jitendra Malik, and Devendra Singh Chaplot.  Goat: Go to any thing, 2023.
  * Chaplot (2019) Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov.  Learning to explore using active neural slam.  In _International Conference on Learning Representations_ , 2019.
  * Chaplot (2020) Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov.  Object goal navigation using goal-oriented semantic exploration.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Chen (2020) Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman.  Soundspaces: Audio-visual navigation in 3d environments.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16_ , pp. 17–36. Springer, 2020.
  * Chen (2019) Tao Chen, Saurabh Gupta, and Abhinav Gupta.  Learning exploration policies for navigation.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/pdf?id=SyMWn05F7.
  * Chen (2022) Wenhu Chen, Pat Verga, Michiel de Jong, John Wieting, and William Cohen.  Augmenting pre-trained language models with qa-memory for open-domain question answering, 2022.  URL https://arxiv.org/abs/2204.04581.
  * Das (2018) Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra.  Embodied question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1–10, 2018.
  * Das (2020) Abhishek Das, Federico Carnevale, Hamza Merzic, Laura Rimell, Rosalia Schneider, Josh Abramson, Alden Hung, Arun Ahuja, Stephen Clark, Gregory Wayne, et al.  Probing emergent semantics in predictive agents via question answering.  In _Proceedings of the 37th International Conference on Machine Learning_ , pp. 2376–2391, 2020.
  * Das (2022) Rajarshi Das, Ameya Godbole, Ankita Naik, Elliot Tower, Robin Jia, Manzil Zaheer, Hannaneh Hajishirzi, and Andrew McCallum.  Knowledge base question answering by case-based reasoning over subgraphs, 2022.  URL https://arxiv.org/abs/2202.10610.
  * Datta (2022) Samyak Datta, Sameer Dharur, Vincent Cartillier, Ruta Desai, Mukul Khanna, Dhruv Batra, and Devi Parikh.  Episodic memory question answering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 19119–19128, 2022.
  * de Jong (2021) Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Fei Sha, and William Cohen.  Mention memory: incorporating textual knowledge into transformers through entity mention attention.  _CoRR_ , abs/2110.06176, 2021.  URL https://arxiv.org/abs/2110.06176.
  * Deitke (2022) Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi.  Procthor: Large-scale embodied ai using procedural generation.  _Advances in Neural Information Processing Systems_ , 35:5982–5994, 2022.
  * Dong (2021) Bin Dong, Fangao Zeng, Tiancai Wang, Xiangyu Zhang, and Yichen Wei.  Solq: Segmenting objects by learning queries.  _Advances in Neural Information Processing Systems_ , 34:21898–21909, 2021.
  * Gan (2022) Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, Daniel L.K. Yamins, James J. DiCarlo, Josh McDermott, Antonio Torralba, and Joshua B. Tenenbaum.  The threedworld transport challenge: A visually guided task-and-motion planning benchmark towards physically realistic embodied ai.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pp. 8847–8854, 2022.  doi: 10.1109/ICRA46639.2022.9812329.
  * Gao (2023) Qiaozi Gao, Govind Thattai, Xiaofeng Gao, Suhaila Shakiah, Shreyas Pansare, Vasu Sharma, Gaurav Sukhatme, Hangjie Shi, Bofei Yang, Desheng Zheng, et al.  Alexa arena: A user-centric interactive platform for embodied ai.  _arXiv preprint arXiv:2303.01586_ , 2023.
  * Gao (2020) Tianyu Gao, Adam Fisch, and Danqi Chen.  Making pre-trained language models better few-shot learners.  _arXiv preprint arXiv:2012.15723_ , 2020.
  * Gao (2022) Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S Sukhatme.  DialFRED: Dialogue-enabled agents for embodied instruction following.  _IEEE Robotics and Automation Letters_ , 7(4):10049–10056, 2022.
  * Gervet (2023) Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot.  Navigating to objects in the real world.  _Science Robotics_ , 8(79):eadf6991, 2023.
  * Gordon (2018) Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi.  Iqa: Visual question answering in interactive environments.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4089–4098, 2018.
  * Greene (2022) Ryan Greene, Ted Sanders, Lilian Weng, and Arvind Neelakantan.  Openai. new and improved embedding model.  2022\.
  * Gupta (2017) Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik.  Cognitive mapping and planning for visual navigation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * Hongjin (2022) SU Hongjin, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al.  Selective annotation makes language models better few-shot learners.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Inoue (2022) Yuki Inoue, Hiroki Ohashi, and Hiroki Ohashi.  Prompter: Utilizing large language model prompting for a data efficient embodied instruction following.  _arXiv preprint arXiv:2211.03267_ , 2022.
  * Kant (2022) Yash Kant, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, Andrew Szot, and Harsh Agrawal.  Housekeep: Tidying virtual households using commonsense reasoning.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIX_ , pp. 355–373. Springer, 2022.
  * Khandelwal (2019) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.  Generalization through memorization: Nearest neighbor language models, 2019.  URL https://arxiv.org/abs/1911.00172.
  * Kim (2023) Byeonghwi Kim, Jinyeon Kim, Yuyeong Kim, Cheolhong Min, and Jonghyun Choi.  Context-aware planning and environment-aware memory for instruction following embodied agents.  In _ICCV_ , 2023.
  * Kolve (2017) Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi.  AI2-THOR: An Interactive 3D Environment for Visual AI.  _arXiv_ , 2017.
  * Krantz (2023) Jacob Krantz, Theophile Gervet, Karmesh Yadav, Austin Wang, Chris Paxton, Roozbeh Mottaghi, Dhruv Batra, Jitendra Malik, Stefan Lee, and Devendra Singh Chaplot.  Navigating to objects specified by images.  _arXiv preprint arXiv:2304.01192_ , 2023.
  * Ku (2020) Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge.  Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 4392–4412, 2020.
  * Kumar (2021) Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik.  Rma: Rapid motor adaptation for legged robots.  2021\.
  * Lewis (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474, 2020.
  * Lin (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pp. 740–755. Springer, 2014.
  * Liu (2022a) Haoyu Liu, Yang Liu, Hongkai He, and Hangfang Yang.  Lebp–language expectation & binding policy: A two-stream framework for embodied vision-and-language interaction task learning agents.  _arXiv preprint arXiv:2203.04637_ , 2022a.
  * Liu (2022b) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen.  What makes good in-context examples for gpt-3?  _DeeLIO 2022_ , pp. 100, 2022b.
  * Liu (2022c) Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen.  What makes good in-context examples for gpt-3?  In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pp. 100–114, 2022c.
  * Liu (2023) Xiaotian Liu, Hector Palacios, and Christian Muise.  Egocentric planning for scalable embodied task achievement.  _arXiv preprint arXiv:2306.01295_ , 2023.
  * Majumder (2023) Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Burch Callison-Burch, and Peter Clark.  Clin: A continually learning language agent for rapid task adaptation and generalization.  _arXiv_ , 2023.
  * Mao (2021) Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen.  Generation-augmented retrieval for open-domain question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 4089–4100, 2021.
  * Min (2021) So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov.  Film: Following instructions in language with modular methods, 2021.
  * Min (2022) So Yeon Min, Hao Zhu, Ruslan Salakhutdinov, and Yonatan Bisk.  Don’t copy the teacher: Data and model challenges in embodied dialogue.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 9361–9368, 2022.
  * Mishra (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 3470–3487, 2022.
  * Mu (2023) Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo.  Embodiedgpt: Vision-language pre-training via embodied chain of thought.  _arXiv preprint arXiv:2305.15021_ , 2023.
  * Murray (2022) Michael Murray, Maya Cakmak, and Maya Cakmak.  Following natural language instructions for household tasks with landmark guided search and reinforced pose adjustment.  _IEEE Robotics and Automation Letters_ , 7(3):6870–6877, 2022.
  * Padmakumar (2021) Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur.  Teach: Task-driven embodied agents that chat, 2021.
  * Pashevich (2021) Alexander Pashevich, Cordelia Schmid, and Chen Sun.  Episodic transformer for vision-and-language navigation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 15942–15952, 2021.
  * Perez (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho.  True few-shot learning with language models.  _Advances in neural information processing systems_ , 34:11054–11070, 2021.
  * Radford (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Ramakrishnan (2020) Santhosh K Ramakrishnan, Ziad Al-Halah, and Kristen Grauman.  Occupancy anticipation for efficient exploration and navigation.  In _European Conference on Computer Vision_ , pp. 400–418. Springer, 2020.
  * Sarch (2022) Gabriel Sarch, Zhaoyuan Fang, Adam W Harley, Paul Schydlo, Michael J Tarr, Saurabh Gupta, and Katerina Fragkiadaki.  Tidee: Tidying up novel rooms using visuo-semantic commonsense priors.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIX_ , pp. 480–496. Springer, 2022.
  * Sarch (2023) Gabriel Sarch, Yue Wu, Michael Tarr, and Katerina Fragkiadaki.  Open-ended instructable embodied agents with memory-augmented large language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , 2023.
  * Savva (2019) Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al.  Habitat: A platform for embodied ai research.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9339–9347, 2019.
  * Schick (2021) Timo Schick, Hinrich Schütze, and Hinrich Schütze.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2339–2352, 2021.
  * Shridhar (2020) Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox.  ALFRED: A benchmark for interpreting grounded instructions for everyday tasks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10740–10749, 2020.
  * Song (2022a) Chan Hee Song, Jihyung Kil, Tai-Yu Pan, Brian M Sadler, Wei-Lun Chao, and Yu Su.  One step at a time: Long-horizon vision-and-language navigation with milestones.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15482–15491, 2022a.
  * Song (2023) Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su.  Llm-planner: Few-shot grounded planning for embodied agents with large language models, 2023.
  * Song (2022b) Haoyu Song, Li Dong, Weinan Zhang, Ting Liu, and Furu Wei.  Clip models are few-shot learners: Empirical studies on vqa and visual entailment.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 6088–6100, 2022b.
  * Trabucco (2022) Brandon Trabucco, Gunnar A Sigurdsson, Robinson Piramuthu, Gaurav S Sukhatme, and Ruslan Salakhutdinov.  A simple approach for visual room rearrangement: 3d mapping and semantic search.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Wang (2023a) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv: Arxiv-2305.16291_ , 2023a.
  * Wang (2023b) Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang.  Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models.  _arXiv preprint arXiv: 2311.05997_ , 2023b.
  * Wei (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_ , 2021.
  * Weihs (2021) Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi.  Visual room rearrangement.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021.
  * Wijmans (2020) Erik Wijmans, Abhishek Kadian, Ari S. Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra.  Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames.  In _ICLR_ , 2020.
  * Wortsman (2019) Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Learning to learn how to learn: Self-adaptive visual navigation using meta-learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6750–6759, 2019.
  * Wu (2023) Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser.  Tidybot: Personalized robot assistance with large language models.  _arXiv preprint arXiv:2305.05658_ , 2023.
  * Wu (2022) Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy.  Memorizing transformers, 2022.  URL https://arxiv.org/abs/2203.08913.
  * Yang (2023) Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Chencheng Jiang, Haoran Tan, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al.  Octopus: Embodied vision-language programmer from environmental feedback.  _arXiv preprint arXiv:2310.08588_ , 2023.
  * Yang (2019) Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi.  Visual semantic navigation using scene priors.  In _Proceedings of (ICLR) International Conference on Learning Representations_ , May 2019.
  * Yenamandra (2023) Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, et al.  Homerobot: Open-vocabulary mobile manipulation.  _arXiv preprint arXiv:2306.11565_ , 2023.
  * Zhang (2021) Yichi Zhang, Joyce Chai, and Joyce Chai.  Hierarchical task learning from language instructions with unified transformers and self-monitoring.  _ACL 2021, Findings._ , 2021.
  * Zhang (2022) Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Yu, Yuwei Bao, and Joyce Chai.  Danli: Deliberative agent for following natural language instructions.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 1280–1298, 2022.
  * Zheng (2022) Kaizhi Zheng, Kaiwen Zhou, Jing Gu, Yue Fan, Jialu Wang, Zonglin Li, Xuehai He, and Xin Eric Wang.  Jarvis: A neuro-symbolic commonsense reasoning framework for conversational embodied agents.  2022\.
  * Zhu (2023) Hao Zhu, Raghav Kapoor, So Yeon Min, Winson Han, Jiatai Li, Kaiwen Geng, Graham Neubig, Yonatan Bisk, Aniruddha Kembhavi, and Luca Weihs.  Excalibur: Encouraging and evaluating embodied exploration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 14931–14942, 2023.

##  Appendix S1 Limitations

Our model has the following limitations:

1\. Task planning from multimodal input: Currently, our LLM receives the
environment’s state only in case of a failure, through VLM feedback.
Integrating the visual state of the environment in a more direct way may
dramatically increase the accuracy of predicted plans. This direction aligns
with recent work (Wang, 2023b; Mu, 2023; Yang, 2023) that uses visual features
as input to language models.

2\. Cost of GPT-4: While GPT-4 is the most accurate Large Language Model, its
high cost necessitates exploring alternatives such as open-source models,
hardware optimization, model compression or distillation of its knowledge to
smaller models, to reduce expenses.

3\. Manual Addition of Domains: Our model supports four domains with shared
examples and prompts, but manual intervention is needed for adding
significantly different domains and tasks. Future developments should focus on
automating the detection and integration of out-of-domain inputs.

##  Appendix S2 Prompts

###  S2.1 Prompt templates for prompt retrieval

In the prompt retrieval experiments, we include four prompt templates to be
retrieved. These templates are shown for TEACh, ALFRED, Dialfred, and the Tidy
Task in Listing S1, Listing S2, Listing S3, Listing S4, respectively.

###  S2.2 In-Context Examples

Samples of the in-context examples are shown for TEACh, ALFRED, Dialfred, and
the Tidy Task in Listing S5, Listing S6, Listing S7, Listing S8, respectively.

##  Appendix S3 Question Asking

###  S3.1 Overview

In the DialFRED benchmark, when HELPER-X is unable to find an object, it is
able to ask one of three question types in order to aid itself. In a real-
world scenario, HELPER-X could take advantage of the LLM’s capability to ask
many types of questions, but the DialFRED benchmark limits us to three:
direction, location, and appearance.

###  S3.2 Question asking pipeline

When HELPER-X does not have an object’s location already in its memory or
multiple instances of an objects exist in the memory, it forms a prompt with
its current context and the API of available questions, as in Listing S9.
Based on the context, HELPER-X then chooses and asks the most appropriate
question. The returned answer and an API of search related actions, alongside
the context and question, are then formed into another prompt, seen in Listing
S10. Finally, this prompt is parsed by HELPER-X into an action script to
search for the object. Examples of this full pipeline for if an object does
not exist in the memory are in Listing S11 and Listing S12.

##  Appendix S4 Pre-conditions

An example of a pre-condition check for a macro-action is provided in Listing
LABEL:precond_example.

##  Appendix S5 Example LLM inputs & Outputs

We provide examples of dialogue input, retrieved examples, and LLM output for
a TEACh sample in Listing LABEL:example1, Listing LABEL:example2, and Listing
LABEL:example3.

##  Appendix S6 Simulation environment

The TEACh dataset builds on the Ai2thor simulation environment (Kolve, 2017).
At each time step the agent may choose from the following actions: Forward(),
Backward(), Turn Left(), Turn Right(), Look Up(), Look Down(), Strafe Left(),
Strafe Right(), Pickup(X), Place(X), Open(X), Close(X), ToggleOn(X),
ToggleOff(X), Slice(X), and Pour(X), where X refers an object specified via a
relative coordinate $(x,y)$ on the egocentric RGB frame. Navigation actions
move the agent in discrete steps. We rotate in the yaw direction by 90
degrees, and rotate in the pitch direction by 30 degrees. The RGB and depth
sensors are at a resolution of 480x480, a field of view of 90 degrees, and lie
at a height of 0.9015 meters. The agent’s coordinates are parameterized by a
single $(x,y,z)$ coordinate triplet with $x$ and $z$ corresponding to movement
in the horizontal plane and $y$ reserved for the vertical direction. The TEACh
benchmark allows a maximum of 1000 steps and 30 API failures per episode.

##  Appendix S7 $Executor$ details

###  S7.1 Semantic mapping and planning

##### Obstacle map

HELPER-X maintains a 2D overhead occupancy map of its environment
$\in\mathbb{R}^{H\times W}$ that it updates at each time step from the input
RGB-D stream. The map is used for exploration and navigation in the
environment.

At every time step $t$, we unproject the input depth maps using intrinsic and
extrinsic information of the camera to obtain a 3D occupancy map registered to
the coordinate frame of the agent, similar to earlier navigation agents
Chaplot (2019). The 2D overhead maps of obstacles and free space are computed
by projecting the 3D occupancy along the height direction at multiple height
levels and summing. For each input RGB image, we run a SOLQ object segmentor
(Dong, 2021) (pretrained on COCO Lin (2014) then finetuned on TEACh rooms) to
localize each of 116 semantic object categories. For failure detection, we use
a simple matching approach from Min (2021) to compare RGB pixel values before
and after taking an action.

##### Object location and state tracking

We maintain an object memory as a list of object detection 3D centroids and
their predicted semantic labels
$\\{[(X,Y,Z)_{i},\ell_{i}\in\\{1...N\\}],i=1..K\\}$, where $K$ is the number
of objects detected thus far. The object centroids are expressed with respect
to the coordinate system of the agent, and, similar to the semantic maps,
updated over time using egomotion. We track previously detected objects by
their 3D centroid $C\in\mathbb{R}^{3}$. We estimate the centroid by taking the
3D point corresponding to the median depth within the segmentation mask and
bring it to a common coordinate frame. We do a simple form of non-maximum
suppression on the object memory, by comparing the euclidean distance of
centroids in the memory to new detected centroids of the same category, and
keep the one with the highest score if they fall within a distance threshold.

For each object in the object memory, we maintain an object state dictionary
with a pre-defined list of attributes. These attributes include: category
label, centroid location, holding, detection score, can use, sliced, toasted,
clean, cooked. For the binary attributes, these are initialized by sending the
object crop, defined by the detector mask, to the VLM model, and checking its
match to each of [f"The {object_category} is {attribute}", f"The
{object_category} is not {attribute}"]. We found that initializing these
attributes with the VLM gave only a marginal difference to initializing them
to default values in the TEACh benchmark, so we do not use it for the TEACh
evaluations. However, we anticipate a general method beyond dataset biases of
TEACh would much benefit from such vision-based attribute classification.

Table S1: Alternative TEACh Execution from Dialog History (EDH) evaluation split. Trajectory length weighted metrics are included in ( parentheses ). SR = success rate. GC = goal condition success rate. Note that Test Seen and Unseen are not the true TEACh test sets, but an alternative split of the validation set used until the true test evaluation is released, as mentioned in the TEACh github README, and also reported by DANLI (Zhang, 2022).  | Validation | Test
---|---|---
| Unseen | Seen | Unseen | Seen
| SR | GC | SR | GC | SR | GC | SR | GC
E.T. | 8.35 (0.86) | 6.34 (3.69) | 8.28 (1.13) | 8.72 (3.82) | 7.38 (0.97) | 6.06 (3.17) | 8.82 (0.29) | 9.46 (3.03)
DANLI |  17.25 (7.16) | 23.88 (19.38) | 16.89 (9.12) | 25.10 (22.56) | 16.71 (7.33) | 23.00 (20.55) |  18.63 (9.41) | 24.77 (21.90)
HELPER |  17.25 (3.22) |  25.24 (8.12) |  19.21 (4.72) |  33.54 (10.95) |  17.55 (2.59) |  26.49 (7.67) | 17.97 (3.44) |  30.81 (8.93)

##### Exploration and path planning

${{HELPER-X}}$ explores the scene using a classical mapping method. We take
the initial position of the agent to be the center coordinate in the map. We
rotate the agent in-place and use the observations to instantiate an initial
map. Second, the agent incrementally completes the maps by randomly sampling
an unexplored, traversible location based on the 2D occupancy map built so
far, and then navigates to the sampled location, accumulating the new
information into the maps at each time step. The number of observations
collected at each point in the 2D occupancy map is thresholded to determine
whether a given map location is explored or not. Unexplored positions are
sampled until the environment has been fully explored, meaning that the number
of unexplored points is fewer than a predefined threshold.

To navigate to a goal location, we compute the geodesic distance to the goal
from all map locations using graph search inoue2022prompter given the top-down
occupancy map and the goal location in the map. We then simulate action
sequences and greedily take the action sequence which results in the largest
reduction in geodesic distance.

###  S7.2 2D-to-3D unprojection

For the $i$-th view, a 2D pixel coordinate $(u,v)$ with depth $z$ is
unprojected and transformed to its coordinate $(X,Y,Z)^{T}$ in the reference
frame:

$(X,Y,Z,1)=\mathbf{G}_{i}^{-1}\left(z\frac{u-c_{x}}{f_{x}},z\frac{v-c_{y}}{f_{y}},z,1\right)^{T}$
(1)

where $(f_{x},f_{y})$ and $(c_{x},c_{y})$ are the focal lengths and center of
the pinhole camera model and $\mathbf{G}_{i}\in SE(3)$ is the camera pose for
view $i$ relative to the reference view. This module unprojects each depth
image $I_{i}\in\mathbb{R}^{H\times W\times 3}$ into a pointcloud in the
reference frame $P_{i}\in\mathbb{R}^{M_{i}\times 3}$ with $M_{i}$ being the
number of pixels with an associated depth value.

##  Appendix S8 Additional details of the Tidy Task

###  S8.1 Metric Definitions in the Tidy Task

The metrics in the original TIDEE paper (Sarch, 2022) require separate human
evaluations on Amazon Mechanical Turk. We define a new set of metrics that
does not require expensive annotations from humans for every evaluation. Below
are detailed descriptions of each of the new metrics:

  1. 1.

Correctly Moved (CM) Average number of correctly moved objects that are out of
place in the scene, and moved by the agent. Higher is better.

  2. 2.

Incorrectly Moved (IM) Average number of incorrectly moved objects that are
not out of place, but were moved by the agent. Lower is better.

  3. 3.

Energy Following ProcThor (Deitke, 2022), for each receptacle object, the
probability that each object type appears on its surface is computed across
the AI2THOR scenes. Here, we compute the total number of times each object
type is on the receptacle type and divide it by the total number of times the
receptacle type appears across the scenes. The energy metric in the Tidy Task
is defined as follows:

$(P_{cleanup}-P_{original})/(P_{dirty}-P_{original})$ (2)

where $P_{cleanup}$, $P_{dirty}$, and $P_{original}$ represent the sum of the
object location probabilities for the cleaned up state of the room, the
dirty/messy state of the room, and the original state of the room with objects
put in-place by human designers, respectively. Lower is better.

  4. 4.

Steps Average number of steps taken by the agent per episode.

###  S8.2 Langauge Instructions for the Tidy Task

Since the Tidy Task does not include natural language instruction annotations,
we formulate the language instruction as the following to give to the HELPER
baseline and HELPER-X: “Tidy up the house. These are the out of place objects:
{detected_out_of_place_objects}. These are the receptacles in the current
scene: {detected_receptacles}”, where {detected_out_of_place_objects} are the
objects classified as out of place, and {detected_receptacles} are any
receptacle detected in the scene by the agent.

To obtain the list of out of place objects, we allow the agents use of the
TIDEE (Sarch, 2022) visual detector to determine whether each object detected
during the mapping phase is out of place. We found that out of place detection
benefits significantly from visual detection in the Tidy Task, and thus we do
not use an LLM for detecting the out of place attribute. Notably, adding the
additional out of place attribute to the objects in the object memory can be
shared across all benchmarks.

Listing S1: Prompt template for TEACh

⬇

You are an adept at translating human dialogues into sequences of actions for
household robots. Given a dialogue between a <Driver> and a <Commander>, you
convert the conversation into a Python program to be executed by a robot.

{API}

Write a script using Python and the InteractionObject class and functions
defined above that could be executed by a household robot.

Here are a few examples of typical inputs and outputs (only for in-context
reference):

{RETRIEVED_EXAMPLES}

Adhere to these stringent guidelines:

1. Use only the classes and functions defined previously. Do not create functions that are not provided above.

2. Make sure that you output a consistent plan. For example, opening of the same object should not occur in successive steps.

3. Make sure the output is consistent with the proper affordances of objects. For example, a couch cannot be opened, so your output should never include the open() function for this object, but a fridge can be opened.

4. The input is dialogue between <Driver> and <Commander>. Interpret the dialogue into robot actions. Do not output any dialogue.

5. Object categories should only be chosen from the following classes: {OBJECT_CLASSES}

6. You can only pick up one object at a time. If the agent is holding an object, the agent should place or put down the object before attempting to pick up a second object.

7. Each object instance should instantiate a different InteractionObject class even if two object instances are the same object category.

Follow the output format provided earlier. Think step by step to carry out the
instruction.

Write a Python script that could be executed by a household robot for the
following:

dialogue: {command}

Python script:

Listing S2: Prompt template for ALFRED

⬇

You are an excellent interpreter of instructions for household tasks. Given a
task overview <High Level Goal> and step to perform <Low Level Goal>, you
break the instructions down into a sequence of robotic actions.

{API}

Write a script using Python and the InteractionObject class and functions
defined above that could be executed by a household robot.

Here are a few examples of typical inputs and outputs (only for in-context
reference):

{RETRIEVED_EXAMPLES}

Adhere to these stringent guidelines:

1. Use only the classes and functions defined previously. Do not create functions that are not provided above.

2. Make sure that you output a consistent plan. For example, opening of the same object should not occur in successive steps.

3. Make sure the output is consistent with the proper affordances of objects. For example, a couch cannot be opened, so your output should never include the open() function for this object, but a fridge can be opened.

4. The input is high level task description and low level subgoals to perform the high level task. Interpret the instructions into robot actions.

5. Object categories should only be chosen from the following classes: {OBJECT_CLASSES}

6. You can only pick up one object at a time. If the agent is holding an object, the agent should place or put down the object before attempting to pick up a second object.

7. Each object instance should instantiate a different InteractionObject class even if two object instances are the same object category.

8. Always focus on solving the high level goal. Low level instructions should only be used to guide and plan better.

9. Before performing each action, check if that action is allowed for a particular receptacle class. A few examples have been given in API documentation.

10. Check if the receptacle needs to be opened before placing the object. If yes, then open the receptacle before placing the object.

Follow the output format provided earlier. Think step by step to carry out the
instruction.

Write a Python script that could be executed by a household robot for the
following:

{command}

Python script:

Listing S3: Prompt template for the Dialfred

⬇

You are an excellent interpreter of instructions for household tasks. Given a
task overview <High Level Goal> and step to perform <Low Level Goal>, you
break the instructions down into a sequence of robotic actions.

{API}

Write a script using Python and the InteractionObject class and functions
defined above that could be executed by a household robot.

Here are a few examples of typical inputs and outputs (only for in-context
reference):

{RETRIEVED_EXAMPLES}

Adhere to these stringent guidelines:

1. Use only the classes and functions defined previously. Do not create functions that are not provided above.

2. Make sure that you output a consistent plan. For example, opening of the same object should not occur in successive steps.

3. Make sure the output is consistent with the proper affordances of objects. For example, a couch cannot be opened, so your output should never include the open() function for this object, but a fridge can be opened.

4. The input is high level task description and low level subgoals to perform the high level task. Interpret the instructions into robot actions.

5. Object categories should only be chosen from the following classes: {OBJECT_CLASSES}

6. You can only pick up one object at a time. If the agent is holding an object, the agent should place or put down the object before attempting to pick up a second object.

7. Each object instance should instantiate a different InteractionObject class even if two object instances are the same object category.

8. Make sure that you are solving both the high level goal and the low level goals. Some instructions may only be present in one or the other, so address everything from both.

9. Before performing each action, check if that action is allowed for a particular receptacle class. A few examples have been given in API documentation.

Follow the output format provided earlier. Think step by step to carry out the
instruction.

Write a Python script that could be executed by a household robot for the
following:

{command}

Python script:

Listing S4: Prompt template for the Tidy Task

⬇

Task: As an AI-driven cleaning robot, you are tasked with employing
commonsense reasoning to identify where to place out of place objects that
aren’t situated appropriately. Given a list of out of place objects, you are
to write a Python program to be executed by a robot that will bring the out of
place objects to a suitable location.

{API}

Write a script using Python and the InteractionObject class and functions
defined above that could be executed by a household robot.

Here are a few examples of typical inputs and outputs (only for in-context
reference):

{RETRIEVED_EXAMPLES}

Adhere to these stringent guidelines:

1. Use only the classes and functions defined previously. Do not create functions that are not provided above.

2. Make sure that you output a consistent plan. For example, opening of the same object should not occur in successive steps.

3. Make sure the output is consistent with the proper affordances of objects. For example, a couch cannot be opened, so your output should never include the open() function for this object, but a fridge can be opened.

4. Object categories should only be chosen from the following classes: {OBJECT_CLASSES}

5. You can only pick up one object at a time. If the agent is holding an object, the agent should place or put down the object before attempting to pick up a second object.

6. Each object instance should instantiate a different InteractionObject class even if two object instances are the same object category.

7. Address each item systematically, one by one.

8. Base your decisions on your ingrained knowledge about the typical placement of day-to-day objects.

Follow the output format provided earlier. Think step by step to carry out the
instruction.

Write a Python script that could be executed by a household robot for the
following:

input: {command}

Python script:

Listing S5: Sample in-context example for TEACh

⬇

Dialogue input:

<Driver> what shall I do today? <Commander> clean all the bowls. <Driver>
where are the bowls? <Commander> start with the one by the stove. <Commander>
left. <Commander> rinse it with water. <Commander> great. <Driver> what next?
<Commander> the next one is in the fridge. <Commander> you need to rinse it
with water also. <Commander> great job. we are finished.

Python script:

target_bowl1 = InteractionObject("Bowl", landmark = "Stove", attributes =
["clean"])

target_bowl1.go_to()

target_bowl1.pickup()

target_bowl1.clean()

target_bowl1.put_down()

target_bowl2 = InteractionObject("Bowl", landmark = "Fridge", attributes =
["clean"])

target_bowl2.go_to()

target_bowl2.pickup()

target_bowl2.clean()

target_bowl2.put_down()

Listing S6: Sample in-context example for ALFRED

⬇

High Level Goal: To heat an apple and place in the black bin.

Low Level Goal:

(1) Turn around and walk to the kitchen island.

(2) Pick up the apple in front of the gold colored plate.

(3) Walk around the kitchen island and to the stove on the right, look above
the stove to

face the microwave.

(4) Place the apple inside the microwave, heat up/cook the apple, take the
apple out of the

microwave.

(5) Turn left, turn left at the fridge, turn left to face the kitchen island,
and look down

at the black bin.

(6) Place the apple in the bin on the right side.

Python script:

target_apple = InteractionObject("Apple", landmark = "CounterTop")

target_apple.go_to()

target_apple.pickup()

target_microwave = InteractionObject("Microwave")

target_microwave.go_to()

target_microwave.open() # open microwave before placing

target_apple.place(target_microwave)

target_microwave.close() # close microwave before toggle on

target_microwave.toggle_on() # toggle on to heat up & cook

target_microwave.toggle_off() # Important! toggle off to end heating. Do not
try to open microwave without toggle off!

target_microwave.open() # open microwave before picking

target_apple.pickup()

target_trashcan = InteractionObject("GarbageCan")

target_trashcan.go_to()

target_apple.place(target_trashcan)

Listing S7: Sample in-context example for Dialfred

⬇

High Level Goal: Put the watch in the bowl on the nightstand.

Low Level Goal:

(1) Turn around, walk around the bookshelf, turn to the desk.

(2) Pick the watch up off of the desk.

(3) Put the watch in the bowl on the desk.

(4) Pick up the bowl off of the desk.

(5) Walk back around the bookshelf, walk over between the two beds.

(6) Put the bowl on the nightstand.

Python script:

target_watch = InteractionObject("Watch", landmark = "Desk")

target_watch.go_to()

target_watch.pickup()

target_bowl = InteractionObject("Bowl", landmark = "Desk")

target_bowl.go_to()

target_watch.place(target_bowl)

target_bowl.go_to()

target_bowl.pickup()

target_nightstand = InteractionObject("SideTable", landmark = "Bed")

target_nightstand.go_to()

target_bowl.place(target_nightstand)

Listing S8: Sample in-context example for the Tidy Task

⬇

input: Tidy up the house. These are the out of place objects: Potato, Knife.
These are

the receptacles in the current scene: DiningTable, Microwave, CoffeeMachine,
CounterTop.

Python script:

# initialize the out of place objects

target_potato = InteractionObject("Potato")

target_knife = InteractionObject("Knife")

# initialize the placement objects to place the out of place object on

target_countertop = InteractionObject("CounterTop") # The best, commonsense
location for both the potato and knife is on the countertop.

# re-position potato to the countertop to tidy it up

target_potato.go_to()

target_potato.pickup()

target_countertop.go_to()

target_potato.place(target_countertop)

# re-position knife to the countertop to tidy it up

target_knife.go_to()

target_knife.pickup()

target_countertop.go_to()

target_knife.place(target_countertop)

Listing S9: Prompt template for Question Selection

⬇

You are an excellent interpreter of human instructions for household tasks.
Given a list of questions you can ask and information about the current
environment and context, you provide a question that should be asked in order
to give the agent useful information.

{API}

Write a script using Python using the class and functions defined above that
could be executed by a household robot.

Adhere to these stringent guidelines:

1. Use only the classes and functions defined previously. Do not create functions that are not provided above.

2. Make sure you choose the question that provides the most information and is most relevant for the situation at hand.

3. Object categories should only be chosen from the following classes: {OBJECT_CLASSES}

Follow the output format provided earlier. Think step by step to carry out the
instruction.

Write a Python script that asks questions to help a household robot in the
following situation:

{context}

Python script:

Listing S10: Prompt template for Answer Parsing

⬇

You are an excellent interpreter of human instructions for household tasks.
Given the current context of the agent, a question that was asked, and an
answer that was given, you must write code for actions the agent should take
based on the answer provided.

{API}

Write a script using Python using the class and functions defined above that
could be executed by a household robot.

Adhere to these stringent guidelines:

1. Use only the classes and functions defined previously. Do not create functions that are not provided above.

2. Make sure you plan the most simple and direct interpretation of the answer given.

3. Prioritize the most specific information given. For example, an actual object name should be deemed more important than a region.

4. If multiple pieces of information are given, ensure you incorporate all of them into the script.

5. Object categories should only be chosen from the following classes: {OBJECT_CLASSES}

Write a Python script that asks questions to help a household robot in the
following situation:

{context}

{question}

{answer}

Python script:

Listing S11: Sample in-context example 1 for Question Asking

⬇

Question Selection Input:

Context: The agent does not know where the ButterKnife is.

Questioning Script:

askForLocation(’ButterKnife’)

Answer Parsing Input:

Context: The agent does not know where the ButterKnife is.

Question Asked: askForLocation(’ButterKnife’)

Answer Returned: The ButterKnife is to your left on the countertop.

Parsed Answer:

# Turn to the left as per the instruction

turn(’left’)

# Search for the butterknife on the counter top

search_near_other_object(’ButterKnife’, ’CounterTop’)

Listing S12: Sample in-context example 2 for Question Asking

⬇

Question Selection Input:

Context: The agent does not know where the SoapBar is.

Questioning Script:

askForLocation(’SoapBar’)

Answer Parsing Input:

Context: The agent does not know where the SoapBar is.

Question Asked: askForLocation(’SoapBar’)

Answer Returned: The SoapBar is to your front right in the garbage can.

Parsed Answer:

# Turn right as the SoapBar is to the front right

turn(’right’)

# Move forward to reach the garbage can

move(’forward’)

# Search for the SoapBar near the garbage can

search_near_other_object(’SoapBar’, ’GarbageCan’)