#  Language Models can Infer Action Semantics for Classical Planners from
Environment Feedback

Wang Zhu Ishika Singh Robin Jia Jesse Thomason
Department of Computer Science
University of Southern California
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Classical planning approaches guarantee finding a set of actions that can
achieve a given goal state when possible, but require an expert to specify
logical action semantics that govern the dynamics of the environment.
Researchers have shown that Large Language Models (LLMs) can be used to
directly infer planning steps based on commonsense knowledge and minimal
domain information alone, but such plans often fail on execution. We bring
together the strengths of classical planning and LLM commonsense inference to
perform domain induction, learning and validating action pre- and post-
conditions based on closed-loop interactions with the environment itself. We
propose PSALM, which leverages LLM inference to heuristically complete partial
plans emitted by a classical planner given partial domain knowledge, as well
as to infer the semantic rules of the domain in a logical language based on
environment feedback after execution. Our analysis on 7 environments shows
that with just one expert-curated example plans, using LLMs as heuristic
planners and rule predictors achieves lower environment execution steps and
environment resets than random exploration while simultaneously recovering the
underlying ground truth action semantics of the domain.

##  1 Introduction

Classical planning requires extensive domain knowledge to produce a sequence
of actions that achieve a specified goal. The domain contains expert-annotated
action semantics that govern the dynamics of the environment. For example,
traditional symbolic methods, like Planning Domain Description Language (PDDL)
solvers, have action semantics annotated in a domain file. Classical planning
algorithms systematically explore the state space based on actions that can be
executed in any given state as per the domain definition. Therefore, the
resultant plan is guaranteed to succeed, if the specified goal is achievable.
However, it is tedious to exhaustively define the domain to enable classical
planning in an environment, and requires a human expert for every new
environment.

We propose a novel domain induction task setting in PDDL, in which an agent
must automatically infer the action semantics of an environment without manual
annotation or error correction. In this setting, the domain information, such
as object properties and action functions headers, are given to an agent. The
agent is then asked to infer the action semantics through interacting with the
environment and learning from resulting feedback. Our proposed setting is
motivated by the longer-term goal of building real-world robots that can
explore a new environment (_e.g_., a kitchen) and learn to perform new tasks
in the environment (_e.g_., cleaning) without human specification of the
environment.

To solve this domain induction problem, we draw inspiration from prior work
that prompts Large Language Models (LLMs) to perform robotic planning tasks.
These methods do not require explicit domain specification, as they assume
implicit encoding of world knowledge in LLMs for generating unseen plans. For
instance, ProgPrompt Singh et al. (2023) generates program-like high-level
plans from LLM. However, LLM generated plans suffer from two major issues: the
plan is not guaranteed to execute successfully, and the environment-specific
prompt makes it hard to transfer similar performance to new environments.
Hence, instead of using the LLM directly as a planner, we use the LLM in
conjunction with a symbolic solver to iteratively explore the environment and
predict the action semantics based on environment feedback.

We propose Predicting Semantics of Actions with Language Models (PSALM), a
novel method that combines language models with symbolic solvers to induce
domain world models. At a high level, our method maintains a probabilistic
memory of learned action semantics, which it iteratively improves by
interacting with the environment. We use LLMs to sample (possibly incorrect)
candidate trajectories and infer action semantics based on the result of
executing those trajectories in the environment. This leverages LLMs’ strong
commonsense reasoning abilities, as well as their ability to generate
syntactically valid formal semantics. Meanwhile, we use a symbolic solver to
search for ways to achieve the final goal state based on our current belief
about the action semantics; this compensates for the LLMs’ comparatively poor
ability to efficiently search through a large state space.

Across 7 diverse environments, we demonstrate the effectiveness and efficiency
of leveraging LLMs for domain induction. PSALM consistently induces correct
domain files, and does so with substantially fewer total execution steps and
numbers of resets than multiple baseline approaches. Overall, this integration
of LLMs and symbolic solvers presents a promising avenue for advancing domain
induction capabilities in robotics.

##  2 Related Works

###  2.1 Classical Planning

Classical task planning algorithms have been widely applied in autonomous
spacecrafts, military logistics, manufacturing, games, and robotics. The
automated STRIPS planner was the first algorithm that operated the Shakey
robot Fikes and Nilsson (1971) in 1970. Classical planners classically require
finite, deterministic, and full state information to generate guaranteed plans
when a path from the initial to the goal state is possible. Some other
frameworks were also shown to be useful for robot planning Carbonell et al.
(1991); Nau et al. (2003). Planning domain description language (PDDL) Jiang
et al. (2019); Fox and Long (2003) and answer set programming (ASP) Brewka et
al. (2011); Lifschitz (2002) are commonly used specification languages for
classical planning domains.

###  2.2 Application of LLMs for Task Planning

Most works assume access to training data knowledge contained in the LLM as a
source of open-domain world understanding, and apply LLMs directly for task
planning. While this method bypasses explicit domain definition as required in
classical planning for plan generation, the plan is not guaranteed to succeed.
Several works have shown that LLMs can act as planners Huang et al. (2022);
Ichter et al. (2022), but such stochastic, generative approaches lose the
success guarantees of classical planners. API-based programmatic plan
generation Singh et al. (2023); Liang et al. (2023), which introduces some
symbolic structure and constraints, has been shown to perform better but still
does not ensure success. Program synthesis for planning has been previously
proposed in LEAPS Trivedi et al. (2021), which generates programs using a
learned latent space. In recent developments Silver et al. (2023); Liu et al.
(2023), PDDL has been used as a translation medium, with LLMs used to generate
either a PDDL plan or goal. Subsequently, a classical planner is used to plan
for the PDDL goal. Generating a PDDL goal eliminates the need to track
execution state, required when generating a plan using LLMs instead. However,
using a classical planner necessitates specification of the full domain. In
this paper, we propose a new problem of domain induction guided by LLMs. We
use LLMs to guide preliminary environment interactions and planning, which
exposes the domain functioning based on execution feedback.

###  2.3 Domain Induction and Memory Augmentation

Knowledge acquisition for task planning through dialog and web access has been
studied in the past. Some works Perera et al. (2015); Amiri et al. (2019)
construct a knowledge base in an open domain to refer to for grounding user
utterance. Recent works have shown LLM based memory augmentation to be
effective. Sarch et al. (2023) builds a memory of language instruction and
corresponding plan to retrieve from for prompting the LLM with a new
instruction, where the retrieved interaction might inform planning. Oswald et
al. (2024) utilizes LLM to generate PDDL actions given detailed text action
descriptions. Closer to our paper, Smirnov et al. (2024) attempts to generate
syntactically correct but not semantically guaranteed PDDL domain using LLMs,
by exhaustively listing syntax errors in the prompt to correct the generated
domains. Han et al. (2024) leverages language feedback from human during
embodied interaction for domain predicates and action semantics prediction.

In this work, we propose a simple and general approach for retrieving domain
via LLM guided interactions with the environment. Our method, PSALM, aims to
accurately recover the correct domain file without requiring human feedback.

Figure 1: An example from the Blocksworld domain: PDDL domain file (left),
PDDL problem file (mid) and PDDL plan (right).

##  3 Problem Formulation and Background

We define the classical planning problem, the planing language PDDL we use,
and the novel domain induction problem in PDDL we study.

###  3.1 Classical planning problem

The basic framework for classical planning is the goal-directed deterministic
planning problem. A classical planning problem $\mathrm{P}$ is formally
defined as a tuple
$\langle\mathcal{D},\mathcal{O},s^{i},\mathcal{S}^{g}\rangle$, where
$\mathcal{D}$ represents the domain (environment), $\mathcal{O}$ represents a
set of objects in the domain, $s^{i}$ denotes the initial state, and
$\mathcal{S}^{g}$ denotes the goal specification, _i.e_., a set of goal states
satisfying the goal conditions. A plan solution to $\mathrm{P}$ is a $T$-step
sequence of actions $a_{1..T}$ (see Fig. 1 right for an example), which once
executed at $s^{i}$ would lead to a state in $\mathcal{S}^{g}$.

###  3.2 PDDL

Planning Definition and Domain Language (PDDL, Aeronautiques et al., 1998), is
a standard encoding language to represent the planning problems. Fig. 1 lists
an example of the PDDL files from the classical Blocksworld domain. The PDDL
domain file (Fig 1 left) defines $\mathcal{D}$, which contains the predicate
definition representing objects properties in the domain, and the symbolic
action set $\mathcal{A}$ in the environment. Each action has an action name
(_e.g_., Put-down), a list of parameters (_e.g_., ?ob), and action semantics.
The action semantics $\Phi_{a}$ of an action $a\in\mathcal{A}$ includes the
preconditions that describe when an action is valid to execute, and the
postconditions (effects) that describe what happens when an action is
executed. Fig. 2 depicts $\Phi_{a}$ as two lists of statements. The PDDL
problem file (Fig 1 mid) defines the objects $\mathcal{O}$, the initial state
$s^{i}$, and the goal states $\mathcal{S}_{g}$.

###  3.3 Domain induction in PDDL

When performing classical planning in a new environment in PDDL, we have to
define the domain file for the new environment. Human experts must carefully
annotate the action semantics to enable the symbolic solver to find solutions.
We propose a novel domain induction task, where agents must automatically find
the action semantics for a new domain without human annotation or correction.
In the domain induction task, the agent knows
$\mathcal{D}\setminus\bigcup_{a\in\mathcal{A}}{\Phi_{a}}$, i.e., everything
about the domain except the action semantics. In addition, the agent has
access to one problem file $\langle\mathcal{O},s^{i},\mathcal{S}^{g}\rangle$,
The goal of the agent is to learn the correct action semantics $\Phi_{a}$ for
each action $a$. During the learning process, the agent interacts with the
environment, but only starting at $s^{i}$, and can reset to $s^{i}$ as many
times as it wants. Once the action semantics is learned, the ground truth
domain file can be constructed and used for by the symbolic solver for
efficient and robust task solving in the domain.

We evaluate the domain induction learning process on three measures. (1) The
accuracy (Acc):
$\sum_{a\in\mathcal{A}}|\hat{\Phi}_{a}\cap\Phi_{a}|/\sum_{a\in\mathcal{A}}|\Phi_{a}|$,
where $\hat{\Phi}_{a}$ is the predicted action semantics for action $a$. We
use Acc to measure how close the agent is towards finding a solution of the
given problem. Though a solution not ensures 100% Acc, matching the ground
truth action semantics is guaranteed for a solution from the symbolic solver.
We care about the follow two measures once we find a solution. (2) The number
of resets (NR): how many times we put the robot back to $s_{i}$; (3) The
number of executed steps (NES): how many total actions we take during the
learning, including the failed ones.

Figure 2: The pipeline of PSALM in four steps: (1) sample trajectories from a
trajectory sampler; (2) execute the trajectories in the environment to get
feedbacks (3) predict action semantics for each action with environment
feedback, and update the memory based on the prediction; (4) sample action
semantics from the memory to construct the domain file for the symbolic solver
to check the success.

##  4 Proposed Approach: PSALM

We propose the framework Predicting Semantics of Actions with Language Models
(PSALM), to learn action semantics through LLM and solver’s interactions with
the environment.

###  4.1 Overview

As shown in Fig. 2, given the task we want to solve, we first use an LLM as
the trajectory sampler to sample trajectories, and execute the trajectories in
the environment to get feedbacks. We then use the LLM again together with a
rule-based parser, as the action semantics predictor to predict action
semantics, _i.e_., preconditions and postconditions, for each action, based on
this environment feedback. We update the memory of the learned action
semantics and finally sample the current belief of the action semantics to the
symbolic solver to check the success.

If the symbolic solver finds a solution, we execute the plan in the
environment to check if the plan succeeds. If the plan reaches the goal in the
environment, we finish the loop; otherwise, we will pass the result of the
failed plan to the LLM to predict the action semantics again. If the symbolic
solver does not find a solution, we will provide some partial candidate
trajectories from the symbolic solver to the trajectory sampler, to seed the
sampled trajectory.

We elaborate on three components in the following section: the trajectory
sampler, the action semantics predictor, and the memory of learned action
semantics.

###  4.2 Trajectory sampler

We sample trajectories by prompting an LLM to generate new trajectories. The
LLM trajectory sampler takes in the domain information and problem information
as input in the prompt, as described in the problem setup. Besides, the memory
information from the sampled preconditions and postconditions is directly
converted to text in the prompt. For example, the postconditions (on-table
?ob) (arm-empty) of the action put-down are converted to natural language The
effects are (on-table ?ob), (arm-empty). As the symbolic solver is performing
a tree-based search algorithm given a time-limit of $W$, if the solver finds
any candidate trajectories, i.e., not a complete solution but a partial
solution stopped by the timer or a dead end, we will leverage the
deterministic search from the symbolic solver, and include the $k$ longest
candidate trajectories as input to the trajectory sampler. The trajectory
sampler will be asked to pick one of the candidate trajectories, and generate
a trajectory starting from it. Additionally, we store previous failed
trajectories, which are used to (1) filter out invalid candidate trajectories
to avoid the trajectory sampler stuck at the same failed ones (2) list up to
$g$ past failed trajectories in the prompt for higher quality sampling from
the language model. We sample $l$ trajectories from the trajectory sampler.
When $l>1$, we predict action semantics for each trajectory separately, and
update the memory with all predictions.

#### Prospection.

To avoid unnecessary steps taken in the environment execution, and improve
trajectory quality, we perform trajectory prospection. For a generated
trajectory $\mathbf{\tau}=a_{1:T}$, we check if is aligned with the current
belief of the environment action semantics for $v$ steps. Concretely, assuming
$v\leq T$, if any action in $a_{1:v}$ does not satisfy the preconditions in
the sampled action semantics, starting from that action, we will keep randomly
sampling one action until the sampled action is valid in the current action
semantics, and repeat this until we have $v$ total actions. Otherwise, if
$a_{1:v}$ is all valid, we execute the original $a_{1:T}$ in the environment.

#### Random sampler.

To compare the LLM sampler with a random one, we assess the random sampler’s
performance, which samples $v$ actions per trajectory. The random sampler also
takes the solver information as input, i.e., it starts from the longest
candidate trajectory if available. Random sampler can also take advantage of
prospection.

###  4.3 Action semantics predictor

We combine LLM-based and rule-based action semantics prediction in PSALM’s
action semantics predictor.

#### LLM-based predictor.

We prompt the LLM to generate the action semantics of each action separately.
Apart from the domain, problem, memory information, for a trajectory
$\mathbf{\tau}=a_{1:T}$ failed at step $t+1$, the LLM action semantics
predictor takes in valid actions $a_{1:t}$, state transitions
$s_{j+1}-s_{j},j=1..t,s_{1}=s^{i}$, and error message for action $a_{t+1}$ as
input in the prompt. We assume the error message is provided by the
environment. When an action fails, the error message points out which
precondition is not satisfied. When no action fails but the plan is still
invalid, the error message points out the goal is not reached. We show in the
analysis that the error message is necessary for predicting action
preconditions. For each action, we prompt the LLM once to predict the
preconditions and once for the postconditions.

#### Rule-based predictor.

As the error message is given, we can parse the output from the environment
feedbacks for rule-based action semantics prediction. The rule-based predictor
parses the error message from the environment output to infer one or more
missing preconditions suggested. It also parses the state transition
descriptions to derive guaranteed postconditions. For each loop, we add the
rule-based action semantics to the memory.

###  4.4 Memory of action semantics

We keep a memory of the learned action semantics. For each action’s action
semantics, we store two lists of predicted statements for preconditions and
postconditions. Each statement $\phi$ has a probability
$p(\phi\in\Phi_{a}|a)$, in short $p(\phi|a)$. We use this probability to
sample the current belief of action semantics, and then construct the current
belief of the domain file for the symbolic solver.

The first time a statement $\phi$ is predicted as part of the action semantics
for $a$, the probability will be assigned to $1$. Afterwards, this probability
will be updated following an exponential forgetting rule. Suppose at time step
$t$, the predicted action semantics of $a$ is $\hat{\Phi}_{a,t}$, the memory
updating rule of statement $\phi$ is

$\displaystyle
p_{t+1}(\phi|a)=\begin{cases}\mathbbm{1}[\phi\in\hat{\Phi}_{a,t+1}],&p_{t}(\phi|a)=0\\\
\gamma_{\phi}p_{t}(\phi|a)+(1-\gamma_{\phi})\mathbbm{1}[(\phi)\in\hat{\Phi}_{a,t+1}],&\text{otherwise}\\\
\end{cases}$

where $\gamma_{\phi}$ is the forgetting factor. If a statement $\phi$ has only
been predicted by LLM, $\gamma_{\phi}=0.8$. Once a statement $\phi$ is
predicted by rule-based predictor, $\gamma_{\phi}=1$, meaning the statement
keeps at probability $1$ in the memory and never decays. Notice that all the
statements predicted in the current time step and in the memory
$\phi\in\bigcup_{t^{\prime}\in\\{1..t+1\\}}\hat{\Phi}_{a,t^{\prime}}$ will be
updated following the rule.

Figure 3: We compare PSALM with multiple variations over 7 domains. We report
on NES and the results suggest (1) LLM as a trajectory sampler greatly reduces
the execution steps; (2) LLM and rule-based action semantics predictors have
complementary benefits; (3) Prospection is helpful overall. TS is short for
trajectory sampler and ASP is short for action semantics predictor.

##  5 Experiments

We discuss the experimental setups, the results, and the ablation studies,
which show that language models can efficiently learn action semantics from
environment feedback.

###  5.1 Experimental setups

All experiments use Fast-Downward
111https://github.com/aibasel/downward/tree/release-22.12.0 planner with a
search time limit of $W=30$ seconds. We use VAL 222https://github.com/KCL-
Planning/VAL as simulator for plan validation and extracting state condition
changes, error messages. We set the maximum number of loops to 1k for pure
random baselines and to 100 for any method involving LLMs, considering the
cost. We report the average over 3 runs for all the methods.

For the main results, we use GPT-4
API333https://platform.openai.com/docs/models/gpt-4 as the language model
agent, with temperature 0, one-shot prompting following Liu et al. (2023). We
list the prompts in Appendix A. We use $v=10$ prospection steps, $l=1$ sampled
trajectory, $k=3$ candidate trajectories, and $g=5$ failed trajectories per
loop. We perform ablation studies over the hyperparameters in the analysis.
The total cost of the API call is around $500.

We experiment on 7 symbolic domains from International Planning Competitions;
each defines 20 tasks that vary in number of the environment objects and
optimal plan length. (1) Barman: The robot is a bartender with 2 hands
preparing cocktails for a customer’s order, using the specified ingredients
and appropriate tools; (2) Blocksworld: The robot reorganizes a collection of
block piles arranged on a table, into a specified configuration while adhering
to the simple physics principles; (3) Floortile: A set of robots painting
color patterns on floor tiles, allowed to move around but not to step on
painted tiles; (4) Grippers: A set of robots with 2 grippers each are given a
task to move objects among different rooms; (5) Storage: The robot lifts and
drops crates initially stored in different areas, into a depot, using a given
set of hoists; (6) Termes: The robot constructs complex structures by
transporting and positioning blocks, as well as using them as a means to move
adjacent blocks; (7) Tyreworld: The robot is assigned with changing flat
tires, which involves tasks such as removing flat tires, inflating the intact
tires, tightening nuts, and returning tools to the boot.

###  5.2 Main results

Fig. 3 compares PSALM with multiple baselines over 7 domains. Everything in
the figure gets 100% accuracy, so we focus on NES. We do not put the pure
random baselines in the figure, since none of the pure random baselines
complete the task within 1k resets.

#### LLM as a sampler greatly reduces the execution steps.

Comparing the blue bars and the purple bars (w/o LLM-TS), we show that the LLM
trajectory sampler greatly reduces the execution steps over all domains. The
random sampler, even with prospection, lacks the commonsense reasoning
knowledge for choosing trajectories likely be solutions of the problem.

#### LLM and rule-based action semantics predictions have complementary
benefits.

Comparing the orange bars (w/o rule-ASP) and the green bars (w/o LLM-ASP),
rule-based action semantics predictor is more important in Floortile, Storage
and Tyreworld, while LLM action semantics predictor is more important in the
other four domains. With the commonsense prior knowledge provided by the LLM,
and the exactness of the information from the rule-based parser, combining
both predictions is always a better solution over all domains.

#### Prospection induces redundant actions sometimes, but is necessary
overall.

For certain domains with a small number of actions $|\mathcal{A}|$, such as
Grippers ($|\mathcal{A}|=3$) and Termes ($|\mathcal{A}|=7$), a large number of
prospection steps causes unnecessary exploration, comparing to the red bars
(w/o prospect.). However, prospection is overall helpful, as the red bar shows
around 1.4 times execution steps on average compared to our method.
Especially, in the Barman domain, prospection reduces more than a half of the
execution steps. On the other hand, notice that all the exploration during
prospection time is simulated by not executed, a large number of prospection
steps is not time-consuming during the trajectory sampling time.

Figure 4: Additional analysis for PSALM. (Left) We vary the type of LLM and
show that PSALM works with GPT-3.5 and Mistral-7B on the Termes domain.
(Middle) Using the LLM prior before trajectory sampling (darker bars) enables
the random baselines to work better compared to not having the prior (lighter
bars), though it can adversely affect the full PSALM method. (Right)
Experiments where we remove the error message from input to the LLM action
semantics predictor. Without error messages, PSALM works only on easy domains.
For the experiments that fail to find a solution of the problem, we put the
accuracy number on top of their bars.

###  5.3 Analysis

We first show PSALM works on some weaker or even public LLMs. We then discuss
the language model prior can effectively enable the random baselines to work,
as the starting point of the dynamics model. Finally, we show that the error
message is important for the action semantics prediction, without which the
domain induction is not possible with the current LLMs for some domains.

#### PSALM can work on less powerful and public LLMs.

Fig. 4 left shows PSALM can work on GPT-3.5 and Mistral-7B, though with more
number of resets (NR) and number of execution steps (NES). Prospection reduces
the gap between the different LLMs, perhaps because it offsets differences in
commonsense reasoning ability.

#### LLM prior enables random baselines.

We try to use LLM to predict the action semantics $\hat{\Phi}_{0,a}$ for each
action $a$, before having any sampled trajectory, and then start the iteration
with the memory of the dynamics model initialized from $\hat{\Phi}_{0,a}$.
This prediction is solely based on LLM’s commonsense reasoning capability.
Fig. 4 middle shows on the Termes domain the LLM prior can enable the random
baseline without prospection to work within 1k resets, and greatly reduce NR
and NES for the random baseline with prospection. However, on more complex
domains such as Barman, the LLM prior does not enable the random baselines,
_i.e_., failing to find a solution within 1k resets, with and without
prospection. The finding implies we can perform domain induction on some
easier domains with limited access, or even one call, of language models. On
the other hand, LLM prior is harmful to PSALM, as it inserts noisy predictions
of action semantics which requires additional resets to erase.

#### Error messages are crucial for domain induction.

For real-world use, error messages may be hard to obtain from the environment.
So, we study whether PSALM can succeed without receiving error messages about
failed actions. Note that without the error message, the action semantics
predictor can only be LLM-based, not rule-based. Fig. 4 right shows that on
slightly complex domains like Termes, PSALM cannot predict the correct
dynamics model, _i.e_., the accuracy is not 100. PSALM works on very simple
domains like Grippers, but it requires 7x more NES to learn. Moreover,
prospection is essential when we have no error message.

###  5.4 Ablation studies

We perform ablation studies on prospection steps, number of sampled
trajectories, number of candidate trajectories, and number of failed
trajectories per run, to show their influence on the PSALM framework. The
results validate our choice of the hyperparameters in the main results.

#### Few prospection steps is enough for PSALM.

We vary prospection steps $v$ from $0,1,5,10$ in Fig, 5(a). Notice the random
baseline does not find a solution when $v=0,1,5$, and reaches an Acc of 78.5.
We conclude the number of prospection steps matters more when the trajectory
sampler is random, while a small number of prospection is enough for language
model trajectory sampler.

(a)

(b)

(c)

(d)

Figure 5: Ablation studies for PSALM. (a). Varying the number of prospection
steps $v$. Few prospection steps is enough for PSALM. (b). Varying the number
of sampled trajectories $l$. One sampled trajectory per run is enough. (c).
Varying the number of candidate trajectories $k$ passed from the symbolic
solver to the LLM trajectory sampler. More candidate trajectories help when
prospection is used. (d). Varying the number of failed trajectories $g$ shown
to the LLM trajectory sampler. A certain amount of failed trajectories is
required for the LLM.

#### One sampled trajectory per run is enough.

We vary the number of sampled trajectories $l$ from $1,3,5$ in Fig. 5(b). For
both LLM and random samplers, We see the total number of steps grows linearly
with number of sampled trajectory, which means LLMs are hard to learn more
information from just sampling more each run, and one sampled trajectory per
run is enough.

#### More candidate trajectories help on prospection.

We vary the number of candidate trajectories $k$ from $0,1,3$ in the LLM
prompt. The results in Fig. 5(c) suggest including candidate trajectory in the
prompt is beneficial, while one candidate trajectory is enough for pure LLM
prediction, more candidate trajectories help LLM with prospection. On the
other hand, as the candidate trajectories can be very long for certain domain
like Barman, and thus result in a very long prompt, we do not experiment with
more than 3 candidate trajectories in the prompt.

#### Choose the number of failed trajectories with care.

We vary the number of failed trajectories $g$ from $0,1,5$ in the LLM prompt.
To select $g$ failed trajectories, we first filter trajectories with no less
than 3 steps, and then sample $g$ trajectories from the filtered ones for the
LLM prompt to avoid generating those trajectories as prefix. The results in
Fig. 5(d) suggest that multiple failed trajectories help LLM action semantics
prediction, but only one failed trajectory could be harmful. We hypothesize
the one failed trajectory in the input might distract the model from following
the candidate trajectories.

##  6 Conclusion and Discussions

In conclusion, we propose a novel domain induction problem in PDDL, where
agents automatically infer the action semantics for a new domain without human
annotation. We introduce a simple but strong framework PSALM, which combines
the commonsense reasoning ability of large language models with the precision
of symbolic solvers for domain induction. To update the action semantics in a
memory, PSALM uses LLMs as agents for trajectory sampling and action semantics
prediction. We demonstrate the effectiveness and efficiency of leveraging LLMs
for domain induction over 7 domains. This integration of LLMs and symbolic
solvers presents a promising avenue for advancing domain induction
capabilities in robotics.

##  7 Limitations

Though PSALM shows to be relatively robust across multiple LLMs, we are not
claim that PSALM can work with any LLM. Besides, as discussed in the analysis,
the error message, which might be hard to get from some real-world
environment, is crucial in the PSALM framework, given the current LLM’s
reasoning ability. Future works may explore methods without the error message
as the input. On the other hand, to apply the domain induction setup to the
real world, the environment predicates and object types have to be annotated
or derived from the environment. The domain induction problem without these
annotations can be a harder. We will leave this extension for future studies.

## Acknowledgments and Disclosure of Funding

WZ and RJ were supported in part by a grant from Open Philanthropy. IS and JT
were supported in part by a grant from the Army Research Lab (ARL) Army AI
Innovations Institute (A2I2), award number W911NF-23-2-0010.

## References

  * Aeronautiques et al. [1998] Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins Sri, Anthony Barrett, Dave Christianson, et al.  Pddl| the planning domain definition language.  _Technical Report, Tech. Rep._ , 1998.
  * Amiri et al. [2019] Saeid Amiri, Sujay Bajracharya, Cihangir Goktolgal, Jesse Thomason, and Shiqi Zhang.  Augmenting knowledge through statistical, goal-oriented human-robot dialog.  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019.
  * Brewka et al. [2011] Gerhard Brewka, Thomas Eiter, and Mirosław Truszczyński.  Answer set programming at a glance.  _Commun. ACM_ , 2011.
  * Carbonell et al. [1991] Jaime Carbonell, Oren Etzioni, Yolanda Gil, Robert Joseph, Craig Knoblock, Steve Minton, and Manuela Veloso.  Prodigy: An integrated architecture for planning and learning.  _SIGART Bull._ , 1991.
  * Fikes and Nilsson [1971] Richard E. Fikes and Nils J. Nilsson.  Strips: A new approach to the application of theorem proving to problem solving.  _Artificial Intelligence_ , 1971.
  * Fox and Long [2003] Maria Fox and Derek Long.  Pddl2.1: An extension to pddl for expressing temporal planning domains.  _ArXiv_ , 2003.
  * Han et al. [2024] Muzhi Han, Yifeng Zhu, Song-Chun Zhu, Ying Nian Wu, and Yuke Zhu.  Interpret: Interactive predicate learning from language feedback for generalizable task planning.  In _Robotics: Science and Systems (RSS)_ , 2024.
  * Huang et al. [2022] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , Proceedings of Machine Learning Research, 2022.
  * Ichter et al. [2022] Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu.  Do as i can, not as i say: Grounding language in robotic affordances.  In _6th Annual Conference on Robot Learning_ , 2022.
  * Jiang et al. [2019] Yu-qian Jiang, Shi-qi Zhang, Piyush Khandelwal, and Peter Stone.  Task planning in robotics: an empirical comparison of pddl- and asp-based systems.  _Frontiers of Information Technology & Electronic Engineering_, 2019.
  * Liang et al. [2023] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng.  Code as policies: Language model programs for embodied control.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 2023.
  * Lifschitz [2002] Vladimir Lifschitz.  Answer set programming and plan generation.  _Artificial Intelligence_ , 2002.
  * Liu et al. [2023] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone.  LLM+P: Empowering large language models with optimal planning proficiency.  _ArXiv preprint_ , 2023.
  * Nau et al. [2003] Dana Nau, Tsz-Chiu Au, Okhtay Ilghami, Ugur Kuter, J William Murdock, Dan Wu, and Fusun Yaman.  Shop2: An htn planning system.  _J. Artif. Intell. Res. (JAIR)_ , 2003.
  * Oswald et al. [2024] James Oswald, Kavitha Srinivas, Harsha Kokel, Junkyu Lee, Michael Katz, and Shirin Sohrabi.  Large language models as planning domain generators.  In _Proceedings of the International Conference on Automated Planning and Scheduling_ , volume 34, pages 423–431, 2024.
  * Perera et al. [2015] Vittorio Perera, Robin Soetens, Thomas Kollar, Mehdi Samadi, Yichao Sun, Daniele Nardi, René van de Molengraft, and Manuela Veloso.  Learning task knowledge from dialog and web access.  _Robotics_ , 2015.
  * Sarch et al. [2023] Gabriel Sarch, Yue Wu, Michael Tarr, and Katerina Fragkiadaki.  Open-ended instructable embodied agents with memory-augmented large language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , 2023.
  * Silver et al. [2023] Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Pack Kaelbling, and Michael Katz.  Generalized planning in pddl domains with pretrained large language models.  _ArXiv preprint_ , 2023.
  * Singh et al. [2023] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg.  Progprompt: Generating situated robot task plans using large language models.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 2023.
  * Smirnov et al. [2024] Pavel Smirnov, Frank Joublin, Antonello Ceravola, and Michael Gienger.  Generating consistent pddl domains with large language models.  _ArXiv preprint_ , 2024.
  * Trivedi et al. [2021] Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J. Lim.  Learning to synthesize programs as interpretable and generalizable policies.  In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , 2021.

##  Appendix A LLM Prompts

We list the prompt template for LLM trajectory sampler (Fig. 6) and LLM action
semantics predictor (Fig. 7), and provide one complete example for each. Fig.
8, 9 is the example for trajectory sampler, and Fig. 10, 11 is the example for
action semantics predictor.

Figure 6: LLM trajectory sampler prompt template Figure 7: LLM action
semantics predictor prompt template Figure 8: LLM trajectory sampler prompt
example Figure 9: LLM trajectory sampler prompt example (cont’) Figure 10: LLM
action semantics predictor prompt example Figure 11: LLM action semantics
predictor prompt example (cont’)