ifaamas [AAMAS ’24]Proc. of the 23rd International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2024)May 6 – 10, 2024 Auckland, New
ZealandN. Alechina, V. Dignum, M. Dastani, J.S. Sichman (eds.) 2024 2024 1101
Massachusetts Institute of Technology Harvard University Massachusetts
Institute of Technology Massachusetts Institute of Technology

# Pragmatic Instruction Following and Goal Assistance via Cooperative
Language-Guided Inverse Planning

Tan Zhi-Xuan∗<EMAIL_ADDRESS>,  Lance Ying∗<EMAIL_ADDRESS>,  Vikash
Mansinghka<EMAIL_ADDRESS>and  Joshua B. Tenenbaum<EMAIL_ADDRESS>
###### Abstract.

People often give instructions whose meaning is ambiguous without further
context, expecting that their actions or goals will disambiguate their
intentions. How can we build assistive agents that follow such instructions in
a flexible, context-sensitive manner? This paper introduces _cooperative
language-guided inverse plan search_ (CLIPS), a Bayesian agent architecture
for pragmatic instruction following and goal assistance. Our agent assists a
human by modeling them as a cooperative planner who communicates _joint plans_
to the assistant, then performs _multimodal_ Bayesian inference over the
human’s goal from actions and language, using large language models (LLMs) to
evaluate the likelihood of an instruction given a hypothesized plan. Given
this posterior, our assistant acts to minimize expected goal achievement cost,
enabling it to pragmatically follow ambiguous instructions and provide
effective assistance even when uncertain about the goal. We evaluate these
capabilities in two cooperative planning domains (Doors, Keys & Gems and
VirtualHome), finding that CLIPS significantly outperforms GPT-4V, LLM-based
literal instruction following and unimodal inverse planning in both accuracy
and helpfulness, while closely matching the inferences and assistive judgments
provided by human raters.

###### Key words and phrases:

Inverse Planning; Bayesian Theory-of-Mind; Instruction Following; Human-Robot
Cooperation; Value Alignment

**footnotetext: Equal contribution.

##  1\. Introduction

Humans act upon the world through our words. We make requests, give
instructions, and communicate information, so that we can better coordinate
with each other Austin (1975). In doing so, we are often parsimonious,
exploiting context to convey our intentions Grice (1975). For example, if
someone says “Can you hold that door?”, it is typically obvious which door
they mean, even though nothing in the sentence distinguishes the door they
just walked past from the one they are heading towards. We understand these
requests because we interpret them _pragmatically_ , in light of the goals and
actions of others. How might we build assistive machines that do the same?

In this paper, we introduce _cooperative language-guided inverse plan search_
(CLIPS), a Bayesian architecture for pragmatic instruction following and goal
assistance (Figure 1). Building upon prior work in inverse planning Baker et
al. (2009); Ramírez and Geffner (2010); Ying et al. (2023a), rational speech
act theory Goodman and Frank (2016); Fried et al. (2018); Sumers et al.
(2023), joint intentionality Tang et al. (2020); Stacy et al. (2021); Wu et
al. (2021), assistance games Hadfield-Menell et al. (2016); Fisac et al.
(2017), and reward learning Ramachandran and Amir (2007); Jeon et al. (2020);
Lin et al. (2022), CLIPS models humans as cooperative planners who communicate
_joint plans_ as instructions. Given this model, CLIPS performs _multimodal_
goal inference from human actions and instructions, using a large language
model (LLM) to score the likelihood of observed utterances Ying et al.
(2023b), and computing a distribution over goals via sequential inverse
planning Zhi-Xuan et al. (2020). This distribution then informs an assistive
policy, which selects helpful actions under uncertainty about the human’s
goal.

We evaluate CLIPS on a suite of multi-step goal assistance problems in a
doors-and-keys gridworld Zhi-Xuan et al. (2020) and the VirtualHome domain
Puig et al. (2018, 2020). In these problems, the assistant must infer the
human’s goal from their actions and possibly ambiguous instructions, then
decide how best to help. Even when leveraging LLMs, standard instruction
following methods struggle with this setting because they disregard pragmatic
context Ahn et al. (2022); Wan et al. (2022); Singh et al. (2023), while
action-only goal inference Baker et al. (2009); Zhi-Xuan et al. (2020); Puig
et al. (2020) ignores linguistic information. Multimodal LLMs have access to
all information, but they fail to ground it in a coherent theory-of-mind
Buschoff et al. (2024); Jin et al. (2024). In contrast, CLIPS is able to use
observed actions and inferred goals to resolve _ambiguous language_ ,
interpret _joint instructions_ , and correct for _incomplete commands_ ,
achieving much higher goal accuracy and cooperative efficiency than GPT-4V,
LLM-based literal instruction following, and unimodal inverse planning, while
correlating strongly with goal inference and assistance judgments provided by
human raters.

Figure 1. Overview of _cooperative language-guided inverse plan search_
(CLIPS). We model a human principal as (a) cooperatively planning a joint
policy for the human and the (robot) assistant. The human is (b) assumed to
take actions from this joint policy while communicating planned actions as an
instruction (_“Can you pass me the red key?”_). Observing this, (c) CLIPS
infers the human’s goal and policy via Bayesian inverse planning. CLIPS then
(d) acts by minimizing expected goal achievement cost, pragmatically
interpreting the ambiguous instruction by picking up _Key 2_. In contrast, a
literal instruction follower might pick up _Key 1_ or _Key 4_ , which are also
red in color.

##  2\. Cooperative Language-Guided Inverse Plan Search

We formulate the setting for CLIPS as a _language-augmented goal assistance
game_ , an extension of assistance games Hadfield-Menell et al. (2016); Fisac
et al. (2017) with linguistic utterances and uncertainty over goals (i.e.
desired terminal states) Saisubramanian et al. (2019). We define this as a
two-player Markov game between a human principal and an assistive agent,
described by the tuple
$(\mathsf{S},\mathsf{U},\mathsf{A}^{{}_{h}},\mathsf{A}^{{}_{r}},\mathsf{C},\mathsf{G},H,P_{s},P_{i})$,
where $\mathsf{S}$ is the set of environment states, $\mathsf{U}$ a space of
utterances $u$ that the human may use to communicate at any step,
$\mathsf{A}^{{}_{h}}$ the set of human actions $a^{{}_{h}}$,
$\mathsf{A}^{{}_{r}}$ the set of assistant actions $a^{{}_{r}}$, $\mathsf{C}$
a set of cost functions
$C:\mathsf{S}\times\mathsf{U}\times\mathsf{A}^{{}_{h}}\times\mathsf{A}^{{}_{r}}\to\mathbb{R}$
that map state-action transitions to real numbers,
$\mathsf{G}\subseteq\mathcal{P}(\mathsf{S})$ a set of possible goals $g$ where
each $g\subseteq S$ is a set of (terminal) states, $H$ a horizon after which
the game automatically terminates, $P_{s}(s^{\prime}|s,a^{{}_{r}},a^{{}_{h}})$
the environment transition distribution, and $P_{i}(s_{i},C,g)$ a distribution
over initial states of the game. As in regular assistance games, the human
knows (i.e. observes) the _true_ cost function $C$ and goal $g$ sampled from
$P_{i}$, but the assistant only observes the initial environment $s_{i}$.
Thus, the assistant has to _infer_ the true goal $g$.

Assistance games admit cooperative equilibria, but they are intractable to
compute in general Fisac et al. (2017), and also assume more game-theoretic
optimality from humans than may be warranted Milli and Dragan (2020). As such,
our goal is to _approximately_ solve the assistance game under reasonable
modeling assumptions about how humans cooperate and communicate. By fixing a
human model, the game becomes a partially observable Markov decision process
(POMDP) from the assistant’s perspective Doshi and Gmytrasiewicz (2009), which
can be solved through approximate methods Nguyen et al. (2011); Macindoe et
al. (2012). We describe this human model, then show how an assistant can
perform Bayesian inference over such a model, using the acquired information
to better assist humans.

###  2.1. Modeling cooperative action and communication

When humans cooperate, we direct our actions towards a shared goal while
expecting that others will do the same. This capacity for _joint
intentionality_ Tomasello and Carpenter (2007) allows us to cooperate rapidly
and flexibly while avoiding nested reasoning about each other’s minds Tang et
al. (2020); Wu et al. (2021). In CLIPS, we exploit this aspect of human
cognition by modeling the human principal as a _cooperative planner_ , who
computes a _joint policy_ $\pi$ for both agents to achieve the goal $g$. The
human follows $\pi$ by taking an action $a^{{}_{h}}_{t}$ at each step $t$, and
assumes the assistant will take an action $a^{{}_{r}}_{t}$ accordingly. These
actions lead to a change in the state $s_{t}$. The human may also speak at any
time $t$ (with the decision denoted by $d_{t}$), issuing a command $c_{t}$
that summarizes their intended policy $\pi$. This command is then rendered in
natural language as an utterance $u_{t}$. The overall generative process is
shown in Figures 2a–b, and can be summarized as follows:

Goal Prior: $\displaystyle g$ $\displaystyle\sim P(g)$ (1) Joint Planning:
$\displaystyle\pi$ $\displaystyle\sim P(\pi|g)$ (2) Action Selection:
$\displaystyle a^{{}_{h}}_{t},a^{{}_{r}}_{t}$ $\displaystyle\sim
P(a^{{}_{h}}_{t},a^{{}_{r}}_{t}|s_{t},\pi)$ (3) Utterance Generation:
$\displaystyle u_{t},c_{t},d_{t}$ $\displaystyle\sim
P(u_{t},c_{t},d_{t}|s_{t},\pi)$ (4) State Transition: $\displaystyle s_{t+1}$
$\displaystyle\sim P(s_{t+1}|s_{t},a^{{}_{h}}_{t},a^{{}_{r}}_{t})$ (5)

$P(g)$ is the assistant’s prior over the human’s goal $g\in\mathsf{G}$, which
we take to be uniform. We can also fold in uncertainty over the human’s cost
function $C\in\mathsf{C}$ into this prior, treating $g$ as a _specification_
that includes both the goal condition and action costs.

(a) One graphical realization of our Bayesian model.

model clips-model($s_{1},G,T$)

$g\sim\textsc{goal-prior}(G)$

$\pi\sim\textsc{policy-init}(g,s_{1})$

for $t\in[1,T]$ do

$\pi\leftarrow\textsc{rths-policy-update}(\pi,g,s_{t})$

$a^{{}_{h}}_{t},a^{{}_{r}}_{t}\sim\textsc{boltzmann-dist}(s_{t},\pi)$

$u_{t},c_{t},d_{t}\sim\textsc{utterance-model}(s_{t},\pi)$

$s_{t+1}\sim\textsc{transition}(s_{t},a^{{}_{h}}_{t},a^{{}_{r}}_{t})$

end for

end model

(b) CLIPS agent-environment model as a probabilistic program.

model utterance-model($s_{t},\pi$)

parameters: $p_{\text{speak}},H,K,\mathcal{E}$

$d_{t}\sim\textsc{bernoulli}(p_{\text{speak}})$

if $d_{t}=\textsc{true}$ then

$a_{t:t+H}\leftarrow\textsc{rollout-policy}(s_{t},\pi,H)$

$\alpha_{1:m}\leftarrow\textsc{extract-salient-actions}(a_{t:t+H})$

$c_{t}\sim\textsc{rand-subset}(\alpha_{1:m},K)$

$u_{t}\sim\textsc{language-model}(c_{t},\mathcal{E})$

else

$c_{t},u_{t}\leftarrow\emptyset,\texttt{""}$

end if

end model

(c) Utterance model $P(u_{t},c_{t},d_{t}|\pi)$ as a probabilistic program.

Command: |  (handover you me key1) where (iscolor key1 blue)
---|---
Utterance: |  Hand me the blue key.
Command: |  (unlock you key1 door1)
Utterance: |  Can you unlock the door for me?
Command: |  (pickup me key1) (unlock you key2 door1) where (iscolor key1 blue) (iscolor door1 green)
Utterance: |  I’m getting the blue key, can you open the green door?
Command: |  (handover you me key1) (handover you me key2) where (iscolor key1 green) (iscolor key2 red)
Utterance: |  Can you pass me the green and red keys?

(d) Paired examples $\mathcal{E}$ of commands $c_{t}$ and utterances $u_{t}$.

Figure 2. Model architecture. In CLIPS, we model the human as a cooperative
planner who computes a joint policy $\pi$ for a goal $g\in G$. The policy
$\pi$ dictates the human’s and assistant’s actions
$a^{{}_{h}}_{t},a^{{}_{r}}_{t}$ at each state $s_{t}$, as well as the command
$c_{t}$ and utterance $u_{t}$ that the human may decide $d_{t}$ to communicate
at step $t$. One realization of this process is depicted in (a), showing a
case where an utterance $u_{3}$ is only made at $t=3$. We implement this
process as probabilistic program, shown in (b). Utterance generation is
modeled by the subroutine in (c), which summarizes salient actions from policy
$\pi$ as a command $c_{t}$, then samples an utterance $u_{t}$ using a (large)
language model prompted with $c_{t}$ and (d) a list of few-shot examples
$\mathcal{E}$ demonstrating how commands are translated into natural language.

To model joint planning, we assume that the human computes a Boltzmann policy
$\pi$ for goal $g$, which defines an approximately rational distribution over
joint actions $a^{{}_{h}}_{t},a^{{}_{r}}_{t}$:

$\pi(a^{{}_{h}}_{t},a^{{}_{r}}_{t}|s_{t})=\frac{\exp\left(-\beta\hat{Q}_{\pi}(s_{t},a^{{}_{h}}_{t},a^{{}_{r}}_{t})\right)}{\sum_{\tilde{a}^{{}_{h}}_{t},\tilde{a}^{{}_{r}}_{t}}\exp\left(-\beta\hat{Q}_{\pi}(s_{t},\tilde{a}^{{}_{h}}_{t},\tilde{a}^{{}_{r}}_{t})\right)}$
(6)

Here, $\beta$ is a rationality parameter controlling the optimality of the
policy (higher $\beta$ is more optimal), over which we may place a prior
$P(\beta)$, and $\hat{Q}_{\pi}(s_{t},a^{{}_{h}}_{t},a^{{}_{r}}_{t})$ is an
estimate of the expected cost of reaching the goal $g$ by taking actions
$a^{{}_{h}}_{t},a^{{}_{r}}_{t}$ at state $s_{t}$. To estimate $\hat{Q}_{\pi}$
efficiently, we extend prior work in online goal inference Zhi-Xuan et al.
(2020), using model-based planning to compute policies on-the-fly. In
particular, we use real-time heuristic search (RTHS) as an anytime planner
Korf (1990); Barto et al. (1995); Koenig and Likhachev (2006), which estimates
$Q$-values in a neighborhood around the current state $s_{t}$ via search
(rths-policy-update in Fig. 2(b)), while using previously estimated $Q$-values
to guide future searches. Unlike earlier methods for inverse planning, this
avoids the intractability of estimating $Q$ for all states and actions
Ramachandran and Amir (2007); Ziebart et al. (2008).111As an additional
simplification, we assume that the human and the assistant take turns while
acting (i.e. $a^{{}_{r}}_{t}$ is a no-op when $a^{{}_{h}}_{t}$ is not, and
vice versa). This reduces the branching factor while planning, but preserves
the optimal solution Boutilier (1996).

Having computed the policy $\pi$, we model human communication as an
approximately _pragmatic_ process. Since the human wants to achieve their
goal, they are likely to convey information about the policy they have in
mind. However, long utterances are costly, and so the human is likely to
convey only a salient summary of the policy, reducing the cost of
communication. We capture these aspects of cooperative communication through a
structured generation procedure that leverages the linguistic competence of
LLMs:

  1. (1)

At each step $t$, the human decides whether to communicate ($d_{t}$) with
probability $p_{\text{speak}}$. If $d_{t}$ is true, we rollout the policy
$\pi$ for $H$ steps, producing a series of planned actions $a_{t:t+H}$.

  2. (2)

The planned actions $a_{t:t+L}$ are filtered down to a domain-specific set of
_salient_ actions $\alpha_{1:m}$, and a random subset of up to $K$ such
actions are incorporated into a command $c_{t}$.

  3. (3)

Finally, the command $c_{t}$ is translated into a natural language utterance
$u_{t}$ using an LLM prompted with few-shot examples (Figure 2(d)). This
defines a likelihood function $P(u_{t}|c_{t})$.

This procedure is implemented as the probabilistic program shown in Figure
2(d). While it does not capture all ways in which a pragmatic speaker might
prefer some utterances over others, key features of pragmatic speech are
modeled: Utterances are about plans, and hence practically useful, but are
also restricted to small ($\leq K$) but salient subsets of those plans,
ensuring they are informative without being costly. Furthermore, using an LLM
as an utterance model $P(u_{t}|c_{t})$ provides broad coverage, enabling CLIPS
to handle much richer utterances than traditional pragmatic models Goodman and
Frank (2016).

###  2.2. Goal inference via inverse planning
from actions and instructions

Using this probabilistic model, our assistant can infer a posterior
distribution over the human’s goals $g$ and policies $\pi$ given a sequence of
observed actions $a^{{}_{h}}_{1:T}$, states $s_{1:T}$ utterance decisions
$d_{1:T}$, and utterances $u_{1:T}$ (where $u_{t}$ is the empty string
whenever $d_{t}=\textsc{false}$). Some care must be taken, however. Since the
assistant is also taking actions $a^{{}_{r}}_{1:T}$ alongside the human
principal, it is tempting to infer the human’s goal by conditioning on _both_
agents’ actions:

$\displaystyle P($ $\displaystyle
g,\pi|s_{1:T},d_{1:T},u_{1:T},a^{{}_{h}}_{1:T},a^{{}_{r}}_{1:T})$
$\displaystyle\propto
P(g,\pi,s_{1:T},d_{1:T},u_{1:T},a^{{}_{h}}_{1:T},a^{{}_{r}}_{1:T})$ (7)
$\displaystyle=P(g,\pi)\textstyle{\prod}_{t=1}^{T}P(s_{t}|s_{t-1},a^{{}_{h},_{r}}_{t-1})P(u_{t},d_{t}|s_{t},\pi)\underline{P(a^{{}_{h}}_{t},a^{{}_{r}}_{t}|s_{t},\pi)}$

For an _external_ observer, this is the appropriate distribution to compute
Shum et al. (2019); Ying et al. (2023b). However, prior work on cooperative
inference also computes Equation 7 when observers are _internal_ to the
environment Tang et al. (2020); Wu et al. (2021), which can lead to
pathologies. For example, the assistant might condition on the fact that it
has not moved so far ($a^{{}_{r}}_{1:T}$ are no-ops), leading it to infer that
the human is pursuing a goal $g$ where the assistant’s help is unnecessary. To
address this, it is crucial to recognize that the assistant is _intervening_
upon the environment through its actions (Figure 2(b), double-lined
nodes).222Alternatively, it is enough to note the assistant’s actions are
selected according to _different policy_ $\pi^{\prime}$ than the inferred
joint policy $\pi$ computed by the human. Thus it is _safer_ for the assistant
not to update its beliefs as if its actions come from $\pi$. As such, we
should detach the evidential connection between the assistant’s actions
$a^{{}_{r}}_{1:T}$ and any causal ancestors, which we denote using Pearl’s do-
operator Pearl (2012):

$\displaystyle P($ $\displaystyle
g,\pi|s_{1:T},d_{1:T},u_{1:T},a^{{}_{h}}_{1:T},\mathbf{do}(a^{{}_{r}}_{1:T}))$
$\displaystyle\propto
P(g,\pi,s_{1:T},d_{1:T},u_{1:T},a^{{}_{h}}_{1:T},\mathbf{do}(a^{{}_{r}}_{1:T}))$
(8)
$\displaystyle=P(g,\pi)\textstyle{\prod}_{t=1}^{T}P(s_{t}|s_{t-1},a^{{}_{h},_{r}}_{t-1})P(u_{t},d_{t}|s_{t},\pi)\underline{P(a^{{}_{h}}_{t}|s_{t},\pi)}$

The difference between Equations 7 and 8 lies in the final underlined term:
Whereas an external observer reweights its beliefs by incorporating the
likelihood of both agents’ actions
$P(a^{{}_{h}}_{t},a^{{}_{r}}_{t}|s_{t},\pi)$, our assistive agent should only
incorporate the likelihood of the human’s action
$P(a^{{}_{h}}_{t}|s_{t},\pi)$.

We compute the distribution in Equation 8 sequentially, as shown in Algorithm
1. An initial belief $b_{0}$ is returned by belief-init, which generates a set
of $N$ weighted samples by either sampling or enumerating over the goal prior
$P(g)$ and policy distribution $P(\pi|g)$ (accounting e.g. for uncertainty
over the rationality parameter $\beta$). Then at each step $t$, the previous
belief $b_{t-1}$ is updated by adjusting the weight $w^{i}$ associated with
each goal $g^{i}$ and policy $\pi^{i}$ sample. To do this, belief-update first
refines the $Q$-value estimates that define the policy $\pi^{i}$ by running
more iterations of RTHS (rths-policy-update in L9). The weights are then
updated to reflect the new state $s_{t}$ and decision to speak $d_{t}$
(L10-11). If $d_{t}$ is true (i.e. $u_{t}$ is observed), we also marginalize
over all commands $c_{t}$ that could have generated $u_{t}$ from $\pi$, and
update $w^{i}$ with the resulting mixture likelihood (L13-14). Finally, we
condition on the human’s action $a^{h}_{t}$, and return the resulting sample
collection (L16-18).

If we enumerate over all possible goals and policy configurations in belief-
init, this is an _exact_ Bayesian inference algorithm. In environments where
there are too many hypotheses for this to be efficient, Algorithm 1 can
readily be extended to a sequential Monte Carlo algorithm Del Moral et al.
(2006); Lew et al. (2023a) in the style of sequential inverse plan search Zhi-
Xuan et al. (2020). For our experiments, however, enumeration is feasible, and
we opt for this approach to avoid variance from sampling.

###  2.3. Pragmatic instruction following as
goal assistance under uncertainty

With Algorithm 1, our assistive agent is able to infer a distribution over the
human’s goal $g$ and policy $\pi$ at each step $t$. How should it use this
information to act, especially if the goal $g$ remains uncertain? In keeping
with our Bayesian approach, we minimize _expected_ cost, acting to help the
human achieve their likely goal as quickly as possible Shah et al. (2020).
Since each inferred policy $\pi^{i}$ corresponds to a
$\hat{Q}_{\pi^{i}}$-value function, the assistant can select actions by
minimizing the expected $\hat{Q}_{\pi^{i}}$ value given its uncertainty over
$\pi^{i}$:

$a^{{}_{r}*}_{t}=\textstyle{\operatorname*{arg\,min}}_{a^{{}_{r}}_{t}}\mathbb{E}_{\pi^{i}}[\hat{Q}_{\pi^{i}}(s_{t},a^{{}_{h}}_{t},a^{{}_{r}}_{t})]$
(9)

We implement this assistance policy with Algorithm 2, which effectively solves
the $Q_{\text{MDP}}$ approximation of the assistive POMDP Littman et al.
(1995); Hauskrecht (2000); Macindoe et al. (2012). At each step $t$, we assume
the assistant has already updated its belief $b_{t-1}$ from the previous step
$t-1$, and also observes the human’s action 333This assumption is natural in
our turn-based setting. When actions are simultaneous the assistant can
minimize expectation over both policies $\pi^{i}$ and human actions
$a^{{}_{h}}_{t}$.. We then iterate over all samples $(w^{i},g^{i},\pi^{i})$,
update the policies $\pi^{i}$ if necessary, and perform a weighted sum of
$\hat{Q}_{\pi^{i}}$-values for all assistant actions $a^{{}_{r}}_{t}$
according to their inferred weights $w^{i}$. This produces a vector of
_assistive_ $Q$-values, which we minimize over to compute the best assistive
action $a^{{}_{r}*}_{t}$.

Algorithm 1 CLIPS belief initialization and update

1:procedure belief-init($N$)

2: $w^{i}\leftarrow 0$; $g^{i}\sim P(g)$; $\pi^{i}\sim P(\pi|g^{i})$ for
$i\in[1,N]$

3: $b_{0}\leftarrow\\{(w^{i},g^{i},\pi^{i})\\}_{i=1}^{N}$; return $b_{0}$

4:end procedure

5:

6:procedure belief-
update($b_{t-1},s_{t-1:t},d_{t},u_{t},a^{{}_{h}}_{t-1:t},a^{{}_{r}}_{t-1}$)

7: $\\{(w^{i},g^{i},\pi^{i})\\}_{i=1}^{N}\leftarrow b_{t-1}$

8: for $i\in[1,N]$ do

9: $\pi^{i}\leftarrow\textsc{rths-policy-update}(\pi^{i},g^{i},s_{t})$

10: $w^{i}\leftarrow w^{i}\cdot
P(s_{t}|s_{t-1},a^{{}_{h}}_{t-1},a^{{}_{r}}_{t-1})$

11: $w^{i}\leftarrow w^{i}\cdot P(d_{t}|s_{t},\pi)$

12: if $d_{t}=\textsc{true}$ then

13: $P(u_{t}|s_{t},\pi)\leftarrow\sum_{c_{t}}P(u_{t}|c_{t})P(c_{t}|s_{t},\pi)$

14: $w^{i}\leftarrow w^{i}\cdot P(u_{t}|s_{t},\pi)$

15: end if

16: $w^{i}\leftarrow w^{i}\cdot P(a^{{}_{h}}_{t}|s_{t},\pi)$

17: end for

18: $w^{i}\leftarrow w^{i}/\sum_{j=1}^{N}w_{j}$

19: $b_{t}\leftarrow\\{(w^{i},g^{i},\pi^{i})\\}_{i=1}^{N}$; return $b_{t}$

20:end procedure

Algorithm 2 CLIPS $Q_{\text{MDP}}$ assistance policy

1:procedure assistance-policy($b_{t-1}$, $s_{t}$, $a^{{}_{h}}_{t}$)

2: for $a^{{}_{r}}_{t}\in\mathcal{A}^{{}_{r}}(s_{t})$ do

3: $\hat{Q}_{\text{assist}}(s_{t},a^{{}_{h}}_{t},a^{{}_{r}}_{t})\leftarrow 0$

4: for $(w^{i},g^{i},\pi^{i})\in b_{t-1}$ do

5: $\pi^{i}\leftarrow\textsc{rths-policy-update}(\pi^{i},g^{i},s_{t})$

6:
$\hat{Q}_{\text{assist}}(s_{t},a^{{}_{h}}_{t},a^{{}_{r}}_{t})\overset{{}_{+}}{\leftarrow}w^{i}\cdot\hat{Q}_{\pi^{i}}(s_{t},a^{{}_{h}}_{t},a^{{}_{r}}_{t})$

7: end for

8: end for

9:
$a^{{}_{r}*}_{t}\leftarrow\arg\min_{a^{{}_{r}}_{t}}Q_{\text{assist}}(s_{t},a^{{}_{h}}_{t},a^{{}_{r}}_{t})$

10: return $a^{{}_{r}*}_{t}$

11:end procedure

##  3\. Experiments

We evaluate our method against a variety of baselines in two domains: a
cooperative gridworld puzzle called multi-agent Doors, Keys & Gems (mDKG)
originally developed in Zhi-Xuan et al. (2020); Ying et al. (2023b), and
VirtualHome (VH), a virtual household simulator Puig et al. (2018, 2020). In
both domains, a human principal and a (robot) assistant cooperate on multi-
step tasks to accomplish the human’s goals. In mDKG, the goals are four
colored gems which are often secured behind doors. Doors can be unlocked by
keys of the same color, and the assistant can help by collecting keys or
unlocking doors (Figure 1). In VirtualHome, the goals are 6–12 household
tasks, including setting up tables and preparing ingredients (Figure 3). Both
domains admit encodings in the Planning Domain Definition Language (PDDL)
McDermott et al. (1998) with some extensions Zhi-Xuan (2022b), which we use as
the environment representation for our planning and inverse planning
algorithms.

Figure 3. Example goal assistance problem in VirtualHome, where the principal
and assistant collaborate to set the dinner table. The principal places three
plates on the table, then says _“Could you get the forks and knives?”_. A
pragmatic assistant has to infer the number of forks and knives from context
(in this case, three each).

To systematically test the ability of assistive agents to pragmatically follow
a human’s instructions and assist with their goals, we developed a dataset of
_goal assistance problems_ for each domain (30 problems in mDKG, 25 problems
in VH). In each problem, the goal is initially unknown to the assistant. While
taking a sequence of actions $a^{{}_{h}}_{1:T}$, the human communicates one or
more instructions $u_{1:T}$ to the assistant. The assistant then has to make
an inference about the human’s goal $g$, and chooses actions to best assist
them. We designed our problems to cover a range of scenarios inspired by human
communication we observed in exploratory studies, varying the information that
can be deciphered from the human’s actions or instructions (see Figure 4 and
Supplementary Information). In mDKG, we restricted the human to only one
instruction $u_{T}$, since more information would render goal inference
trivial. In VH, however, the goal space was larger, allowing us to construct
10 problems with multiple utterances such as _“Can you get the plates?”_
followed by _“Bring the bowls too!”_

###  3.1. Model configuration

In our experiments, we configured CLIPS to compute $Q$-values using the real-
time adaptive A* variant of RTHS Koenig and Likhachev (2006), and a
$\operatorname{Gamma}$ prior over the rationality parameter $\beta$. For the
utterance model, we used either OpenAI’s 6.7B curie model Brown et al. (2020)
or the davinci-002 as our LLMs, due to the more diffuse probabilities they
provided as base models. For goal assistance, we ran Algorithm 2 in offline
mode, fixing the inferred posterior after observing $a_{1:T}^{h}$ and
$u_{1:T}$, then selecting actions by minimizing expected $Q$-values with
respect to the fixed posterior. This was evaluated against a simulated human
agent which followed a joint policy to the true goal, unless it was apparent
that the assistant was not doing the same. While Algorithm 2 can also be run
in online mode, we fixed the posterior to better match the information that
our human raters were provided.

###  3.2. Baselines

To evaluate the benefit of pragmatic instruction following over either a
literal interpretation of instructions, or goal inference from only one
modality, we included the following baselines:

####  3.2.1. Unimodal Inverse Planning

We implemented action-only and language-only inverse planning (IP) baselines
as ablations of CLIPS, where the action-only baseline is similar to sequential
inverse plan search Zhi-Xuan et al. (2020) for inference and Watch-and-Help
Puig et al. (2020) for assistance. In these baselines, the assistive agent
uses the same belief update shown in Algorithm 1, except that it conditions on
only actions $a^{{}_{h}}_{1:t}$ or only the utterance $u_{t}$. In cases with
no actions, CLIPS and the language-only baseline are equivalent.

####  3.2.2. LLM-Based Literal Listener

The literal listener baseline interprets the instruction $u_{t}$ in state
$s_{t}$ without further information about the human’s actions or goals. To
implement this baseline, we adapt the utterance model in Figure 2(d) to sample
from the space of all assistant-directed commands $c_{t}$ that are possible in
state $s_{t}$. In particular, we enumerate over all assistant actions
$\mathcal{A}^{{}_{r}}$, and select a subset of up to $K$ salient actions to
form a command $c_{t}$. This defines a distribution over commands
$P(c_{t}|s_{t})$ which _does not_ depend the principal’s policy $\pi$ or goal
$g$. Given the command $c_{t}$, we use an LLM (text-davinci-002 for mDKG,
davinci-002 for VH) as an utterance likelihood $P(u_{t}|c_{t})$. Assuming a
uniform prior over commands, we can perform enumerative inference to compute
the posterior $P(c_{t}|u_{t},s_{t})$ given an instruction $u_{t}$ in state
$s_{t}$.

Given a command $c_{t}$, the assistant still needs to generate an assistive
plan. We did this in two ways: The _naive_ literal assistant interprets a
command $c_{t}$ by randomly selecting one concrete grounding (e.g. _“Can you
pick up the red key?”_ could mean picking up the red key closest to the human
or some other key), then planning to achieve that concrete goal. In contrast,
the _efficient_ literal assistant tries to directly satisfy the command
$c_{t}$ in the most efficient way444Note that the naive/efficient distinction
was not meaningful in the VirtualHome problems, where utterance ambiguity was
not due to grounding ambiguity. (e.g. _“Could you unlock the blue door?”_ is
satisfied by unlocking the blue door closest to the assistant). For both
variants, the assistant can either satisfy the most likely command
$c_{t}^{*}$, or sample a command from the full distribution
$P(c_{t}|u_{t},s_{t})$. We report results for the latter after averaging over
10 samples, using systematic sampling to reduce variance.

####  3.2.3. Multimodal LLM (GPT-4V)

For the mDKG domain, we used GPT-4 with Vision (GPT-4V) Achiam et al. (2023)
as a purely neural baseline, prompting it with the same set of rules and
instructions that we provided to our human raters, along with the final frame
of each animated visual that we showed to humans (similar to Figure 1(b)).
This allowed us to probe the degree to which multimodal LLMs are capable of
intuitive pragmatic reasoning given spatially-grounded actions and verbal
instructions Buschoff et al. (2024). Due to cost and rate limits, we report
zero-shot performance with a temperature of zero.

More details about our models and datasets can be found in the Supplementary
Information (https://osf.io/v8ru7/). Source code is available at
https://github.com/probcomp/CLIPS.jl.

Table 1. Performance of CLIPS vs. baseline methods, measured in terms accuracy (posterior probability of true goal, precision and recall for assistance options), helpfulness (plan length and human cost relative to CLIPS), and human similarity (correlation of goal inferences and assistance options with mean human ratings) Metrics are averaged across the dataset per domain, with standard errors reported in brackets. | Accuracy | Helpfulness | Human Similarity
---|---|---|---
Method | $P(g_{\text{true}})$ | Assist. Prec. | Assist. Rec. | Rel. Plan Length | Rel. Human Cost | Goal Cor. | Assist. Cor.
Doors, Keys & Gems
Humans | 0.67 (0.04) | 0.83 (0.02) | 0.85 (0.01) | – | – | – | –
CLIPS (Ours) | 0.74 (0.05) | 0.97 (0.03) | 0.97 (0.03) | 1.00 (0.00) | 1.00 (0.00) | 0.93 (0.01) | 0.96 (0.01)
Lang. Only IP | 0.55 (0.05) | 0.90 (0.06) | 0.83 (0.06) | 1.26 (0.10) | 1.18 (0.07) | 0.74 (0.01) | 0.83 (0.01)
Action Only IP | 0.31 (0.04) | 0.43 (0.09) | 0.40 (0.09) | 1.68 (0.15) | 1.46 (0.08) | 0.15 (0.01) | 0.45 (0.01)
Literal Efficient | – | 0.65 (0.08) | 0.54 (0.07) | 1.55 (0.11) | 1.58 (0.10) | – | 0.47 (0.01)
Literal Naive | – | 0.58 (0.04) | 0.47 (0.02) | 1.54 (0.07) | 1.52 (0.05) | – | 0.54 (0.01)
GPT-4V | 0.29 (0.08) | 0.39 (0.08) | 0.37 (0.08) | – | – | 0.10 (0.01) | 0.11 (0.01)
VirtualHome
CLIPS (Ours) | 0.63 (0.04) | 0.87 (0.04) | 1.00 (0.00) | 1.00 (0.00) | 1.00 (0.00) | – | –
Lang. Only IP | 0.37 (0.05) | 0.59 (0.05) | 0.96 (0.03) | 1.33 (0.07) | 1.35 (0.08) | – | –
Action Only IP | 0.25 (0.03) | 0.61 (0.07) | 0.84 (0.07) | 1.30 (0.07) | 1.33 (0.07) | – | –
Literal Listener | – | 0.64 (0.06) | 0.70 (0.07) | 1.48 (0.08) | 1.54 (0.09) | – | –

###  3.3. Human judgments

As an additional standard for comparison, we conducted a study with 100 human
participants from the US through Prolific (mean age = 39.8, 59 men, 38 women,
2 non-binary), presenting each of them with 15 goal assistance problems from
the mDKG domain. In each problem, participants saw the actions taken by the
human principal and observed the instruction at the end. Participants were
then asked to select the set of gems that they thought were likely to be
principal’s goal, and then indicate how the robot agent should best assist the
principal. This gave 50 goal and assistance ratings for each problem (93%
power for Cohen’s d=0.5 at $\alpha$=0.05).

For the assistance question, we presented each participant with a set of
_assistance options_ , corresponding to either picking up keys or unlocking
doors. This allowed us to query participants’ understanding of how to provide
assistance without having them solve the entire problem. To compare these
human-provided assistance options with CLIPS and our baselines, we extracted
the corresponding actions from the assistive plans generated by each method,
and (where applicable) estimated the marginal probability of a particular
assistance option occurring in the assistive plan via sampling.

###  3.4. Performance metrics

We evaluated CLIPS and the baseline methods in terms of their goal and
assistance accuracy, the helpfulness and efficiency of the generated plans,
and their similarity to human judgments. To evaluate accuracy, we calculated
the probability $P(g_{\text{true}})$ assigned to the true goal for the non-
literal methods, as well as precision and recall for selecting the optimal
assistance options that were consistent with the instruction. To evaluate
helpfulness and efficiency, we computed the length of the generated plan and
the total action cost incurred by the human principal, relativized to CLIPS.
Finally, to evaluate human similarity in mDKG, we calculated the correlation
between each method’s outputs with the average ratings provided by our
participants. We calculated Pearson’s $r$ as well its variability from 1000
bootstrapped samples of the human dataset.

(a) Ambiguous Predicates. The human player asks for a red key when three are
available. CLIPS resolves this ambiguity, inferring that the human wants _Key
2_ to reach the yellow gem.

(b) Ambiguous Indexicals. The human player refers to a door using the
indexical “this”. CLIPS interprets this pragmatically, inferring that _Door 3_
is intended even though _Door 2_ is equally close to the human.

(c) Partial Instructions. The human player asks for a single red key, but
requires two to reach the red gem. CLIPS infers this and picks up both keys
(_Key 4_ and _Key 5_), unlike the literal listener baselines.

(d) Uncertain Goals. The human player asks for a yellow key while going to
pick up a red key. CLIPS pragmatically infers their goal to be one of the
three gems on the left, and hence picks up the left key (_Key 3_).

(e) Joint Instructions.The human player asks for a blue door to be unlocked
while indicating that they will pick up a red key. The CLIPS assistant infers
that it should unlock _Door 4_ so that the human can unlock _Door 5_.

(f) Safe Assistance. The human player asks the robot to unlock a red door when
there are two such doors. The CLIPS assistant reasons that unlocking _Door 3_
is safer as it leads to _both_ blue and green gems, unlike _Door 2_.

Figure 4. Goal assistance problems in Doors, Keys & Gems. Each sub-figure
contains a visual (_left_), instruction (_bottom left_), goal posteriors
produced by each method (_top right_), and the probability of a key or door
appearing in the assistive plans generated by each method (_bottom right_).
Our pragmatic goal assistance method, CLIPS, best matches the goal inferences
and assistance options produced by human raters (averaged across raters). In
contrast, language and action-only inverse planning (Lang. IP & Act. IP) have
higher goal uncertainty, the literal baselines fail to resolve instruction
ambiguity, and GPT-4V often produces incoherent responses.

###  3.5. Results

The results of our experiments are presented in Table 1. In terms of accuracy,
CLIPS assigned higher probability to the true goal than the baselines (note
that 100% is not possible since goals cannot always be distinguished, as in
Fig. 4(d)), while achieving close to perfect precision and recall in selecting
the optimal assistance options. Indeed, CLIPS outperformed the average human
in this regard. In contrast, unimodal inverse planning was much less accurate
at inferring the true goal, and all baselines selected the correct assistance
options at significantly lower rates. This affected the efficiency of
assistance: CLIPS was 1.26 to 1.68 times faster in achieving the human’s goal
than the baselines, and produced plans that were 1.18 to 1.58 times less
costly for the human.

For human similarity in the mDKG domain, we found that both the goal
inferences and the assistance options produced by CLIPS correlated highly with
human ratings, achieving Pearson’s $r$ of 0.93 (95% CI: 0.91–0.94) and 0.96
(95% CI: 0.95–0.96) respectively. In contrast, unimodal inverse planning
produced goal inferences that were highly dissimilar from humans. Correlation
with human-selected assistance options was also poor. These results
demonstrate the importance of accounting for pragmatic context. Out of all
methods, GPT-4V performed the worst, possibly because of the spatial reasoning
and multi-step planning required for coherent goal inference in mDKG. Among
participants themselves, we found that the median Pearson’s $r$ of each
participant’s ratings with the mean rating was 0.85 (IQR: 0.71–0.94) for goal
inferences and 0.87 (IQR: 0.65–0.95) for assistance options, indicating that
our dataset of human judgments is a reliable measure of average human
performance.

In Figure 4, we compare the results of CLIPS against human judgements and
baselines on six illustrative goal assistance problems, providing qualitative
observations in the captions. The results show how CLIPS closely mirrors human
judgements for both goal inference and assistance, resolving ambiguity in
predicates (Fig. 4(a)) and indexicals (Fig. 4(b)) while successfully
completing partial instructions (Fig. 4(c)) and interpreting joint
instructions (Fig. 4(e)). In comparison, the unimodal and literal baselines
make less confident inferences or fail to assist appropriately, while GPT-4V
provides incoherent answers. We also find that both humans and CLIPS are able
to assist appropriately when there is significant goal uncertainty (Fig.
4(d)), and even when the optimal assistive plans for each goal make diverging
recommendations (Fig. 4(f)), illustrating the importance of uncertainty-aware
assistance.

##  4\. Related Work

##### Theory of Mind as Bayesian Inverse Planning

Bayesian Theory of Mind (BToM) posits that humans reason about the actions and
mental states of others through (approximately) Bayesian inference Baker et
al. (2011, 2017). In particular, Bayesian inverse planning can model how
humans infer the goals of others by assuming that agents act rationally to
achieve their goals Gergely and Csibra (2003); Baker et al. (2009); Shum et
al. (2019); Jara-Ettinger et al. (2019). To efficiently solve these inference
problems, prior work uses sequential Monte Carlo methods Del Moral et al.
(2006); Lew et al. (2023a) to perform online goal inference over model-based
planning agents Zhi-Xuan et al. (2020); Alanqary et al. (2021). CLIPS builds
upon this paradigm, extending recent work by Ying et al. (2023b) on multi-
agent inverse planning.

##### Rational Speech Acts

Rational Speech Act (RSA) theory frames communication as a recursive process,
where speakers make pragmatic utterances that optimize relevance to listeners,
and listeners interpret speaker utterances in light of the pragmatic goals
that underlie those utterances Goodman and Frank (2016); Fried et al. (2018);
Sumers et al. (2023). CLIPS instantiates the RSA framework in the context of
cooperative planning.

##### Multimodal Goal Inference and Reward Learning.

Goal inference can be framed as online inverse reinforcement learning (IRL),
where the aim is to to learn a reward or cost function explaining the agent’s
behavior in a single episode Jara-Ettinger (2019). CLIPS can thus be viewed as
a language-informed IRL method, though IRL is primarily applied offline given
a dataset of expert demonstrations Williams et al. (2018); Tung et al. (2018);
Fu et al. (2019). Particularly relevant is _reward-rational implicit choice_
Jeon et al. (2020), a framework for multimodal Bayesian reward learning from
heterogeneous human feedback. CLIPS can be seen as an application of this
framework for specific modalities, but extended to the cooperative setting.

##### Decentralized Cooperation and Joint Intentionality

When multiple agents cooperate on a task, they usually need to track each
other’s mental states to make decisions. Common approaches to this involve
recursive reasoning over each other’s goals and plans, which can quickly grow
intractable Doshi and Gmytrasiewicz (2009); Camerer et al. (2004). To address
this challenge, computational cognitive scientists have developed models of
_joint intentionality_ Tomasello and Carpenter (2007), where cooperating
agents conceive of themselves as a _group agent_ with a common goal in mind
Stacy et al. (2021); Tang et al. (2022). This approach allows groups of agents
to converge on shared goals Tang et al. (2020) and efficient task
decompositions Wu et al. (2021) in a decentralized manner with limited
recursion. CLIPS uses this insight to model joint planning and pragmatic
instruction generation in humans.

##### Value Alignment and Assistance Games.

CLIPS is a solution strategy for language-augmented goal assistance games, an
extension of the assistance game formalism for human-AI value alignment
Hadfield-Menell et al. (2016); Russell (2021). In contrast to prior work, our
framework leverages joint intentionality when modeling human principals,
thereby requiring less recursion than iterated best-response Hadfield-Menell
et al. (2016), while avoiding the intractability of equilibrium solutions
Fisac et al. (2017); Milli and Dragan (2020).

##### Instruction Following with Language Models.

CLIPS is a form of grounded instruction following from natural language Tellex
et al. (2011); Squire et al. (2015); Misra et al. (2016), using LLMs to score
the likelihood of an utterance given a grounded command. While many recent
studies have employed LLMs for translating language into actions Ahn et al.
(2022); Singh et al. (2023) or task specifications Kwon et al. (2023); Yu et
al. (2023); Liu et al. (2022), we opt for an explicitly Bayesian approach,
allowing our method to integrate information from both actions and
instructions in a principled, reliable, and modular manner.

##  5\. Discussion and Future Work

In this paper, we introduced cooperative language-guided inverse plan search
(CLIPS) as a Bayesian architecture for pragmatic instruction following and
goal assistance. By using a structured cognitive model of how speakers produce
both actions and utterances given their goals, CLIPS is able to integrate
information from observed actions and ambiguous instruction, inferring a
distribution over speaker’s goals and intentions. Using this distribution, a
CLIPS assistant is then able to assist the speaker in achieving their goal
through expected cost minimization. Through our experiments, we show that the
proposed architecture produces human-like outputs on goal inference and
assistance tasks. Compared with the baselines, CLIPS also makes more accurate
goal inferences, and chooses actions that result in more efficient plans.

While CLIPS demonstrates compelling theoretical properties and strong
empirical performance, a number of challenges remain before it can be applied
to scenarios with a larger number of goals and assistance options. As noted in
Section 2.2, the Bayesian inference strategy we adopt is fully enumerative,
but this approach breaks down once the space of possible goals, plans, and
commands grows sufficiently large. Recent advances in probabilistic
programming could overcome these bottlenecks. For example, more sophisticated
sequential Monte Carlo strategies could be used to flexibly propose and update
hypotheses, focusing computation on only the most likely sets of latent
variables Zhi-Xuan et al. (2020); Lew et al. (2023a). These strategies can be
applied to perform _constrained decoding_ from large language models Lew et
al. (2023b); Willard and Louf (2023), thereby allowing LLMs to be used as
sound proposal distributions over a grammar for commands. This would benefit
from LLMs’ performance at few-shot semantic parsing and translation Shin et
al. (2021), while preserving the Bayesian nature of our framework.

The assistance policy we introduced in Algorithm 2 could also be extended in a
number of ways. In many assistive settings, uncertainty is sufficiently high
that it makes sense for the assistant to take _information gathering actions_.
Indeed, this is what people typically do when we are confused: we ask
questions. To enable this ability, assistants should perform _belief-space
planning_ over the set of possible goal beliefs Sunberg and Kochenderfer
(2018); Macindoe et al. (2012), thinking ahead about which actions reveal more
information about the principal’s goals. Such planning would alleviate issues
that can arise due to the $Q_{\text{MDP}}$ approximation Littman et al.
(1995); Hauskrecht (2000) used by Algorithm 2. In the longer run, assistants
could be augmented with natural language outputs, enabling them to ask
clarifying questions by planning ahead in a white-box, interpretable manner.
If successful, this would constitute a considerable step towards trustworthy
assistive AI that effectively collaborates and communicates with humans.

00footnotetext: Errata: Along with typographical fixes, this version corrects
minor errors in the GPT-4V assistance results that are present in the version
of this paper published in the ACM Digital Library.

This work was funded in part by the DARPA Machine Common Sense, AFOSR, and ONR
Science of AI programs, along with the MIT-IBM Watson AI Lab, and gifts from
Reid Hoffman and the Siegel Family Foundation. Tan Zhi-Xuan is supported by an
Open Philanthropy AI Fellowship.

## References

  * (1)
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  GPT-4 Technical Report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022\.  Do as I can, not as I say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ (2022).
  * Alanqary et al. (2021) Arwa Alanqary, Gloria Z Lin, Joie Le, Tan Zhi-Xuan, Vikash K Mansinghka, and Joshua B Tenenbaum. 2021.  Modeling the mistakes of boundedly rational agents within a Bayesian theory of mind.  _arXiv preprint arXiv:2106.13249_ (2021).
  * Austin (1975) John Langshaw Austin. 1975.  _How to do things with words_. Vol. 88.  Oxford University Press.
  * Aversa et al. (2016) Davide Aversa, Sebastian Sardina, and Stavros Vassos. 2016.  Pruning and preprocessing methods for inventory-aware pathfinding. In _2016 IEEE Conference on Computational Intelligence and Games (CIG)_. IEEE, 1–8.
  * Baker et al. (2011) Chris Baker, Rebecca Saxe, and Joshua Tenenbaum. 2011.  Bayesian Theory of Mind: Modeling Joint Belief-Desire Attribution. In _Proceedings of the Annual Meeting of the Cognitive Science Society, 33 (33)_.
  * Baker et al. (2017) Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B Tenenbaum. 2017.  Rational quantitative attribution of beliefs, desires and percepts in human mentalizing.  _Nature Human Behaviour_ 1, 4 (2017), 1–10.
  * Baker et al. (2009) Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum. 2009.  Action understanding as inverse planning.  _Cognition_ 113, 3 (2009), 329–349.
  * Barto et al. (1995) Andrew G Barto, Steven J Bradtke, and Satinder P Singh. 1995.  Learning to act using real-time dynamic programming.  _Artificial intelligence_ 72, 1-2 (1995), 81–138.
  * Boutilier (1996) Craig Boutilier. 1996.  Planning, learning and coordination in multiagent decision processes. In _TARK_ , Vol. 96. Citeseer, 195–210.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Buschoff et al. (2024) Luca M. Schulze Buschoff, Elif Akata, Matthias Bethge, and Eric Schulz. 2024.  Visual cognition in multimodal large language models.  arXiv:2311.16093 [cs.LG]
  * Camerer et al. (2004) Colin F Camerer, Teck-Hua Ho, and Juin-Kuan Chong. 2004.  A cognitive hierarchy model of games.  _The Quarterly Journal of Economics_ 119, 3 (2004), 861–898.
  * Choi and Kim (2014) Jaedeug Choi and Kee-Eung Kim. 2014.  Hierarchical bayesian inverse reinforcement learning.  _IEEE transactions on cybernetics_ 45, 4 (2014), 793–805.
  * Cusumano-Towner et al. (2019) Marco F Cusumano-Towner, Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. 2019.  Gen: a general-purpose probabilistic programming system with programmable inference. In _Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation_. ACM, 221–236.
  * Del Moral et al. (2006) Pierre Del Moral, Arnaud Doucet, and Ajay Jasra. 2006.  Sequential Monte Carlo samplers.  _Journal of the Royal Statistical Society Series B: Statistical Methodology_ 68, 3 (2006), 411–436.
  * Doshi and Gmytrasiewicz (2009) Prashant Doshi and Piotr J Gmytrasiewicz. 2009.  Monte Carlo sampling methods for approximating interactive POMDPs.  _Journal of Artificial Intelligence Research_ 34 (2009), 297–337.
  * Fisac et al. (2017) Jaime F Fisac, Monica A Gates, Jessica B Hamrick, Chang Liu, Dylan Hadfield-Menell, Malayandi Palaniappan, Dhruv Malik, S Shankar Sastry, Thomas L Griffiths, and Anca D Dragan. 2017.  Pragmatic-Pedagogic Value Alignment.  _arXiv preprint arXiv:1707.06354_ (2017).
  * Fried et al. (2018) Daniel Fried, Jacob Andreas, and Dan Klein. 2018.  Unified Pragmatic Models for Generating and Following Instructions. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , Marilyn Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, New Orleans, Louisiana, 1951–1963.  https://doi.org/10.18653/v1/N18-1177
  * Fu et al. (2019) Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. 2019.  From language to goals: Inverse reinforcement learning for vision-based instruction following.  _arXiv preprint arXiv:1902.07742_ (2019).
  * Gergely and Csibra (2003) György Gergely and Gergely Csibra. 2003.  Teleological reasoning in infancy: The naıve theory of rational action.  _Trends in cognitive sciences_ 7, 7 (2003), 287–292.
  * Giora (2004) Rachel Giora. 2004.  On the graded salience hypothesis.  (2004).
  * Goodman and Frank (2016) Noah D Goodman and Michael C Frank. 2016.  Pragmatic language interpretation as probabilistic inference.  _Trends in cognitive sciences_ 20, 11 (2016), 818–829.
  * Grice (1975) Herbert P Grice. 1975.  Logic and Conversation.  In _Speech Acts_. Brill, 41–58.
  * Hadfield-Menell et al. (2016) Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. 2016.  Cooperative inverse reinforcement learning. In _Advances in neural information processing systems_. 3909–3917.
  * Haslum et al. (2005) Patrik Haslum, Blai Bonet, Héctor Geffner, et al. 2005\.  New admissible heuristics for domain-independent planning. In _AAAI_ , Vol. 5. 9–13.
  * Hauskrecht (2000) Milos Hauskrecht. 2000.  Value-function approximations for partially observable Markov decision processes.  _Journal of artificial intelligence research_ 13 (2000), 33–94.
  * Held and Karp (1971) Michael Held and Richard M Karp. 1971.  The traveling-salesman problem and minimum spanning trees: Part II.  _Mathematical programming_ 1, 1 (1971), 6–25.
  * Helmert and Domshlak (2009) Malte Helmert and Carmel Domshlak. 2009.  Landmarks, critical paths and abstractions: what’s the difference anyway?. In _Proceedings of the International Conference on Automated Planning and Scheduling_ , Vol. 19. 162–169.
  * Hernández et al. (2015) Carlos Hernández, Tansel Uras, Sven Koenig, Jorge A Baier, Xiaoxun Sun, and Pedro Meseguer. 2015.  Reusing cost-minimal paths for goal-directed navigation in partially known terrains.  _Autonomous Agents and Multi-Agent Systems_ 29 (2015), 850–895.
  * Hopkins et al. (2023) Aspen K Hopkins, Alex Renda, and Michael Carbin. 2023.  Can LLMs Generate Random Numbers? Evaluating LLM Sampling in Controlled Domains. In _ICML 2023 Workshop: Sampling and Optimization in Discrete Space_.
  * Jara-Ettinger (2019) Julian Jara-Ettinger. 2019.  Theory of mind as inverse reinforcement learning.  _Current Opinion in Behavioral Sciences_ 29 (2019), 105–110.
  * Jara-Ettinger et al. (2019) Julian Jara-Ettinger, Laura Schulz, and Josh Tenenbaum. 2019.  The Naive Utility Calculus as a unified, quantitative framework for action understanding.  _PsyArXiv_ (2019).
  * Jeon et al. (2020) Hong Jun Jeon, Smitha Milli, and Anca Dragan. 2020.  Reward-rational (implicit) choice: A unifying formalism for reward learning.  _Advances in Neural Information Processing Systems_ 33 (2020), 4415–4426.
  * Jin et al. (2024) Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B Tenenbaum, and Tianmin Shu. 2024.  MMToM-QA: Multimodal Theory of Mind Question Answering.  _arXiv preprint arXiv:2401.08743_ (2024).
  * Kecskes et al. (2004) Istvan Kecskes et al. 2004\.  The role of salience in processing pragmatic units.  _Acta Linguistica Hungarica (Since 2017 Acta Linguistica Academica)_ 51, 3-4 (2004), 309–324.
  * Kirk et al. (2023) Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2023.  Understanding the effects of RLHF on LLM generalisation and diversity.  _arXiv preprint arXiv:2310.06452_ (2023).
  * Koenig and Likhachev (2006) Sven Koenig and Maxim Likhachev. 2006.  Real-Time Adaptive A*. In _Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems_. 281–288.
  * Koenig and Sun (2009) Sven Koenig and Xiaoxun Sun. 2009.  Comparing real-time and incremental heuristic search for real-time situated agents.  _Autonomous Agents and Multi-Agent Systems_ 18 (2009), 313–341.
  * Korf (1990) Richard E Korf. 1990.  Real-time heuristic search.  _Artificial intelligence_ 42, 2-3 (1990), 189–211.
  * Kwon et al. (2023) Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. 2023.  Reward Design with Language Models. In _The Eleventh International Conference on Learning Representations_.  https://openreview.net/forum?id=10uNUgI5Kl
  * Lew et al. (2023a) Alexander K Lew, George Matheos, Tan Zhi-Xuan, Matin Ghavamizadeh, Nishad Gothoskar, Stuart Russell, and Vikash K Mansinghka. 2023a.  SMCP3: Sequential Monte Carlo with probabilistic program proposals. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 7061–7088.
  * Lew et al. (2023b) Alexander K Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K Mansinghka. 2023b.  Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs.  _arXiv preprint arXiv:2306.03081_ (2023).
  * Lin et al. (2022) Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan. 2022.  Inferring rewards from language in context.  _arXiv preprint arXiv:2204.02515_ (2022).
  * Littman et al. (1995) Michael L Littman, Anthony R Cassandra, and Leslie Pack Kaelbling. 1995.  Learning policies for partially observable environments: Scaling up.  In _Machine Learning Proceedings 1995_. Elsevier, 362–370.
  * Liu et al. (2022) Jason Xinyu Liu, Ziyi Yang, Benjamin Schornstein, Sam Liang, Ifrah Idrees, Stefanie Tellex, and Ankit Shah. 2022.  Lang2LTL: Translating Natural Language Commands to Temporal Specification with Large Language Models. In _Workshop on Language and Robotics at CoRL 2022_.
  * Macindoe et al. (2012) Owen Macindoe, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. 2012.  POMCoP: Belief Space Planning for Sidekicks in Cooperative Games.. In _AIIDE_.
  * McDermott et al. (1998) Drew McDermott, Malik Ghallab, Adele Howe, Craig Knoblock, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. 1998.  PDDL - The Planning Domain Definition Language.
  * Milli and Dragan (2020) Smitha Milli and Anca D Dragan. 2020.  Literal or pedagogic human? analyzing human model misspecification in objective learning. In _Uncertainty in artificial intelligence_. PMLR, 925–934.
  * Misra et al. (2016) Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. 2016.  Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions.  _The International Journal of Robotics Research_ 35, 1-3 (2016), 281–300.
  * Murphy and Russell (2001) Kevin Murphy and Stuart Russell. 2001.  Rao-Blackwellised particle filtering for dynamic Bayesian networks.  In _Sequential Monte Carlo methods in practice_. Springer, 499–515.
  * Nguyen et al. (2011) Truong-Huy Nguyen, David Hsu, Wee-Sun Lee, Tze-Yun Leong, Leslie Kaelbling, Tomas Lozano-Perez, and Andrew Grant. 2011.  Capir: Collaborative action planning with intention recognition. In _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , Vol. 7. 61–66.
  * Pearl (2012) Judea Pearl. 2012.  The do-calculus revisited. In _Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence_. 3–11.
  * Puig et al. (2018) Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. 2018.  Virtualhome: Simulating household activities via programs. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 8494–8502.
  * Puig et al. (2020) Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. 2020.  Watch-and-help: A challenge for social perception and human-ai collaboration.  _arXiv preprint arXiv:2010.09890_ (2020).
  * Ramachandran and Amir (2007) Deepak Ramachandran and Eyal Amir. 2007.  Bayesian Inverse Reinforcement Learning.. In _IJCAI_ , Vol. 7. 2586–2591.
  * Ramírez and Geffner (2010) Miguel Ramírez and Hector Geffner. 2010.  Probabilistic plan recognition using off-the-shelf classical planners. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 24.
  * Russell (2021) Stuart Russell. 2021.  Human-Compatible Artificial Intelligence.  _Human-like machine intelligence_ (2021), 3–23.
  * Saisubramanian et al. (2019) Sandhya Saisubramanian, Kyle Hollins Wray, Luis Pineda, and Shlomo Zilberstein. 2019.  Planning in stochastic environments with goal uncertainty. In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 1649–1654.
  * Sclar et al. (2023) Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023.  Minding Language Models’ (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 13960–13980.  https://doi.org/10.18653/v1/2023.acl-long.780
  * Shah et al. (2020) Ankit Shah, Shen Li, and Julie Shah. 2020.  Planning with uncertain specifications (puns).  _IEEE Robotics and Automation Letters_ 5, 2 (2020), 3414–3421.
  * Shin et al. (2021) Richard Shin, Christopher H Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021.  Constrained language models yield few-shot semantic parsers.  _arXiv preprint arXiv:2104.08768_ (2021).
  * Shum et al. (2019) Michael Shum, Max Kleiman-Weiner, Michael L Littman, and Joshua B Tenenbaum. 2019.  Theory of minds: Understanding behavior in groups through inverse planning. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 6163–6170.
  * Singh et al. (2023) Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023.  Progprompt: Generating situated robot task plans using large language models. In _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 11523–11530.
  * Squire et al. (2015) Shawn Squire, Stefanie Tellex, Dilip Arumugam, and Lei Yang. 2015.  Grounding English commands to reward functions. In _Robotics: Science and Systems_.
  * Stacy et al. (2021) Stephanie Stacy, Chenfei Li, Minglu Zhao, Yiling Yun, Qingyi Zhao, Max Kleiman-Weiner, and Tao Gao. 2021.  Modeling communication to coordinate perspectives in cooperation.  _arXiv preprint arXiv:2106.02164_ (2021).
  * Sumers et al. (2023) Theodore R Sumers, Mark K Ho, Thomas L Griffiths, and Robert D Hawkins. 2023.  Reconciling truthfulness and relevance as epistemic and decision-theoretic utility.  _Psychological Review_ (2023).
  * Sunberg and Kochenderfer (2018) Zachary Sunberg and Mykel Kochenderfer. 2018.  Online algorithms for POMDPs with continuous state, action, and observation spaces. In _Proceedings of the International Conference on Automated Planning and Scheduling_ , Vol. 28. 259–263.
  * Tang et al. (2022) Ning Tang, Siyi Gong, Minglu Zhao, Chenya Gu, Jifan Zhou, Mowei Shen, and Tao Gao. 2022.  Exploring an Imagined “We” in human collective hunting: Joint commitment within shared intentionality. In _Proceedings of the annual meeting of the cognitive science society_ , Vol. 44.
  * Tang et al. (2020) Ning Tang, Stephanie Stacy, Minglu Zhao, Gabriel Marquez, and Tao Gao. 2020.  Bootstrapping an Imagined We for Cooperation.. In _CogSci_.
  * Tellex et al. (2011) Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. 2011.  Understanding natural language commands for robotic navigation and mobile manipulation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 25. 1507–1514.
  * Thompson (1935) William R Thompson. 1935.  On the theory of apportionment.  _American Journal of Mathematics_ 57, 2 (1935), 450–456.
  * Tomasello and Carpenter (2007) Michael Tomasello and Malinda Carpenter. 2007.  Shared intentionality.  _Developmental science_ 10, 1 (2007), 121–125.
  * Tung et al. (2018) Hsiao-Yu Tung, Adam W Harley, Liang-Kang Huang, and Katerina Fragkiadaki. 2018.  Reward learning from narrated demonstrations. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 7004–7013.
  * Wan et al. (2022) Yanming Wan, Jiayuan Mao, and Josh Tenenbaum. 2022.  HandMeThat: Human-Robot Communication in Physical and Social Environments.  _Advances in Neural Information Processing Systems_ 35 (2022), 12014–12026.
  * Willard and Louf (2023) Brandon T Willard and Rémi Louf. 2023.  Efficient Guided Generation for LLMs.  _arXiv preprint arXiv:2307.09702_ (2023).
  * Williams et al. (2018) Edward C Williams, Nakul Gopalan, Mine Rhee, and Stefanie Tellex. 2018.  Learning to parse natural language to grounded reward functions with weak supervision. In _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 4430–4436.
  * Wu et al. (2021) Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman-Weiner. 2021.  Too Many Cooks: Bayesian Inference for Coordinating Multi-Agent Collaboration.  _Topics in Cognitive Science_ 13, 2 (2021), 414–432.
  * Yates (1948) Frank Yates. 1948.  Systematic sampling.  _Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences_ 241, 834 (1948), 345–377.
  * Ying et al. (2023a) Lance Ying, Katherine M Collins, Megan Wei, Cedegao E Zhang, Tan Zhi-Xuan, Adrian Weller, Joshua B Tenenbaum, and Lionel Wong. 2023a.  The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputs.  _arXiv preprint arXiv:2306.14325_ (2023).
  * Ying et al. (2023b) Lance Ying, Tan Zhi-Xuan, Vikash Mansinghka, and Joshua B Tenenbaum. 2023b.  Inferring the goals of communicating agents from actions and instructions. In _Proceedings of the AAAI Symposium Series_ , Vol. 2. 26–33.
  * Yu et al. (2023) Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. 2023\.  Language to Rewards for Robotic Skill Synthesis.  _arXiv preprint arXiv:2306.08647_ (2023).
  * Zhi-Xuan (2022a) Tan Zhi-Xuan. 2022a.  GenGPT3.jl: GPT-3 as a generative function in Gen.jl.  https://github.com/probcomp/GenGPT3.jl
  * Zhi-Xuan (2022b) Tan Zhi-Xuan. 2022b.  _PDDL. jl: An Extensible Interpreter and Compiler Interface for Fast and Flexible AI Planning_.  Ph.D. Dissertation. Massachusetts Institute of Technology.
  * Zhi-Xuan et al. (2022) Tan Zhi-Xuan, Nishad Gothoskar, Falk Pollok, Dan Gutfreund, Joshua B Tenenbaum, and Vikash K Mansinghka. 2022.  Solving the Baby Intuition Benchmark with a hierarchically Bayesian theory-of-mind.  _arXiv preprint arXiv:2208.02914_ (2022).
  * Zhi-Xuan et al. (2020) Tan Zhi-Xuan, Jordyn Mann, Tom Silver, Josh Tenenbaum, and Vikash Mansinghka. 2020.  Online Bayesian Goal Inference for Boundedly Rational Planning Agents.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Ziebart et al. (2008) Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. 2008.  Maximum entropy inverse reinforcement learning.. In _AAAI_ , Vol. 8. Chicago, IL, USA, 1433–1438.

##  Appendix A Goal Assistance Problems

Table A1 and Table A2 describe the scenarios and conditions for our goal
assistance problems in multi-agent Doors, Keys & Gems (mDKG) and VirtualHome
(VH) respectively. VH scenarios were encoded in PDDL McDermott et al. (1998)
for use with our inverse planning system, but they can be easily be translated
to VirtualHome’s evolving graph simulator or Unity simulator Puig et al.
(2018).

Table A1. Goal assistance problems in Doors, Keys & Gems. 30 problems were divided into those that required assistance with either picking up keys or unlocking doors. Actions and utterances were varied according to the following factors. Factor | Condition |  Description
---|---|---
Instruction Uncertainty | Ambiguous |  Ambiguous instruction that can be associated with many possible items. (e.g. _“Can you get me the red key?”_ when multiple red keys are available.)
Partial |  The principal only requests a subset of the items they require to achieve their goal. (e.g. _“Can you get me the red key?”_ when both red and blue keys to reach its goal.)
Agent Involvement | Assistant Actions |  The instruction involves only actions that the assistant should take. (e.g. _“Can you pick up that key?”_ , _“Please unlock the red doors.”_)
Joint Actions |  The instructions contain information about both agents’ actions. (e.g. _“I’m getting the red key, can you get the blue key?”_)
Goal Uncertainty | Low Uncertainty |  The actions and instructions combined are largely sufficient to uniquely determine the goal. (e.g. In Figure 4(b), only the red gem is likely.)
High Uncertainty |  The principal’s goal remains ambiguous even after observing both their actions and instructions. (e.g. In Figure 4(d), the three gems on the left are plausible goals.)
Table A2. Goal assistance problems in VirtualHome. 25 problems were divided
into (a) three scenarios, which correspond to different initial states of the
environment, and had different sets of possible goals. Actions and utterances
(b) were varied across several conditions.

Scenario | Goal Category ($k$) | Description
---|---|---
Food Preparation | Salads (3) | Place a chef’s knife & ingredients (onion, cucumber/potato & tomato/chicken) on the table.
(6 goals) | Stews (3) | Place an onion, carrot, a main (salmon, chicken, or potato) & wine on the table.
Table Setting | Plates (4) | Place $n$ plates on the table, where $n\in[1,4]$.
(12 goals) | Plates & Cutlery (4) | Place $n$ plates, $n$ forks & $n$ knives on the table, where $n\in[1,4]$.
| Plates & Bowls (4) | Place $n$ plates, $n$ forks, $n$ knives & $n$ bowls on the table, where $n\in[1,4]$.
Drink Pairing | Wine (3) | Place wine and $n$ wineglasses on the table, $n\in[1,3]$.
(12 goals) | Wine Pairing (3) | Place wine, cheese, $n$ wineglasses & $n$ forks on the table, $n\in[1,3]$.
| Juice (3) | Place juice & $n$ waterglasses on the table, $n\in[1,3]$.
| Juice Pairing (3) | Place juice, $n$ waterglasses, $n$ cupcakes & $n$ forks on the table, $n\in[1,3]$.
(a) Scenarios and goals.

Factor | Condition |  Description
---|---|---
Instruction Uncertainty | Ambiguous Number |  Instruction which mentions an ambiguous number of items to be collected. (e.g. _“Can you help with these plates?”_)
Ambiguous Type |  Instruction which is ambiguous about the type of item to be collected. (e.g. _“Can you get two glasses?”_ , _“And I need the veggies for the stew.”_)
Utterance Count | Single |  The principal gives a single instruction to the assistant. (e.g. _“Go get the forks and knives.”_ , _“Can you get some wine?”_)
Multiple |  The principal gives an instruction to the assistant, takes a few more actions, then gives another instruction. (e.g. _“Could you find two glasses?”_ followed by _“Get some forks as well.”_)
(b) Factors and conditions.

##  Appendix B Model Configuration

In this section, we provide further details about how CLIPS and each of our
baselines was configured for each domain.

###  B.1. CLIPS & Unimodal Inverse Planning

We describe the configuration for CLIPS and unimodal inverse planning
together, since the only difference between the former and the latter is
whether both action likelihoods $P(a^{{}_{h}}_{t}|s_{t},\pi)$ and utterance
likelihoods $P(u_{t}|s_{t},\pi)$ are incorporated into the weight $w$
associated with each hypothesized goal $g$, cost function $C$ and policy $\pi$
in Algorithm 1.

#### Goal and Cost Priors

In both domains, we assumed a uniform goal prior $P(g)$ over the set of
possible goals (4 in mDKG, 6–12 in VH). In mDKG, we considered four possible
cost functions $C\in\mathsf{C}$, and placed a uniform prior $P(C)$ over such
cost functions. This made CLIPS more robust to the possibility that the human
would prefer to do less (or more) work, asking the robot to pick up keys or
unlock doors that would be sub-optimal if action costs were equal between both
agents. The costs for each action were as follows:

  * •

Human pick-up: 5. Robot pick-up: 1. Wait cost: 0.6. All other action costs: 1.

  * •

Human pick-up: 5. Robot pick-up: 1. Wait cost: 0.6. Other human costs: 2.
Other robot costs: 1.

  * •

Human pick-up: 5. Robot pick-up: 1. Robot unlock: 5. Wait cost: 0.6. All other
action costs: 1.

  * •

Human pick-up: 5. Robot pick-up: 1. Robot unlock: 5. Wait cost: 0.6. Other
human costs: 2. Other robot costs: 1.

In VH, we avoided the need the perform online inference over multiple cost
functions by using a slightly more flexible utterance model that allowed for
the possibility that the principal might ask the assistant to carry out sub-
optimal actions (e.g. asking the assistant to acquire _all_ remaining items,
even though the optimal joint plan would have both agents share the work more
equally). This allowed us to use a _single_ cost function that assigned unit
cost to all actions for both agents. Future work should investigate how more
flexible models for multi-agent task division (e.g. Wu et al. (2021)) could be
integrated with CLIPS, along with offline learning of cost function priors.

#### Policy Configuration

In Bayesian inverse planning, an algorithm is required to compute the joint
policy $\pi$ for each possible goal. While it is possible to do this using a
domain-general planning algorithm and search heuristic, we configured our
planners in a domain-specific manner to achieve greater speed and reliability.

##### Doors, Keys & Gems

In mDKG, we used the Real-Time Adaptive A* (RTAA*) variant Koenig and
Likhachev (2006) of real-time heuristic search Korf (1990) to incrementally
compute the value function $V(s)$ for the policy $\pi$. Given $V(s)$, one can
then compute $Q(s,a)=C(s,a)+\sum_{s^{\prime}}P(s^{\prime}|s,a)V(s^{\prime})$.
Each time rths-policy-update is called, we perform A* search up to a search
budget of $2^{18}$ nodes from each state $s_{n}$ that neighbors the current
state $s$, using the current estimate $\hat{V}$ of the value function as the
search heuristic. Once A* search terminates at state $s_{f}$ (either by
reaching the goal or using up the budget), we update the value function for
_all_ states $s_{i}$ in the interior of the search tree as follows:

$\hat{V}(s_{i})\leftarrow\hat{V}(s_{f})-C(s_{n},s_{f})+C(s_{n},s_{i})$

where $C(s_{n},s_{i})$ is the cost of getting from $s_{n}$ to $s_{i}$ by
traversing the search tree. As shown in Koenig and Likhachev (2006), as long
as $\hat{V}$ is initialized to a _consistent_ (under-)estimate of the true
value function, subsequent $\hat{V}$ values will remain consistent. We ensured
this by (lazily) initializing $\hat{V}$ with a minimum spanning tree heuristic
Held and Karp (1971) over the graph of keys and doors required to reach the
goal, using a preprocessing step used in inventory-aware pathfinding Aversa et
al. (2016) to extract necessary keys and doors. To further accelerate
planning, we pruned actions that were irrelevant to the goal (e.g. picking up
unnecessary keys), and implemented cost-minimal path reuse from the tree-
adaptive variant of RTAA* Hernández et al. (2015), allowing us to terminate A*
search whenever an existing path to the goal was found. Collectively, these
algorithmic choices allowed us to estimate the optimal value function $V^{*}$
in close to real-time for most timesteps, despite the difficulty of optimal
multi-agent planning.

##### VirtualHome

In VH, we also experimented with using RTAA* to compute the policy $\pi$.
However, this unpredictably increased the latency of inverse planning when the
search budget was too high, and led to unstable $Q$-value estimates when the
search budget was too low. Instead we found that it was sufficient to compute
an analytic lower bound $\hat{V}$ to the optimal value function $V^{*}$ by
counting the number of remaining items and movements required to achieve the
goal. Using $\hat{V}$ was both faster (no search was required), and produced
more consistent estimates of the relative $Q$-values $Q(s,a)-V(s)$ (also known
as the _advantage function_) for each state $s$ and goal $g$, resulting in
more stable action likelihoods $P(a|s,\pi)$ and goal inferences. However, this
required hand-deriving a lower bound on $V^{*}$. In the future, we hope to
explore the use of domain-general lower bounds on $V^{*}$ using admissible
planning heuristics Haslum et al. (2005); Helmert and Domshlak (2009), and to
implement more consistent value function updates based on the LSS-LRTA*
variant of real-time heuristic search Koenig and Sun (2009), allowing us to
stably estimate $V^{*}$ with a low search budget.

##### Rationality Modeling

As in hierarchical Bayesian approaches to inverse reinforcement learning Choi
and Kim (2014) and theory-of-mind Zhi-Xuan et al. (2022), we placed a Gamma
prior over the rationality parameter $\beta$ for the Boltzmann policy,
discretized across 33 values from $2^{-3}$ to $2^{5}$ by incrementing the
exponent with a step size of $0.25$. In mDKG, we used a $\text{Gamma}(0.5,1)$
prior, while in VH, we used a $\text{Gamma}(1,2)$ to match the less optimal
actions taken in that domain. For efficient inference, we used Rao-
Blackwellization Murphy and Russell (2001), marginalizing over all values of
$\beta$ when computing the action likelihood $P(a^{{}_{h}}|s_{t},\pi)$, then
analytically updating the local posterior
$P(\beta|a^{{}_{h}}_{1:t},s_{1:t},\pi,g)$.

#### Utterance Model

In the CLIPS utterance model, salient actions $\alpha_{1:m}$ are first
extracted from the joint policy $\pi$, before being sub-sampled into a command
$c$, which is then translated into an utterance $u$. In our implementation, we
enumerated over commands to reduce variance. This section describes how we
extracted salient actions, the structure of commands, and the command
enumeration strategy for each domain.

##### Extracting Salient Actions

Following work in linguistics Giora (2004); Kecskes et al. (2004) and RSA
theory Goodman and Frank (2016), we assumed that some types of actions were
more likely to be communicated because of their _salience_ to both the speaker
and the listener. To model this, after rolling out the joint policy $\pi$ for
up to $H$ steps, we filtered out all non-salient actions, leaving only the
salient actions $\alpha_{1:m}$ to be communicated. In VH, to account for the
possibility of the principal delegating _all_ remaining items to the assistant
(instead of equally sharing the load), we enforced that the principal could
not collect new items when rolling out the policy.

As in Ying et al. (2023b), we manually defined action salience: In mDKG,
pickup, unlock, and handover actions and iscolor predicates were salient,
while in VH, grab actions and itemtype predicates were salient. In general,
however, action salience could be learned, either from a corpus, or by offline
estimation of which actions tend be more _unique_ and hence _informative_ for
a particular goal and plan.

##### Command Enumeration

Formally, we define a command $c:=([a,...],\\{p,...\\},\\{(v,\tau),...\\})$ as
a sequence of (possibly lifted) actions $[a,...]$, a set of associated
predicates $\\{p,...\\}$ that qualify an action’s arguments, and a set
$\\{(v,\tau),...\\}$ of free variables $v$ with associated types $\tau$ which
appear as an argument in at least one action in $[a,...]$. For example, the
command (handover robot human ?key1) where (iscolor ?key1 blue) has one
partially lifted action $a=$ (handover robot human ?key1), along with a
predicate $p=$ (iscolor ?key1 blue) that describes the variable $v=$ ?key1 as
blue, where the type of ?key1 is $\tau=$ key.

To generate commands from a set of a salient actions $\alpha_{1:m}$ and
associated predicates $p_{1:m}$, we enumerated over all subsets of salient
action-predicate pairs of up to size $K=3$ in mDKG and $K=4$ in VH, before
pruning away combinations of actions that were less likely. In mDKG, we did
this by excluding commands with more than two distinct action types, commands
where _all_ actions were taken by the speaker/principal, and also commands
which involved chains of dependent actions (e.g. picking up a key, followed by
handing over the same key). In VH, we excluded all commands that involved
_any_ action by the speaker (since our VH dataset did not contain joint
instructions). Following enumeration of these (ground) commands, we then
lifted the commands, replacing concrete object names with corresponding typed
variables. We also replaced all agent identifiers such as human and robot with
corresponding indexicals (me and you). Duplicate lifted commands were then
removed.

These choices were made partly to limit the number of commands enumerated.
Future extensions to CLIPS that make use of constrained LLM decoding Lew et
al. (2023b); Shin et al. (2021) could avoid the limitations involved in
command enumeration, directly translating an utterance $u$ into a much larger
space of possible commands (e.g. a context-free grammar over commands).

##### Language Models

In mDKG, we used OpenAI’s 6.7B curie model Brown et al. (2020) as our
LLM555Unfortunately, as of 4 January 2024, OpenAI has shutdown access to curie
and the original GPT-3 family of models, making it no longer possible to
reproduce our results exactly. Nonetheless, we expect similar results when
using davinci-002 as a replacement model. , while in VH, we used their more
recently released davinci-002 model. Both of these were base models, providing
more diffuse probabilities that avoid issues with over-confidence and mode
collapse that can arise in LLMs finetuned on instructions or human preferences
Hopkins et al. (2023); Kirk et al. (2023).

LLMs were instantiated as generative functions in the Gen probabilistic
programming system Cusumano-Towner et al. (2019) using the GenGPT3.jl library
Zhi-Xuan (2022a), allowing us to easily score the probability
$P(u|c,\mathcal{E})$ of an utterance $u$ given a command $c$ and few-shot
examples $\mathcal{E}$ provided in the prompt. Each utterance was scored
against all enumerated commands in parallel via a batched API request. In
mDKG, we provided 18 examples of commands being translated to utterances,
while in VH, we provided 12 examples. Examples can be found in the source
code.

#### Assistance Policy

When running the $Q_{\text{MDP}}$ assistance policy shown in Algorithm 2, we
filtered out goal hypotheses with a weight $w^{i}$ of less than $0.02$, so as
to avoid the cost of updating the policies $\pi^{i}$ for unlikely goals. In
mDKG, we updated the policies using RTAA* with a node budget $2^{1}6$ and a
time limit of $10\text{s}$. In VH, we updated $Q$-values using the analytic
lower bound $\hat{V}$ mentioned earlier.

We evaluated the assistance policy against a simulated human principal, which
was designed to (i) follow a “ground truth” plan whenever possible, (ii)
switch to following an approximately optimal joint policy when the following
the original plan was not possible, (iii) fallback to a single-agent policy
upon detection that the assistant was not following the joint policy to the
true goal (i.e. not being helpful). To detect this, the principal compared how
likely the assistant was to be following a joint policy to the true goal
(Boltzmann, with $\beta=1$) vs. a random baseline, switching to the fallback
policy once it was 10 times more likely for the assistant to be acting at
random.

###  B.2. LLM-Based Literal Listener

Our literal listener baselines consisted of two components: literal command
inference, and literal assistance. We describe each below.

#### Literal Command Inference

Command inference was implemented similarly to inference over the CLIPS
utterance model. We first enumerated over a set of possible commands in the
current state $s$, then used a LLM to score the probability $P(u|c,s)$ of the
observed utterance $u$ for each command $c$. Normalizing these probabilities
produced a posterior over commands $P(c|u,s)$.

##### Utterances

In mDKG, we made command inference easier by replacing joint instructions
(e.g. _“I’m getting the red key, can you get the blue key?”_) with assistant-
directed instructions (e.g. _“Can you get the blue key?”_). In VH, we handled
multiple utterances $u_{t_{1}}$, $u_{t_{2}}$ by merging the inferred commands
$c=c_{t_{1}}\cup c_{t_{2}}$. This created a product distribution
$P(c|u_{t_{1}},u_{t_{2}},s_{t_{1}},s_{t_{2}})=P(c_{t_{1}}|u_{t_{1}},s_{t_{1}})P(c_{t_{2}}|u_{t_{2}},s_{t_{2}})$
over merged commands.

##### Command Enumeration

To enumerate commands, we first enumerated all salient actions that were
reachable from the current state $s$. We then enumerated all subsets the set
of salient actions of up to size $K=3$ in mDKG an $K=4$, pruning away certain
combinations as in the CLIPS utterance model. Commands were then lifted, and
duplicate commands were removed.

##### Language Models

We used text-davinci-002 for mDKG. Given the larger space of commands for the
literal listener, we found that using a RLHF finetuned model such as text-
davinci-002 ensured that more posterior probability mass concentrated on the
intuitively correct set of commands. To simulate this effect in our VH
experiments after text-davinci-002 access was shutdown by OpenAI, we used
davinci-002 with a temperature of 0.5. 12 few-shot examples were used for both
mDKG and VH.

#### Literal Assistance

Literal assistance involved several steps: (i) Systematically sampling a
command $c$ from the inferred distribution $P(c|u,s)$; (ii) Converting the
command $c$ to an equivalent goal formula $g_{c}$; (iii) Generating an
assistive plan that first achieved $g_{c}$, then the true goal $g$.

##### Sampling Commands

As described in the main text, we estimated the _mean_ performance of literal
assistance by drawing $m=10$ samples from the distribution $P(c|u,s)$ and
averaging our metrics across these samples. To do this in a low-variance
manner, we first sorted the commands by their inferred probabilities. We then
used systematic sampling Yates (1948), selecting a command $c_{i}$ as long the
random variable $x_{i}$ fell within its corresponding probability bin, where
$x_{i}=(i-1)/m+x$ for $i\in[1,m]$, and $x\sim U(0,1/m)$.

##### Mapping Commands to Goals

Given a sampled command $c=([a,...],\\{p,...\\},\\{(v,\tau),...\\})$, we
mapped each action $a$ to a set of goal predicates $\\{p_{a},...\\}$ that it
achieves. By combining these predicates with the qualifier predicates
$\\{p,...\\}$, then adding an existential quantifier for any variables, we
created a PDDL goal formula $g_{c}$ that could be used by a planning
algorithm. For example, the command (handover robot human ?key1) where
(iscolor ?key1 blue) would be converted to the goal formula (exists (?key1 -
key) (and (pickedup-by robot ?key1) (has human ?key1) (iscolor ?key1 blue))).

Some commands had multiple possible groundings. In mDKG (but not in VH), these
groundings corresponded to plans with different lengths. As such, we
implemented both a _naive_ and _efficient_ form of literal assistance. In
naive assistance, we generated plans for all possible groundings of the lifted
command $c$, and averaged performance metrics across these plans. For example,
if key2 was blue, then a grounding of (handover robot human ?key1) where
(iscolor ?key1 blue) might be (handover robot human key2), which translates to
the the goal (and (pickedup-by robot key2) (has human key2)). In efficient
assistance, we simply generated a plan for existential goal formula implied by
the lifted command, automatically finding the most efficient way to satisfy
that formula.

##### Plan Generation

We generated an assistive plan in two phases. In the first phase, we generated
an optimal _joint_ plan to the commanded goal $g_{c}$ using A* search with a
node budget of $2^{16}$ (for commands with handover actions, this might
involve both agents walking towards a convenient meeting point). If finding a
joint plan was unsuccessful, we fell back to computing a plan where only the
assistant was allowed to move. In the second phase, we then enforced that only
the principal was allowed to move (in mDKG) or pick up items (in VH), and
computed the remaining plan to achieve the principal’s true goal $g$ (again,
using A* search with a node budget of $2^{16}$).

###  B.3. Multimodal LLM (GPT-4V)

We used the public preview of GPT-4V (gpt-4-1106-vision-preview) as a baseline
model in our mDKG experiments. We used zero-shot prompting, providing the
model with effectively the same instructions as given to the human raters,
along with the final frame of the GIF animation shown to humans (Figure B1).
Responses were generated with a temperature of 0.

You’re watching a human and a robot collaborating on a treasure game shown
below. The human player wants to collect exactly one of the four target gems.
The two players cooperate to achieve the human’s goal. During the game, the
human may communicate with the robot to coordinate their actions. Any
communication that happens is shown below the visuals. The rules of the game
are as follows: • The human player’s goal is to collect one target gem. • The
robot plays an assistive role, and cannot pick up gems on its own. • The
players can move on the white squares. • The players have a full view of the
map at all times. • Keys are used to unlock doors with the same color (e.g.
red keys only unlock red doors). • Each key can only be used once. Keys
disappear after being used. • One player can pass keys to the other if they’re
next to each other. • Players can occupy the same square and walk past each
other. • The keys and doors are labeled. The labels are shown on the top right
corner of each cell. stimulus.png In the scenario above, the human player took
several actions, then gave the instruction: ”Can you hand me that key?” Given
this information, what gem is most likely to be the human agent’s goal? Which
key(s) should the robot pick up to best assist the human? Please respond in
the following JSON format: { ”goal”: [red/yellow/blue/green], ”assist”: [key1,
key2, etc.] }

Figure B1. GPT-4V zero-shot prompt for an example key assistance problems in
Doors, Keys & Gems.

##  Appendix C Human Experiments

Here we provide additional details about our human experiments for the mDKG
domain. Participants first completed a tutorial that explained the task and
experimental interface, then answered 5 comprehension questions before
proceeding to the main experiment. In the main experiment, they were shown 15
stimuli in randomized order, which involved either keys or doors as assistance
options (Figure C1).

To incentivize accurate but calibrated responses, participants were rewarded
for accurately guessing the true goal or selecting the optimal assistance
option. Specifically, they earned $1/N$ bonus points if they selected $N$
goals out of which one was the true goal, but $0$ points if none of their
selected goals was the true goal. They earned one additional point if they
selected exactly the right assistance options. Participants were paid US$1 for
every 30 bonus points they earned, on top of a base pay of US$16/hr (US$4 per
experiment).

(a) Key assistance problem.

(b) Door assistance problem.

Figure C1. Web interface for human experiments. Participants had to select
checkboxes for all gems they thought were likely to be human principal’s goal,
and select all keys or doors they thought the robot assistant should pick-up
or unlock.

(a) Goal Inference: Human ($y$-axis) vs. model ($x$-axis) goal inferences.

(b) Assistance: Human ($y$-axis) vs. model ($x$-axis) assistance judgments.

Figure C2. Human-model correlation plots for Doors, Keys & Gems. In (a), we
show scatter plots comparing average human goal inferences vs. model goal
inferences across all stimuli. In (b), we compare average human assistance
judgments vs. model outputs by overlaying a scatter plot over a binarized
confusion matrix. Error bars correspond to 95% confidence intervals for the
mean human response. At the bottom-right of each plot we report Pearson’s $r$
(95% bootstrap CI below).

In Figure C2, we show correlation plots that provide a more comprehensive view
of how well each model matches human responses. As Figure C2a shows, CLIPS
best reproduces human goal inferences. In contrast, unimodal inverse planning
tends to be under-confident relative to humans, since they only incorporate
evidence from one modality rather than two. GPT-4V tends to be over-confident
on stimuli where humans are not (and vice versa), resulting in a very low
correlation overall.

To compare human assistance judgments with model outputs, Figure C2b shows
scatter plots laid over binarized confusion matrices, generated by
thresholding human and model responses at 0.5, then plotting bin counts as a
heatmap. Since many of our models return either 1 or 0 for assistance, this
provides a better way to visualize human-model agreement. CLIPS best matches
human judgments, followed by language only inverse planning. The other models
often fail to select an assistance option in cases where humans do (deeper
colored top-left quadrants). GPT-4V selects options in many cases where humans
do not (bottom-left quadrant), achieving the lowest correlation.

##  Appendix D GPT-4V Responses

To better illustrate the capabilities and limitations of multimodal LLMs, we
show several example responses by GPT-4V in Figure D1.

(a) Ambiguous predicates.

GPT4V Response:
Based on the image and the rules provided, the human player is asking for the
red key, which suggests that their goal is to collect the red gem. The red gem
is behind the red door labeled ”1,” and the red key is required to unlock it.
Since the human player can only collect one target gem, and they are asking
for the red key, it is most likely that the red gem is their goal. The robot
should pick up the red key to assist the human player. The robot cannot pick
up gems, so it only needs to focus on the keys that will help the human player
achieve their goal. Here is the response in the requested JSON format: {
”goal”: [”red”], ”assist”: [”key1”] }

(b) Ambiguous number.

GPT4V Response:
Based on the image and the information provided, the human player has asked
the robot to pass the red keys. This implies that the human player is aiming
for a gem that requires passing through a red door. In the image, the red gem
is behind a red door, so it is the most likely target for the human player.
The robot should pick up the keys that will allow the human to reach the red
gem. The red gem is behind door 3, which requires key 3. However, to reach key
3, the robot first needs to unlock door 1 with key 1. Therefore, the robot
should pick up key 1 and then key 3 to assist the human player. Here is the
response in the requested JSON format: { ”goal”: [”red”], ”assist”: [”key1”,
”key3”] }

(c) Joint instructions.

GPT4V Response:
Based on the information provided and the layout of the game board, the human
player has expressed an intention to pick up the blue key. This key can unlock
the blue door, which is blocking the path to the green gem. Since the human
player’s goal is to collect one target gem and they are going for the blue
key, it is reasonable to assume that the green gem is their intended target.
The robot, in an assistive role, has been asked to pick up a red key. There
are two red keys (key4 and key5) and two red doors on the map. However, since
the human is going for the green gem, the robot should pick up the red key
that will assist in this goal. Key4 is the red key that unlocks the red door
on the path to the green gem. Therefore, the robot should pick up key4. Here
is the JSON response with the likely goal and the key the robot should pick
up: { ”goal”: [”green”], ”assist”: [”key4”] }

Figure D1. Example responses by GPT-4V. GPT-4V produces two paragraphs of
reasoning text, then gives its final response in JSON.

As can be seen from Figure D1, GPT-4V is highly inaccurate in its ability to
perceive objects in the provided image and determine their spatial
relationships, sometimes confabulating object labels that do not exist (Fig.
D1a–b), failing to notice which gems are locked behind which doors (Fig.
D1b–c), and failing to perceive which items are along a path to the goal (Fig.
D1c). This weakness at spatial perception makes current multimodal LLMs
inadequate for assistive tasks like ours, which require coherent spatial
reasoning.

However, the reasoning text produced by GPT-4V appears to demonstrate internal
means-ends consistency (keys are required to unlock doors of the same color,
which in turn allow the human to reach the gems behind them). As prior work
has shown Sclar et al. (2023), this suggests that if an abstract symbolic
representation is provided to a state-of-the-art LLM with the appropriate
high-level relations (spatial connectivity etc.), they may be able to perform
significantly better. Nonetheless, deriving an abstract symbolic
representation from the underlying gridworld is itself a form of complex
spatial reasoning Aversa et al. (2016), which likely presents a challenge to
contemporary LLMs.

A full set of GPT-4V’s responses can be found in the Supplementary Information
(https://osf.io/v8ru7/).

##  Appendix E Alternate Assistance Methods

In the main text, we reported results for an _offline_ version of the
$Q_{\text{MDP}}$ assistance policy in Algorithm 2, which keeps the goal
posterior fixed after observing $a_{1:T}^{h}$ and $u_{1:T}$, rather than
continuing to update the posterior as the human principal takes more actions.
Here we report results for the _online_ version of $Q_{\text{MDP}}$
assistance. We also report results for an alternative assistance method,
$\bar{\pi}_{\text{MDP}}$, which is constructed by averaging over _policies_
$\pi$ for each inferred goal $g$: Rather than minimizing the expected
$Q$-value as in $Q_{\text{MDP}}$ assistance Littman et al. (1995),
$\bar{\pi}_{\text{MDP}}$ first samples a goal $g$ from the posterior, then
follows the policy $\pi$ for that goal. This can be viewed as Thompson
sampling Thompson (1935) over policies.

(a) Uncertain Goals. When goal uncertainty is high, $Q_{\text{MDP}}$
assistance acts confidently to to minimize expected goal achievement cost,
unlike $\bar{\pi}_{\text{MDP}}$ assistance, which is less confident.

(b) Ambiguous Indexicals. Given uncertainty over the goal,
$\bar{\pi}_{\text{MDP}}$ randomizes over which door is most efficient to
unlock. This leads to a higher chance of failure to reach the green gem.

(c) Safe Assistance. When the most efficient plan for the most likely goal
causes another goal to be unreachable, $\bar{\pi}_{\text{MDP}}$ is less safe
than $Q_{\text{MDP}}$ assistance, which acts to ensure that both likely goals
are reachable.

(d) Failed Assistance. In the one mDKG problem where CLIPS produces inaccurate
goal inferences, $Q_{\text{MDP}}$ assistance fails, whereas
$\bar{\pi}_{\text{MDP}}$ sometimes assists the human appropriately by
randomizing over goals.

Figure E1. Examples comparing (offline) $Q_{\textmd{MDP}}$ and $\bar{\pi}_{\textmd{MDP}}$ assistance in Doors, Keys & Gems. In $Q_{\textmd{MDP}}$ assistance, we repeatedly take the best action that minimizes the expected goal achievement cost with respect to the posterior over goal specifications. In $\bar{\pi}_{\textmd{MDP}}$ assistance, we randomly select a single goal specification $g$ from the posterior, and follow the optimal joint policy for that goal. Table E1. Comparison of assistance methods measured in terms accuracy (precision and recall for assistance options), helpfulness (plan length and human cost relative to CLIPS with offline $Q_{\text{MDP}}$ assistance), and human similarity (correlation of assistance options with mean human ratings) Metrics are averaged across the dataset per domain, with standard errors reported in brackets. |  | Accuracy | Helpfulness | Human Sim.
---|---|---|---|---
Inference | Assistance | Assist. Prec. | Assist. Rec. | Rel. Plan Length | Rel. Human Cost | Assist. Cor.
Doors, Keys & Gems
CLIPS |  $Q_{\text{MDP}}$ (Offline) | 0.97 (0.03) | 0.97 (0.03) | 1.00 (0.00) | 1.00 (0.00) | 0.96 (0.01)
Lang. Only IP |  $Q_{\text{MDP}}$ (Offline) | 0.90 (0.06) | 0.83 (0.06) | 1.26 (0.10) | 1.18 (0.07) | 0.83 (0.01)
Action Only IP |  $Q_{\text{MDP}}$ (Offline) | 0.43 (0.09) | 0.40 (0.09) | 1.68 (0.15) | 1.46 (0.08) | 0.45 (0.01)
CLIPS |  $Q_{\text{MDP}}$ (Online) | 1.00 (0.00) | 1.00 (0.00) | 0.99 (0.01) | 0.99 (0.00) | 0.97 (0.00)
Lang. Only IP |  $Q_{\text{MDP}}$ (Online) | 0.90 (0.06) | 0.83 (0.06) | 1.26 (0.10) | 1.18 (0.07) | 0.83 (0.01)
Action Only IP |  $Q_{\text{MDP}}$ (Online) | 0.88 (0.06) | 0.90 (0.06) | 1.15 (0.07) | 1.09 (0.03) | 0.83 (0.01)
CLIPS | $\bar{\pi}_{\text{MDP}}$ | 0.91 (0.03) | 0.91 (0.03) | 1.09 (0.04) | 1.08 (0.02) | 0.96 (0.00)
Lang. Only IP | $\bar{\pi}_{\text{MDP}}$ | 0.79 (0.05) | 0.77 (0.05) | 1.31 (0.09) | 1.19 (0.04) | 0.85 (0.01)
Action Only IP | $\bar{\pi}_{\text{MDP}}$ | 0.72 (0.04) | 0.41 (0.05) | 1.58 (0.12) | 1.45 (0.05) | 0.58 (0.01)
VirtualHome
CLIPS |  $Q_{\text{MDP}}$ (Offline) | 0.87 (0.04) | 1.00 (0.00) | 1.00 (0.00) | 1.00 (0.00) | –
Lang. Only IP |  $Q_{\text{MDP}}$ (Offline) | 0.59 (0.05) | 0.96 (0.03) | 1.33 (0.07) | 1.35 (0.08) | –
Action Only IP |  $Q_{\text{MDP}}$ (Offline) | 0.61 (0.07) | 0.84 (0.07) | 1.30 (0.07) | 1.33 (0.07) | –
CLIPS |  $Q_{\text{MDP}}$ (Online) | 0.90 (0.04) | 1.00 (0.00) | 1.12 (0.15) | 1.13 (0.17) | –
Lang. Only IP |  $Q_{\text{MDP}}$ (Online) | 0.59 (0.05) | 0.96 (0.03) | 1.33 (0.07) | 1.35 (0.08) | –
Action Only IP |  $Q_{\text{MDP}}$ (Online) | 0.84 (0.06) | 0.92 (0.06) | 1.03 (0.04) | 1.04 (0.05) | –
CLIPS | $\bar{\pi}_{\text{MDP}}$ | 0.59 (0.06) | 0.65 (0.06) | 1.22 (0.06) | 1.33 (0.07) | –
Lang. Only IP | $\bar{\pi}_{\text{MDP}}$ | 0.45 (0.05) | 0.57 (0.06) | 1.37 (0.08) | 1.48 (0.09) | –
Action Only IP | $\bar{\pi}_{\text{MDP}}$ | 0.46 (0.06) | 0.45 (0.05) | 1.61 (0.14) | 1.77 (0.16) | –

##### Qualitative analysis

Figure E1 illustrates how (offline) $Q_{\text{MDP}}$ assistance differs from
$\bar{\pi}_{\text{MDP}}$. While $\bar{\pi}_{\text{MDP}}$ assistance better
reproduces the gradedness seen in human assistance judgments, this comes at
the cost of reduced safety and helpfulness. By following a policy $\pi$ to
goal $g$ according to how likely $g$ is, $\bar{\pi}_{\text{MDP}}$ fails to
aggregate information across multiple uncertain goals when deciding the best
action to take. This is most clearly illustrated in Figure E1c, where CLIPS
assigns higher probability to the blue gem over the green gem.
$Q_{\text{MDP}}$ assistance safely unlocks _Door 3_ instead of _Door 2_ ,
since doing so will ensure that both the blue gem and green gem are reachable.
In contrast, $\bar{\pi}_{\text{MDP}}$ unlocks _Door 2_ on average 66% of the
time, which is unsafe. However, as Figure E1d shows, $\bar{\pi}_{\text{MDP}}$
is slightly better when CLIPS produces the wrong goal inferences (due to the
sub-optimality of the human’s actions with respect to the true goal). In this
single failure case, $Q_{\text{MDP}}$ confidently executes the wrong actions,
whereas $\bar{\pi}_{\text{MDP}}$ assistance is sometimes successful.

##### Quantitative analysis

We present the performance of different assistance methods in Table E1.
Compared to offline $Q_{\text{MDP}}$ assistance, $\bar{\pi}_{\text{MDP}}$
assistance was less accurate, and produced longer plans on average. This
highlights the importance of minimizing expected cost under uncertainty,
rather than just sampling a likely goal to follow. As might be expected,
online $Q_{\text{MDP}}$ assistance was more accurate, since the posterior over
goals was updated as new actions were observed. This significantly improved
the performance of action-only inverse planning (note that language-only
inverse planning is the same regardless of assistance method, since the human
principal communicates no further instructions). However, multimodal goal
inference via CLIPS combined with online $Q_{\text{MDP}}$ assistance was still
the most accurate, since it leveraged linguistic information from the
beginning of assistance.

Apart from one failure case in VirtualHome where the assistant and simulated
human principal were caught in a livelock, CLIPS with online $Q_{\text{MDP}}$
assistance was slightly more helpful on average than offline assistance.
Nonetheless, offline assistance was almost as helpful. This suggests that once
enough actions are observed to disambiguate the principal’s instruction, CLIPS
can still assist effectively even if it has partial to no observability of the
principal’s subsequent actions.