# Language Models, Agent Models, and World Models:
The LAW for Machine Reasoning and Planning

Zhiting Hu (UCSD), Tianmin Shu∗ (JHU)
<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal contribution

###### Abstract

Despite their tremendous success in many applications, large language models
often fall short of consistent reasoning and planning in various (language,
embodied, and social) scenarios, due to inherent limitations in their
inference, learning, and modeling capabilities. In this position paper, we
present a new perspective of machine reasoning, Law, that connects the
concepts of language models, agent models, and world models, for more robust
and versatile reasoning capabilities. In particular, we propose that world and
agent models are a better abstraction of reasoning, that introduces the
crucial elements of deliberate human-like reasoning, including beliefs about
the world and other agents, anticipation of consequences, goals/rewards, and
strategic planning. Crucially, language models in Law serve as a backend to
implement the system or its elements and hence provide the computational power
and adaptability. We review the recent studies that have made relevant
progress and discuss future research directions towards operationalizing the
Law framework.

##  1 Introduction

Large language models (LLMs) are among the most powerful intelligent machines
people have built to date. They are adept at generating natural language
continuations from a given text (or multi-modal) input. Natural language is a
flexible means for humans to describe the world, express thoughts, and
communicate with each other. LLMs, trained with the vast text humans have ever
produced, inherit much of the knowledge conveyed through natural language,
including the causal structure of the world (expressed in phrases like “a
bottle is pushed, water pours out”), reasonings about various subjects,
scientific theories, beliefs, cultural norms, etc.

On the other hand, LLMs often fall short of consistent reasoning and planning
and sometimes fail surprisingly in tasks that humans find easy. Figure 1 shows
such examples in different reasoning scenarios. These failure examples
highlight several fundamental limitations of machine reasoning based on LLMs:

First, natural language text is often ambiguous and imprecise. One of the key
reasons for this ambiguity and imprecision is that the rich context, which
humans rely on when producing the text, is often missing. This context
includes the specific perceptual and social situations the human agents were
in, their mental states (e.g., intentions, beliefs, and thinking processes),
and world commonsense. Thus LLMs, which learn only to simulate the surface
text without modeling the underlying context, lack grounding on the physical,
social, and mental experiences.

Another core limitation of LLMs arises from the inefficiency of language as
the medium for carrying out reasoning in certain situations (Figure 1,
embodied reasoning). For instance, articulating all subtle differences between
two leaves might require an extensive text paragraph. In contrast, generating
an image that visually represents these leaves can be far more efficient,
requiring just a few pixels. Similarly, using other sensory modalities (e.g.,
videos) is often more straightforward than relying on language to describe
intuitive physics, such as predicting fluid flow based on its viscosity and
the surrounding obstacles.

Figure 1: LLMs, such as GPT4, fail in simple tasks of language, embodied, and
social reasoning. Erroneous parts in the answers are highlighted in  red.

These limitations are further exacerbated by the inference procedures of LLMs.
They reason by generating text autoregressively, token-by-token, from left to
right in a single pass, reminiscent of humans’ System-I intuitive thinking.
Humans’ System-II reasoning stands in stark contrast to LLM reasoning. In
particular, humans possess a mental model of the world. The “world model” in
our minds enables us to simulate actions and their effects on the world’s
state for robust reasoning during complex tasks (Tolman, 1948; Briscoe, 2011;
Battaglia et al., 2013; Allen et al., 2020; Pramod et al., 2020). For example,
when planning to achieve a goal, we use our internal world model to think
about different actions we could take and predict possible outcomes for each
choice. This prediction of outcomes in turn helps refine the action plan for
better attaining the goal. This decision-making process is governed by an
“agent model” on top of the world model. Further, in social reasoning tasks,
human agents additionally use their beliefs about other agents. For example,
during a conversation, an agent needs to infer others’ intentions and their
potential reactions to decide the most appropriate things to say. Therefore,
humans achieve their goals and successfully interact with one another through
deliberate planning guided by their internal models of the world and other
agents.

Figure 2: Left: Language models and world/agent models are usually studied in
different contexts. Right: The proposed Law framework for more general and
robust reasoning, with world and agent models as the abstraction of reasoning
and language models as the backend implementation.

Human agents also exhibit richer learning mechanisms than LLMs. As shown in
Figure 1 (embodied/social reasoning), LLMs trained merely with large-scale
text corpora lack fundamental real-world experience, such as tracking and
interacting with objects, understanding real-world physics and spatiotemporal
relationships, sensing and tracking the world states, and recognizing other
agents’ behaviors, etc. Human agents bypass these limitations by learning
through interaction with the environment. For instance, we acquire new
knowledge by attempting tasks and receiving feedback (e.g., a chef refines
their culinary skills by experimenting with different ingredients and tasting
the outcomes), or simply by exploring the surroundings randomly (e.g., a child
learns about different textures and sensations by randomly picking up various
objects).

In sum, current LLM reasoning and planning face key limitations in inference
(autoregressive generation), learning (imitation from data corpora without
real-world interaction), and modeling (inefficiency of language and its lack
of grounding). In this position paper, we present a new perspective toward
more general and robust machine reasoning across language, embodied, social,
and other broad scenarios. In particular, inspired by the above discussion, we
propose a unified Law framework for machine reasoning that connects the
concepts of language models, agent models, and world models (Figure 2, right).

Specifically, the concepts of world and agent models have their roots in
cognitive science and developmental psychology (e.g., Tolman, 1948; Premack
and Woodruff, 1978; Johnson-Laird, 1983, 2010; Gentner and Stevens, 2014;
Nortmann et al., 2015; Maus et al., 2013; Forrester, 1971; Gopnik and Wellman,
1994; Gergely and Csibra, 2003; Spelke and Kinzler, 2007; Battaglia et al.,
2013; Baker et al., 2009; Jara-Ettinger et al., 2016; Baker et al., 2017). As
mentioned earlier, a world model (§2.2) is a mental representation that agents
use to understand and predict the external world around them; an agent model
(§2.3) contains a world model and also other crucial components, including the
agent’s goals as well as its beliefs of the current world state and other
agents. These components together shape the agent’s cognitive processes,
enabling deliberate reasoning and planning. In the fields of artificial
intelligence and machine learning, world and agent models have typically been
studied in the context of reinforcement learning and robotics (e.g.,
Toussaint, 2003; Schulkin, 2012; Ha and Schmidhuber, 2018; Berkenkamp et al.,
2017; Clavera et al., 2018; Zhang et al., 2019; Kaiser et al., 2019; Moerland
et al., 2023; LeCun, 2022). For instance, recent studies show world modeling
enables agents to make effective action plans in specific games and embodied
control problems (Schrittwieser et al., 2020; Hafner et al., 2020).

In this paper, we highlight the enormous new opportunities of integrating
language models with world and agent models, for more general reasoning
capabilities not possible with the individual formulations alone. In
particular, compared to the current paradigm of LM-based reasoning, we posit
that world and agent models are a better abstraction of machine reasoning, as
they natively encompass the essential components for human-like
reasoning—e.g., beliefs, goals, anticipation of consequences, and deliberate
planning (Figure 2, right). In this framework, LMs are one of the ways for
implementing world/agent models or the individual components. That is, LMs
serve as the backend that operationalizes the framework. Compared to
conventional implementations, LMs provide the computational power and
adaptability necessary for handling vastly diverse reasoning scenarios. On the
other hand, the new role of LMs in the Law reasoning framework also highlights
their limitations and inspires future research for improvement.

In the following sections, we first give a brief background of the three
models, respectively (§2). We then present the new Law framework of reasoning
(§3), where we review the emerging studies related to each element in the
framework, and discuss the roadmap for addressing the various challenges
inherent in existing approaches and achieving more advanced machine reasoning
and planning.

##  2 Preliminary: The Three Models

###  2.1 Language Models (LMs)

A modern neural LM processes text by learning to predict the next word $x_{t}$
given the preceding text sequence $x_{1:t-1}$:

$P(x_{t}|x_{1:t-1}).$ (1)

Pretrained with massive text (and multi-modal) data corpora, LLMs, such as
(Chat)GPTs (Brown et al., 2020; OpenAI, 2023), Gemini (Google, 2023), and
Llama (Touvron et al., 2023a, b), have exhibited emergent reasoning abilities
in a wide range of language tasks, including question answering, math
reasoning, code generation, conversation, and others.

###  2.2 World Models (WMs)

The knowledge of the world is extremely broad, ranging from how a ball would
fall and bounce off the ground, to how the price of a stock would rise and
fall. In the context of embodied tasks (where the world model concept is
usually studied), a world model can typically be formulated as state
transition probabilities, which characterizes a generative, casual mechanism
of how the world state changes after an agent’s actions:

$\mathcal{T}(s^{\prime}|s,a),$ (2)

where $s$ is the current world state, $a$ is an action taken by an agent, and
$s^{\prime}$ is the next state after the action.

The need for a world model to conduct commonsense physical reasoning
(Battaglia et al., 2013; Ullman et al., 2017; Smith et al., 2019) and problem-
solving such as tool use and model-based planning (Allen et al., 2020) has
long been argued for in cognitive science. There has also been recent evidence
from neuroscience suggesting that our brains use a physics engine as a world
model to simulate the future (Pramod et al., 2020). Similarly, there has been
increasing interest in building a world model for physical scene understanding
(Wu et al., 2017; Li et al., 2020; Allen et al., 2022) and model-based
reinforcement learning (Berkenkamp et al., 2017; Clavera et al., 2018; Zhang
et al., 2019; Kaiser et al., 2019; Hafner et al., 2020; Moerland et al., 2023)
and planning (Toussaint et al., 2018; Li et al., 2019; Jatavallabhula et al.,
2021). These prior works have demonstrated that the use of world models can
enable more data-efficient learning and better generalization in unseen
scenarios.

###  2.3 Agent Models (AMs)

Figure 3: When an agent infers the mental state of another agent, it needs to
build a mental model of another agent. This can be formulated as a level-1
agent model reasoning about a level-0 agent model.

We not only need to understand the world around us but also make intelligent
decisions to achieve our goals by interacting with the world. Moreover, we
also have to understand and interact with other agents. On the one hand, we
understand the distinction between physical entities and an agent and have
represented them in fundamentally different ways since infancy Spelke and
Kinzler (2007); Gergely and Csibra (2003). On the other hand, we can also
appreciate the fact an agent’s behavior is contained by the world and that
models of agents can not be separate from models of the world. A minimum
definition of an agent model includes the following components:

Goal and reward. An agent has its goal, which defines a reward function that
guides the agent’s goal-directed behavior. Sometimes, the reward function also
includes the cost of the agent’s actions.

Belief. For an agent that has only partial observation of the world (e.g., a
robot can only sense the objects around it), it has only incomplete
information about the world state. Therefore, it needs to form a belief about
what the true world state could be.

World model. An agent has its world model in its mind, which may or may not be
the same as the actual world. For instance, where we imagine a basketball will
land after we throw it may be different from where it will actually land.

Planning. Given an agent’s mental state (goal, reward, and belief), its
rational behavior can be modeled as planning which searches for actions that
maximize its reward or reach its goal by simulating ahead using the world
model in its mind.

There are two levels of use of agent models:

In embodied tasks, an agent model represents how an embodied agent optimizes
its actions to maximize its accumulated reward based on its belief of the
current world state and the physical constraints defined in its world model.
For instance, given the command of “give me a cup,” a robot needs to find the
cup as quickly as possible (goal and reward) based on where it believes the
cup could be (belief) and the shortest path to reach the likely locations
without hitting any obstacles (world model). We term this type of agent model
level-0 agent model. There have been works on using LMs to build level-0 agent
models for language agents (e.g., Andreas, 2022; Sumers et al., 2023; Deng et
al., 2023) and embodied agents (e.g., Huang et al., 2022; Ahn et al., 2022; Li
et al., 2022b).

In social reasoning tasks, we utilize the models of other agents to reason
about their behaviors. This capacity is commonly referred to as Theory of Mind
(Premack and Woodruff, 1978), which involves forming mental models of other
agents and conducting causal reasoning to interpret other agents’ behaviors in
terms of their mental states (such as goals and beliefs). We term the agent
models that reason about other agents, level-1 agent models (Figure 3). For
instance, to understand a person’s searching behavior, we need to infer what
goal (the object they are looking for) and belief (where they believe the
object is) may lead to the plan (the observed behavior) of that person.
Systems designed to interact with humans, such as assistive robots (e.g.,
Dautenhahn, 2007; Hadfield-Menell et al., 2016; Patel and Chernova, 2022; Puig
et al., 2023), AI teachers (e.g., Wang et al., 2021), autonomous vehicles
(e.g., Chandra et al., 2020), and cooperative embodied agents (e.g., Bara et
al., 2021; Sclar et al., 2022), must be able to understand and cooperate with
humans in a grounded, physical world. Therefore, there is a critical need for
AI systems to develop robust social reasoning that combines social commonsense
(via level-1 agent models) and physical commonsense (via world models). Recent
studies have revealed the lack of human-level social reasoning in LMs (e.g.,
Sap et al., 2022; Jin et al., 2023; Ullman, 2023; Shapira et al., 2023;
Moghaddam and Honey, 2023). We hypothesize that it is possible to enhance LMs’
social reasoning capacity by building explicit world models and level-1 agent
models. We may even enable recursive social reasoning (e.g., Gmytrasiewicz and
Doshi, 2005; Goodman and Frank, 2016; Hadfield-Menell et al., 2016; Tejwani et
al., 2022; Schulz et al., 2023; Jha et al., 2023)) via higher-level agent
models.

##  3 The LAW Framework

###  3.1 Reasoning with World and Agent Models, on the Language Model Backend

####  3.1.1 Limitations of Reasoning with Language Models

LLMs have exhibited strong reasoning abilities in many language tasks. Recent
LM reasoning approaches further boost their performance by guiding LMs to
generate intermediate reasoning steps. For example, Chain-of-Thought (CoT)
(Wei et al., 2022) prompts the LMs to generate step-by-step derivations before
producing the final answer. More recent approaches introduce more
sophisticated structures into the reasoning process, such as decomposing a
target question into a series of subquestions (Zhou et al., 2022; Xie et al.,
2023a), using beam or tree-structured search to find better reasoning chains
(Yao et al., 2023a; Jung et al., 2022; Zhu et al., 2022; Liu et al., 2023a),
and adding self-verification steps for rectifying reasoning errors (Ouyang et
al., 2023; Shinn et al., 2023; Madaan et al., 2023; Weng et al., 2022).

Compared to LLM reasoning, deliberate human reasoning relies on the internal
world model which allows human brains to play out different reasoning steps
and their effects on the world state. Take the example of playing BlocksWorld,
which involves generating an action plan to rearrange blocks to a target
configuration (Figure 1, embodied reasoning). To devise such an action plan,
humans imagine different potential actions (e.g., “pick up the red block”),
simulate the state (i.e., block configuration) after each action using the
world model, and assess its likelihood of achieving the desired outcome. We
then refine our action plan by choosing the most promising steps. Similarly,
when solving a math problem, we explore different possible derivation steps
and their resultant states (i.e., intermediate conclusions derived so far),
evaluate how each state is closer to the final solution, and choose the best
derivation path accordingly. In both cases, the internal world model plays a
key role by allowing us to explore multiple possibilities, simulate their
outcomes, and iteratively refine the reasoning trace.

Inspired by human reasoning, we can pinpoint several essential components that
are missing in the current reasoning with LLMs, including: (1) explicit
modeling of the world state (e.g., block configuration, intermediate math
conclusions); for example, as in Figure 1 (embodied reasoning), CoT typically
generates a sequence of actions without describing the block configuration
after each step, often leading to inconsistent action plans (such as those
yielding invalid states); (2) an internal WM for simulating future states,
which is a foundation of human reasoning; (3) a reward mechanism to assess and
guide the reasoning towards the desired states; and (4) due to the above,
balance between exploration (of possible reasoning options not considered yet)
vs. exploitation (of the best reasoning steps identified so far), to
efficiently navigate the vast reasoning space and find the optimal reasoning
trace.

####  3.1.2 Reasoning with World and Agent Models using Language Model Backend

The above limitations call for a new conceptual framework of machine
reasoning. Instead of reasoning directly with LMs, we propose that world and
agent models are a better abstraction for carrying out robust and versatile
reasoning. With the explicit, built-in components, including beliefs,
anticipation of outcomes, and goals/rewards, a reasoning formulation based on
world and agent models inherently overcomes the aforementioned limitations.
Given a problem, the agent model performs deliberate planning in the reasoning
space based on its beliefs about the current state and other agents as well as
its prediction of future states resulting from various actions (through the
world model), all directed by the agent’s goal. The agent decides on the next
step or an action plan by maximizing its reward while adhering to constraints
due to its beliefs and world model. Under this abstraction, crucially, LLMs
are used as the backbone for implementing the system or its components.
Therefore, the system incorporates the computation power and flexibility of
LLMs for processing the diverse noisy scenarios in the real world, and the
structured abstraction of world and agent models to enable robust, efficient,
and versatile reasoning capabilities in language, embodied, social, and other
problems. In the remainder of this section, we review recent works that have
made meaningful progress relevant to the proposed framework. We discuss the
limitations of the current LLM backend and outline the future research
directions later.

Figure 4: Illustration of RAP (Hao et al., 2023a) for reasoning in BlocksWorld
and math problems. (Figure from Hao et al. (2023a)).

##### LMs as Both World and Agent Models.

Perhaps of most relevance to the Law framework is Reasoning-via-Planning (RAP)
(Hao et al., 2023a) which introduced the idea of world and agent modeling into
the reasoning problems previously handled by LLMs directly (Figure 4).
Specifically, RAP repurposes an LLM as a world model by prompting the LLM to
predict the next state $s_{t+1}$ of reasoning after applying a reasoning step
$a_{t}$ to the current state $s_{t}$ (e.g., predicting new conclusions after a
derivation step for a math problem, as described above). Similarly, the same
LLM is prompted to act as an agent model that produces an action after each
state. As a result, a reasoning trace consists of a sequence of interleaved
states and reasoning steps $(s_{0},a_{0},s_{1},\dots,a_{T-1},s_{T})$. This
differs from the previous reasoning methods, such as CoT as mentioned above,
where the reasoning focuses on generating a sequence of only actions, e.g.,
($a_{0}=$ ‘‘pickup red block’’, $a_{1}=$ ‘‘stack on blue block’’, …). Similar
as in (Li et al., 2022a), augmenting the reasoning with the (predicted) world
states helps the LM with a more grounded and coherent inference. The full
reasoning trace is simulated by the LLM itself (as a reasoning agent with an
internal world model) without interacting with the external real environment.
This resembles humans contemplating a possible plan in their minds.

More crucially, the capability of simulating future states (due to the
introduction of the world model) allows the incorporation of principled
planning algorithms for strategic exploration in the vast reasoning space. RAP
uses the classic Monte Carlo Tree Search (MCTS) (Kocsis and Szepesvári, 2006;
Coulom, 2007) for finding high-reward reasoning traces with a balance between
exploration and exploitation. Note that strategic search with MCTS was also
used in previous successful systems such as AlphaGo (Silver et al., 2016). In
problems like chess and Go, perfect world models exist (e.g., each move
deterministically leads to a subsequent chess state). Real-world reasoning
problems are more challenging due to the complex uncertain state dynamics. RAP
and its follow-ups (e.g., Wang et al., 2023b) show the benefits of structuring
LLM reasoning with future state prediction and strategic planning.

As a general way to construct generative models, probabilistic programs have
also been used for constructing world models and agent models for physical
(e.g., Gothoskar et al., 2021) and social reasoning (e.g., Zhi-Xuan et al.,
2020). A recent work (Wong et al., 2023) leverages the code-writing capacity
of LMs to translate natural language descriptions about the world and other
agents to probabilistic programs of the world and other agents. This provides
an alternative use of LMs in the constructing world and agent models, in which
LMs serve as a flexible interface between language and thought (about the
world and other agents).

##### LMs as the Planner in Agent Models.

There have been many works in building embodied agents using LMs. The most
common use of LMs is to generate plans based on prompts that specify the
state, task, and even memory. While empirical results on LMs’ planning
capacity have been promising (Huang et al., 2022), the plans generated by LMs
often fail to robustly solve long-horizon planning problems in complex,
partially observable. To address this limit, recent works have been using LMs
in an interactive planning paradigm, providing feedback from the environment
and reflection on past actions as additional prompts for LMs to adjust their
plan generation for future steps. Such an interactive planning paradigm has
achieved success in both single-agent planning (e.g., Dasgupta et al., 2023;
Wang et al., 2023d) and multiagent collaboration (e.g., Mandi et al., 2023;
Zhang et al., 2023). Finetuning LMs on specific domains has also been
demonstrated to be beneficial for improving their planning capacities on the
trained tasks. Specifically, the finetuned LMs exhibit a certain level of
compositional generalization within the same domain (Li et al., 2022b).
However, it remains unclear how much of the acquired planning capacity during
finetuning can be generalized to novel domains. Moreover, when using the LMs
for reasoning about the plans of other agents (i.e., as the planners in other
agents’ models), we can see an improved Theory of Mind capacity compared to
using LMs to directly infer other agents’ mental states (Jin et al., 2023).
This suggests that while LMs on their own still lack social reasoning
capacity, they can serve as a component in agent models to achieve better
model-based social reasoning. Lastly, beyond embodied agents, LMs can also
simulate social behaviors in abstract environments mimicking a simplified
society (Park et al., 2023). Without the need to generate physically grounded
actions, LMs can synthesize high-level but also more sophisticated social
behaviors.

##### LMs as the Goal/Reward in Agent Models.

Although LMs have demonstrated promising planning abilities, for many embodied
tasks (such as low-level robot control), conventional methods still have
better performance. Instead of using LMs to produce the final plans, recent
works have studied the possibility of using LMs as a component in an agent
model, most notable for generating goals (Xie et al., 2023b) or rewards (Yu et
al., 2023; Kwon et al., 2023; Ma et al., 2023). Goal and reward specifications
grounded to a physical robot body for intended tasks can be difficult and
typically require expert knowledge. However, the in-context learning capacity
of LMs can provide an easier way to translate language descriptions about the
intended tasks to accurate goal and reward specifications following a few
provided examples.

##### LMs as the Belief in Agent Models.

To the best of our knowledge, there has not been much work on explicating
modeling beliefs using LMs as a separate module. However, there have been
evaluations of LMs’ ability to encode belief representations about the world
states (e.g., Li et al., 2021), showing promising but imperfect results.
Additionally, there has been empirical evidence showing LMs’ lack of ability
to infer other agents’ beliefs using Theory of Mind benchmarks (Sap et al.,
2022; Jin et al., 2023; Ullman, 2023; Shapira et al., 2023). For future work,
it could be valuable to explicitly model belief update for an agent model as a
separate module using LMs, similar to LMs as the planner, goal, or reward.

###  3.2 Enhancing the Language Model Backend

The new perspective of reasoning under the Law framework also reveals a number
of directions for enhancing the LM backend, in order to better operationalize
the reasoning system or its modules. In particular, LMs need to learn by not
only imitating existing data corpora but also all different forms of
experience (Hu and Xing, 2022), such as interacting with external environments
and other agents, to gain a more robust and comprehensive understanding of the
physical and social world. Moreover, as discussed previously, language is
often not the most efficient medium for expressing all information during
reasoning (e.g., describing a world state in world modeling). This calls for
multi-modal understanding and generation capabilities in the backend model, to
support more versatile and grounded world and agent modeling during reasoning.
As we discuss below, recent studies have begun exploring these areas, yet
there is still considerable room for further advancements.

##### Learning with Embodied Experiences.

Learning from pure text is unlikely to be sufficient to acquire much of the
knowledge of the physical world and develop robust embodied skills. Recent
works have explored the possibility of enhancing LMs’ world knowledge with
embodied experiences. Recent works have proposed different ways to collect
embodied experiences, including random exploration (Xiang et al., 2023),
accomplishing specified goals (Xiang et al., 2023; Zeng et al., 2023; Wang et
al., 2023c), and proposing new tasks for an LM itself via an auto-curriculum
(Wang et al., 2023a). These diverse embodied experiences can unlock new ways
to train language models to acquire knowledge about the world, with objectives
beyond instruction finetuning and simple human preference feedback (e.g.,
RLHF, Ouyang et al., 2022).

Given collected embodied experiences, we can finetune LMs for domain-specific
tasks (e.g., Li et al., 2022b) and only use the resulting models for the
target domains. However, it is also possible to preserve LMs’ original
language skills while injecting the additional embodied knowledge into the
LMs, as studied in (Xiang et al., 2023). Finally, instead of finetuning LMs,
Wang et al. (2023a) have also explored the possibility of constructing an
ever-growing repository of skills through memory.

##### Learning with Social Interactions.

In addition to learning from embodied experiences, we hypothesize that LMs can
also benefit from social learning. For instance, LMs can learn from (1)
observing human demonstrations for performing embodied tasks, (2) observing
human social interactions, and (3) interacting with humans or other models
(including LMs, Liu et al. (2023c)). Such social learning experiences would
not only help LMs acquire world knowledge from humans and other LMs but also
develop better agent models that can support stronger social reasoning.

##### Multimodal World Modeling.

As discussed earlier, language has only a limited capacity to describe the
world state and its dynamics. Therefore, there is a need for multimodal
processing for world models (and agent models). One way to achieve this is to
learn multimodal models, such as GPT-4V, LLaVA (Liu et al., 2023b), and Gemini
(Google, 2023). While these models could be powerful tools for many tasks
(especially for multimodal understanding), they are limited to act as world
models due to the inability of generating images/videos for describing world
states sequentially.

Recent advances in generative models such as diffusion models have provided a
new way of modeling the world – learning a video generator that can predict
the future frames conditioned on action commands (e.g., Yang et al., 2023; Hu
et al., 2023). Such video prediction-based world models can simulate the
detailed change in the world state, allowing motion planning that is unable to
be achieved by world models constructed by LMs alone. However, training long-
horizon video prediction models that can generalize to novel scenarios is
difficult. It can also not be efficient, as the frame-level simulation is only
necessary for low-level motion control, whereas, for high-level task planning,
abstract state representations are adequately sufficient. Therefore, we can
envision a multi-level multimodel world model, simulating the world at an
abstract level (e.g., symbolic state representations such as scene graphs) and
fine-grained level (e.g., pixels or other types of raw sensory data).

##### Tool Using.

Enabling LMs to use external tools (e.g., functions, APIs, other models)
serves as another way to augment LMs with multimodal capabilities (AutoGPT,
2022; OpenAI, 2022). Emerging research has been done on building LM agents
that use tools for completing various tasks, through finetuning LMs with tool-
use demonstration data (Schick et al., 2023; Patil et al., 2023), in-context
learning (Yao et al., 2023b; Paranjape et al., 2023), tool embedding (Hao et
al., 2023b), and others. Most works still rely on LMs to perform direct
reasoning and determine the application of tools within the process. We expect
the world/agent model abstraction will facilitate enhanced tool-using
capabilities.

##  4 Discussions

We presented the Law framework as a new perspective of formulating machine
reasoning. Integrating the crucial elements of belief, future anticipation,
goals/reward, and strategic planning, Law aims at more robust and versatile
reasoning capabilities beyond the current reasoning with language models.
Aspects of the Law framework are aligned with recent proposals about building
world models (LeCun, 2022) and agent models (Andreas, 2022). Crucially, Law
introduces an integrated framework that combines three models in a cognitively
grounded way for solving a broad range of tasks. We have discussed how
existing language models may serve as the backend for reasoning with world and
agent worlds. We have also proposed possible ways to enhance the world and
agent modeling capacity of the language model backend, including new training
paradigms and the augmentation of multimodality capabilities.

We recognize that the Law framework has its limitations. First, the language
model backend implies symbolic representations in a discrete space. We have
discussed the possibility of augmenting this space with additional continuous
latent spaces modeled by other modalities (e.g., the latent space for a
diffusion model that simulates pixel-level world state transitions). However,
it may also be possible to use a single continuous latent space for a world
model or an agent model. While we hypothesize that symbolic representations
from language models may help us to learn the causal structures of the world
and agents as demonstrated by existing LMs, it remains unclear whether
continuous latent representations can achieve the same capacity (Ha and
Schmidhuber, 2018; Hafner et al., 2019; Anand et al., 2019; Ermolov and Sebe,
2020; LeCun, 2022). Second, it is possible that the current world and agent
modeling may not capture all knowledge about the world and agents. For
instance, we assume that agent behaviors are driven by goals or rewards.
However, behaviors can be driven by other variables, such as social norms.
Lastly, this paper does not discuss the inherent limits of Transformer
architectures (e.g., Dziri et al., 2023). We believe that further studies on
understanding the learning mechanism of Transformers can be complementary to
and beneficial for the development of machine reasoning.

## References

  * Ahn et al. (2022) M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ , 2022.
  * Allen et al. (2020) K. R. Allen, K. A. Smith, and J. B. Tenenbaum.  Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning.  _PNAS_ , 2020.
  * Allen et al. (2022) K. R. Allen, Y. Rubanova, T. Lopez-Guevara, W. Whitney, A. Sanchez-Gonzalez, P. Battaglia, and T. Pfaff.  Learning rigid dynamics with face interaction graph networks.  _arXiv preprint arXiv:2212.03574_ , 2022.
  * Anand et al. (2019) A. Anand, E. Racah, S. Ozair, Y. Bengio, M.-A. Côté, and R. D. Hjelm.  Unsupervised state representation learning in atari.  _Advances in neural information processing systems_ , 32, 2019.
  * Andreas (2022) J. Andreas.  Language models as agent models.  _arXiv preprint arXiv:2212.01681_ , 2022.
  * AutoGPT (2022) AutoGPT.  Autogpt, 2022.  URL https://autogpt.net.
  * Baker et al. (2009) C. L. Baker, R. Saxe, and J. B. Tenenbaum.  Action understanding as inverse planning.  _Cognition_ , 113(3):329–349, 2009.
  * Baker et al. (2017) C. L. Baker, J. Jara-Ettinger, R. Saxe, and J. B. Tenenbaum.  Rational quantitative attribution of beliefs, desires and percepts in human mentalizing.  _Nature Human Behaviour_ , 1(4):1–10, 2017.
  * Bara et al. (2021) C.-P. Bara, S. CH-Wang, and J. Chai.  Mindcraft: Theory of mind modeling for situated dialogue in collaborative tasks.  In _Conference on Empirical Methods in Natural Language Processing_ , 2021.
  * Battaglia et al. (2013) P. W. Battaglia, J. B. Hamrick, and J. B. Tenenbaum.  Simulation as an engine of physical scene understanding.  _PNAS_ , 2013.
  * Berkenkamp et al. (2017) F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause.  Safe model-based reinforcement learning with stability guarantees.  _Advances in neural information processing systems_ , 30, 2017.
  * Briscoe (2011) R. E. Briscoe.  Mental imagery and the varieties of amodal perception.  _Pacific Philosophical Quarterly_ , 92(2):153–173, 2011.
  * Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chandra et al. (2020) R. Chandra, A. Bera, and D. Manocha.  Stylepredict: Machine theory of mind for human driver behavior from trajectories.  _arXiv preprint arXiv:2011.04816_ , 2020.
  * Clavera et al. (2018) I. Clavera, J. Rothfuss, J. Schulman, Y. Fujita, T. Asfour, and P. Abbeel.  Model-based reinforcement learning via meta-policy optimization.  In _Conference on Robot Learning_ , pages 617–629. PMLR, 2018.
  * Coulom (2007) R. Coulom.  Efficient selectivity and backup operators in monte-carlo tree search.  In _Computers and Games: 5th International Conference, CG 2006, Turin, Italy, May 29-31, 2006. Revised Papers 5_ , pages 72–83. Springer, 2007\.
  * Dasgupta et al. (2023) I. Dasgupta, C. Kaeser-Chen, K. Marino, A. Ahuja, S. Babayan, F. Hill, and R. Fergus.  Collaborating with language models for embodied reasoning.  _arXiv preprint arXiv:2302.00763_ , 2023.
  * Dautenhahn (2007) K. Dautenhahn.  Socially intelligent robots: dimensions of human–robot interaction.  _Philosophical transactions of the royal society B: Biological sciences_ , 362(1480):679–704, 2007.
  * Deng et al. (2023) X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su.  Mind2web: Towards a generalist agent for the web.  _arXiv preprint arXiv:2306.06070_ , 2023.
  * Dziri et al. (2023) N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jian, B. Y. Lin, P. West, C. Bhagavatula, R. L. Bras, J. D. Hwang, et al.  Faith and fate: Limits of transformers on compositionality.  _arXiv preprint arXiv:2305.18654_ , 2023.
  * Ermolov and Sebe (2020) A. Ermolov and N. Sebe.  Latent world models for intrinsically motivated exploration.  _Advances in Neural Information Processing Systems_ , 33:5565–5575, 2020.
  * Forrester (1971) J. W. Forrester.  Counterintuitive behavior of social systems.  _Theory and decision_ , 2(2):109–140, 1971.
  * Gentner and Stevens (2014) D. Gentner and A. L. Stevens.  _Mental models_.  Psychology Press, 2014.
  * Gergely and Csibra (2003) G. Gergely and G. Csibra.  Teleological reasoning in infancy: The naıve theory of rational action.  _Trends in cognitive sciences_ , 7(7):287–292, 2003.
  * Gmytrasiewicz and Doshi (2005) P. J. Gmytrasiewicz and P. Doshi.  A framework for sequential planning in multi-agent settings.  _Journal of Artificial Intelligence Research_ , 24:49–79, 2005.
  * Goodman and Frank (2016) N. D. Goodman and M. C. Frank.  Pragmatic language interpretation as probabilistic inference.  _Trends in cognitive sciences_ , 20(11):818–829, 2016.
  * Google (2023) Google.  Gemini: A family of highly capable multimodal models.  Technical report, Google, 2023.
  * Gopnik and Wellman (1994) A. Gopnik and H. M. Wellman.  The theory theory.  In _An earlier version of this chapter was presented at the Society for Research in Child Development Meeting, 1991._ Cambridge University Press, 1994.
  * Gothoskar et al. (2021) N. Gothoskar, M. Cusumano-Towner, B. Zinberg, M. Ghavamizadeh, F. Pollok, A. Garrett, J. Tenenbaum, D. Gutfreund, and V. Mansinghka.  3dp3: 3d scene perception via probabilistic programming.  _Advances in Neural Information Processing Systems_ , 34:9600–9612, 2021.
  * Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber.  World models.  _arXiv preprint arXiv:1803.10122_ , 2018.
  * Hadfield-Menell et al. (2016) D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. Dragan.  Cooperative inverse reinforcement learning.  In _Advances in neural information processing systems_ , 2016.
  * Hafner et al. (2019) D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson.  Learning latent dynamics for planning from pixels.  In _International conference on machine learning_ , pages 2555–2565. PMLR, 2019.
  * Hafner et al. (2020) D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba.  Mastering atari with discrete world models.  _arXiv preprint arXiv:2010.02193_ , 2020.
  * Hao et al. (2023a) S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu.  Reasoning with Language Model is Planning with World Model.  _arXiv preprint arXiv:2305.14992_ , 2023a.
  * Hao et al. (2023b) S. Hao, T. Liu, Z. Wang, and Z. Hu.  Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings.  _arXiv preprint arXiv:2305.11554_ , 2023b.
  * Hu et al. (2023) A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado.  Gaia-1: A generative world model for autonomous driving.  _arXiv preprint arXiv:2309.17080_ , 2023.
  * Hu and Xing (2022) Z. Hu and E. P. Xing.  Toward a ’Standard Model’ of Machine Learning.  _Harvard Data Science Review_ , 4(4), oct 27 2022.  https://hdsr.mitpress.mit.edu/pub/zkib7xth.
  * Huang et al. (2022) W. Huang, P. Abbeel, D. Pathak, and I. Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In _International Conference on Machine Learning_ , pages 9118–9147. PMLR, 2022.
  * Jara-Ettinger et al. (2016) J. Jara-Ettinger, H. Gweon, L. E. Schulz, and J. B. Tenenbaum.  The naïve utility calculus: Computational principles underlying commonsense psychology.  _Trends in cognitive sciences_ , 20(8):589–604, 2016.
  * Jatavallabhula et al. (2021) K. M. Jatavallabhula, M. Macklin, F. Golemo, V. Voleti, L. Petrini, M. Weiss, B. Considine, J. Parent-Lévesque, K. Xie, K. Erleben, et al.  gradsim: Differentiable simulation for system identification and visuomotor control.  _arXiv preprint arXiv:2104.02646_ , 2021.
  * Jha et al. (2023) K. Jha, T. A. Le, C. Jin, Y.-L. Kuo, J. B. Tenenbaum, and T. Shu.  Neural amortized inference for nested multi-agent reasoning.  _arXiv preprint arXiv:2308.11071_ , 2023.
  * Jin et al. (2023) C. Jin, Y. Wu, J. Cao, J. Xiang, Y.-L. Kuo, Z. Hu, T. Ullman, A. Torralba, J. Tenenbaum, and T. Shu.  Mmtom-qa: Multimodal theory of mind question answering.  In _NeurIPS 2023 Foundation Models for Decision Making Workshop_ , 2023.
  * Johnson-Laird (1983) P. N. Johnson-Laird.  _Mental models: Towards a cognitive science of language, inference, and consciousness_.  Harvard University Press, 1983.
  * Johnson-Laird (2010) P. N. Johnson-Laird.  Mental models and human reasoning.  _PNAS_ , 2010.
  * Jung et al. (2022) J. Jung, L. Qin, S. Welleck, F. Brahman, C. Bhagavatula, R. L. Bras, and Y. Choi.  Maieutic prompting: Logically consistent reasoning with recursive explanations.  _arXiv preprint arXiv:2205.11822_ , 2022.
  * Kaiser et al. (2019) L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, et al.  Model-based reinforcement learning for atari.  _arXiv preprint arXiv:1903.00374_ , 2019.
  * Kocsis and Szepesvári (2006) L. Kocsis and C. Szepesvári.  Bandit based monte-carlo planning.  In _Machine Learning: ECML 2006: 17th European Conference on Machine Learning Berlin, Germany, September 18-22, 2006 Proceedings 17_ , pages 282–293. Springer, 2006.
  * Kwon et al. (2023) M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh.  Reward design with language models.  _arXiv preprint arXiv:2303.00001_ , 2023.
  * LeCun (2022) Y. LeCun.  A path towards autonomous machine intelligence.  _Open Review_ , 2022.
  * Li et al. (2021) B. Z. Li, M. Nye, and J. Andreas.  Implicit representations of meaning in neural language models.  _arXiv preprint arXiv:2106.00737_ , 2021.
  * Li et al. (2022a) B. Z. Li, M. Nye, and J. Andreas.  Language modeling with latent situations.  _arXiv preprint arXiv:2212.10012_ , 2022a.
  * Li et al. (2022b) S. Li, X. Puig, C. Paxton, Y. Du, C. Wang, L. Fan, T. Chen, D.-A. Huang, E. Akyürek, A. Anandkumar, et al.  Pre-trained language models for interactive decision-making.  _Advances in Neural Information Processing Systems_ , 35:31199–31212, 2022b.
  * Li et al. (2019) Y. Li, H. He, J. Wu, D. Katabi, and A. Torralba.  Learning compositional koopman operators for model-based control.  _arXiv preprint arXiv:1910.08264_ , 2019.
  * Li et al. (2020) Y. Li, T. Lin, K. Yi, D. Bear, D. Yamins, J. Wu, J. Tenenbaum, and A. Torralba.  Visual grounding of learned physical models.  In _International conference on machine learning_ , pages 5927–5936. PMLR, 2020.
  * Liu et al. (2023a) B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone.  Llm+ p: Empowering large language models with optimal planning proficiency.  _arXiv preprint arXiv:2304.11477_ , 2023a.
  * Liu et al. (2023b) H. Liu, C. Li, Q. Wu, and Y. J. Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023b.
  * Liu et al. (2023c) R. Liu, R. Yang, C. Jia, G. Zhang, D. Zhou, A. M. Dai, D. Yang, and S. Vosoughi.  Training socially aligned language models in simulated human society.  _arXiv preprint arXiv:2305.16960_ , 2023c.
  * Ma et al. (2023) Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar.  Eureka: Human-level reward design via coding large language models.  _arXiv preprint arXiv:2310.12931_ , 2023.
  * Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_ , 2023.
  * Mandi et al. (2023) Z. Mandi, S. Jain, and S. Song.  Roco: Dialectic multi-robot collaboration with large language models.  _arXiv preprint arXiv:2307.04738_ , 2023.
  * Maus et al. (2013) G. W. Maus, J. Fischer, and D. Whitney.  Motion-dependent representation of space in area mt+.  _Neuron_ , 78(3):554–562, 2013.
  * Moerland et al. (2023) T. M. Moerland, J. Broekens, A. Plaat, C. M. Jonker, et al.  Model-based reinforcement learning: A survey.  _Foundations and Trends® in Machine Learning_ , 2023\.
  * Moghaddam and Honey (2023) S. R. Moghaddam and C. J. Honey.  Boosting theory-of-mind performance in large language models via prompting.  _arXiv preprint arXiv:2304.11490_ , 2023.
  * Nortmann et al. (2015) N. Nortmann, S. Rekauzke, S. Onat, P. König, and D. Jancke.  Primary visual cortex represents the difference between past and present.  _Cerebral Cortex_ , 25(6):1427–1440, 2015.
  * OpenAI (2022) OpenAI.  Chatgpt plugins.  https://openai.com/blog/chatgpt-plugins, 2022.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Ouyang et al. (2023) S. Ouyang, Z. Zhang, B. Yan, X. Liu, J. Han, and L. Qin.  Structured chemistry reasoning with large language models.  _arXiv preprint arXiv:2311.09656_ , 2023.
  * Paranjape et al. (2023) B. Paranjape, S. Lundberg, S. Singh, H. Hajishirzi, L. Zettlemoyer, and M. T. Ribeiro.  Art: Automatic multi-step reasoning and tool-use for large language models.  _arXiv preprint arXiv:2303.09014_ , 2023.
  * Park et al. (2023) J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein.  Generative agents: Interactive simulacra of human behavior.  In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ , pages 1–22, 2023.
  * Patel and Chernova (2022) M. Patel and S. Chernova.  Proactive robot assistance via spatio-temporal object modeling.  _arXiv preprint arXiv:2211.15501_ , 2022.
  * Patil et al. (2023) S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez.  Gorilla: Large language model connected with massive apis.  _arXiv preprint arXiv:2305.15334_ , 2023.
  * Pramod et al. (2020) R. Pramod, M. Cohen, K. Lydic, J. Tenenbaum, and N. Kanwisher.  Evidence that the brain’s physics engine runs forward simulations of what will happen next.  _Journal of Vision_ , 20(11):1521–1521, 2020\.
  * Premack and Woodruff (1978) D. Premack and G. Woodruff.  Does the chimpanzee have a theory of mind?  _Behavioral and brain sciences_ , 1(4):515–526, 1978.
  * Puig et al. (2023) X. Puig, T. Shu, J. B. Tenenbaum, and A. Torralba.  Nopa: Neurally-guided online probabilistic assistance for building socially intelligent home assistants.  _arXiv preprint arXiv:2301.05223_ , 2023.
  * Sap et al. (2022) M. Sap, R. LeBras, D. Fried, and Y. Choi.  Neural theory-of-mind? on the limits of social intelligence in large lms.  _arXiv preprint arXiv:2210.13312_ , 2022.
  * Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_ , 2023.
  * Schrittwieser et al. (2020) J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al.  Mastering atari, go, chess and shogi by planning with a learned model.  _Nature_ , 588(7839):604–609, 2020.
  * Schulkin (2012) J. Schulkin.  _Action, perception and the brain: Adaptation and cephalic expression_.  Springer, 2012.
  * Schulz et al. (2023) L. Schulz, N. Alon, J. S. Rosenschein, and P. Dayan.  Emergent deception and skepticism via theory of mind.  In _ICML 2023: First Workshop on Theory of Mind in Communicating Agents (ToM 2023)_ , 2023.
  * Sclar et al. (2022) M. Sclar, G. Neubig, and Y. Bisk.  Symmetric machine theory of mind.  In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 19450–19466. PMLR, 17–23 Jul 2022.
  * Shapira et al. (2023) N. Shapira, M. Levy, S. H. Alavi, X. Zhou, Y. Choi, Y. Goldberg, M. Sap, and V. Shwartz.  Clever hans or neural theory of mind? stress testing social reasoning in large language models.  _arXiv preprint arXiv:2305.14763_ , 2023.
  * Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. R. Narasimhan, and S. Yao.  Reflexion: Language agents with verbal reinforcement learning.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Silver et al. (2016) D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  _nature_ , 529(7587):484–489, 2016.
  * Smith et al. (2019) K. Smith, L. Mei, S. Yao, J. Wu, E. Spelke, J. Tenenbaum, and T. Ullman.  Modeling expectation violation in intuitive physics with coarse probabilistic object representations.  _Advances in neural information processing systems_ , 32, 2019.
  * Spelke and Kinzler (2007) E. S. Spelke and K. D. Kinzler.  Core knowledge.  _Developmental science_ , 10(1):89–96, 2007.
  * Sumers et al. (2023) T. Sumers, S. Yao, K. Narasimhan, and T. L. Griffiths.  Cognitive architectures for language agents.  _arXiv preprint arXiv:2309.02427_ , 2023.
  * Tejwani et al. (2022) R. Tejwani, Y.-L. Kuo, T. Shu, B. Katz, and A. Barbu.  Social interactions as recursive mdps.  In _Conference on Robot Learning_ , pages 949–958. PMLR, 2022.
  * Tolman (1948) E. C. Tolman.  Cognitive maps in rats and men.  _Psychological review_ , 55(4):189, 1948.
  * Toussaint (2003) M. Toussaint.  Learning a world model and planning with a self-organizing, dynamic neural system.  _Advances in neural information processing systems_ , 16, 2003.
  * Toussaint et al. (2018) M. A. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum.  Differentiable physics and stable modes for tool-use and manipulation planning.  2018\.
  * Touvron et al. (2023a) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. (2023b) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, and Y. e. a. Babaei.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Ullman (2023) T. Ullman.  Large language models fail on trivial alterations to theory-of-mind tasks.  _arXiv preprint arXiv:2302.08399_ , 2023.
  * Ullman et al. (2017) T. D. Ullman, E. Spelke, P. Battaglia, and J. B. Tenenbaum.  Mind games: Game engines as an architecture for intuitive physics.  _Trends in cognitive sciences_ , 21(9):649–665, 2017.
  * Wang et al. (2023a) G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv:2305.16291_ , 2023a.
  * Wang et al. (2021) Q. Wang, K. Saha, E. Gregori, D. Joyner, and A. Goel.  Towards mutual theory of mind in human-ai interaction: How language reflects what students perceive about a virtual teaching assistant.  In _Proceedings of the 2021 CHI conference on human factors in computing systems_ , pages 1–14, 2021.
  * Wang et al. (2023b) X. Wang, C. Li, Z. Wang, F. Bai, H. Luo, J. Zhang, N. Jojic, E. P. Xing, and Z. Hu.  Promptagent: Strategic planning with language models enables expert-level prompt optimization.  _arXiv preprint arXiv:2310.16427_ , 2023b.
  * Wang et al. (2023c) X. Wang, Z. Wang, J. Liu, Y. Chen, L. Yuan, H. Peng, and H. Ji.  Mint: Evaluating llms in multi-turn interaction with tools and language feedback.  _arXiv preprint arXiv:2309.10691_ , 2023c.
  * Wang et al. (2023d) Z. Wang, S. Cai, A. Liu, X. Ma, and Y. Liang.  Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents.  _arXiv preprint arXiv:2302.01560_ , 2023d.
  * Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ , 2022.
  * Weng et al. (2022) Y. Weng, M. Zhu, S. He, K. Liu, and J. Zhao.  Large language models are reasoners with self-verification.  _arXiv preprint arXiv:2212.09561_ , 2022.
  * Wong et al. (2023) L. Wong, G. Grand, A. K. Lew, N. D. Goodman, V. K. Mansinghka, J. Andreas, and J. B. Tenenbaum.  From word models to world models: Translating from natural language to the probabilistic language of thought.  2023\.
  * Wu et al. (2017) J. Wu, E. Lu, P. Kohli, B. Freeman, and J. Tenenbaum.  Learning to see physics via visual de-animation.  _NeurIPS_ , 2017.
  * Xiang et al. (2023) J. Xiang, T. Tao, Y. Gu, T. Shu, Z. Wang, Z. Yang, and Z. Hu.  Language Models Meet World Models: Embodied Experiences Enhance Language Models.  _arXiv preprint arXiv:2305.10626_ , 2023.
  * Xie et al. (2023a) Y. Xie, K. Kawaguchi, Y. Zhao, X. Zhao, M.-Y. Kan, J. He, and Q. Xie.  Decomposition enhances reasoning via self-evaluation guided decoding.  _arXiv preprint arXiv:2305.00633_ , 2023a.
  * Xie et al. (2023b) Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh.  Translating natural language to planning goals with large-language models.  _arXiv preprint arXiv:2302.05128_ , 2023b.
  * Yang et al. (2023) M. Yang, Y. Du, K. Ghasemipour, J. Tompson, D. Schuurmans, and P. Abbeel.  Learning interactive real-world simulators.  _arXiv preprint arXiv:2310.06114_ , 2023.
  * Yao et al. (2023a) S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models.  2023a.
  * Yao et al. (2023b) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao.  ReAct: Synergizing reasoning and acting in language models.  In _International Conference on Learning Representations (ICLR)_ , 2023b.
  * Yu et al. (2023) W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, et al.  Language to rewards for robotic skill synthesis.  _arXiv preprint arXiv:2306.08647_ , 2023.
  * Zeng et al. (2023) A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang.  Agenttuning: Enabling generalized agent abilities for llms.  _arXiv preprint arXiv:2310.12823_ , 2023.
  * Zhang et al. (2023) H. Zhang, W. Du, J. Shan, Q. Zhou, Y. Du, J. B. Tenenbaum, T. Shu, and C. Gan.  Building cooperative embodied agents modularly with large language models.  _arXiv preprint arXiv:2307.02485_ , 2023.
  * Zhang et al. (2019) M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine.  Solar: Deep structured representations for model-based reinforcement learning.  In _International conference on machine learning_ , pages 7444–7453. PMLR, 2019.
  * Zhi-Xuan et al. (2020) T. Zhi-Xuan, J. Mann, T. Silver, J. Tenenbaum, and V. Mansinghka.  Online bayesian goal inference for boundedly rational planning agents.  _Advances in neural information processing systems_ , 33:19238–19250, 2020.
  * Zhou et al. (2022) D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and E. Chi.  Least-to-most prompting enables complex reasoning in large language models.  _arXiv preprint arXiv:2205.10625_ , 2022.
  * Zhu et al. (2022) X. Zhu, J. Wang, L. Zhang, Y. Zhang, R. Gan, J. Zhang, and Y. Yang.  Solving math word problem via cooperative reasoning induced language models.  _arXiv preprint arXiv:2210.16257_ , 2022.