Entropy of Causal Horizons
Eric M Howard1
1Department of Physics and Astronomy, Macquarie University
Email<EMAIL_ADDRESS>
Received **** 2016
Copyright © 2016 by author(s) and Scientific Research Publishing Inc.
This work is licensed under the Creative Commons Attribution International
License (CC BY).
http://creativecommons.org/licenses/by/4.0/

Abstract
We analyze spacetimes with horizons and study the thermodynamic aspects of
causal horizons, suggesting that the resemblance between gravitational and
thermodynamic systems has a deeper quantum mechanical origin. We find that the
observer dependence of such horizons is a direct consequence of associating a
temperature and entropy to a spacetime. The geometrical picture of a horizon
acting as a one-way membrane for information flow can be accepted as a natural
interpretation of assigning a quantum field theory to a spacetime with
boundary, ultimately leading to a close connection with thermodynamics.
Keywords
black hole entropy; causal horizon;black hole horizon; black hole
thermodynamics; Rindler spacetime; Rindler horizon; quantum gravity;
thermodynamics of spacetime

1\. Introduction

The interaction between gravity and thermodynamics has become a subject of
growing interest since the discovery of Bekenstein-Hawking entropy.[2] The
analogy between black hole mechanics and thermodynamics led Bekenstein [3] and
Hawking to argue that black holes should be viewed as true thermodynamic
systems, characterised by entropy and temperature.[4]

Several authors have worked on extending black hole entropy to the more
general case where the connection between entropy and area would generalize to
any horizon, like accelerated horizons in Rindler spacetime. One intriguing
conclusion was the idea that horizon entropy arises in a natural way just
because gravity is emergent from thermodynamics. The new perspective motivated
Jacobson,[5][6] Padmanabhan [7] [8] and Verlinde to suggest an interpretation
of gravity as a thermodynamic phenomenon.[9]

While the concepts of temperature and entropy remain associated to spacetimes
with horizons, a theory that fully explains the quantum structure of spacetime
should ultimately remain independent of any microscopic description of the
horizons themselves. The hard part here is the lack of understanding about the
nature of the microstates whose counting would produce the horizon entropy,
while the geometrical picture [10] of the black hole entropy would still
remain a good approximation for the theory.

The notion of entropy is closely connected to the accessibility of
information, therefore an observer dependent concept. If a geometrical
interpretation of gravity is accepted, surfaces acting as one-way membranes
for information will exist, leading to a deeper connection with
thermodynamics. An inevitable connection between one-way membranes that arise
in a spacetime with horizons and thermodynamical entropy is already obvious
here.[11]

The interesting fact is that so far we have knowledge only about two types of
physical processes that can lead to distinct regions of spacetime, some of
them completely disconnected from each other so that information from one
spacetime patch will not be accessible to an observer located within another
patch: light and gravity. Therefore, some strong connection between light,
gravity and thermodynamics must exist. The AdS/CFT conjecture is intimately
connected to this argument.

The extreme conditions for quantum, relativistic and gravitational regimes:
the very small, the very heavy and the very fast, seem to intersect at a
fundamental level, leading to a more general principle that so far seems to
have eluded our complete understanding. A theory formulating a full
description of the quantum structure of the spacetime should be responsible to
explain all three extreme conditions and in this context, should involve
themodynamics principles in its basic arguments.

If the connection between thermodynamics and spacetimes with horizons has a
fundamental meaning, the argument should be able to be extended to any general
horizon, therefore we can assign entropy to any horizon, irrespective of
whether the horizons are observer independent (black holes) or observer and
coordinate dependent (de Sitter and Rindler horizons).

Our argument here is that the entropy of spacetimes arise because there is
information hidden behind the horizon, generating different spacetime regions,
causally disconnected by an entangling null surface acting as a one-way
information barrier. An exploration of the Quantum Field Theory of such a
spacetime should suggest a natural association with a temperature. The
inertial observer will ascribe the entropy given by the logarithm of the
number of internal states. Since a horizon is capable to block information
from an observer, it is feasible to associate an entropy with the horizon.
Here we show that the entropy of an event in flat empty space depends on the
observer that measures it. We argue that this dependence originates in a pure
quantum mechanical effect of the fact that for an accelerated observer, the
event is hidden behind the Rindler horizon.

2\. Horizon entropy and observers

The idea of entanglement entropy has emerged as a measure of entanglement in
order to explain the internal correlations within a quantum system. In quantum
field theory the entanglement between a spatial region on a fixed Cauchy
surface region and its complement, is found to be divergent because of the
presence of short range correlations near the entangling surface that
separates the spacetime regions.[12]

Entropy can be viewed as a measure of missing or unavailable information about
a system. It would be expected that some entropy to be associated with the
event horizon, since it hides information. The horizon has an “entanglement
entropy” associated with the quantum field nearby, produced by tracing out its
interior inaccessible modes and ultimately correlated with the exterior
accessible (to the observer) modes of the field. For a fixed background
spacetime continuum, the entanglement entropy is infinite and a short distance
cut-off on the field degrees of freedom needs to be introduced in order to
obtain a finite entropy If the cut-off is near the Planck scale, the entropy
is of the same order of magnitude as the horizon itself.

The divergence of the entanglement entropy requires an ultraviolet cut-off at
Planck scale. In order to regulate this divergence, gravity should be treated
as a quantum field. As gravity is non-renormalizable, the entanglement entropy
should also be non-renormalizable. Furthermore, a complete theory of quantum
gravity should explain the cut-off and solve the divergence of the
entanglement entropy together with the ultraviolet problem. [13]

Sorkin [14] [15] has proposed the idea that entanglement entropy offers a
microscopic explanation for Bekenstein-Hawking entropy. In Quantum field
theory, space can be partitioned into subregions. The degrees of freedom can
be isolated inside the horizon from the surrounding spacetime, allowing a
state on the Cauchy surface to be a pure state like the vacuum, whereas the
region of spacetime outside the horizon may contain mixed states such as a
thermal states of non-zero temperature.

In this way, the horizon will posses entropy, while offering a clear division
of degrees of freedom outside the horizon that are observable, and degrees of
freedom behind the horizon that are not perceived by any observer.

At a conceptual level, the entanglement entropy does not represent the
ignorance of the observer about state of the spacetime region behind the
horizon but a measure of the observer’s ignorance about the state outside the
horizon that is due to the observer’s inability to measure the states behind
the horizon. The increase of entropy on the horizon is a effect of the
entanglement as well as simple relativistic causality.

The mathematical form of the association of temperature with the horizon is
universal and it does not distinguish between different types of horizons
(Rindler horizon in flat space, Schwarzschild black hole event horizon, de
Sitter horizon). The fact that the temperature and entropy both arise in the
same way, suggest a strong direct association of entropy with all horizons.
[16]

We extend this concept to Rindler horizons or any type of horizons that
exhibit causal order. We know that Rindler coordinates describe an observer
moving along a constant acceleration path.[17] In Minkowski spacetime, Rindler
coordinates depict the near-horizon limit of a black hole. In AdS spacetime,
Rindler coordinates divide AdS in two causally disconnected regions, known as
left and right (in causal contact with the observer) Rindler wedges.
Thermodynamical entropy is here produced by variables that are observable
(energy of a system), whereas other variables (exact positions of particles)
remain unobservable.

As a consequence, the entropy ascribed to a localized system by different
observers is not necessarily the same. The entropy of an event in flat empty
space will be differently ascribed by an inertial and a Rindler observer. The
entropy seen by a Rindler observer should have a bound given by by
$E_{Rindler}/T$, where $E_{Rindler}$ is the energy of an object associated
with an event observed by the Rindler observer and $T$ the temperature
measured by the observer.

For an inertial observer, the vacuum state is given by the Minkowski vacuum
$|0\rangle$. The excitation of the vacuum that could be due to the presence of
an object is given by the density matrix

$\rho={1\over n}\sum_{i=1}^{n}|1;i\rangle\langle 1;i|,$ (1)

The perturbation of the vacuum state is in an undetermined microstate. The
$|1;i\rangle$ are Minkowski states represented by the $n$ possible microstates
of the perturbation.

The measured entropy will also contain an additional contribution introduced
by the acceleration of the Rindler observer. [18] The measured energy
$E_{Rindler}=\delta E$ is given by the difference in Killing energies when the
perturbation is present and when it is not. The entropy of each state will be
divergent. The Rindler object entropy $S_{Rindler}$ is the similar difference
$\delta S$ between the two entropies in the right Rindler wedge, in causal
contact with the observer.

The entropy as measured by an inertial observer is given by

$S=-\mbox{Tr}\rho\ln\rho=\ln n.$ (2)

with $\rho$ being the density matrix $\rho_{Rindler}$ equivalent for a Rindler
observer.

For a perturbation with the energy $\delta E$, the density matrix $\rho_{R1}$
for a Rindler observer can be calculated by integrating the unobservable
modes. The entropy will become $S_{1}=-{\rm Tr}\ (\rho_{R1}\ln\rho_{R1})$. The
inertial vacuum state density matrix in Rindler frame is $\rho_{R0}$ with
entropy $S_{0}=-{\rm Tr}\ (\rho_{01}\ln\rho_{R0})$.

For the Rindler observer, the difference in entropies $\delta S=S_{1}-S_{0}$
is the entropy of the vacuum state. Near the horizon, for $\kappa\to\infty$ we
have

$\delta S=\beta\delta E=\frac{2\pi}{\kappa}\delta E$ (3)

where $\beta=T^{-1}$.

The Rindler observer will not measure the same entropy of the perturbation.
Following Sorkin hypothesis for black hole entropy and apply to all causal
horizons, [19] the increase of entanglement entropy is a direct consequence of
the fact that a Rindler horizon is a causal barrier that separates two
causally disconnected regions of spacetime. The whole spacetime is visible to
the inertial observer, therefore in causal contact with the observer.

The entropy measured by a Rindler observer is given by the right Rindler wedge
in causal contact with the observer, when the object crosses the Rindler
horizon. The entropies associated with this perturbation by a Minkowski and
Rindler observer behave in different ways.

If we trace the Minkowski vacuum $|0\rangle\langle 0|$ over the left Rindler
wedge, the thermal density matrix $\rho_{R0}$ will contain all the information
that a Rindler observer can access in the Minkowski vacuum:

$\rho_{R0}=\mbox{Tr}_{left}|0\rangle\langle 0|.$ (4)

We consider a new density matrix $\rho_{R1}$ corresponding to the Rindler
description of the state $\rho_{Rindler}$, containing information about the
object added to the Minkowski vacuum $|0\rangle$ and further compare it to
$\rho_{R0}$. The density matrix $\rho_{R0}$ represents the thermal
equilibrium. The new density matrix $\rho_{R1}$ is

$\rho_{R1}=\mbox{Tr}_{left}\rho_{Rindler}=\mbox{Tr}_{left}\frac{1}{n}\sum_{i=1}^{n}|1;i\rangle\langle
1;i|.$ (5)

If $\rho_{R1}=\rho_{R0}+\delta\rho$, for $\kappa\to\infty$ we only take into
account states with $\delta\rho/\rho_{R0}\ll 1$. The difference in entropy is

$\displaystyle-\delta S$ $\displaystyle=$ $\displaystyle{\rm Tr}\
(\rho_{R1}\ln\rho_{R1})-{\rm Tr}\ (\rho_{R0}\ln\rho_{R0})\simeq{\rm Tr}\
(\delta\rho\ln\rho_{R0})$ (6) $\displaystyle=$ $\displaystyle{\rm Tr}\
(\delta\rho(-\beta H))=-\beta{\rm Tr}\
\left((\rho_{R1}-\rho_{R0})H\right)\equiv-\beta\delta E$

with Tr $\delta\rho\approx 0$ and $\rho_{0}=Z^{-1}\exp(-\beta H)$. Here $H$ is
the Hamiltonian in Rindler frame. The difference $\delta E$ is defined as a
difference in expectation values of the Rindler Hamiltonian between the two
states.

If the perturbation on the Rindler density matrix $\rho_{R0}$ is very small,
we have the new density matrix $\rho_{R1}=\rho_{R0}+\delta\rho$, with
“$\delta\rho\ll\rho_{R0}$”. In the limit of large numbers $ne^{-E/T}\gg 1$,
the approximation still holds.

The difference in entropies $\delta S$ using a Taylor expansion $\rho_{R0}$ is

$\delta
S\approx-\mbox{Tr}\left[\delta\rho{\delta(\rho\ln\rho)\over\delta\rho}\Big{|}_{\rho=\rho_{R0}}\right]=-\mbox{Tr}[\delta\rho(1+\ln\rho_{R0})]=-\mbox{Tr}[\delta\rho\ln\rho_{R0}],$
(7)

with $\mbox{Tr}\rho_{R1}=\mbox{Tr}\rho_{R0}$.

As the initial density matrix is thermal, we have
$\rho_{R0}=e^{-H/T}/(\mbox{Tr}e^{-H/T})$ and

$\delta
S\approx-\mbox{Tr}[\delta\rho(-H/T)]={\mbox{Tr}[H(\rho_{R1}-\rho_{R0})]\over
T}={\delta E\over T},$ (8)

with $\mbox{Tr}(\delta\rho)=0$.

If by introducing a perturbation, the energy increases by $\delta E$, but the
entropy can’t be increased by more than the amount of thermal energy,
independent of the nature of the excitation. This energy bound arises from
tracing over the left Rindler wedge, being a direct consequence of the
existence of the horizon itself, as a causal barrier between spacetime
regions.

The energy bound will be given by

$\delta S<\delta S_{max}={\delta E\over T}.$ (9)

For a Rindler observer, when quantum mechanical effects are taken into
account, the entropy and energy are linked by the Bekenstein bound, associated
with thermodynamics, the holographic principle and the covariant entropy bound
of quantum gravity. Bekenstein conjectured the entropy bound to be a
universally valid principle for complete systems in nature. Using quantum
statistics, Bekenstein showed that the entropy bound holds for complete
systems with non-interacting massless quantum fields in flat spacetime, idea
criticized by several authors (Unwin, Page, Unruh, Wald, Pelath).

If a system violates this bound (highly entropic object), for a perturbation
with large number of internal states, described by a large entropy with fixed
energy, the entropy and energy appear independent to an inertial observer. The
interesting fact is that this bound does not depend on the gravitational
constant, suggesting a geometric nature of the horizon entropy and ultimately
an emergent origin of gravity.

An observer that falls into a black hole will have access to a different
amount of information compared to another observer that is stationary outside
the horizon. Similarly, a Rindler observer will have access to different
regions of spacetime in comparison to the inertial observer. The inertial
observer measures zero temperature and entropy for the vacuum state, whereas a
Rindler observer measures a finite temperature and non-zero divergent entropy.
Entropy and temperature are apparently observer dependent, due to the effects
of the quantum field on the horizon.

If the field configuration of the vacuum state behind the horizon is traced
over, a density matrix $\rho$ for the field configuration outside the horizon
can be detected. The behaviour of the horizon as a one way membrane blocking
all the information to the outside observer suggests an observer dependent
entropy that can be associated with any causal horizon.[20] The entropy arises
from quantum cross-correlations which exist across the horizon. The quantum
state of the horizon is a correlated state with a component inside the horizon
and another one outside. If the observer traces over the states behind the
horizon, the spacetime outside the horizon will be given by a density
matrix.[21] The spacetime will contain two isolated regions, separated by the
surface horizon whereas the description of each spacetime region will be given
by tracing over the other complementary region. The thermal density matrix and
therefore the temperature of the horizon can be obtained from the integration
of modes that are hidden by the horizon.

The increase in entropy is due to a separate contribution originated in the
quantum field within the spacetime near the horizon, concept closely related
to the “entanglement entropy” generated by correlations between quantum fields
on each side of the horizon. The entropy measures the lack of information
about the measurement outcomes implied by the Heisenberg uncertainty relation
and at the same time measures the amount of entanglement between the two sides
of the horizon. In other words, the von Neumann entropy describes the observer
statistical ignorance of the specific microstate (a translation of the
classical entropy of Gibbs and Boltzmann) while expressing the amount of
entanglement.[22] This idea supports a possible equivalence between
interpretations of von Neumann entanglement entropy and Boltzmann statistical
entropy concepts.

An analysis of the thermodynamics across a horizon discloses a strong relation
between thermality and von Neumann entropy describing entanglement. The
observer interacts with the quantum field observables defining the quantum
state as a density matrix. The entropy will quantify the uncertainty, due to
entanglement of measurements of the quantum field seen by an accelerated
observer. S. Lloyd has already proposed the idea that quantum uncertainty
gives rise to entanglement, showing that the Heisenberg uncertainty principle
and quantum entanglement and are two inextricably connected phenomena.
Jacobson derived Einstein’s equations from the vacuum entanglement entropy, on
the understanding of a connection between the semi-classical Einstein
equations and a maximal vacuum entanglement hypothesis.[23]

If the horizon entropy is given by the Bekenstein-Hawking formula, then the
second law of thermodynamics leads to Einstein’s equations. More recently,
Ryu-Takayanagi have shown that there is a deeper relationship between
entanglement and gravity in the AdS/CFT conjecture, where the entanglement
entropy between a CFT patch and its neighbourhood is given a minimal AdS area
with the same boundary as the patch. In the context of the AdS/CFT
correspondence, entanglement entropy is defined on the AdS boundary and
proportional to the area of a minimal surface in the bulk of the AdS
spacetime.

In Rindler spacetime, the observer will have access to the right Rindler wedge
and observe in a natural way all the thermal fluctuations, as the observer
attributes a density matrix to a quantum state only after integrating out the
unobservable modes. Here, the thermal fluctuations have a quantum mechanical
origin, suggesting an artificial distinction between quantum and thermal
properties applied to the horizon that should disappear in a correct complete
theory of quantum gravity.

As a direct consequence, any matter plunging towards the horizon will take an
infinite amount of time in order to cross the horizon for the outside
observer.[24] On the other hand, the quantum effects will smear the location
of the horizon. Any description of the location of the horizon has to take
into account these quantum fluctuations. The physics loses its predictive
power about what happens to the information encoded at the horizon. [25] Both
location of the horizon and the crossing time are unpredictable to the outside
observer.

The Rindler observer (or an observer outside a black hole horizon) attributes
a finite temperature to the inertial vacuum and associates an entropy change
to a horizon while the inertial observer (or an observer falling through the
horizon) sees none of these phenomena.

The entropy of matter crossing the horizon is $\delta S=\delta E/T_{\rm
horizon}$ where the horizon plays the role of a causal barrier for the Rindler
observer, with inaccessible internal degrees of freedom and temperature
$T_{\rm horizon}$.

The concept of entropy is associated with the missing information but this
does not lead to non zero entropy, leading to the “information loss problem”.
[26] The horizon hides information only from the point of view of an external
observer. From this perspective, the Rindler entropy is observer dependent.

The connection between entanglement entropies and areas of extremal surfaces
leads us to the conclusion that the thermodynamic properties of a system need
to be conceived as observer dependent, depending on the spacetime region where
the system is perceived. The entanglement entropy for a quantum field theory
will obey the area law. The quantum correlations between the interior and the
exterior of the horizon will account for the black hole entropy.

Entanglement entropy was already proposed as a probe of the architecture of
the spacetime in quantum gravity. The point here is that the quantum source of
black hole entropy is associated with the thermal behaviour of the spacetime,
suggesting a deeper connection between entanglement and thermodynamics. [27]
As a consequence, a consistent and complete theory of quantum gravity should
be built in a formalism using “histories” and “path integral” formulations of
quantum mechanics rather than the spatial (“3 + 1”) canonical quantization. In
fact, the area law itself, that gives a geometric character to entropy, brings
the unexpected link between gravity, spacetime geometry, thermodynamics and
quantum field theory. The horizon will play the role of an entangling membrane
that separates the degrees of freedom between the the two sides of a black
hole. [28]

This idea is also backing up induced gravity models like the eternal AdS black
hole, in AdS/CFT conjecture, with the horizon entropy being defined as the
entanglement entropy between the microscopic degrees of freedom of the
boundary CFT and its thermofield double. [29] The area law is also confirmed
by loop quantum gravity [30] and spin foam theory, [31] with horizon entropy
defined by the entanglement entropy of spin-network connections crossing an
entangling surface.

3\. Conclusions

The paradigm described in this paper, relying on a study of general causal
horizons,[32] is that spacetime itself (not just the black hole interior or
the event horizon) has entropy and can be described in geometric, macroscopic
terms (area law) and thermodynamic concepts, as a manifestation of a deeper
quantum nature of spacetime that at this time eludes us. The obvious
consequence of associating the entropy of causal horizons to the missing or
hidden information is the fact that the entropy is proportional to horizon
area, with respect to the observer that perceives the horizon.

The thermodynamic route to gravity brings many conceptual ramifications. In
such a paradigm where the entanglement of quantum fields can be considered the
source of entropy and has a geometrical nature,[33] any consistent quantum
gravity theory should take into account the thermodynamic properties of
spacetime. Entropy encodes all information about the dynamics of spacetime and
finding its extreme will lead to Einstein field equations, as Jacobson
demonstrated. In a thermodynamic description, the presence of entropy leads to
equipartition of energy among the degrees of freedom and consistent field
equations. [34]

An observer plunging into a black hole will not perceive its event horizon in
the same way as an observer orbiting the black hole. Therefore observers in
different states of motion will have access to distinct patches of spacetime.
The plunging observer measures distinct features compared to the observer near
the black hole. At first glance, entropy appears to be an observer dependent
notion. At the same time, each observer is located in a different region of
the spacetime and will measure the properties of the horizon, from within a
distinct spacetime patch and relative to the quantum field associated with
that region. A local Rindler observer perceives a local causal horizon
together with its thermal behaviour associated to an event. A connection
between quantum fluctuations and thermal fluctuations must exist here.

This hypothesis is intimately related to the idea that the field
equations,[35] therefore gravitational dynamics are associated to horizon
thermodynamics. For a spacetime with horizons, the quantum field theory for
this spacetime will possess a horizon with the properties of a black body of
finite temperature. A connection between horizon thermality and gravity
becomes obvious here.

From a different perspective, the emergence of classical spacetime continuum
[36] seems directly related to the quantum entanglement of degrees of freedom
in a non-perturbative description of quantum gravity. [37] [38] A good
understanding of how classical spacetime emerges from an underlying quantum
system [39]would give us insight into the full quantum nature of spacetime.

## References

  * [1]
  * [2] J. D. Bekenstein. Black Holes And Entropy, Phys. Rev. D 7, 2333, 1973; 23, 287, 1981; 49, 1912, 1994.
  * [3] J. D. Bekenstein. Quantum information and quantum black holes, arXiv:gr-qc/0107049.
  * [4] J. D. Bekenstein. Holographic bound from second law, arXiv:gr-qc/0007062.
  * [5] T. Jacobson. A Note On Hartle-Hawking Vacua, Phys. Rev. D 50, 6031, 1994, arXiv:gr-qc/9407022.
  * [6] T. Jacobson. Comment on *Accelerated detectors and temperature in (anti) de Sitter spaces, Class. Quant. Grav. 15, 251, 1998, arXiv:gr-qc/9709048.
  * [7] T. Padmanabhan. Classical and Quantum Thermodynamics of horizons in spherically symmetric spacetimes, Class.Quan.Grav. 19, 5387, 2002, arXiv:gr-qc/0204019.
  * [8] T. Padmanabhan and Apoorva Patel. Semiclassical quantization of gravity I: Entropy of horizons and the area spectrum, 2003, arXiv:hep-th/0305165.
  * [9] Padmanabhan T. Entropy of Static Spacetimes and Microscopic Density of States, Class.Quan.Grav., 21, 4485, 2004, arXiv:gr-qc/0308070
  * [10] E M Howard. Geometric aspects of Extremal Kerr black hole entropy, Journal of Modern Physics, 2013, 4, 3, pp. 357-363, arXiv:gr-qc/1511.00594
  * [11] K.S. Thorne, R.H. Price and D.A. Macdonald. Black Holes: The Membrane Paradigm, Yale University Press, London, 1986; F.J. Alexander, U. H. Gerlach, Phys.Rev. D44, 3887-3898, 1991.
  * [12] Birrell N.D and Davies P.C.W. Quantum fields in curved space. Cambridge University Press, Cambridge, 1982.
  * [13] L. Bombelli, R. K. Koul, J. Lee and R. D. Sorkin. A Quantum Source of Entropy for Black Holes, Phys. Rev. D 34, 373, 1986.
  * [14] R.D. Sorkin. The Statistical Mechanics of Black Hole Thermodynamics, in R.M. Wald (ed.) Black Holes and Relativistic Stars, (U. of Chicago Press, 1998), pp. 177-194, arXiv:gr-qc/9705006.
  * [15] R.D. Sorkin. Toward an Explanation of Entropy Increase in the Presence of Quantum Black Holes, Phys. Rev. Lett. 56 1885-1888, 1986.
  * [16] R. Bousso. A Covariant Entropy Conjecture, JHEP 9907, 004, 1999, arXiv:hep-th/9905177.
  * [17] W. G. Unruh and R. M. Wald. Acceleration Radiation And Generalized Second Law Of Thermodynamics, Phys. Rev. D 25, 942, 1982.
  * [18] W. G. Unruh and N. Weiss. Acceleration Radiation In Interacting Field Theories, Phys. Rev. D 29, 1656, 1984.
  * [19] E M Howard. Causal structure of general relativistic spacetimes, AIP Conf. Proc. 1246, 127, 2010, arXiv:gr-qc/1601.06864
  * [20] T. Jacobson, R. Parentani. Horizon Entropy, Found.Phys. 33, 323-348, 2003, arXiv:gr-qc/0302099.
  * [21] S. N. Solodukhin. Entanglement entropy of black holes, Living Rev. Rel. 14, 8, 2011, arXiv:hep-th/1104.3712.
  * [22] Padmanabhan T. Gravitation: Foundations and Frontiers, Cambridge University Press UK, 2010.
  * [23] Kothawala D. The thermodynamic structure of Einstein tensor, arXiv:1010.2207.
  * [24] R. M. Wald. The thermodynamics of black holes, Living Rev. Rel. 4, 6, 2001, arXiv:gr-qc/9912119.
  * [25] V. P. Frolov and I. Novikov. Dynamical origin of the entropy of a black hole, Phys. Rev. D 48, 4545, 1993, arXiv:gr-qc/9309001.
  * [26] S. Carlip. Entropy from Conformal Field Theory at Killing Horizons, Class. Quant. Grav. A17, 923, 2002, arXiv:gr-qc/9906126.
  * [27] T. Jacobson. Thermodynamics of Spacetime: The Einstein Equation of State Phys. Rev. Lett. 75, 1260, 1995, arXiv:gr-qc/9504004.
  * [28] A. O. Barvinsky, V. P. Frolov and A. I. Zelnikov. Wave Function Of A Black Hole And The Dynamical Origin Of Entropy, Phys. Rev. D 51, 1741, 1995, arXiv:gr-qc/9404036.
  * [29] R. Laflamme. Geometry And Thermofields, Nucl. Phys. B 324, 233, 1989.
  * [30] H. Sahlmann. Black hole horizons from within loop quantum gravity, Phys. Rev. D 84, 044049, 2011, arXiv:gr-qc/1104.4691.
  * [31] S. Alexandrov, M. Geiller and K. Noui. Spin Foams and Canonical Quantization, SIGMA 8, 055, 2012, arXiv:gr-qc/1112.1961.
  * [32] E M Howard. Causal Stability Conditions for General Relativistic Spacetimes, Journal of Physical Science and Application 2 (7), 258-264, 2012, arXiv:gr-qc/1601.05609
  * [33] C. G. Callan, Jr. and F. Wilczek. On geometric entropy, Phys. Lett. B 333, 55, 1994, arXiv:hep-th/9401072.
  * [34] J. D. Bekenstein. Holographic bound from second law of thermodynamics, Phys. Lett. B 481, 339, 2000, arXiv:hep-th/0003058.
  * [35] L.D. Landau, E.M. Lifshitz. The Classical Theory of Fields, Fourth edition, ergamon, 1975.
  * [36] G. ’t Hooft. Dimensional Reduction In Quantum Gravity, arXiv:gr-qc/9310026.
  * [37] L. Susskind. The World as a hologram, J. Math. Phys. 36, 6377, 1995, arXiv:hep-th/9409089.
  * [38] W. Fischler and L. Susskind. Holography and cosmology, 1998, arXiv:hep-th/9806039.
  * [39] Hawking S.W. Particle creation by black holes, Commun.Math.Phys, 43, 199, 1975
  * [40]


# Plan, Eliminate, and Track —
Language Models are Good Teachers for Embodied Agents.

Yue Wu  So Yeon Min  Yonatan Bisk  Ruslan Salakhutdinov  Amos Azaria  Yuanzhi
Li  Tom M. Mitchell  Shrimai Prabhumoye

###### Abstract

Pre-trained large language models (LLMs) capture procedural knowledge about
the world. Recent work has leveraged LLM’s ability to generate abstract plans
to simplify challenging control tasks, either by action scoring, or action
modeling (fine-tuning). However, the transformer architecture inherits several
constraints that make it difficult for the LLM to directly serve as the agent:
e.g. limited input lengths, fine-tuning inefficiency, bias from pre-training,
and incompatibility with non-text environments. To maintain compatibility with
a low-level trainable actor, we propose to instead use the knowledge in LLMs
to simplify the control problem, rather than solving it.

We propose the Plan, Eliminate, and Track (PET) framework. The Plan module
translates a task description into a list of high-level sub-tasks. The
Eliminate module masks out irrelevant objects and receptacles from the
observation for the current sub-task. Finally, the Track module determines
whether the agent has accomplished each sub-task. On the AlfWorld instruction
following benchmark, the PET framework leads to a significant 15% improvement
over SOTA for generalization to human goal specifications.

AlfWorld, Text Games, Large Language Models, In-context prompting, Language
Models, Zero-shot, Few-shot

##  1 Introduction

Humans can abstractly plan their everyday tasks without execution; for
example, given the task “Make breakfast”, we can roughly plan to first pick up
a mug and make coffee, before grabbing eggs to scramble. Embodied agents,
endowed with this capability will generalize more effectively by leveraging
common-sense reasoning.

Recent work (Huang et al., 2022a, b; Ahn et al., 2022; Yao et al., 2020) has
used LLMs (Bommasani et al., 2021) for abstract planning for embodied or
gaming agents. These have shown incipient success in extracting procedural
world knowledge from LLMs in linguistic form with posthoc alignment to
executable actions in the environment. However, they treat LLMs as the actor,
and focus on adapting LLM outputs to executable actions either through fine-
tuning (Micheli & Fleuret, 2021) or constraints (Ahn et al., 2022). Using LLM
as the actor works for pure-text environments with limited interactions (Huang
et al., 2022b; Ahn et al., 2022) (just consisting of “picking/placing”
objects), but limits generalization to other modalities. In addition, the
scenarios considered have been largely simplified from the real world. Ahn et
al. (2022) provides all available objects and possible interactions at the
start and limits tasks to the set of provided objects/interactions. Huang et
al. (2022b) limits the environment to objects on a single table.

Figure 1: PET framework. Plan module uses LLM to generate a high-level plan.
Eliminate Module uses a QA model to mask irrelevant objects in observation.
Track module uses a QA model to track the completion of sub-tasks.

On the other hand, to successfully “cut some lettuce” in a real-world room,
one has to “find a knife”, which can be non-trivial since there can be
multiple drawers or cabinets (Chaplot et al., 2020; Min et al., 2021; Blukis
et al., 2021). A more realistic scenario leads to a diverse, complicated set
of tasks or large and changing action space. Furthermore, the text description
of the observation increases as a function of the number of receptacles and
objects the agent sees. Combined with growing roll-outs, the state becomes too
verbose to fit into any LLM.

In this work, we explore alternative mechanisms to leverage the prior
knowledge encoded in LLMs without impacting the trainable nature of the actor.
We propose a 3-step framework (Figure 1): Plan, Eliminate, and Track (PET).
Plan module simplifies complex tasks by breaking them down into sub-tasks. It
uses a pre-trained LLM to generate a list of sub-tasks for an input task
description employing example prompts from the training set similar to Huang
et al. (2022a); Ahn et al. (2022). The Eliminate module addresses the
challenge of long observations. It uses a zero-shot QA language model to score
and mask objects and receptacles that are irrelevant to the current sub-task.
The Track module uses a zero-shot QA language model to determine if the
current sub-task is complete and moves to the next sub-task. Finally, the
Action Attention agent uses a transformer-based architecture to accommodate
for long roll-out and variable length action space. The agent observes the
masked observation and takes an action conditioned on the current sub-task.

We focus on instruction following in indoor households on the AlfWorld
(Shridhar et al., 2020b) interactive text environment benchmark. Our
experiments and analysis demonstrate that LLMs not only remove 40% of task-
irrelevant objects in observation through common-sense QA, but also generate
high-level sub-tasks with 99% accuracy. In addition, multiple LLMs may be used
in coordination with each other to assist the agent from different aspects.

Our contributions are as follows:

  1. 1.

PET: A novel framework for leveraging pre-trained LLMs with embodied agents;
our work shows that each of P, E, T serves a complementary role and should be
simultaneously addressed to tackle control tasks.

  2. 2.

An Action Attention agent that handles the changing action space for text
environments.

  3. 3.

A 15% improvement over SOTA for generalization to human goals via sub-task
planning and tracking.

##  2 Related Work

#### Language Conditioned Policies

A considerable portion of prior work studies imitation learning (Tellex et
al., 2011; Mei et al., 2016; Nair et al., 2022; Stepputtis et al., 2020; Jang
et al., 2022; Shridhar et al., 2022; Sharma et al., 2021) or reinforcement
learning (Misra et al., 2017; Jiang et al., 2019; Cideron et al., 2020; Goyal
et al., 2021; Nair et al., 2022; Akakzia et al., 2020) policies conditioned on
natural language instruction or goal (MacMahon et al., 2006; Kollar et al.,
2010). While some prior research has used pre-trained language embeddings to
improve generalization to new instructions (Nair et al., 2022), they lack
domain knowledge that is captured in LLMs. Our PET framework enables planning,
progress tracking, and observation filtering through the use of LLMs, and is
designed to be compatible with any language conditional policies above.

#### LLMs for Control

LLMs have recently achieved success in high-level planning. Huang et al.
(2022a) shows that pre-trained LLMs can generate plausible plans for day-to-
day tasks, but the generated sub-tasks cannot be directly executed in an end-
to-end control environment. Ahn et al. (2022) solves the executability issue
by training an action scoring model to re-weigh LLM action choices and
demonstrates success on a robot. However, LLM scores work for simple
environments with actions limited to pick/place (Ahn et al., 2022), but fails
with environments with more objects and diverse actions (Shridhar et al.,
2020b). Song et al. (2022) uses GPT3 to generate step-by-step low-level
commands, which are then executed by respective control policies. the work
improves Ahn et al. (2022) with more action diversity and on-the-fly re-plan.
In addition, all the above LLMs require few-shot demonstrations of up to 17
examples, making the length of the prompt infeasible for AlfWorld. Micheli &
Fleuret (2021) fine-tuned a GPT2-medium model on expert trajectories in
AlfWorld and demonstrated impressive evaluation results. However, LM fine-
tuning requires a fully text-based environment, consistent expert
trajectories, and a fully text-based action space. Such requirements greatly
limit the generalization to other domains, and even to other forms of task
specification. We show that our PET framework achieves better generalization
to human goal specifications which the agents were not trained on.

#### Hierarchical Planning with Natural Language

Due to the structured nature of natural language, Andreas et al. (2017)
explored associating each task description to a modular sub-policy. Later
works extend the above approach by using a single conditional policy (Mei et
al., 2016), or by matching sub-tasks to templates (Oh et al., 2017). Recent
works have shown that LLMs are proficient high-level planners (Huang et al.,
2022a; Ahn et al., 2022; Lin et al., 2022), and therefore motivates us to
revisit the idea of hierarchical task planning with progress tracking. To our
knowledge, PET is the first work combining a zero-shot subtask-level LLM
planner and zero-shot LLM progress tracker with a low-level conditional sub-
task policy.

#### Text Games

Text-based games are complex, interactive simulations where the game state and
action space are in natural lanugage. They are fertile ground for language-
focused machine learning research. In addition to language understanding,
successful play requires skills like memory and planning, exploration (trial
and error), and common sense. The AlfWorld (Shridhar et al., 2020b) simulator
extends a common text-based game simulator, TextWorld Côté et al. (2018a), to
create text-based analogs of each ALFRED scene.

#### Agents for Large Action Space

He et al. (2015) learns representation for state and actions with two
different models and computes the Q function as the inner product of the
representations. While this could generalize to large action space, they only
considered a small number of actions.

Fulda et al. (2017); Ahn et al. (2022) explore action elimination in the
setting of affordances. Zahavy et al. (2018) trains a model to eliminate
invalid actions on Zork from external environment signals. However, the
functionality depends on the existence of external elimination signal.

##  3 Plan, Eliminate, and Track

In this section, we explain our 3-step framework: Plan, Eliminate, and Track
(PET). In Plan module ($\mathcal{M}_{\textbf{P}}$), a pre-trained LLM
generates a list of sub-tasks for an input task description using samples from
the training set as in-context examples. The Eliminate module
($\mathcal{M}_{\textbf{E}}$) uses a zero-shot QA language model to score and
mask objects and receptacles that are irrelevant to the current sub-task. The
Track module ($\mathcal{M}_{\textbf{T}}$) uses a zero-shot QA language model
to determine if the current sub-task is complete and moves to the next sub-
task. Note that Plan is a generative task and Eliminate and Track are
classification tasks.

We also implement an attention-based agent (Action Attention), which scores
each permissible action and is trained on imitation learning on the expert.
The agent observes the masked observation and takes an action conditioned on
the current sub-task.

#### Problem Setting

We define the task description as $\mathcal{T}$, the observation string at
time step $t$ as $\mathcal{O}^{t}$, and the list of permissible actions
$\left\\{a_{i}^{t}|a_{i}^{t}\text{ can be executed}\right\\}$ as $A^{t}$. For
each observation string $\mathcal{O}^{t}$, we define the receptacles and
objects within the observation as $r_{i}^{t}$ and $o_{i}^{t}$ respectively.
The classification between receptacles and objects is defined by the
environment (Shridhar et al., 2020b). For a task $\mathcal{T}$, we assume
there exists a list of sub-tasks $\mathcal{S}_{\mathcal{T}}=\\{s_{1},\ldots
s_{k}\\}$ that solves $\mathcal{T}$.

###  3.1 Plan

Figure 2: Plan Module (Sub-task Generation). 5 full examples are chosen from
the training set based on RoBERTa embedding similarity with the task query
description. Then the examples are concatenated with the task query to get the
prompt. Finally, we prompt the LLM to generate the desired sub-tasks.

Tasks in the real world are often complex and need more than one step to be
completed. Motivated by the ability of humans to plan high-level sub-tasks
given a complex task, we design the Plan module ($\mathcal{M}_{\textbf{P}}$)
to generate a list of high-level sub-tasks for a task description
$\mathcal{T}$.

Inspired by the contextual prompting techniques for planning with LLMs (Huang
et al., 2022a), we use an LLM as our plan module $\mathcal{M}_{\textbf{P}}$.
For a given task description $\mathcal{T}$, we compose the query question
$\mathcal{Q}_{\mathcal{T}}$ as “What are the middle steps required to
$\mathcal{T}$?”, and require $\mathcal{M}_{\textbf{P}}$ to generate a list
sub-tasks $\mathcal{S}_{\mathcal{T}}=\\{s_{1},\ldots s_{k}\\}$.

Specifically, we select the top $5$ example tasks $\mathcal{T}^{E}$ from the
training set based on RoBERTa (Liu et al., 2019) embedding similarity with the
query task $\mathcal{T}$. We then concatenate the example tasks with example
sub-tasks in a query-answer format to build the prompt
$\mathcal{P}_{\mathcal{T}}$ for $\mathcal{M}_{\textbf{P}}$ (Fig. 2):

$\mathcal{P}_{\mathcal{T}}=\mathtt{concat}(\mathcal{Q}_{\mathcal{T}^{E}_{1}},\mathcal{S}_{\mathcal{T}^{E}_{1}},\ldots,\mathcal{Q}_{\mathcal{T}^{E}_{5}},\mathcal{S}_{\mathcal{T}^{E}_{5}},\mathcal{Q}_{\mathcal{T}})$

An illustration of our prompt format is shown in Figure 2, where
$\mathcal{T}=$“heat some apple and put it in fridge”, and
$\mathcal{Q}_{\mathcal{T}^{E}_{1}}=$“What are the middle steps required to put
two spraybottles on toilet”, $\mathcal{S}_{\mathcal{T}^{E}_{1}}=$“take a
spraybottle, place the spraybottle in/on toilet, take a spraybottle, place the
spraybottle in/on toilet”. The expected list of sub-tasks to achieve this task
$\mathcal{T}$ is $s_{1}=$‘take an apple’, $s_{2}=$‘heat the apple’, and
$s_{3}=$‘place the apple in/on fridge’

Figure 3: Eliminate Module (Receptacle Masking). We use a pre-trained QA model
to filter irrelevant receptacles/objects in the observation of each scene. As
we can see, the original observation is too long and the receptacles shown in
red are not relevant for task completion. These receptacles are filtered by
the QA model making the observation shorter.

###  3.2 Eliminate

Typical Alfworld scenes can start with around 15 receptacles, each containing
up to 15 objects. In some close-to-worst cases, there can be around 30 open-
able receptacles (e.g. a kitchen with many cabinets and drawers), and it
easily takes an agent with no prior knowledge more than 50 steps for the agent
to find the desired object (repeating the process of visiting each receptacle,
opening it, closing it). We observe that many receptacles and objects are
irrelevant to specific tasks during both training and evaluation, and can be
easily filtered with common-sense knowledge about the tasks. For example, in
Fig. 3 the task is to heat some apple. By removing the irrelevant receptacles
like the coffeemachine, garbagecan, or objects like knife, we could
significantly shorten our observation. We therefore propose to leverage
commonsense knowledge captured by large pre-trained QA models to design our
Eliminate module $\mathcal{M}_{\textbf{E}}$ to mask out irrelevant receptacles
and objects.

For task $\mathcal{T}$, we create prompts in the format
$\mathcal{P}_{r}=$“Your task is to: $\mathcal{T}$. Where should you go to?”
for receptacles and $\mathcal{P}_{o}=$“Your task is to: $\mathcal{T}$. Which
objects will be relevant?” for objects. Using the pre-trained QA model
$\mathcal{M}_{\textbf{E}}$ in a zero-shot manner, we compute score
$\mu_{o_{i}}=\mathcal{M}_{\textbf{E}}(\mathcal{P}_{o},o_{i})$ for each object
$o_{i}$ and $\mu_{r_{i}}=\mathcal{M}_{\textbf{E}}(\mathcal{P}_{o},r_{i})$ for
each receptacle $r_{j}$ in observation at every step. $\mu$ represents the
belief score of whether the common-sense QA model believes the
object/receptacle is relevant to $\mathcal{T}$. We then remove $o_{i}$ from
observation if $\mu_{o_{i}}<\tau_{o}$, and remove $r_{i}$ if
$\mu_{r_{i}}<\tau_{r}$. Threshold $\tau_{o},\tau_{r}$ are hyper-parameters.

Figure 4: Track Module (Progress Tracking). At every step, we take the last 3
steps of roll-out as context and append a query (about whether the current
sub-task is completed) to get the prompt. A pre-trained QA model generates a
Yes/No answer to the prompt. For the answer “Yes”, we update the tracker to
the next sub-task.

###  3.3 Track

For the agent to utilize the high-level plan, it first needs to know which
sub-task to execute. A human actor typically starts from the first item and
check-off the tasks one by one until completion. Therefore, similar to Section
3.2, we use a pre-trained QA model to design the Track module
$\mathcal{M}_{\textbf{T}}$ to perform zero-shot sub-task completion
detection.111Note that the current system design does not allow re-visiting
finished sub-tasks, so the agent has no means to recover if it undoes its
previous sub-task at test time.

Specifically, as illustrated in Figure 4, for sub-task list
$\mathcal{S}_{\mathcal{T}}=\\{s_{1},\ldots s_{k}\\}$, we keep track of a
progress tracker $p$ (initialized at $1$) that indicates the sub-task the
agent is currently working on $(s_{p})$. We then compose the context as the
last $d$ steps of the agent observation for the current sub-task and the
question as “Did you finish the task of $s_{p}$?”. For efficiency, we set
$d:=\min(d+1,3)$ at each step. Note that $d$ is reset to $1$ whenever the
progress tracker updates. Hence, the template
$\mathcal{P}_{a}=\mathtt{concat}(\mathcal{O}^{t-d},\ldots,\mathcal{O}^{t-1},$
“Did you finish the task of $s_{p}$?”$)$. We feed $\mathcal{P}_{a}$ to a pre-
trained zero-shot QA model $\mathcal{M}_{\textbf{T}}$ and compute the
probability of tokens ‘Yes’ and ‘No’ as follows:
$p_{\mathcal{M}_{\textbf{T}}}(``Yes"|\mathcal{P}_{a})$ and
$p_{\mathcal{M}_{\textbf{T}}}(``No"|\mathcal{P}_{a})$. If
$p_{\mathcal{M}_{\textbf{T}}}(``Yes"|\mathcal{P}_{a})>p_{\mathcal{M}_{\textbf{T}}}(``No"|\mathcal{P}_{a})$
then we increment the tracker $p$ to track the next sub-task.

If the tracking ends prematurely, meaning that
$p>len(\mathcal{S}_{\mathcal{T}})$ but the environment has not returned
“done”, we fall back to conditioning with $\mathcal{T}$. We study the rate of
pre-mature ends in Section 4.4 in terms of precision and recall.

Figure 5: Agent (Action Attention). Action Attention block is a transformer-
based framework that computes a key $K_{i}$ for each permissible action and
output action scores as dot-product between key and query $Q$ from the
observations.

###  3.4 Agent

Since the number of permissible actions can vary a lot by the environment, the
agent needs to handle arbitrary dimensions of action space. While Shridhar et
al. (2020b) addresses this challenge by generating actions token-by-token,
such a generation process leads to degenerate performance even on the training
set.

We draw inspiration from the field of text summarization, where models are
built to handle variable input lengths. See et al. (2017) generates a summary
through an attention-like “pointing” mechanism that extracts the output word
by word. Similarly, an attention-like “pointing” model could be used to select
an action from the list of permissible actions.

#### Action Attention

We are interested in learning a policy $\pi$ that outputs the optimal action
among permissible actions. We eschew the long rollout/ large action space
problems by (1) representing observations by averaging over history, and (2)
individually encoding actions (Fig 5). In our proposed action attention
framework, we first represent historical observations $H^{t}$ as the average
of embeddings of all individual observations through history (Eq. 1), and
$H^{A}$ as the list of embeddings of all the current permissible actions (Eq.
2). Then, in Eq. 3, we compute the query $Q$ using a transformer with a
“query” head ($\mathcal{M_{Q}}$) on task embedding ($H^{t}$), the current
observation embedding ($\mathcal{O}^{t}$), and the list of action embeddings
($H^{A}$). In Eq. 4 we compute the key $K_{i}$ for each action $a_{i}$ using
the same transformer with a “key” head ($\mathcal{M_{K}}$) on task embedding
($H^{t}$), the current observation embedding ($\mathcal{O}^{t}$), and
embedding of action ($a_{i}$).

Finally, we compute the dot-product of the query and keys as action scores for
the policy $\pi$ (Eq. 5).

$\displaystyle H^{t}$
$\displaystyle=\text{avg}_{j\in[1,t-1]}\text{Embed}(\mathcal{O}^{j})$ (1)
$\displaystyle H^{A}$
$\displaystyle=\left[\text{Embed}(a_{1}^{t}),...,\text{Embed}(a_{n}^{t})\right]$
(2) $\displaystyle Q$
$\displaystyle=\mathcal{M_{Q}}\left(\text{Embed}(\mathcal{T}),H^{t},\text{Embed}(\mathcal{O}^{t}),H^{A}\right)$
(3) $\displaystyle K_{i}$
$\displaystyle=\mathcal{M_{K}}\left(\text{Embed}(\mathcal{T}),H^{t},\text{Embed}(\mathcal{O}^{t}),\text{Embed}(a_{i}^{t})\right)$
(4) $\displaystyle\pi$ $\displaystyle=\text{softmax}\left([Q\cdot
K_{i}|i\in\text{all permissible actions}]\right)$ (5)

##  4 Experiments and Results

We present our experiments as follows. First, we explain the environment setup
and baselines for our experiments. Then we compare PET to the baselines on
different splits of the environment. Finally, we conduct ablation studies and
analyze the PET framework part by part. We show that PET generalizes better to
human goal specification under efficient behavior cloning training.

###  4.1 Experimental Details

| Template Goal Specification | Human Goal Specification
---|---|---
Model | seen | unseen | seen | unseen
BUTLER + DAgger* (Shridhar et al., 2020b) | 40 | 35 | 8 | 3
BUTLER + BC (Shridhar et al., 2020b) | 10 | 9 | - | -
GPT (Micheli & Fleuret, 2021) | 91 |  95 | 42 | 57
PET + Action Attention (Ours) | 70 | 67.5 | 52.5 | 60
Table 1:  Comparison of different models in terms of completion rate per
evaluation split (seen and unseen), with and without human annotated goals.
PET under-performs GPT on Template goal specifications but generalizes better
to human goal specifications. * We include the performance of BUTLER with
DAgger for completeness. All other rows are trained without interaction with
the environment, MLE for GPT and behavior cloning for BUTLER+BC and PET.

#### AlfWorld Environment

ALFWorld (Shridhar et al., 2020b) is a set of TextWorld environments (Côté et
al., 2018b) that are parallels of the ALFRED embodied dataset (Shridhar et
al., 2020a). ALFWorld includes 6 task types that each require solving multiple
compositional sub-goals. There are 3553 training task instances ({tasktype,
object, receptacle, room}), 140 in-distribution evaluation task instances
(seen split - tasks themselves are novel but take place in rooms seen during
training) and 134 out-of-distribution evaluation task instances (unseen split
- tasks take place in novels rooms). An example of the task could be: “Rinse
the egg to put it in the microwave.” Each training instance in AlfWorld comes
with an expert, from which we collected our training demonstration.

#### Human Goal Specification

The crowd-sourced human goal specifications for evaluation contain 66 unseen
verbs and 189 unseen nouns (Shridhar et al., 2020b). In comparison, the
template goals use only 12 ways of goal specification. In addition, the
sentence structure for human goal specification is more diverse compared to
the template goals. Therefore, human goal experiments are good for testing the
generalization of models to out-of-distribution scenarios.

#### Pre-trained LMs.

For the Plan module (sub-task generation), we experimented with the open-
source GPT-Neo-2.7B (Black et al., 2021), and an industry-scale LLM with 530B
parameters (Smith et al., 2022). For the Eliminate module (receptacle/object
masking), we choose Macaw-11b (Tafjord & Clark, 2021), which is reported to
have common sense QA performance on par with GPT3 (Brown et al., 2020) while
being orders of magnitudes smaller. We use a decision threshold of $0.4$ for
Macaw score below which the objects are masked out. For the Track module
(progress tracking), we use the same Macaw-11b model as the Eliminate module
answer to Yes/No questions.

#### Actor Model Design.

Our Action Attention agent ($\mathcal{M_{Q}}$ and $\mathcal{M_{K}}$) is a
12-layer transformer with 12 heads and hidden dimension 384. The last layer is
then fed into two linear heads to generate $K$ and $Q$. For embedding of
actions and observations, we use pre-trained RoBERTa-large (Liu et al., 2019)
with embedding dimension 1024. For sub-task generation, we use ground-truth
sub-tasks for training, and generated sub-tasks from Plan module for
evaluation.

#### Experimental Setup.

Unlike the original benchmark (Shridhar et al., 2020b), we experiment with
models trained with behavior cloning. Although Shridhar et al. (2020b) observe
that models benefit greatly from DAgger training, DAgger assumes an expert
that is well-defined at all possible states, which is inefficient and
impractical. In our experiments, training is 100x slower with DAgger compared
to behavior cloning (3 weeks for DAgger v.s. 6 hours for Behavior Cloning). In
addition, we demonstrate that our models surpass the DAgger training
performance of the BUTLER (Shridhar et al., 2020b) agents trained with DAgger,
even when our agent does not have the option to interact with the environment.

#### Baselines.

Our first baseline is the BUTLER::BRAIN (BUTLER) agent (Shridhar et al.,
2020b), which consists of an encoder, an aggregator, and a decoder. At each
time step $t$, the encoder takes initial observation $s^{0}$, current
observation $s^{t}$, and task string $s_{\text{task}}$ and generates
representation $r^{t}$. The recurrent aggregator combines $r^{t}$ with the
last recurrent state $h^{t-1}$ to produce $h^{t}$, which is then decoded into
a string $a^{t}$ representing action. In addition, the BUTLER agent uses beam
search to get out of stuck conditions in the event of a failed action. Our
second baseline GPT (Micheli & Fleuret, 2021) is a fine-tuned GPT2-medium on
3553 demonstrations from the AlfWorld training set. Specifically, the GPT is
fined-tuned to generate each action step word-by-word to mimic the rule-based
expert using the standard maximum likelihood loss.

###  4.2 Overall Results on Template and Human Goals

We compare the performance of action attention assisted by PET with BUTLER
(Shridhar et al., 2020b) and fine-tuned GPT (Micheli & Fleuret, 2021) in Table
1. For human goal specifications, PET outperforms SOTA (GPT) by 25% on seen
and 5% on the unseen split.

Although PET under-performs GPT on Template goal specifications, GPT requires
fine-tuning on fully text-based expert trajectory and thus loses adaptability
to different environment settings. Qualitatively, on human goal specification
tasks, where the goal specifications are out-of-distribution, GPT often gets
stuck repeating the same action after producing a single wrong move. On the
other hand, since the Plan module of PET is not trained on the task, it
generalizes to the variations for human goal specifications as shown in
Section 4.5. Quantitatively, GPT suffers from a relative $50\%$ performance
drop transferring from template to human-goal specifications, whereas PET
incurs only a $15\sim 25\%$ drop.

The setting closest to PET is BUTLER with behavior cloning (BUTLER + BC).
Since BUTLER + BC performs poorly, we also include DAgger training results.
Nevertheless, action attention assisted by PET outperforms BUTLER with DAgger
by more than 2x while being much more efficient. (Section 4.1)

| Template Goals | Human Goals
---|---|---
LLM | seen | unseen | seen | unseen
GPT-2 (Radford et al., 2019) | 94.29 (0.97) | 87.31 (0.94) | 10.07 (0.62) | 7.98 (0.58)
GPT-Neo-2.7B (Black et al., 2021) | 99.29 (1.00) | 96.27 (0.98) | 4.70 (0.82) | 9.16 (0.80)
MT-NLG (Smith et al., 2022) | 98.57 (0.99) |  100 (1.00) | 40.04 (0.94) | 49.3 (0.94)
Table 2:  Evaluation of different LLMs for Plan module in terms of accuracy and RoBERTa embedding cosine similarity (in brackets) against ground-truth sub-tasks, per evaluation split (seen and unseen), with and without human annotated goals. The MT-NLG with 530B parameters achieves the overall best performance on all dataset splits and greatly exceeds the performance of smaller models on hard tasks with human goal specification. In addition, MT-NLG generates sub-tasks with almost perfect embedding similarity for all tasks. Model Ablations | seen | unseen
---|---|---
Action Attention | 25 | 9
Action Attention + Eliminate | 25 | 11
Action Attention + Plan & Track | 35 | 15
Action Attention + PET | 52.5 | 27.5
Table 3:  Comparison of different ablations of PET trained on a sampled set of
140 demonstrations from the training set, in terms of completion rate per
evaluation split (seen and unseen). Applying Eliminate module alone has an
insignificant effect on overall performance compared to Plan & Track. However,
applying Eliminate module on sub-tasks together with Plan & Track results in a
much more significant performance improvement.

###  4.3 Ablations for Plan, Eliminate, and Track

In Table 3, we analyze the contribution of each PET module by sequentially
adding each component to the action attention agent on 140 training
trajectories sampled from the training set. The data set size is chosen to
match the size of the seen validation set, for an efficient and sparse
setting. Note that we treat Plan and Track as a single module for this
ablation since they cannot work separately.

Adding Plan and Track greatly improves the completion rate relatively by
$60\%$, which provides evidence to our hypothesis that solving some embodied
tasks step-by-step reduces the complexity. We observe a relatively
insignificant $3\%$ improvement in absolute performance when adding Eliminate
without sub-task tracking. On the other hand, when applying Eliminate to sub-
tasks with Plan and Track, we observe more than $60\%$ relative improvement
over just Plan and Track alone. We, therefore, deduce that Plan and Track
boost the performance of Eliminate during evaluation, since it is easier to
remove irrelevant objects when the objective is more focused on sub-tasks.

###  4.4 Automated Analysis of PET modules

#### Plan Module

We experiment with different LLMs such as GPT2-XL (Radford et al., 2019), GPT-
Neo-2.7B (Black et al., 2021), and the 530B parameter MT-NLG (Smith et al.,
2022) models. Table 2 reports the generation accuracy and the RoBERTa (Liu et
al., 2019) embedding cosine similarity against ground-truth sub-tasks. We
observe that all LLMs achieve high accuracy on template goal specifications,
where there is no variation in sentence structures. For human goal
specification, MT-NLG generates subtasks similar to ground truth in terms of
embedding similarity, while the other smaller models perform significantly
worse.

#### Eliminate module

We evaluate the zero-shot receptacle/object masking performance of Macaw on
the three splits of AlfWorld. In Fig 6, we illustrate the AUC curve of the
relevance score that the model assigns to the objects v.s. objects that the
rule-based expert interacted with when completing each task. Since the Macaw
QA model is queried in a zero-shot manner, it demonstrates consistent masking
performance on all three splits of the environment, even on the unseen split.
In addition, we note that object receptacle accuracy is generally lower than
object accuracy because of the counter-intuitive spawning locations described
in Section 4.5. In our experiments, a decision threshold of $0.4$ has a recall
of $0.91$ and reduces the number of objects in observation by $40\%$ on
average.

Figure 6: Plot of AUC scores of zero-shot relevance identification across all
tasks in the Alfworld-Thor environment, with the Macaw-11b model. The ground
truth is obtained as receptacles/objects accessed by the rule-based expert.
Top: Receptacle relevance identification. Bottom: Object relevance
identification. The QA model achieves an average AUC-ROC score of $65$ for
receptacles and $76$ on objects.

#### Track module

Since sub-task alignment information is not provided by the environment, we
explore an alternative performance metric for the detection of the event of
completion. Ideally, a sub-task tracker should record the last sub-task as
“finished” if and only if the environment is “fully solved” by the expert. As
an agreement measure, we report a precision of 0.99 and a recall of 0.78 for
Macaw-11B and a precision of 0.96 and a recall of 0.96 for Macaw-large. The
larger model (Macaw-11b) is more precise but misses more detection, therefore
limiting the theoretical performance to 78%. The smaller model is much less
accurate according to human evaluation but does not limit the overall model
performance in theory. In our experiments, we find that both models produce
similar overall results, which may suggest that the overall results could be
improved with LLMs doing better on both precision and recall.

Human Goal Specification Examples
---
Task |  Chill a cup and place it in the cabinet.
GT |  cool the mug$\rightarrow$place the mug in/on coffeemachine
Gen |  chill the mug$\rightarrow$return the mug to coffeemachine
Task |  Take the pencil from the desk, put it on the other side of the desk
GT |  take a pencil$\rightarrow$place the pencil in/on shelf
Gen |  pick up the white pencil on the desk$\rightarrow$put the white pencil on another spot on the desk
Table 4:  Failure examples from the Plan module on human goal specifications
(Task), ground-truth (GT) v.s. generated (Gen). In the first example,
generated plan differs from the ground truth but the meaning agrees. In the
second example, the generated plan largely differs from the ground truth due
to the mistake in human goal specification — “another side on the desk”
instead of “shelf”.

###  4.5 Qualitative Analysis

#### Plan Module

We show two types of failure examples for sub-task generation in Table 4. The
first type of error is caused by generating synonyms of the ground truth, and
the second type of error is caused by inaccuracies in the human goal
specifications. Note that our Action Attention framework uses RoBERTa (Liu et
al., 2019) embedding for sub-tasks, known to be robust to synonym variations.

#### Eliminate Module

We observe that the main source of elimination error occurs when the module
incorrectly masks a receptacle that contains the object of interest so the
agent fails to find such receptacles. This is often because some objects in
the AI2Thor simulator do not spawn according to common sense. As noted in the
documentation of the
environment222ai2thor.allenai.org/ithor/documentation/objects/object-types/,
objects like Apple or Egg has a chance of spawning in unexpected receptacles
like GarbageCan, or TVStand. However, such generations in AI2Thor are unlikely
in real deployment; thus, the “mistakes” of our Eliminate module are
reasonable.

#### Track Module

Experimentally, we find that sub-task planning/tracking is particularly
helpful for tasks that require counting procedures. As shown in Table 5, PET
breaks the task of “Place two soapbar in cabinet” into two repeating set of
sub-tasks: “take soapbar$\rightarrow$place soapbar in/on cabinet”. Sub-task
planning and tracking, therefore, simplify the hard problem of counting.

##  5 Conclusion, Limitations, and Future Work

In this work, we propose the Plan, Eliminate, and Track (PET) framework that
uses pre-trained LLMs to assist an embodied agent in three steps. Our PET
framework requires no fine-tuning and is designed to be compatible with any
goal-conditional embodied agents.

In our experiments, we combine PET with a novel Action Attention agent that
handles the dynamic action space in AlfWorld. Our Action Attention agent
greatly outperforms the BUTLER baseline. In addition, since the PET framework
is not trained to fit the training set tasks, it demonstrates better
generalization to unseen human goal specification tasks. Finally, our ablation
studies show the Plan and Track modules together improve the performance of
Eliminate module to achieve the best performance.

Our results show that LLMs can be a good source of common sense and procedural
knowledge for embodied agents, and multiple LLMs may be used in coordination
with each other to further improve effectiveness.

One of the major limitations of our current system design is that the Track
module (progress tracker) does not re-visit finished sub-tasks. If for
example, the agent is executing sub-tasks [picked up a pan, put the pan on
countertop], and it picked up a pan but put it in the fridge (undo pickup
action). Since the progress tracker does not take into consideration previous
progress being undone, the system may break in this situation. Future work can
focus on adding sub-task-level dynamic re-planning to address this limitation
or explore other ways in which LLMs can assist the learning of the policy
(i.e., reading an instruction manual about the environment).

## References

  * Ahn et al. (2022) Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jang, E., Ruano, R. J., Jeffrey, K., Jesmonth, S., Joshi, N. J., Julian, R., Kalashnikov, D., Kuang, Y., Lee, K.-H., Levine, S., Lu, Y., Luu, L., Parada, C., Pastor, P., Quiambao, J., Rao, K., Rettinghouse, J., Reyes, D., Sermanet, P., Sievers, N., Tan, C., Toshev, A., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Xu, S., Yan, M., and Zeng, A.  Do as i can, not as i say: Grounding language in robotic affordances, 2022\.  URL https://arxiv.org/abs/2204.01691.
  * Akakzia et al. (2020) Akakzia, A., Colas, C., Oudeyer, P.-Y., Chetouani, M., and Sigaud, O.  Grounding language to autonomously-acquired skills via goal generation.  _arXiv preprint arXiv:2006.07185_ , 2020.
  * Andreas et al. (2017) Andreas, J., Klein, D., and Levine, S.  Modular multitask reinforcement learning with policy sketches.  In _International Conference on Machine Learning_ , pp. 166–175. PMLR, 2017.
  * Black et al. (2021) Black, S., Gao, L., Wang, P., Leahy, C., and Biderman, S.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021.  URL https://doi.org/10.5281/zenodo.5297715.  If you use this software, please cite it using these metadata.
  * Blukis et al. (2021) Blukis, V., Paxton, C., Fox, D., Garg, A., and Artzi, Y.  A persistent spatial semantic representation for high-level natural language instruction execution, 2021.  URL https://arxiv.org/abs/2107.05612.
  * Bommasani et al. (2021) Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel, K., Goodman, N., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D. E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M., Krishna, R., Kuditipudi, R., Kumar, A., Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li, X. L., Li, X., Ma, T., Malik, A., Manning, C. D., Mirchandani, S., Mitchell, E., Munyikwa, Z., Nair, S., Narayan, A., Narayanan, D., Newman, B., Nie, A., Niebles, J. C., Nilforoshan, H., Nyarko, J., Ogut, G., Orr, L., Papadimitriou, I., Park, J. S., Piech, C., Portelance, E., Potts, C., Raghunathan, A., Reich, R., Ren, H., Rong, F., Roohani, Y., Ruiz, C., Ryan, J., Ré, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K., Tamkin, A., Taori, R., Thomas, A. W., Tramèr, F., Wang, R. E., Wang, W., Wu, B., Wu, J., Wu, Y., Xie, S. M., Yasunaga, M., You, J., Zaharia, M., Zhang, M., Zhang, T., Zhang, X., Zhang, Y., Zheng, L., Zhou, K., and Liang, P.  On the opportunities and risks of foundation models, 2021.  URL https://arxiv.org/abs/2108.07258.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chaplot et al. (2020) Chaplot, D. S., Gandhi, D., Gupta, A., and Salakhutdinov, R.  Object goal navigation using goal-oriented semantic exploration, 2020\.  URL https://arxiv.org/abs/2007.00643.
  * Cideron et al. (2020) Cideron, G., Seurin, M., Strub, F., and Pietquin, O.  Higher: Improving instruction following with hindsight generation for experience replay.  In _2020 IEEE Symposium Series on Computational Intelligence (SSCI)_ , pp. 225–232. IEEE, 2020.
  * Côté et al. (2018a) Côté, M.-A., Kádár, A., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Hausknecht, M., Asri, L. E., Adada, M., et al.  Textworld: A learning environment for text-based games.  In _Workshop on Computer Games_ , pp. 41–75. Springer, 2018a.
  * Côté et al. (2018b) Côté, M.-A., Kádár, A., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Hausknecht, M., Asri, L. E., Adada, M., et al.  Textworld: A learning environment for text-based games.  In _Workshop on Computer Games_ , pp. 41–75. Springer, 2018b.
  * Fulda et al. (2017) Fulda, N., Ricks, D., Murdoch, B., and Wingate, D.  What can you do with a rock? affordance extraction via word embeddings.  _arXiv preprint arXiv:1703.03429_ , 2017.
  * Goyal et al. (2021) Goyal, P., Niekum, S., and Mooney, R.  Pixl2r: Guiding reinforcement learning using natural language by mapping pixels to rewards.  In _Conference on Robot Learning_ , pp. 485–497. PMLR, 2021.
  * He et al. (2015) He, J., Chen, J., He, X., Gao, J., Li, L., Deng, L., and Ostendorf, M.  Deep reinforcement learning with a natural language action space.  _arXiv preprint arXiv:1511.04636_ , 2015.
  * Huang et al. (2022a) Huang, W., Abbeel, P., Pathak, D., and Mordatch, I.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents, 2022a.  URL https://arxiv.org/abs/2201.07207.
  * Huang et al. (2022b) Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., Sermanet, P., Brown, N., Jackson, T., Luu, L., Levine, S., Hausman, K., and Ichter, B.  Inner monologue: Embodied reasoning through planning with language models, 2022b.  URL https://arxiv.org/abs/2207.05608.
  * Jang et al. (2022) Jang, E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C., Levine, S., and Finn, C.  Bc-z: Zero-shot task generalization with robotic imitation learning.  In _Conference on Robot Learning_ , pp. 991–1002. PMLR, 2022.
  * Jiang et al. (2019) Jiang, Y., Gu, S. S., Murphy, K. P., and Finn, C.  Language as an abstraction for hierarchical deep reinforcement learning.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Kollar et al. (2010) Kollar, T., Tellex, S., Roy, D., and Roy, N.  Toward understanding natural language directions.  In _2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , pp. 259–266. IEEE, 2010.
  * Lin et al. (2022) Lin, B. Y., Huang, C., Liu, Q., Gu, W., Sommerer, S., and Ren, X.  On grounded planning for embodied tasks with language models.  _arXiv preprint arXiv:2209.00465_ , 2022.
  * Liu et al. (2019) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * MacMahon et al. (2006) MacMahon, M., Stankiewicz, B., and Kuipers, B.  Walk the talk: Connecting language, knowledge, and action in route instructions.  _Def_ , 2(6):4, 2006.
  * Mei et al. (2016) Mei, H., Bansal, M., and Walter, M. R.  Listen, attend, and walk: Neural mapping of navigational instructions to action sequences.  In _Thirtieth AAAI Conference on Artificial Intelligence_ , 2016.
  * Micheli & Fleuret (2021) Micheli, V. and Fleuret, F.  Language models are few-shot butlers.  _arXiv preprint arXiv:2104.07972_ , 2021.
  * Min et al. (2021) Min, S. Y., Chaplot, D. S., Ravikumar, P., Bisk, Y., and Salakhutdinov, R.  Film: Following instructions in language with modular methods, 2021.
  * Misra et al. (2017) Misra, D., Langford, J., and Artzi, Y.  Mapping instructions and visual observations to actions with reinforcement learning.  _arXiv preprint arXiv:1704.08795_ , 2017.
  * Nair et al. (2022) Nair, S., Mitchell, E., Chen, K., Savarese, S., Finn, C., et al.  Learning language-conditioned robot behavior from offline data and crowd-sourced annotation.  In _Conference on Robot Learning_ , pp. 1303–1315. PMLR, 2022.
  * Oh et al. (2017) Oh, J., Singh, S., Lee, H., and Kohli, P.  Zero-shot task generalization with multi-task deep reinforcement learning.  In _International Conference on Machine Learning_ , pp. 2661–2670. PMLR, 2017.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I.  Language models are unsupervised multitask learners.  2019\.
  * See et al. (2017) See, A., Liu, P. J., and Manning, C. D.  Get to the point: Summarization with pointer-generator networks.  _arXiv preprint arXiv:1704.04368_ , 2017.
  * Sharma et al. (2021) Sharma, P., Torralba, A., and Andreas, J.  Skill induction and planning with latent language.  _arXiv preprint arXiv:2110.01517_ , 2021.
  * Shridhar et al. (2020a) Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D.  Alfred: A benchmark for interpreting grounded instructions for everyday tasks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10740–10749, 2020a.
  * Shridhar et al. (2020b) Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M.  Alfworld: Aligning text and embodied environments for interactive learning.  _arXiv preprint arXiv:2010.03768_ , 2020b.
  * Shridhar et al. (2022) Shridhar, M., Manuelli, L., and Fox, D.  Cliport: What and where pathways for robotic manipulation.  In _Conference on Robot Learning_ , pp. 894–906. PMLR, 2022.
  * Smith et al. (2022) Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., Zheng, E., Child, R., Aminabadi, R. Y., Bernauer, J., Song, X., Shoeybi, M., He, Y., Houston, M., Tiwary, S., and Catanzaro, B.  Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model.  _CoRR_ , abs/2201.11990, 2022.  URL https://arxiv.org/abs/2201.11990.
  * Song et al. (2022) Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L., and Su, Y.  Llm-planner: Few-shot grounded planning for embodied agents with large language models.  _arXiv preprint arXiv:2212.04088_ , 2022.
  * Stepputtis et al. (2020) Stepputtis, S., Campbell, J., Phielipp, M., Lee, S., Baral, C., and Ben Amor, H.  Language-conditioned imitation learning for robot manipulation tasks.  _Advances in Neural Information Processing Systems_ , 33:13139–13150, 2020.
  * Tafjord & Clark (2021) Tafjord, O. and Clark, P.  General-purpose question-answering with macaw.  _arXiv preprint arXiv:2109.02593_ , 2021.
  * Tellex et al. (2011) Tellex, S., Kollar, T., Dickerson, S., Walter, M., Banerjee, A., Teller, S., and Roy, N.  Understanding natural language commands for robotic navigation and mobile manipulation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 25, pp. 1507–1514, 2011.
  * Yao et al. (2020) Yao, S., Rao, R., Hausknecht, M., and Narasimhan, K.  Keep calm and explore: Language models for action generation in text-based games, 2020.  URL https://arxiv.org/abs/2010.02903.
  * Zahavy et al. (2018) Zahavy, T., Haroush, M., Merlis, N., Mankowitz, D. J., and Mannor, S.  Learn what not to learn: Action elimination with deep reinforcement learning.  _Advances in neural information processing systems_ , 31, 2018.

##  Appendix A Example Trajectory

Sub-task | Observation + Action
---|---
take a soapbar |  Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, and a towelholder 1.
Action: You go to toilet 1.
On the toilet 1, you see nothing.
Action: You go to sinkbasin 1.
On the sinkbasin 1, you see nothing.
Action: You go to sinkbasin 2.
On the sinkbasin 2, you see nothing.
Action: You go to garbagecan 1.
On the garbagecan 1, you see nothing.
Action: You go to countertop 1.
On the countertop 1, you see a soapbar 2, and a soapbar 1.
Action: You take soapbar 1 from countertop 1.
place the soapbar in/on cabinet |  You pick up the soapbar 1 from the countertop 1.
Action: You go to cabinet 1.
The cabinet 1 is closed. You open cabinet 1
Action: You open cabinet 1.
The cabinet 1 is open. In it, you see a cloth 1.
Action: You put soapbar 1 in/on cabinet 1.
take a soapbar |  You put the soapbar 1 in/on the cabinet 1.
Action: You close cabinet 1.
You close the cabinet 1.
Action: You go to countertop 1.
On the countertop 1, you see a soapbar 2.
Action: You take soapbar 2 from countertop 1.
place the soapbar in/on cabinet |  You pick up the soapbar 2 from the countertop 1
Action: You go to cabinet 1.
The cabinet 1 is closed. You open cabinet 1
Action: You open cabinet 1.
The cabinet 1 is open. In it, you see a cloth 1.
Action: You put soapbar 1 in/on cabinet 1.
Table 5:  Example trajectory of PET agent on the task “Place two soapbar in
cabinet” from unseen evaluation split. The Plan module breaks down the task
into two repeating procedures, and the Track module accurately identifies when
to switch to the next sub-task, thus avoiding the challenging task of
counting. We also observe the Eliminate module at work, removing all
irrelevant objects at steps 2,3,4,5, showing nothing.