# Supplementary Materials for:
A Re-examination of Ellipticity Corrections for Seismic Phases

Stuart Russell1, John F. Rudge1, Jessica C.E. Irving2, Sanne Cottaar1
1\. University of Cambridge, 2. University of Bristol

This document includes:

1. Derivation of ellipticity corrections for seismic phases.

2. Derivation of the ellipticity of figure with radius for a rotating body in hydrostatic equilibrium.

3. Tables of minimum and maximum ellipticity corrections for seismic phases on Earth.

4. Information regarding tables of ellipticity correction coefficients.

##  1 Derivation of Ellipticity Corrections

The following derivation obtains the result presented in the theory section of
the main publication. The derivation is essentially that of Dziewonski &
Gilbert (1976) coupled with the reformulation of Doornbos (1988).

In spherical coordinates, a point can be defined at a radius, $r$, co-
latitude, $\vartheta$, and longitude, $\phi$. A surface of constant velocity
can then be defined as

$R(\vartheta,\phi)=r+\delta r(\vartheta,\phi)$ (1)

where $r$ is the spherically averaged value of $R(\vartheta,\phi)$

$r=\frac{1}{4\pi}\int_{S}R(\vartheta,\phi)\,\mathrm{d}\Omega.$ (2)

The velocity, $v$, on this surface is constant

$v(R(\vartheta,\phi))=v(r+\delta r(\vartheta,\phi))=\text{constant}$ (3)

and by a Taylor expansion

$v(r+\delta r(\vartheta,\phi))=v(r)+\delta r\,v^{\prime}(r)+\mathcal{O}(\delta
r^{2})=\text{constant}.$ (4)

The velocity at $r$ can be defined as the velocity at the surface, $R$, plus a
small perturbation, $\delta v$

$v(r,\vartheta,\phi)=v(R(\vartheta,\phi))+\delta v(r,\vartheta,\phi).$ (5)

By substituting (4) into (5) it is possible to define $\delta v$

$\delta v(r,\vartheta,\phi)=-v^{\prime}{}(r)\delta r(\vartheta,\phi).$ (6)

The relative perturbation of a surface is defined by

$\varsigma(r,\vartheta,\phi)\equiv r^{-1}\delta r(\vartheta,\phi),$ (7)

and can be related to a planet’s ellipticity of figure $\epsilon$ by

$\varsigma(r,\vartheta,\phi)=\epsilon(r)\left(1/3-\cos^{2}\vartheta\right)=-\frac{2}{3}\epsilon(r)P_{2}^{0}(\cos\vartheta).$
(8)

where $P_{l}^{m}(x)$ denotes associated Legendre polynomials of degree $l$ and
order $m$. For seismic waves propagating within an elliptical Earth it is
necessary to analyse the effect of $\delta v$ on the travel time of a ray. The
angular slowness of a ray, $p$, is constant. For a point on a ray with
epicentral coordinates, $(r,\theta)$,

$dt/d\theta=p=rv^{-1}\sin i=\eta\sin i$ (9)

where $i$ is the incident angle (angle from vertical) of the ray, and $\eta$
is the ray slowness defined by

$\eta\equiv\frac{r}{v}.$ (10)

The vertical slowness, which varies along the ray, is defined as

$q\equiv\eta\cos i\equiv(\eta^{2}-p^{2})^{1/2}.$ (11)

If a point moves a small distance, $\delta d$, along the ray path then the
radial distance moved is $\delta r=\delta d/\cos i$. Therefore by using (9)
and (11) to relate $p$, $\eta$ and $q$, the time taken, $\delta t$, is

$\delta t=v^{-1}\delta d=v^{-1}\cos i\,\delta r=\eta r^{-1}\cos i\,\delta
r=\,qr^{-1}\delta r.$ (12)

Let the following notation represent the jump in value of a function, $z(r)$,
across a surface, $r$:

$\left[z(r)\right]_{-}^{+}=\lim_{s\to 0^{+}}\left[z(r+s)-z(r-s)\right].$ (13)

It is then possible to express the effect of moving a discontinuity in $v$ by
a small amount $\delta r$. This is done by taking the difference of (12)
evaluated for a ray travelling between $r+\delta r$ and $r$ in the case of
both the upper and lower layers (propagating in the other layer between
$r+\delta r$ and $r$ is the simplest way to imagine the effect of moving the
discontinuity between the two layers). For transmission this is

$\delta t=-r^{-1}\delta
r\left[q\right]_{-}^{+}=-\varsigma\left[q\right]_{-}^{+}$ (14)

and for reflection is

$\delta t=\mp r^{-1}\delta
r\left(q_{0}+q_{1}\right)=\mp\varsigma\left(q_{0}+q_{1}\right)$ (15)

where the preceding minus/plus refers to top-side/bottom-side reflections,
respectively, and $\eta_{0}$ and $\eta_{1}$ represent the value of $\eta$
before and after interaction with the boundary. In the case where there is no
conversion of wave type due to interaction with the boundary, equation (15)
simplifies to

$\delta t=\mp 2\varsigma q.$ (16)

The source depth must also be deformed to account for the effects of
ellipticity, and this is done using the above equations, treating the source
depth as a discontinuity where the wave does not propagate in the upper medium
(or lower medium in the case of phases with an upwards take-off angle, such as
those preceded by a p or s segment).

The following equation is a result from Julian & Anderson (1968) (their
equation 4) for the effect on travel time, $\delta t$, of a small perturbation
in velocity, $\delta v$, on a ray, by integrating between the bottoming point
of the ray, $r_{p}$, and the radius of the Earth, $a$

$\delta t=-\int_{\text{ray path}}v^{-2}\delta
v\,\mathrm{d}d=-2\int_{r_{p}}^{a}\eta v^{-2}q^{-1}\delta v\,\mathrm{d}r.$ (17)

The above can be generalised for a source at $r_{0}$ and a receiver at $r_{1}$
(for a receiver at the surface $r_{1}=a$)

$\delta t=-\sum_{n=0}^{1}{\int_{r_{p}}^{r_{n}}\eta v^{-2}q^{-1}\delta
v\,\mathrm{d}r},$ (18)

which on substituting (6) and (7) becomes

$\delta t=\sum_{n=0}^{1}{\int_{r_{p}}^{r_{n}}\eta v^{-2}q^{-1}\varsigma
v^{\prime}r\,\mathrm{d}r}.$ (19)

Dziewonski & Gilbert (1976) transform equation (19) into an integral over
distance

$\delta t=p^{-1}\int_{0}^{\Delta}\eta^{3}\varsigma
v^{\prime}\,\mathrm{d}\theta,$ (20)

where $\Delta$ is the total epicentral distance. However, for ray paths that
propagate close to the centre of the planet, equation (20) can be problematic
due to a discontinuity in $\theta$ that occurs at the centre of the planet. We
instead rewrite (19) using the vertical slowness, noting the derivatives,

$\displaystyle\frac{\mathrm{d}q}{\mathrm{d}\eta}=\frac{\eta}{q},$ (21)
$\displaystyle\frac{\mathrm{d}\eta}{\mathrm{d}r}=\frac{1}{v}-\frac{r}{v^{2}}v^{\prime},$
(22)

from which it follows that (19) can be written as

$\delta t=\sum_{i}\left\\{{\int_{q_{0}}^{q_{1}}\frac{\eta v^{\prime}}{1-\eta
v^{\prime}}\ \varsigma\ \mathrm{d}q}\right\\}_{i}$ (23)

which is an integration over vertical slowness for each continuous section,
$i$, of a ray path where $q_{0}$ and $q_{1}$ are the values of $q$ at the
start and end of each ray segment, with $q_{1}>q_{0}$. (23). It is convenient
to introduce the parameter, $\xi$, from Bullen (1963):

$\xi=\frac{\eta}{r}\left(\frac{\mathrm{d}\eta}{\mathrm{d}r}\right)^{-1}=\frac{\mathrm{d}\ln
r}{\mathrm{d}\ln\eta}.$ (24)

Equation (23) therefore becomes

$\delta t=\sum_{i}\left\\{{\int_{q_{0}}^{q_{1}}\left(\xi-1\right)\varsigma\
\mathrm{d}q}\right\\}_{i}$ (25)

which is similar to the formulation of Doornbos (1988). The parameter $\xi$ is
constant when using the Mohorovicic or Bullen law $v(r)=Ar^{B}$ for
interpolation within a velocity layer (Buland & Chapman, 1983).

Using the cosine rule for the surface of a sphere, the co-latitude,
$\vartheta$, at any distance, $\theta$, on a ray path with a given azimuth,
$\zeta$, can be determined

$\cos\vartheta=\cos\vartheta_{0}\cos\theta+\sin\vartheta_{0}\sin\theta\cos\zeta$
(26)

Equation (26) is in the correct form to apply the Addition Theorem for
Legendre Functions (Arfken, 1985)

$P_{2}^{0}(\cos\vartheta)=\sum_{m=0}^{2}P_{2,m}(\cos\vartheta_{0})P_{2,m}(\cos\theta)\cos
m\zeta,$ (27)

where

$P_{l,m}(\cos\theta)=\left[(2-\delta_{m,0})\frac{(l-m)!}{(l+m)!}\right]^{1/2}P_{l}^{m}(\cos\theta).$
(28)

$P_{l,m}(x)$ are the Schmidt semi-normalised associated Legendre polynomials,
where the normalising weights are such that the Addition Theorem is
simplified.

Using equations (8) and (27) it follows that

$\varsigma=-\frac{2}{3}\epsilon\sum_{m=0}^{2}P_{2,m}(\cos\vartheta_{0})P_{2,m}(\cos\theta)\cos
m\zeta.$ (29)

Substituting equation (29) into equation (23) gives

$\delta t=-\frac{2}{3}\sum_{m=0}^{2}P_{2,m}(\cos\vartheta_{0})\cos
m\zeta{\int_{q_{0}}^{q_{1}}\left(\xi-1\right)\epsilon P_{2,m}(\cos\theta)\
\mathrm{d}q}$ (30)

It is possible to include the effects of phase and distance in calculable
coefficients, $\sigma_{m}$. Rewriting equation (30) as

$\delta t=\sum_{m=0}^{2}\sigma_{m}P_{2,m}(\cos\vartheta_{0})\cos m\zeta$ (31)

where the ellipticity correction coefficients, $\sigma_{m}$, also include the
effects of displacing discontinuities in $v$ as per equations (14) and (15)

$\sigma_{m}=\sum_{i}\left\\{{\int_{q_{0}}^{q_{1}}\left(\xi-1\right)\epsilon\lambda_{m}(\theta)\
\mathrm{d}q}\right\\}_{i}-\sum_{j}\left\\{\epsilon\lambda_{m}(\theta)\left[q\right]_{-}^{+}\right\\}_{j}\mp\sum_{k}\left\\{\epsilon\lambda_{m}(\theta)q\right\\}_{k}$
(32)

where $i$ represents a sum over continuous regions of the ray path and $j$ and
$k$ represent sums over all transmitting and reflecting boundaries,
respectively, and

$\lambda_{m}(\theta)=-\frac{2}{3}P_{2,m}(\cos\theta).$ (33)

Equation (32) is for a phase that remains one wave type along its entire path;
it is important, that in the case where phase conversions at boundaries take
place, that the equations for transmission and reflection through boundaries
are applied correctly. For diffracted phases, the change in arc length due to
ellipticity is second order in $\epsilon$ (Kennett & Gudmundsson, 1996) and
can be ignored. For diffracted phases it is therefore only necessary to apply
equation (32) to the up-going and down-going parts of the ray path.

##  2 Ellipticity of Surfaces of Constant Density

For any planet with ellipticity due to rotation, the relative perturbation of
a surface, $\delta r$, from the reference radius of the surface, $r$, can be
related to co-latitude, $\vartheta$, by equation (8). The ellipticity of
figure for a surface of equal density, $\epsilon$, varies with depth and is
dependent on the density distribution within a planet and its angular
velocity. It is necessary to calculate $\epsilon$ before ellipticity
corrections for seismic waves can be calculated for that velocity model. The
following is only a brief overview of the method to calculate $\epsilon$ and a
full derivation can be found in Bullen (1975). An application to simple Earth
models can be found in Bullen & Haddon (1973).

Given a density distribution of a body with radius, $\rho(r)$, it is possible
to calculate the mass, $M(r)$, within a surface of radius $r$ for a spherical
body by an integral of volume:

$M(r)=\int\rho\,\mathrm{d}V=4\pi\int_{0}^{r}\rho r^{2}\,\mathrm{d}r.$ (34)

The moment of inertia, $I(r)$, of a body with radial density distribution,
$\rho(r)$, can be calculated by treating the body as the sum of thin spherical
shells:

$I(r)=\frac{2}{3}\pi\int\rho r^{2}\,\mathrm{d}V=\frac{8}{3}\pi\int_{0}^{r}\rho
r^{4}\,\mathrm{d}r.$ (35)

The mean moment of inertia coefficient, $y(r)$, for the mass within each
surface of constant density is then calculable

$y(r)=\frac{I(r)}{M(r)r^{2}}.$ (36)

Using the Darwin-Radau approximation, which assumes hydrostatic equilibrium,
the Radau parameter, $\upeta$, and $y$ are related by

$\frac{2}{5}(1+\upeta)^{1/2}=1-\frac{3}{2}y.$ (37)

Radau’s parameter is related to ellipticity by a differential equation:

$\upeta=\frac{r}{\epsilon}\frac{\mathrm{d}\epsilon}{\mathrm{d}r}$ (38)

which can be solved by separation of variables

$\epsilon=c\exp\left({\int\frac{\upeta}{r}\,\mathrm{d}r}\right)$ (39)

where $c$ is a constant of integration. Using the Darwin-Radau approximation
it is possible to estimate $\epsilon$ at the surface. Let the subscript $a$
represent variables at the surface.

$\upeta_{a}=\frac{5h}{2\epsilon_{a}}-2$ (40)

where $h$ is a convenient geodynamical parameter representing the ratio of
rotational and gravitational forces at the equator on the surface of a body,

$h=\frac{a^{3}\Omega^{2}}{GM},$ (41)

where $a$ is the radius of the body, $\Omega$ is the angular velocity of the
body, $M$ is the mass of the body (calculated using equation (34)) and $G$ is
the universal gravitational constant. Having calculated $\epsilon_{a}$ it is
possible to calculate the constant of integration, $c$ in equation (39).

This formulation allows the easy calculation of $\epsilon$ for any surface of
equal density within a rotating body with a given density distribution where
Darwin-Radau’s equation holds. This assumes that the body is in hydrostatic
equilibrium and is an ellipsoid of revolution. This assumption does not hold
for all bodies, notably the gas giants of our own solar system, however the
practical difficulties of doing seismology on such bodies means that the need
for ellipticity corrections in such a scenario is unlikely. For the vast
majority of bodies where seismology is likely to be performed, this
formulation is sufficient.

##  3 Minimum and Maximum Ellipticity Corrections on Earth

Table S1 shows the minimum and maximum ellipticity corrections for various
seismic phases in PREM (Dziewonski & Anderson, 1981). These correspond to
those shown in Figure 3 in the main publication and were calculated by a
systematic search with source depths from 0 km to 700 km, source latitudes
from -90° to 90° azimuths from 0° to 360° and all integer distances for which
that phase has a ray theoretical arrival predicted by ObsPy TauP (Crotwell et
al., 1999; Beyreuther et al., 2010) in PREM.

Table S1: Minimum and maximum ellipticity corrections for commonly used
seismic phases in PREM. Corrections for depth phases can be calculated but are
not shown here.
Phase | Distance range | Minimum correction (s) | Maximum Correction (s)
---|---|---|---
p (up-going) | 0° - 12° | -0.44 | 0.24
P | 0° - 98° | -1.01 | 0.97
PP | 0° - 196° | -1.75 | 1.94
PcP | 0° - 98° | -1.49 | 0.97
Pdiff | 96° - 158° | -1.86 | 0.97
PKPab | 143° - 177° | -2.19 | 1.39
PKPbc | 143° - 152° | -2.24 | 1.32
PKPdf | 116° - 180° | -2.68 | 1.34
PKKPab | 235° - 257° | -1.37 | 1.82
PKKPbc | 235° - 282° | -2.19 | 1.89
PKKPdf | 210° - 360° | -3.88 | 1.94
P’P’ab | 289° - 355° | -4.36 | 2.79
P’P’bc | 289° - 305° | -4.45 | 2.64
PKiKP | 0° - 152° | -2.33 | 1.32
s (up-going) | 0° - 11° | -0.78 | 0.43
S | 0° - 102° | -1.83 | 1.81
SS | 0° - 205° | -3.18 | 3.62
ScS | 0° - 102° | -2.72 | 1.81
Sdiff | 100° - 162° | -3.53 | 1.81
SKSac | 60° - 141° | -3.19 | 1.93
SKSdf | 106° - 180° | -3.92 | 1.96
SKKSac | 60° - 271° | -3.34 | 2.50
SKKSdf | 201° - 360° | -5.11 | 2.56
S’S’ac | 122° - 283° | -5.59 | 3.87
SKiKS | 0° - 141° | -3.56 | 1.93
PS | 18° - 141° | -2.43 | 2.35
PcS | 0° - 62° | -2.11 | 1.18
PKSab | 129° - 142° | -2.39 | 1.61
PKSbc | 129° - 147° | -2.78 | 1.63
PKSdf | 110° - 180° | -3.30 | 1.65
PKKSab | 213° - 222° | -2.28 | 2.04
PKKSbc | 214° - 276° | -2.32 | 2.19
PKKSdf | 205° - 360° | -4.49 | 2.25
SP | 12° - 141° | -2.46 | 2.35
ScP | 0° - 62° | -2.11 | 1.18
SKPab | 130° - 142° | -2.33 | 1.61
SKPbc | 130° - 147° | -2.72 | 1.63
SKPdf | 111° - 180° | -3.30 | 1.65
SKKPab | 215° - 222° | -2.22 | 2.04
SKKPbc | 215° - 276° | -2.34 | 2.19
SKKPdf | 206° - 360° | -4.49 | 2.25

##  4 Tables of ellipticity correction coefficients

While direct calculation of ellipticity corrections for a given ray path is
expected to be the most accurate and convenient method of calculating
ellipticity corrections, in applications where large numbers of sources and
receivers are used it may be substantially more efficient to interpolate pre-
calculated tables of coefficients. We have produced tables of coefficients
similar to those of Kennett & Gudmundsson (1996) as available in their
ELCOR.dat file.

The original ELCOR.dat contains coefficients sampled at 5 degree distance
intervals and irregular depth intervals (0 km, 100 km, 200 km, 300 km, 500 km,
700 km) for a range of seismic phases in the ak135 velocity model (Kennett et
al., 1995). The phase names, order and dimensions of this file are hardcoded
in the original ellip package and therefore replacement tables must contain
the same phases in the same order with the same number of entries per phase in
order to be compatible.

One complication of the original ELCOR.dat is that it contains coefficients
even where there is not a ray-theoretical arrival and this is highlighted in
the ellipcor.help file of the ellip package: ‘The presence of a value in the
tables does not imply a physical arrival at all distance, depth combinations.
Where necessary extrapolation has been used to ensure satisfactory results.’
Exactly what is meant by ‘necessary extrapolation’ is not apparent.

Using EllipticiPy, we have produced a replacement ELCOR.dat file, called
ELCOR_replacement.dat, that is designed to act as a direct replacement. These
coefficients are for the ak135 model and are sampled at the same distance and
depth points as the original file. The various assumptions and procedures used
to produce this file are:

  * •

For points where a phase name has several possible ray paths (e.g. for upper
mantle P wave triplications) coefficients for the first arriving ray path are
provided.

  * •

Branches of core phases are separated as distinct phases, except for P′P′ and
S′S′ which are explained below.

  * •

For points where we do not expect a ray-theoretical arrival but there is
reason to believe that the original values from ELCOR.dat are correct (phases
below 180 degrees and good agreement where there are arrivals) we have used
the original values.

  * •

For points where we do not expect a ray-theoretical arrival but there is
reason to believe that the original values from ELCOR.dat are in doubt, we
have linearly extrapolated from the closest calculable values.

  * •

In the original ELCOR.dat, tables for P′P′ and S′S′ exceed the expected
distance ranges for those phases that bottom in the outer core, and in parts
of the given range correspond to the df (inner core) branches of the phase. In
order to keep ELCOR_replacement.dat compatible, this file must have the same
number of entries for each phase as the original ELCOR.dat file. Therefore for
P′P′ the coefficients given are for P′P′df and for S′S′ they are for S′S′ac at
the lower end of the distance range and for S′S′df at the upper end of the
range. The coefficients for P′P′bc and P′P′df are continuous, as are those for
S′S′ac and S′S′df. The only branch not represented by these phase entries is
P′P′ab.

  * •

PnS as we understand it (a P wave that descends to and diffracts along the
Moho before continuing downwards as an S wave) does not occur totally within
the distance range given in the original ELCOR.dat, nor do the coefficients
given match those calculated for this phase. As we do not know what ray path
these coefficients are for and cannot reproduce them at the exact distances,
we omit this phase in ELCOR_replacement.dat.

One major disadvantage of using pre-calculated tables is that the exact ray
path used to calculate the coefficients is not known to the user. Therefore if
in doubt, we encourage all users to use EllipticiPy directly as ObsPy allows
easy visualisation of ray paths to confirm that the coefficients are for the
intended ray path. EllipticiPy also contains a function which produces
tabulated coefficients for a given phase and source depth. Coefficients are
produced for the full range of ray parameters for which the phase exists,
using the same discretisation in slowness as used for the velocity model by
ObsPy TauP. Such coefficients can be pre-calculated and then used in a
standard interpolation routine.

ELCOR_replacement.dat is available at this project’s GitHub repository,
https://github.com/StuartJRussell/EllipticiPy, along with the main python
package.

## References

  * Arfken (1985) Arfken, G., 1985.  The addition theorem for spherical harmonics., in Mathematical Methods for Physicists, pp. 693–695, Academic Press, New York, NY.

  * Beyreuther et al. (2010) Beyreuther, M., Barsch, R., Krischer, L., Megies, T., Behr, Y., & Wassermann, J., 2010.  ObsPy: A Python Toolbox for Seismology, Seismol. Res. Lett., 81(3), 530–533.
  * Buland & Chapman (1983) Buland, R. & Chapman, C., 1983.  The computation of seismic travel times, Bull. Seismol. Soc. Am., 73(5), 1271–1302.
  * Bullen & Haddon (1973) Bullen, K. & Haddon, R., 1973.  The ellipticities of surfaces of equal density inside the earth, Phys. Earth Planet. In., 7(2), 203–212.
  * Bullen (1963) Bullen, K. E., 1963.  An introduction to the theory of seismology, Cambridge University Press, Cambridge, 3rd edn.
  * Bullen (1975) Bullen, K. E., 1975.  The Earth’s Density, Dordrecht: Springer Netherlands.
  * Crotwell et al. (1999) Crotwell, P., Owens, T., & Ritsema, J., 1999.  The TauP Toolkit: Flexible Seismic Travel-time and Ray-path Utilities, Seismol. Res. Lett., 70(2).
  * Doornbos (1988) Doornbos, D., 1988.  Asphericity and ellipticity corrections, in Seismological Algorithms, pp. 75–85, Academic Press, New York, NY.
  * Dziewonski & Gilbert (1976) Dziewonski, A. & Gilbert, F., 1976.  The effect of small, aspherical perturbations on travel times and a re-examination of the corrections for ellipticity, Geophys. J. R. Astr. Soc., 44, 7–17.
  * Dziewonski & Anderson (1981) Dziewonski, A. M. & Anderson, D., 1981.  Preliminary Reference Earth Model, Phys. Earth Planet. In., 25(4), 297–356.
  * Julian & Anderson (1968) Julian, B. R. & Anderson, D. L., 1968.  Travel times, apparent velocities and amplitudes of body waves, Bull. Seismol. Soc. Am., 58, 339–366.
  * Kennett & Gudmundsson (1996) Kennett, B. & Gudmundsson, O., 1996.  Ellipticity corrections for seismic phases, Geophys. J. Int., 127(1), 40–48.
  * Kennett et al. (1995) Kennett, B., Engdahl, E., & Buland, R., 1995.  Constraints on seismic velocities in the Earth from traveltimes, Geophys. J. Int., 122(1), 108–124.


# Can only LLMs do Reasoning?:
Potential of Small Language Models in Task Planning

Gawon Choi and Hyemin Ahn Gawon Choi and Hyemin Ahn are with Graduate School
of Artificial Intelligence, Ulsan National Institute of Science and
Technology, Ulsan, Korea.  {gawon5325<EMAIL_ADDRESS>
###### Abstract

In robotics, the use of Large Language Models (LLMs) is becoming prevalent,
especially for understanding human commands. In particular, LLMs are utilized
as domain-agnostic task planners for high-level human commands. LLMs are
capable of Chain-of-Thought (CoT) reasoning, and this allows LLMs to be task
planners. However, we need to consider that modern robots still struggle to
perform complex actions, and the domains where robots can be deployed are
limited in practice. This leads us to pose a question: If small LMs can be
trained to reason in chains within a single domain, would even small LMs be
good task planners for the robots? To train smaller LMs to reason in chains,
we build ‘COmmand-STeps datasets’ (COST) consisting of high-level commands
along with corresponding actionable low-level steps, via LLMs. We release not
only our datasets but also the prompt templates used to generate them, to
allow anyone to build datasets for their domain. We compare GPT3.5 and GPT4
with the finetuned GPT2 for task domains, in tabletop and kitchen
environments, and the result shows that GPT2-medium is comparable to GPT3.5
for task planning in a specific domain. Our dataset, code, and more output
samples can be found in https://github.com/Gawon-Choi/small-LMs-Task-Planning

##  I INTRODUCTION

In recent robotics, the use of Large Language Models(LLMs) has become
increasingly prevalent, particularly in understanding human instructions. Due
to the huge scale of LLMs, which are commonly over 100 billion parameters, it
becomes imperative to deploy LLMs on external servers, and this makes the
robot highly dependent on internet speed, and any latency directly harms the
efficiency.

In this paper, we examine if robots need ‘large-scale’ language models (LMs)
in practice. There are existing methodologies such as RT-2 [1] and SayCan [2],
and they argue the advanced capabilities of the LLMs make robots possible to
accurately interpret abstract and ambiguous human commands. However, we claim
that the capabilities of LLMs are too extensive compared to the actual
diversity and complexity of tasks that modern robots can execute. This low
task complexity is also reflected in the LLMs-based methods, which evaluate
their robots in controlled environments that are engineered to look like a
real office or kitchen.

We also argue that the linguistic complexity of commands for robots is quite
lower than what LLMs can handle. When comparing the dataset complexity used in
pretraining LLMs with those employed in training robots, the discrepancy
between the low complexity of commands for robots and the advanced LLMs
capability becomes evident.

Figure 1: Overview of our proposed method. When the user specifies one’s
domain, first, our method extract the dataset for task planning, which
consists of high-level commands and low-level actionable steps, from LLMs. The
dataset is used to train small LMs which will be deployed in the robot.

One might claim that the LLMs’ ability to perform Chain-of-Thought (CoT)
reasoning [3] remains crucial. CoT reasoning refers to step-by-step reasoning,
and it induces LLMs to describe intermediate reasoning steps to achieve the
goal. Recent studies [4, 5] demonstrated that designing prompts for LLMs to
describe their reasoning process enhances the overall performance in the
Natural Language Processing (NLP) benchmark [6]. LLMs can reason in chains
domain-agnostic, however, we need to recall that the domains where robots can
be deployed are limited in practice.

This leads us to pose a question: If small LMs can be trained to reason in
chains, even within a single domain, would even smaller LMs be good task
planners for the robots? To answer this question, we aim to finetune smaller
LMs for CoT reasoning in a single domain. However, we face the issue of a lack
of open-source datasets to train CoT reasoning. To address this issue, we
utilize LLMs to build the datasets for CoT reasoning. We employ GPT3.5 [7] to
generate our ‘COmmand-STeps Dataset’ (COST dataset) consisting of abstract
commands and actionable steps that break down the command into a sequence of
actions to achieve the goal, such as CoT reasoning. We finetune small LMs
(i.e., GPT2) with the dataset to generate action steps for the user’s command.

We release not only our dataset but also the prompt templates used to generate
it. It allows anyone to build datasets for their environment and conditions,
and to use them for finetuning small LMs in their domain. We also compare
these finetuned small LMs with LLMs in tabletop and kitchen environments. From
this comparison, we find that GPT3.5 and finetuned GPT2-medium are comparable
for planning the robot actions for each environment.

Our contributions are as follows: (1) We design the prompt templates to
generate the task planning dataset for any domain. (2) We analyze the
linguistic complexity of robotics datasets and a dataset for pretraining LLMs
[8], and discover a gap in complexity between the datasets. (3) By comparing
LLMs with finetuned small LMs with our dataset, we find the potential of small
LMs as the task planner.

##  II RELATED WORKS

###  II-A Robotics with Large Language Models

The rapid growth of LLMs is also having an impact on robotics. Recent research
in scene understanding demonstrated that using LLMs outperforms using vision
language models (VLMs) [9]. Furthermore, LLMs lead to higher accuracy in
situational awareness tasks, by using text inputs to measure the ambiguity of
user commands [10]. A few studies aim to design robots that can flexibly
understand high-level user commands and infer current states through
communication, by using LLMs [11, 12, 13].

One of the most common applications of LLMs in robotics is task planning,
which breaks down high-level commands into a sequence of low-level action
steps. Some research enabled robots to generate plans in Pythonic code style
using LLMs, given input user commands as predefined Python function library
for robot actions [12, 14]. Also, there is research using LLMs as an
intermediate model to ensure robust task planning [15, 16]. Several other
studies also employed LLMs for robot task planning purposes [17, 18].

Recent studies revealed that LLMs’ performance is improved by prompting to
reason in chains, known as Chain-of-Thought (CoT) [3], and engineering the
prompts. [19, 5, 20, 21]. Robot task planning shares similarities with CoT
reasoning, where the reasoning process is broken down into intermediate steps
to achieve a goal. Based on this, some researchers tried applying CoT to task
planning. For instance, RT-2 [1] employs large-scale VLMs to summarize the
precise intent behind ambiguous user commands and generate sequences of action
tokens. the Socratic model [22], SayCan [2], and PaLM-E [23] utilize LLMs to
generate intermediate action steps based on abstract and ambiguous user
commands. A common feature of these studies is that LLMs are used as a CoT
reasoner to enable robots to plan to execute high-level commands in any
domain.

However, do robots require such large-scale models capable of handling a wide
range of NLP tasks? Existing studies often directly apply LLMs or LLM-based
VLMs to robots, enabling them to perform high-level reasoning for commands
from any environment. But in reality, modern robots still have limits in
hardware to perform complex tasks in the real-world. While the ultimate goal
is to execute complex tasks across various domains, it is equally important to
consider the practical limitations of modern robots and choose appropriate LMs
accordingly. Therefore, we narrow our focus to tasks that are within the
current capabilities of robots and explore the possibility of smaller LMs.

###  II-B Language Models with Chain-of-Thought

Recently, there has been a dramatic increase in the size of LMs, which we
refer to as large-scale LMs (LLMs). It is demonstrated that LLMs performance
is significantly enhanced by training LLMs to explain their reasoning process
[19], and also engineering input prompts for LLMs to reason in chains [5, 3,
20, 21]. These training and engineering methodologies are known as Chain-of-
Thought (CoT) reasoning and CoT prompting, respectively. Researchers mainly
focused on applying CoT prompting only to LLMs, since they assume that only
large-scale LMs have the capabilities to describe intermediate reasoning
steps. In practice, larger models absolutely outperform smaller models on CoT
reasoning [5, 3, 4, 20], and from this, the spotlight has been on LLMs,
resulting in less effort to efficiently improve the smaller LMs performance on
CoT reasoning.

Nonetheless, a few studies explored the potential of small LMs on CoT
reasoning [24, 25, 26, 27]. Magister et al. [24] explored the trade-off
between model size and reasoning capabilities via knowledge distillation [28],
using LLMs as the teacher model and smaller LMs [29] as the student model.
This study employed outputs from the teacher model as the training set to
finetune the small LMs for CoT reasoning. The results showed that small LMs
can learn CoT reasoning, as LLMs. To train small LMs to explain the reasoning
process, Kang et al. [25] narrowed the models down to task-specific, and
generated rationales for answers via knowledge distillation from LLMs. Ma et
al. [26] also tried to train small LMs to generate rationales via knowledge
distillation from LLMs, by narrowing the domain down to scientific QA. Fu et
al. [27] specialized LMs for a particular task and gave up some generalization
capability of LMs.

Our approach shares similarities with these previous studies, the primary
difference is that we focus on limiting both the task domain and the
environment domain in which the robot can be deployed in practice. By training
small LMs, such as GPT2-medium and GPT2-base [30], whose parameters size are
around 600 times smaller than GPT3.5 [7], for only a single task and
environment, we anticipate that small LMs can be CoT reasoners, and also serve
as task planners in robotics. Our goal is to develop small LMs capable of
generating actionable steps for robots to understand and execute high-level
commands within a ‘practical’ environment where robots ‘actually’ can be
deployed, by limiting the domains that LMs train on. We aim to demonstrate the
potential of small LMs in task planning.

##  III PROPOSED APPROACH

We build datasets to finetune small LMs for task planning in a single domain,
such as kitchen and tabletop environments, via knowledge distillation from
GPT3.5 [7], by CoT prompting on it. Our dataset consists of high-level
commands and corresponding low-level actionable steps. To allow users to build
a dataset for their domain, we propose prompt templates that can be adaptable
to any environment and conditions. For our dataset, we also use the prompt
templates with our environment and conditions. Our workflow is as follows:

  * 1.

Domain-specific objects generation

    1. (a)

Create a list of objects that can exist in a given domain (i.e., kitchen), by
prompting LLMs.

  * 2.

High-level command generation

    1. (a)

Generate 20 high-level commands by prompting LLMs with the objects created in
the previous step.

    2. (b)

Repeat the process 2-(a) n times until the desired number of commands is
generated. Through this repeat, we get a total of $20*\textit{n}$ commands.

  * 3.

Generating a sequence of action steps for each high-level command

    1. (a)

Generate a sequence of actionable steps to execute the high-level command, by
CoT prompting LLMs.

    2. (b)

Repeat the process 3-(a) $20*\textit{n}$ times, until all high-level commands
have a corresponding action steps.

  * 4.

Fine-tuning small language models

    1. (a)

Finetune the small LMs to generate intermediate actionable steps for the high-
level commands, using the command-steps pairs dataset.

The overall generation process of our dataset is summarized in Figure 2. We
name our dataset as COST, COmmand-STeps dataset.

Figure 2: A flow chart describing how our COmmand-STep dataset (COST) is
generated. Detailed description can be found in Sec. III.

###  III-A Objects List Generation

We designed a prompt template for generating objects that can exist in a given
domain. Our prompt template allows the user to generate a list of objects in
whatever domain they plan to deploy the robot (i.e., Kitchen). If the user
already has the list of objects for their own domain, one can skip this step.

To create objects suitable for robot manipulation, we describe the hardware
specification of the robot in the prompt. To guide LLMs in generating
appropriate objects, we also instruct LLMs to briefly describe the use of each
object. Additionally, we give a few output samples in the prompts to represent
what we desire as output.

Figure 3 shows our prompt template for object generation. In this prompt,
engineering [Condition of the object] and the {sample objects} is crucial. For
instance, if the user fills the prompt template for the kitchen without any
[Condition of the object], and just gives ‘knife’ as A sample object, LLMs
focus only on kitchen tools. However, if the user provides conditions such as
‘the object can be a cooking ingredient like a carrot.’, and gives balanced
output samples, ‘potato’ and ‘knife’, LLMs generate tools or ingredients as
objects, in balance. What matters is that the users can clearly describe the
object features they want to generate for their prompt.

List up {count} objects that the robot can PICK UP, GRAB, and use in the
“{domain}”. For each object, describe why that object could be used by the
robot in that place.
[Situation]
# There is the robot in the {domain}.
[Condition of the object]
# Describe the conditions of objects you want to generate.
[Information about the robot]
# Describe briefly the hardware specification of the robot.
[Outputs]
1\. {sample objects}
: {A sample description of how the object could be used}
2\.

Figure 3: The prompt template to generate an object list in the given domain.
The green lines are where the user needs to fill in. The prompt requires the
user to specify their domain, the number of objects to be generated, the
conditions of the objects, information about the robot, and output samples.

I want to collect examples of instructions that a person might give to a robot
in {domain}. You need to generate {count} sets of instructions using the
[Object list] provided. Each set should include $<$USED_OBJECTS$>$ and
$<$INSTRUCTION$>$. Please adhere to the guidelines mentioned in [Condition of
the instructions generated] below.
I’ve given you an [Output template] to follow. You should replace the
placeholders $<$USED_OBJECTS$>$ and $<$INSTRUCTION$>$, using the [Example of
output] as a reference.
[Conditions of the commands to be generated]
* Each $<$INSTRUCTION$>$ should be a concise single sentence.
* The instructions should be constructed by following these steps:
Step 1. Select n or more objects to use from the [Object list].
Step 2. Create an instruction using the selected objects from step 1.
[Example of output]
({Sample objects used in the command}) {A sample command}
[Output template]
($<$USED_OBJECTS$>$), $<$INSTRUCTION$>$
[Note]
# Describe notification which you’d like LLMs to consider when generating
commands.
[Object list]
# Input your objects list in here.
——————————– [Output]
1\.

Figure 4: The prompt template to generate high-level commands using the object
list. We first describe the task that LLM should do, and present conditions
for generating, output examples, an output template and give an object list as
an input. The green lines are where the user needs to fill in.

You have been given ”Objects in front of the robot” and ”Command”. Your task
is to generate ”Action Steps”.
An [Output template] will be provided for your output, and $<$MASK$>$ is a
placeholder for content. Please see the [Information about the robot] and
[Note] section for additional guidance.
[Information about the robot]
# Describe physical capability of your robot.
[Output template]
Action Steps= $<$MASK$>$
[Example of Input and Output]
*Input:
Command= {A sample command}
Objects in front of the robot={A sample object list}
*Output:
Action Steps=
Step 1. {A sample step 1} (ACTION : {Action in step1}, TARGET: {Target in step
1 action})
…
Step n. {A sample step n} (ACTION : {Action in step n}, TARGET: {Target in
step n action})
[Note]
# Describe conditions which you’d like LLMs to consider when generating steps.
——————————– *Input:
Command= {A input command}
Objects in front of the robot= {A input objects list}
*Output:

Figure 5: The prompt template for generating action steps for each high-level
command. The blue lines are only for the prompt where the available objects
are fixed, and the black lines are common to prompts in Fig 5 and 6. The green
lines are where the user needs to fill in.

You have been given a ”Command” by one person to a robot. Your task is to
generate two outputs: 1. ”Action Steps” 2. ”Required Objects”
An [Output template] will be provided for your output, and $<$MASK$>$ is a
placeholder for content. Please see the [Note] section for additional
guidance.
[Output template]
Action Steps= $<$MASK$>$
Required Objects= $<$MASK$>$
[Example of Input and Output]
*Input:
Command= {A sample command}
*Output:
Action Steps=
Step 1. {A sample step 1} (ACTION : {Action in step1}, TARGET: {Target in step
1 action})
…
Step n. {A sample step n} (ACTION : {Action in step n}, TARGET: {Target in
step n action})
Required Objects= {A sample list of required objects}
[Note]
# Describe conditions which you’d like LLMs to consider when generating steps.
——————————– *Input:
Command= {A input command}
*Output:

Figure 6: The prompt template for generating action steps for each high-level
command. The blue lines are only for the prompt where the available objects
are not fixed, and the black lines are common to prompts in Fig 5 and 6. The
green lines are where the user needs to fill in.

###  III-B High-level Commands Generation

‘High-level command’ refers to the abstract command that includes a lot of
implicit contents and, in this study, also refers to the command that requires
multiple action steps to achieve the goal. We generate more than 20,000 high-
level commands for our COST datasets. Generating all commands by inputting a
single prompt is challenging due to the limitation of LLMs’ maximum output
length and the tendency of noise to increase towards the latter part of the
output. Therefore, we generate only 20 high-level commands per a single prompt
input, and repeatedly call the GPT3.5 API until the desired number of commands
are generated.

Figure 4 shows our prompt template to generate high-level commands. To fix the
output format, we provide the output template and instruct LLMs to fill in the
placeholders in the template, by referring to the guidelines presented in
[31]. To guide LLMs to use only the objects in [Object list], we give
[Conditions of the commands to be generated] as a guideline in the prompt,
which suggests the following generation order: First, select objects from
[Object list], then, generate a high-level command only using the selected
objects. If we do not give the guideline, LLMs will often create new objects
that are not included in [Object list] when generating commands. However, it
is challenging to eliminate the possibility of new objects creation, because
LLMs are probability models. If the user desires to avoid generating commands
using objects outside the given list, a post-processing will be required. As
we fixed the output format through prompting, a post-processing would be
relatively straightforward. However, we rather recommend using new objects
that LLMs generate by breaking the rules in the prompt. In our experiment, we
keep adding new objects to the [Object List] to be used in the next prompt
input. This guarantees us to generate more varied high-level commands from
LLMs, but the number of objects in the list keeps increasing. So, we randomly
sample N objects from the entire object list as the [Object list] for each
prompt input.

The users can control ‘How abstractly generate the commands’ via [Example of
output]. If the user wants highly abstract commands, it is recommended to
avoid clarifying the name of used objects in {A sample command}, such as ‘Make
a sandwich.’. On the other hand, if the user prefers relatively explicit
commands, it is recommended to clarify the name of used objects, such as ‘Make
a sandwich with bread, tomato, and cheese, after cutting the tomato.’.

###  III-C Action Steps Generation

Figure 5 and 6 are our prompt templates to generate ‘Action steps’ for each
high-level command. Action steps are defined as the process of breaking down
high-level commands into intuitive and simple actionable steps for the robot,
to achieve the goal of the command [2]. Based on this definition, we can
consider action steps as CoT reasoning, in that it is a process of breaking
down high-level instruction into intermediate steps to achieve the answer (or
goal). Each action step consists of a short description of what the robot
needs to do at this stage, along with ‘ACTION’ which the robot should perform,
and ‘TARGET’ which the robot should aim. Specifying ACTION and TARGET helps
avoid noise, such as a step with more than one action.

By prompting, we instruct the LLMs to generate a sequence of actionable steps
required for the robot to execute a high-level command. We design the prompts
to input only one high-level command at a time, and to output a list of action
steps associated with the input command. Since we should generate action steps
lists for every high-level command, we repeatedly called GPT3.5 API until we
got an action steps list for every command.

We adopt guidelines [31] that fill in placeholders to standardize the output
format. We design two versions of the prompt to cover different cases: When
the objects to be used in action steps are fixed or flexible. The former
prompt, Figure 5, is designed to generate action steps using only given
objects, and the latter prompt, Figure 6, is designed to generate not only the
action steps but also the objects required by the action steps, for the input
command.

We use the prompt in Figure 5 to build the COST dataset for a tabletop domain.
For general real-world environments, such as kitchens, we use the prompt in
Figure 6. In tabletop, robot actions are limited to pick-and-place, so we
provide the [Information about the robot] section only for the prompt in
Figure 5 to describe the physical capabilities of the robot, to ensure that
LLMs generate Action steps consisting of actions that the robot can perform in
practice.

The crucial parts that are common to both versions of the prompt are [Example
of input and output] and [Note]. For instance, to ensure action steps in a
tabletop environment consist of only pick-and-place actions and to avoid
consecutive picking actions, the user should design an example that reflects
all of these conditions. It is also crucial to mention in [Note] the
conditions that LLMs should follow when generating action steps. For instance,
if one needs to generate action steps by limiting the number of objects the
robot can hold at once, this rule can be described in [Note].

###  III-D Fine-tuning Small Language Models

We build our COST datasets for the tabletop and kitchen environments, by
prompting LLMs using prompt templates Fig 4-6. We generate datasets for just
two domains, but note that our prompt templates allows anyone to generate a
dataset for their task domain. Using the dataset, we fine-tune LMs to generate
action steps. Then we verified if small LMs can plan to execute the high-level
command, when the environment and task domain are limited.

##  IV DATASET

###  IV-A Comparison of Datasets for Robotics and LLMs

We analyze the complexity between the text datasets in robotics and the
dataset used for pretraining LLMs. Based on the result of the analysis, we aim
to verify whether the large-scale LMs with more than 100 billion parameters
were indeed essential for processing the commands by the robot.

We use Lima [8] as a representative of the dataset for pretraining LLMs, and
use RT-1 [32], RoboVQA [33], BridgeData V1 and V2 [34, 35], and the Language
Table dataset [36] for simulator as the robotics datasets. We only utilized
text data from these robotics datasets for our comparison.

To compare the readability levels among datasets, we employed the Flesch-
Kincaid grade level formula as a metric. The Flesch-Kincaid test is the metric
used to determine what school grade is the document appropriate to read. The
metric score is calculated based on the number of words per sentence and the
number of syllables per word. The result of 1 to 18 means the document is
suitable for reading in grades 1st to 18th. Kincaid in Table I shows that
while robotics datasets are suitable for reading at 1st to 3rd grade, Lima
requires a much higher readability level. To measure the complexity per
sentence, we calculated the depth of the syntactic tree. The syntactic tree is
a tree diagram that depicts the syntactic structure of a sentence. If the
depth of the syntactic tree is large, it means that the sentence consists of
many phrases and clauses. In Table I, Syntactic tree denotes the average depth
of the syntactic tree for all sentences in the dataset, with the numbers in
parentheses indicating variance. It shows that the sentences in Lima are
structurally deeper than all other robotics datasets. Lima also has the
largest depth’s variance than the robotics dataset, and it can be interpreted
as Lima consisting of the most diverse sentence structures. We also analyzed
the vocabulary size by counting the number of unique tokens (types) in each
dataset. #Types in Table I shows that Lima contains a larger vocabulary than
the robotics datasets.

Datasets | Complexity
---|---
Kincaid | Syntactic tree | #Types
Lima | 7.91 | 5.56(4.4) | 29962
RT1 | 1.89 | 4.3(0.82) | 51
RoboVQA | 1.86 | 4.44(1.58) | 2624
Bridge | 2.7 | 5.36(1.42) | 2678
LT(sim) | 3.54 | 5.28(1.24) | 843
Kitchen, Command | 3.07 | 4.6(0.8) | 2239
Kitchen, Steps | 1.32 | 3.84(0.71) | 2979
Tabletop, Command | 4.81 | 5.71(1.94) | 1410
Tabletop, Steps | 1.79 | 4.45(2.09) | 990
TABLE I: Dataset Complexity Comparison. The gray cell is for our datasets. For more details on Lima and robotics datasets, see Sec. 4.1, and for more details on our datasets, see Sec. 4.2. Datasets | Part-of-Speech
---|---
NN | VB | AD
Lima | 10.59 | 8.08 | 5.34
RT1 | 3.08 | 0.85 | 2.33
RoboVQA | 3.63 | 2.37 | 2.08
Bridge | 3.58 | 3.3 | 1.95
LT(sim) | 2.3 | 1.57 | 0.75
Kitchen, Command | 5.34 | 4.79 | 3.54
Tabletop, Command | 3.96 | 3.29 | 2.49
TABLE II: Average entropy per PoS. The gray cell is for our datasets. For more
details on the Lima and robotics datasets, see Sec. 4.1, and for more details
on our datasets, see Sec. 4.2.

We examine the bias in vocabulary usage in each part of speech (PoS) for each
dataset. To analyze it, we calculated the entropy for the occurrence frequency
of words per PoS. We interpret that higher entropy means that diverse words
are evenly used, and lower entropy means that only a few specific words are
frequently used. The formula for calculating the entropy $H(\textit{W})$ is as
follows:

$\displaystyle H(\textit{W})=-\sum_{\textit{w}\in\textit{W}}p(\textit{w})\cdot
logp(\textit{w})$ (1) $\displaystyle p(\textit{w})=\frac{\text{Occurrence
number of {w}}}{\text{Total number of tokens}}$ (2)

where W is a set of total words (types) in each PoS in the document (dataset),
w is a word in W, and $p(\textit{w})$ is occurrence probability of w in
certain PoS in the document.

Table II shows average entropy per PoS. NN, VB, and AD mean a set of noun
families, a set of verb families, and a set of adjective and adverb families,
respectively. Across all PoS, the entropy for the robotics dataset is
significantly lower compared to Lima, and we analyze it because the robotics
datasets consist mostly of commands that can be executed with limited actions
and objects in a given domain. The results show that robotics datasets utilize
only a few specific words noticeably more often.

Table I and II show that the dataset for pretraining LLMs is more challenging
to read, consists of diverse and complex sentences, and requires more
vocabulary, compared to robotics datasets. Through the results, we can
identify a gap in complexity between robotics datasets and datasets for
pretraining LLMs. Based on this, we expect that smaller LMs also be able to
comprehend input commands and plan effectively for the tasks, if it can be
demonstrated that a small LMs can learn CoT reasoning for robotics planning
within a single domain.

###  IV-B Our Dataset Generated by LLMs

Objects= [’Bowl’, ’Whisk’, ’Table’, ’Egg’, ’Apple’]
Command= Beat the egg in a bowl.
Action Step=
Step 1. PICK up the egg. (ACTION: Pick $\mid$ TARGET: Egg)
Step 2. CRACK the egg into the bowl. (ACTION: Crack $\mid$ TARGET: Bowl)
Step 3. PICK up the whisk. (ACTION: Pick $\mid$ TARGET: Whisk)
Step 4. BEAT the egg in the bowl using the whisk. (ACTION: Beat $\mid$ TARGET:
Whisk, Egg, Bowl)

Figure 7: A sample of our COST dataset for kitchen domain. For a detailed
description of the sample, please refer to Sec. 7

We find that the commands in robotics are simpler than what LLMs learn, from
the result, we expect that small LMs can learn well for task planning. To
finetune the planning process on small LMs, we need our command-steps datasets
of similar complexity to datasets for robotics. So, we build the COST dataset
using GPT3.5 [7]. Please refer to Sec. III. for the detailed process of
building this dataset.

The gray cells in Table I and Table II indicate the complexity of our dataset.
‘Kitchen, Command’ and ‘Tabletop, Command’ refer to the high-level command set
for each environment, and ‘Kitchen, Steps’ and ‘Tabletop, Steps’ refer to the
steps set for each environment, respectively. Table I shows that the
complexity of our command set is similar to that of robotics datasets.
However, looking at the #Types in table I and the entropy per PoS in Table II,
we can see that our command set has a larger and more diverse vocabulary than
robotics datasets. Considering that this vocabulary is all for one domain, we
can say that our datasets contain richer vocabulary for a single domain than
the robotics datasets. We can also see that the steps set contains easier
sentences than our command set and robotics datasets.

Figure 7 depicts sample data in the kitchen domain. Objects is an objects list
in front of the robot. In order to train LMs to select only the ‘necessary’
items from a given objects, Objects contains not only the necessary items to
execute the command, but also unnecessary items. Objects are followed by
Command and then Action Steps. Command is an input instruction to the robots,
and Action Steps is a set of intermediate steps to execute the command.

##  V EXPERIMENTS

###  V-A Example Results

Input:  Objects= [’Yellow semicircle block’, ’Red circle block’, ’Red
semicircle block’, ’Yellow bowl’, ’Red bowl’] Command= Arrange semicircle
blocks in the same colored bowl as the picked block. Action steps= ——————————-
Output:
Step 1. PICK up the red semicircle block. (ACTION: Pick $|$ TARGET: Red
semicircle block) Step 2. PLACE the red semicircle block in the red bowl.
(ACTION: Place $|$ TARGET: Red bowl) Step 3. PICK up the yellow semicircle
block. (ACTION: Pick $|$ TARGET: Yellow semicircle block) Step 4. PLACE the
yellow semicircle block in the yellow bowl. (ACTION: Place $|$ TARGET: Yellow
bowl)

Figure 8: An example result in tabletop domain. When the Input is fed into the
model, it predicts what comes after the Input and return the Output.

Input:  Objects= [’Cucumber’, ’Lettuce’, ’Tray’, ’Sink’, ’Knife’, ’Cup’]
Command= Slice the washed lettuce and place it on tray. Action steps=
——————————- Output:
Step 1. PICK up the lettuce. (ACTION: Pick $|$ TARGET: Lettuce) Step 2. MOVE
to the sink. (ACTION: Move $|$ TARGET: Sink) Step 3. WASH the lettuce under
running water. (ACTION: Wash $|$ TARGET: Lettuce) Step 4. PICK up the knife.
(ACTION: Pick $|$ TARGET: Knife) Step 5. SLICE the lettuce using the knife.
(ACTION: Slice $|$ TARGET: Knife, Lettuce) Step 6. PLACE the sliced lettuce on
the tray. (ACTION: Place $|$ TARGET: Sliced lettuce, On the tray)

Figure 9: An example result in kitchen domain. When the Input is fed into the model, it predicts what comes after the Input and return the Output. Models | Success Rate (%)
---|---
GPT2-base | 50
GPT2-medium | 85
GPT3.5 | 70 (90)
GPT4 | 95 (100)
TABLE III: The success rate for 20 test commands for the tabletop domain. For
GPT3.5 and GPT4, The numbers denote the result from an prompt from [22], and
from our prompt, respectively.

Figure 8 and 9 show examples of results from finetuned GPT2-medium with our
dataset for the tabletop and kitchen domain, respectively. The inputs are an
unseen command and the random objects including items required to execute the
command.

To evaluate small LMs finetuned with our tabletop dataset, we aim to apply
them on the tabletop simulator. The possible actions in the tabletop simulator
are only pick-and-place, so, we restrict all actions in the tabletop dataset
to pick or place. For the kitchen dataset, we constrain the possible actions
to pick, place, move, and utilize the objects.

###  V-B Results in the Tabletop Simulator

Figure 10: The sample result of finetuned GPT2-medium in the tabletop
simulator. In the first text box, the black text is the input to the model,
and the blue text is the output of the model. We post-process the output to
input it in CLIPort [37]. The ‘Initial State’ image refers to the start scene
on the simulator, and the ‘Final State’ image refers to the end scene after
performing all Action steps.

We aim to compare the results of small LMs finetuned on our tabletop dataset,
with the results of Socratic models (SMs) [22], which plan in chain for the
command using LLMs. For this, we finetune GPT2-base and GPT2-medium [30] on
our tabletop dataset, and experiment on the tabletop simulator with the
finetuned LMs outputs and the LLMs outputs. To apply generated Action steps to
the tabletop simulator, it needs to go through CLIPort, so we need to post-
process them to fit CLIPort input format. Figure 10 shows how we post-process
them.

We evaluate LMs for 20 test data, which consist of unseen commands and random
objects. The order of the experiments is as follows: (1) Randomly spawn
objects on the table in the simulator. (2) Input a command to the LMs, and get
a sequence of intermediate actionable steps. (3) Post-process the output to
fit in CLIPort input format, and enter it into CLIPort. (4) The robot performs
the action steps based on the output of CLIPort.

We compare the initial and final scenes after performing all action steps, and
manually judged if the robot successfully executed the input commands. We used
the GPT3.5 and GPT4 outputs on the prompts in [22] as the SMs results, and
also verified the outputs from our prompt in Figure 5.

Table III presents the success rate per model for the 20 test sets. GPT2-base
has a noticeably lower success rate, and GPT4 has a noticeably higher success
rate, but the notable point is that the GPT2-medium achieves a higher success
rate than GPT3.5 on SMs. However, GPT3.5 on our prompt achieves a higher
success rate than GPT2-medium. Consequently, despite the large gap in the
number of parameters, GPT2-medium and GPT3.5 are comparable in tabletop
environments, and GPT3.5 has a bit of unstable performance because it depends
on prompting.

###  V-C User Study

Figure 11: Bar graph for survey results. The y-axis represents a score scale
from 1(Very poor) to 7(Excellent).

To evaluate whether small LMs finetuned on our datasets can plan effectively
for high-level commands as LLMs, we conducted a survey. The survey includes
input-outputs samples for the kitchen domain. We employ GPT2-base and
GPT2-medium as small LMs, and employ GPT3.5 and GPT4 as LLMs. To get the
outputs under the same conditions as in small LMs, we use the LLMs’ output
returned from inputting the prompt 4. Participants receive the input objects,
input commands, and each model’s outputs for the input, without being informed
of which model each output was from, and asked to rate each output on a scale
of 1 to 7. The scoring criteria are as follows: First, whether the output
steps are appropriate to execute the input command. Second, the output steps
are represented in small enough steps for the robot to follow. Third, the
output steps can be performed using the appropriate things in given objects.
The survey consisted of 3 different input sets and their outputs. We built
three versions of the survey with different sample sets and surveyed 40 people
per survey. The results of the three surveys are combined to get the overall
results.

Figure 11 is the bar graph for the survey result. It shows that GPT2-base has
a markedly lower score, and GPT4 has a markedly higher score. However,
interestingly, there is little difference in scores between GPT2-medium and
GPT3.5, and even GPT2-medium scored more consistently higher. The p-value
between GPT2-medium and GPT3.5 is around 0.42, which means that there is no
statistical difference between the results of the two models. Consequently, in
a given domain, GPT2-medium fine-tuned for planning can reason in chains with
nearly equivalent performance to GPT3.5.

##  VI CONCLUSIONS

In this work, we suggest the question for the discrepancy between the low
complexity of commands in robotics and the advanced capability of LLMs, and
explore the potential for task planning on small LMs in a single domain. To
validate the question, first, we build the COmmand-STeps dataset (COST) on a
single domain and finetuned small LMs with it. Then, we experiment to compare
the finetuned small LMs with LLMs in not only tabletop simulator but also user
study for the kitchen environment. As a result, we find that domain-specific
GPT2-medium and domain-agnostic GPT3.5 are comparable for planning the robot
actions in the given environment. We argue that LLMs may not always be the
best choice for task planning in robotics, and we also hope that researchers
will positively consider using small LMs to fit the environment in which they
deploy their robots.

## References

  * [1] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid _et al._ , “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control,” in _Conference on Robot Learning_. PMLR, 2023, pp. 2165–2183.
  * [2] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian _et al._ , “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances,” in _Conference on robot learning_. PMLR, 2023, pp. 287–318.
  * [3] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou _et al._ , “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 24 824–24 837, 2022.
  * [4] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou _et al._ , “Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them,” in _Findings of the Association for Computational Linguistics: ACL 2023_ , 2023, pp. 13 003–13 051.
  * [5] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann _et al._ , “PaLM: Scaling Language Modeling with Pathways,” _Journal of Machine Learning Research_ , vol. 24, no. 240, pp. 1–113, 2023.
  * [6] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso _et al._ , “Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models,” _Transactions on Machine Learning Research_ , 2023.
  * [7] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray _et al._ , “Training language models to follow instructions with human feedback,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 27 730–27 744, 2022.
  * [8] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu _et al._ , “LIMA: Less Is More for Alignment,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [9] W. Chen, S. Hu, R. Talak, and L. Carlone, “Leveraging Large (Visual) Language Models for Robot 3D Scene Understanding,” _arXiv preprint arXiv:2209.05629_ , 2022.
  * [10] J. Park, S. Lim, J. Lee, S. Park, M. Chang, Y. Yu, and S. Choi, “CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agents,” _IEEE Robotics and Automation Letters_ , 2023.
  * [11] Z. Mandi, S. Jain, and S. Song, “RoCo: Dialectic Multi-Robot Collaboration with Large Language Models,” _arXiv preprint arXiv:2307.04738_ , 2023.
  * [12] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “ChatGPT for Robotics: Design Principles and Model Abilities,” _Microsoft Auton. Syst. Robot. Res_ , vol. 2, p. 20, 2023.
  * [13] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, “TidyBot: personalized robot assistance with large language models,” in _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2023, pp. 3546–3553.
  * [14] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “ProgPrompt: Generating Situated Robot Task Plans using Large Language Models,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 11 523–11 530.
  * [15] J. Ruan, Y. Chen, B. Zhang, Z. Xu, T. Bao, H. Mao, Z. Li, X. Zeng, R. Zhao _et al._ , “TPTU: Task Planning and Tool Usage of Large Language Model-based AI Agents,” in _NeurIPS 2023 Foundation Models for Decision Making Workshop_ , 2023.
  * [16] K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf, “SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning,” in _Conference on Robot Learning_. PMLR, 2023, pp. 23–72.
  * [17] Z. Wu, Z. Wang, X. Xu, J. Lu, and H. Yan, “Embodied Task Planning with Large Language Models,” _arXiv preprint arXiv:2307.01848_ , 2023.
  * [18] Y. Ding, X. Zhang, C. Paxton, and S. Zhang, “Task and Motion Planning with Large Language Models for Object Rearrangement,” in _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2023, pp. 2086–2092.
  * [19] S. Narang, C. Raffel, K. Lee, A. Roberts, N. Fiedel, and K. Malkan, “WT5?! Training Text-to-Text Models to Explain their Predictions,” _arXiv preprint arXiv:2004.14546_ , 2020.
  * [20] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large Language Models are Zero-Shot Reasoners,” _Advances in neural information processing systems_ , vol. 35, pp. 22 199–22 213, 2022.
  * [21] A. Lampinen, I. Dasgupta, S. Chan, K. Mathewson, M. Tessler, A. Creswell, J. McClelland, J. Wang, and F. Hill, “Can language models learn from explanations in context?” in _Findings of the Association for Computational Linguistics: EMNLP 2022_ , 2022, pp. 537–563.
  * [22] A. Zeng, M. Attarian, K. M. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. S. Ryoo, V. Sindhwani, J. Lee _et al._ , “Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language,” in _The Eleventh International Conference on Learning Representations_ , 2022.
  * [23] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu _et al._ , “PaLM-E: An Embodied Multimodal Language Model,” in _Proceedings of the 40th International Conference on Machine Learning_ , 2023, pp. 8469–8488.
  * [24] L. C. Magister, J. Mallinson, J. Adamek, E. Malmi, and A. Severyn, “Teaching Small Language Models to Reason,” in _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 2023, pp. 1773–1781.
  * [25] M. Kang, S. Lee, J. Baek, K. Kawaguchi, and S. J. Hwang, “Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [26] Y. Ma, H. Jiang, and C. Fan, “Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA,” _arXiv preprint arXiv:2308.04679_ , 2023.
  * [27] Y. Fu, H. Peng, L. Ou, A. Sabharwal, and T. Khot, “Specializing Smaller Language Models towards Multi-Step Reasoning,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 10 421–10 430.
  * [28] G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” _stat_ , vol. 1050, p. 9, 2015.
  * [29] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” _The Journal of Machine Learning Research_ , vol. 21, no. 1, pp. 5485–5551, 2020.
  * [30] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language Models are Unsupervised Multitask Learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [31] J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,” _arXiv preprint arXiv:2302.11382_ , 2023.
  * [32] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu _et al._ , “RT-1: Robotics Transformer for Real-World Control at Scale,” _arXiv preprint arXiv:2212.06817_ , 2022.
  * [33] P. Sermanet, T. Ding, J. Zhao, F. Xia, D. Dwibedi, K. Gopalakrishnan, C. Chan, G. Dulac-Arnold, N. Joshi, P. Florence _et al._ , “RoboVQA: Multimodal Long-Horizon Reasoning for Robotics,” in _NeurIPS 2023 Foundation Models for Decision Making Workshop_ , 2023.
  * [34] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine, “Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets,” _arXiv preprint arXiv:2109.13396_ , 2021.
  * [35] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du _et al._ , “BridgeData V2: A Dataset for Robot Learning at Scale,” in _Conference on Robot Learning_. PMLR, 2023, pp. 1723–1736.
  * [36] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence, “Interactive Language: Talking to Robots in Real Time,” _IEEE Robotics and Automation Letters_ , 2023.
  * [37] M. Shridhar, L. Manuelli, and D. Fox, “CLIPort: What and Where Pathways for Robotic Manipulation,” in _Conference on Robot Learning_. PMLR, 2022, pp. 894–906.