# Investigating the impact of spin effects at the high-energy neutrino-nucleon
interactions while it crosses the Earth’s core

R. Francener  Instituto de Física Gleb Wataghin - UNICAMP, 13083-859,
Campinas, SP, Brazil.  D.R. Gratieri  Escola de Engenharia Industrial
Metalúrgica de Volta Redonda, Universidade Federal Fluminense (UFF),
27255-125, Volta Redonda, RJ, Brazil  Instituto de Física Gleb Wataghin -
UNICAMP, 13083-859, Campinas, SP, Brazil.  G. Torrieri  Instituto de Física
Gleb Wataghin - UNICAMP, 13083-859, Campinas, SP, Brazil.  Institute of
Physics, Jan Kochanowski University, Ul. Uniwersytecka 7, 25-406 Kielce,
Poland.

###### Abstract

In this work, we investigate the impact of assuming the polarization of the
Earth’s outer core on the propagation of neutrinos that cross the all the
Earth. We taking into account the spin-dependent structure functions to
describe the polarized neutrino-nucleon cross-section, and also on the
neutrino absorption while it crosses all of the Earth. We found that adding
spin information and simultaneously assuming polarization of Earth’s outer
core impacts the probability of neutrino absorption in the energy range of 10
- 100 TeV and for upward neutrino direction. However, the magnitude of the
effect is small and should be comparable with the magnitude of the errors
associated with the IceCube neutrino data.

Neutrino Absorption, Polarized Targets, Earth’s Outer Core.

##  I Introduction

Interactions with polarized nuclei have gained great attention in theoretical
and experimental physics community since the results of the European Muon
Collaboration (EMC) from the late eighties EMC_1988 ; EMC_1989 . Such results
pointed out that “the total quark spin constitutes only a small fraction
$\Delta\Sigma(Q^{2})$ of the proton’s spin”. This result became known as the
“Proton Spin Crisis” (PSC). Usually, the nucleon spin is assumed to be given
in terms of the sum of the spin contributions from quarks, gluons and orbital
magnetic moment from both quarks and gluons,

$\displaystyle\frac{1}{2}=\frac{1}{2}\Delta\Sigma(Q^{2})+\Delta
G(Q^{2})+L_{q}(Q^{2})+L_{g}(Q^{2}),$ (1)

where $\Delta G(Q^{2})$ is the gluon contribution to the nucleon spin,
$L_{q}(Q^{2})$ and $L_{g}(Q^{2})$ are the Orbital Angular Momentum (OAM)
contributions from quarks and gluons, respectively. The quark contribution to
nucleon spin, $\Delta\Sigma(Q^{2})$, can also be understood in terms of the
sum in all quarks of integral in x of helicity distributions $(\Delta
q^{i}(x,Q^{2}))$ EMC_1988 ; EMC_1989 . In all cases, in the Naive Parton
Model, the quantity $\Delta q^{i}(x,Q^{2})dx$ is the number of polarized
(anti)quarks of type $q$ carrying a momentum fraction between $x$ and $x+dx$.
The index $i=u,d,c,s,b,t$ stands for each quark flavor. Within the Quantum
Cromodynamics (QCD) there are both perturbative and non-perturbative
corrections, in such way that the structure functions associated with the
nucleons are obtained from the partonic density functions through the
Factoriazation Theorem Collins:1989gx . For a complete review, see
Workman:2022ynf . For the theoretical formalism, we point to reference pol ,
which we follow closely. See also pol2 . Explicitly, the result from the EMC
collaboration pointed out that $\Delta\Sigma(Q^{2})=0.14\pm 0.23$. Current
analysis from COMPASS collaboration COMPASS  report that about $31\%\pm 11\%$
of proton spin comes from quarks, for $Q^{2}=3\;\mathrm{GeV}^{2}$. Actually,
the literature seems to converge for the longitudinally spin polarization
scattering, but there still a puzzle for transversely polarized nuclear
targets interacting with transversely polarized projectile nucleons ($\approx
40\%$) and charged leptons ($\approx 5-10\%$) Spin_quark_sea . To illustrate
the actual scenario, in Fig. 1 we present the contribution to proton’s spin
due to quarks measured by several collaborations EMC_1988 ; EMC_1989 ; COMPASS
; COMPASS3 ; compass2 ; clas ; hermes2 ; SMC ; e143 ; e142 ; e154 ; jlab  as
function of the respective momentum scale, $Q^{2}$. For comparison, our
results assuming the predictions from KATAO ; KATAO2 ; KATAO3 ; florian ;
florian2  are also shown. At the present level of experimental accuracy, the
scalling of polarized PDFs was still not clearly seen Workman:2022ynf . In
fact, the structure functions associated with the quarks are relatively the
most known ones for the both unpolarized and polarized cases. Moreover, a
recent work from JAM Collaboration Cocuzza:2022ovz  based on the STAR data
Ethier:2017zbq  in the range of $0.01\leq x\leq 0.3$ and $Q^{2}=10$ GeV 2,
presents for the first time results favoring the nonzero helicity sea
assymmetry, with $\Delta{\cal{X}}^{2}/N_{dat}\approx 1\sigma$. Such results
are from a global analysis of both unpolarized and polarized PDFs. Concerning
the gluon spin, we know today Workman:2022ynf  that at small $x$ and large
$Q^{2}$ the gluon density function, $g(x,Q^{2})$ , is considerably larger than
the density functions associated to the quarks, which implies that at the high
energy regime the nucleon can be understood as a collective of gluons. Hence,
it is straightforward to expect some degree of contribution from gluons to the
nucleon spin. Indeed, in Spin_quark_sea , which is a review of the topic, it
is stated that while the contribution from the valence quarks saturates at the
high-energy limit, the gluon contribution is expect to reach $\approx 50\%$ at
the present accelerator energies. This is in agreement with recent results
from lattice QCD lattice  and also and experimental analyzes exp_gluon .
Moreover, since $\Delta\Sigma(Q^{2})$ and $\Delta G(Q^{2})$ are observables,
overall spin conservation $(S_{p}=1/2)$ can be used, and implies in large
contribution from OAM to the proton spin Scopetta:1999ue . Indeed, in
Ratcliffe  it is shown that orbital angular momentum is generated in the
partonic dynamical evolution as it is given by the DGLAP equations
Gribov:1972ri ; Altarelli:1977zs ; Dokshitzer:1977sg .

Figure 1:  Compilation of experimental results for the quarks contribution to
proton spin from EMC_1988 ; EMC_1989 ; COMPASS ; COMPASS3 ; compass2 ; clas ;
hermes2 ; SMC ; e143 ; e142 ; e154 ; jlab . For comparison, the prediction
from KATAO KATAO ; KATAO2 ; KATAO3  and DSSV florian ; florian2  are also
shown.

To better describe the quark and gluon content and aspects of the three-
dimensional structure of the nucleon, there are generalized parton
distribution (GPDs) and transverse momentum dependent distributions (TMDs)
introduction_gpd ; introduction_gpd2 . Such distributions are complementary
and aim to describe the transverse plane of nucleon propagation. Another
important class of parton distributions are the longitudinal spin-dependent
ones, which describe the asymmetry between quarks with opposite spins in the
nucleon. Recently, several experimental collaborations have focused their
efforts on measuring the (longitudinal) spin-dependent structure functions in
collisions of charged leptons with polarized hydrogen, deuterium and helium-3
nuclei e143 ; hermes ; compass2 ; clas ; jlab . Current measurements focus on
modest kinematic ranges ($x>10^{-3}$ and $Q^{2}<10^{2}\;\mathrm{GeV}$). With
existing data on spin asymmetry and spin-dependent structure functions,
different authors have built fits that parameterize these data florian ;
florian2 ; KATAO ; KATAO2 ; KATAO3  and allow extrapolations beyond observed
kinematic ranges. One of the main goals of the future Electron-Ion Collider
(EIC) eic  is to improve our understanding of the helicity distributions of
quarks and gluons inside nucleons and heavy nuclei. With this new collider it
is intended to increase the current observation range of these distributions.
While $x$ should be decreased to approximately $5\cdot 10^{-5}$, $Q^{2}$
should be increased to approximately $10^{3}\;\mathrm{GeV}^{2}$ (see Fig. 10
in eic ). Typical IceCube events occur with $x\sim 10^{-2,\;-3}$ and
$Q^{2}\sim 10^{3,\;4}$, very close to the future EIC data, making the
necessary extrapolation much closer and with a high confidence level.

In this work we present a study of the impact of polarization of hadronic
targets on Deep Inelastic Scattering (DIS) of muonic neutrinos and
antineutrinos, and we apply the obtained result to the neutrinos absorption by
the Earth. Such a study is strongly motivated by a probable polarization of
the Earth’s outer core, whith is generated by a turbulent flow of liquid metal
geodynamics . We will verify if through the interaction of neutrinos with the
Earth it is possible to estimate the polarization of the outer core. This
study is also motivated by the recent IceCube measurement of the cross section
of muonic neutrinos by the Earth’s absorption measure . This measurement
indicates that the cross section, in the observed energy range
($6.3-980\;\mathrm{TeV}$), is about 1.3 times the cross section predicted by
the standard model modelop . The interaction of neutrinos with the Earth can
be measured through the attenuation of the incident neutrino flux that crosses
the Earth and is measured by the IceCube, as illustrated in Fig. 2. The
IceCube detector can measure High Energy Neutrino Sample, with energies above
$60\;\mathrm{TeV}$. In this energy range the predominant interaction is deeply
inelastic scattering formaggio . For the neutrino to reach the detector, it is
necessary that it does not interact via charged current when crossing the
Earth. Neutral current interaction only decreases the energy of the beam.
Although in this work we focus on the analysis of the interaction of muon
neutrino, the results are also very similarly applicable for electron
neutrino. In the energy limit of the neutrino much larger than the mass of the
lepton produced, electron and muon neutrinos have the same cross section with
hadronic targets. However, for the analysis of the absorption of electron
neutrino it is necessary to also include the effects of interaction with
electrons (Glashow resonance) glashow ; gandhi . For recent work on neutrino
absorption considering Glashow resonance, see Refs. IceCube2 ; victor ;
victor2 .

Figure 2:  Attenuation of neutrino flux by Earth’s absorption.

##  II Formalism

A correct description of the proton spin from quarks is of particular interest
for neutrino physics. Neutrino and antineutrino are left-handed and right-
handed chiral eigenstates, respectively. So, (anti)neutrino capturing the
(right)left-hand component of the quark wavefunction, and any change between
distribution of the quarks right and left-hand in the nucleons will impact
specifically (anti)neutrino scattering and absorption cross section. At this
point one must notice that besides the fact that weak neutrino-nucleon
interactions are given in terms of chiral states. Thus, the information about
how much of the nucleon spin is due to the quarks, i. e., the value of
$\Delta\Sigma(Q^{2})$, is not taken into account in the most common procedure
of calculation of neutrino-nucleon cross-section at the deep inelastic regime
111At the high neutrino energies, where the description of the nucleon target
in terms of form factors is no longer available.. At sufficiently high
energies, due to asymptotic freedom Gross:1973id ; Politzer:1973fx , it is
possible to describe the neutrino-nucleon interaction in terms of the neutrino
scattering on free quarks that constitute the nucleon. The assumption of the
equal distribution of left and right quark spins leads to the average of the
initial polarization state. As both the charged current (CC) and neutral
current (NC) DIS processes are inclusive reactions, they take into account all
the possible final hadronic states, and a sum over all the final state
polarization possibilities is also applied halzen . In pol , the formalism to
include spin effects at (anti)neutrino-nucleon interaction is presented.

In the unpolarized DIS (CC), the neutrino $\nu_{l}$ (antineutrino
$\bar{\nu}_{l}$) with energy $E_{\nu}$ interacts by exchanging a virtual boson
$W^{\pm}$ of four - momentum $q$ ($q^{2}=-Q^{2}$). The initial lepton becomes
the associated charged lepton $l^{\pm}(=e,\mu,\tau)$ with energy $E^{\prime}$
and the hadron goes to an unknown state of invariant mass $W$, characterized
by $W^{2}>m_{N}^{2}$, with $m_{N}$ being the mass of the hadronic target. In
terms of Bjorken’s $x$, the inelasticity $y=(E_{\nu}-E^{\prime})/E_{\nu}$ and
the virtuality of the exchanged boson $Q^{2}$, the unpolarized double
differential cross section of neutrino DIS is given by reno

$\displaystyle\begin{aligned}
\frac{\mathrm{d}\sigma^{\nu(\bar{\nu})}}{\mathrm{d}x\mathrm{d}y}=\frac{G_{F}^{2}E_{\nu}m_{N}}{\pi}\left(\frac{M_{W}^{2}}{Q^{2}+M_{W}^{2}}\right)^{2}\left\\{\left(y^{2}x+\frac{m_{l}^{2}y}{2E_{\nu}m_{N}}\right)F_{1}(x,Q^{2})+\right.\left(1-y-\frac{m_{l}^{2}}{4E_{\nu}^{2}}-\frac{m_{N}xy}{2E_{\nu}}\right)F_{2}(x,Q^{2})+\\\
+(-)\left(xy-\frac{xy^{2}}{2}-\frac{m_{l}^{2}y}{4E_{\nu}m_{N}}\right)F_{3}(x,Q^{2})\left.+\frac{m_{l}^{2}(m_{l}^{2}+Q^{2})}{E_{\nu}^{2}m_{\nu}^{2}x}F_{4}(x,Q^{2})-\frac{m_{l}^{2}}{E_{\nu}m_{N}}F_{5}(x,Q^{2})\right\\}\,\,\,,\end{aligned}$
(2)

where $G_{F}$ is the Fermi’s constant, $M_{W}$ the $W^{\pm}$ boson mass,
$m_{l}$ the mass of the lepton produced and $F_{i}$ are the spin independent
structure functions. In the parton model, $F_{2}(x,Q^{2})$ is interpreted in
terms of the sum of the helicity distributions of the quarks that have the
flavor that can interact with the neutrino paschos . $F_{1}(x,Q^{2})$ can be
written in terms of $F_{2}(x,Q^{2})$ with the Callan-Gross relation and
$F_{3}(x,Q^{2})$ is associated with quark - antiquark asymmetry. In this paper
we use the CTEQ18 parameterization cteq18  for quark distributions, which uses
the DGLAP evolution equations. In the high energy limit, which we are
interested in, we also assume that the Albright - Jarlskog relations aj  hold.
The standard variables of DIS are connected by $Q^{2}=2E_{\nu}m_{N}xy$, in the
rest frame of target.

Moreover, when we consider the polarized hadronic target, the cross section of
the Eq. 2 is modified by a factor dependent on the hadronic spin, given by pol
; pol2

$\displaystyle\begin{aligned}
\frac{\mathrm{d}\Delta\sigma^{\nu(\bar{\nu})}}{\mathrm{d}x\mathrm{d}y}=\frac{G_{F}^{2}ME_{\nu}}{\pi}\left(\frac{M_{W}^{2}}{Q^{2}+M_{W}^{2}}\right)^{2}\lambda_{N}\left\\{(-)\left[-yx(2-y)+\frac{2x^{3}y^{3}m_{l}^{2}}{Q^{2}}\right]\right.g_{1}(x,Q^{2})+(-)\left(\frac{4x^{3}y^{2}m_{l}^{2}}{Q^{2}}\right)g_{2}(x,Q^{2})+\\\
+\frac{2xym_{l}^{2}}{Q^{2}}\left(1-y-\frac{x^{2}y^{2}m_{l}^{2}}{Q^{2}}\right)g_{3}(x,Q^{2})+\left[-1+y-\frac{2x^{2}ym_{l}^{2}}{Q^{2}}\left(1-\frac{3y}{2}-\frac{x^{2}y^{2}m_{l}^{2}}{Q^{2}}\right)\right]g_{4}(x,Q^{2})+\\\
\left.+\left(-y^{2}x+\frac{2x^{4}y^{3}m_{l}^{2}}{Q^{2}}\right)g_{5}(x,Q^{2})\right\\}\,\,\,,\end{aligned}$
(3)

where $\lambda_{N}$ is the helicity of the hadronic target and
$g_{i}(x,Q^{2})$ are the spin - dependent structure functions. Unlike the
$F_{i}(x,Q^{2})$ functions, the $g_{i}(x,Q^{2})$ functions in the parton model
are written with the difference of the helicity distributions of the each
quarks, describing the net amount of quarks with spin in a given direction
pol2 . In the limit of energies of interest in this work, we assume the
validity of the Dicus relation dicus . Such a relation allows us to write
$g_{4}$ in terms of $g_{5}$, similar to the Callan-Gross relation,
$g_{4}(x,Q^{2})=2xg_{5}(x,Q^{2})$. Both relationships emerged from the
observation that, when we neglect the masses involved, helicity is conserved
in the quark-gluon coupling. The polarized structure functions are described
in pol , where it is shown that the contributions from $g_{2}(x,Q^{2})$ and
$g_{3}(x,Q^{2})$ to the cross-sections are suppressed by factors like
$m^{2}_{l}/Q^{2}$, which could be measured in the neutrino factories. However,
in this work we are initially interested in $Q^{2}\geq\approx m^{2}_{W}$,
which is the typical value for the neutrino-nucleon interactions at the
IceCube neutrino observatory. Hence, we can disregard contributions from
$g_{2}(x,Q^{2})$ and $g_{3}(x,Q^{2})$. Also, in the same limit, as the same
structures appears in in both polarized and unpolarized cases, the polarized
contribution to the neutrino-nucleon cross-section can be obtained from Eq. 2
replacing $F_{1}(x,Q^{2})\rightarrow-g_{5}(x,Q^{2})$,
$F_{2}(x,Q^{2})\rightarrow-g_{4}(x,Q^{2})$, and $F_{3}(x,Q^{2})\rightarrow
2g_{1}(x,Q^{2})$.

Figure 3: The thickness of the Earth in centimeters of the water for nucleons
unpolarized and with outer core partially polarized. We use the PREM model
prem .

An accurate description of the neutrino - nucleon cross section, as well as
the distribution of matter in the interior of the Earth, is fundamental to
estimate the absorption of neutrinos that cross the Earth, given that these
are fundamental ingredients for the calculation. The probability of the
neutrino crossing without being absorbed can be quantified with gandhi

$\displaystyle
P_{Shad}(E_{\nu},\theta_{z})=\mathrm{exp}\left[-N_{A}\sigma(E_{\nu})\int^{r(\theta_{z})}_{0}\rho_{N}(r)\mathrm{d}r\right]\,\,\,,$
(4)

where $N_{A}$ is the Avogadro numbers, $\rho(r)$ the Earth’s density profile
and $r(\theta_{z})=-2R_{Earth}\mathrm{cos}\;\theta_{z}$ is the total distance
travelled by neutrino. In this work we use the PREM model prem  for the
description of the Earth’s profile density. In Fig. 3 we show the thickness of
matter traversed by the neutrino as a function of the zenith angle. We see two
distinct cases: in the continuous black line we show the thickness of matter
crossed without considering polarization. While the dashed curves show the
thickness of matter traversed unpolarized (red) and polarized (below)
considering a hypothetical case of 30% polarization in the outer core (blue).
The PREM model indicates that the outer core is located between 1221.5 km and
3480.0 km, which implies that the neutrino crosses this potentially polarized
layer only if it hits with $\mathrm{cos}\;(\theta_{z})$ less than $-0.84$.

|  |
---|---|---
(a) |  | (b)
|  |
(c) |  | (d)
Figure 4: Cross section for muon (a) neutrino and (b) antineutrino DIS with
the isoscalar target. In (c) and (d) we present the same results for tau
neutrino. We consider two distinct cases: unpolarized target and polarized
with DSSV florian  and KATAO KATAO  parametrizations of spin - dependent
structure functions.

##  III Results

Initially, we present in Fig. 4 the cross section of (a) muon and (c) tau
neutrino; and (b) muon and (d) tau antineutrino with a isoscalar target as a
function of (anti)neutrino energy. We calculate the cross sections for
unpolarized and polarized targets with helicity $\lambda_{N}=-1$. For the
calculation of the polarized cross sections, we use two different
parameterizations of the spin-dependent structure functions, DSSV florian  and
KATAO KATAO . Both parameterizations lead to similar results. The impact of
target polarization on the cross section becomes less significant with
increasing energy, practically disappearing for energies above
$10^{7}\;\mathrm{GeV}$. For lower energies of incoming neutrino,
$10^{3}-10^{4}\;\mathrm{GeV}$, the unpolarized and polarized cross sections
differ by a multiplicative factors of $0.7-1.3$, depending on the
parameterization of the spin - dependent structure functions and whether the
beam is neutrino or antineutrino.

|  |
---|---|---
(a) |  | (b)
Figure 5: Difference between unpolarized and polarized differential cross sections normalized by unpolarized differential cross sections for (a) neutrino and (b) antineutrino incident. We calculate using the KATAO KATAO  and CTEQ18 cteq18  parameterization, for neutrino and antineutrino incident with energies of $10^{3}\;\mathrm{GeV}$.  |
---|---
(a) | (b)
|
(c) | (d)
Figure 6: Probability of the antineutrino, (a) and (b), and neutrino, (c) and
(d), crossing the Earth without interacting via charged current as a function
of energy and $\mathrm{cos}\;(\theta_{z})$ considering the unpolarized Earth,
(a) and (c), and with 100% polarization in the outer core, (b) and (d).

To better trace the origin of the polarization effect on the nutrino-nucleon
interaction (CC), we present Fig. 5. In it we quantify the difference between
the unpolarized (UU) and polarized (LL) differential cross section, normalized
with the unpolarized differential cross section. $\theta_{z}$ is the angle
between the direction of arrival of the neutrino (antineutrino) with the spin
of the isoscalar target, considering again $\lambda_{N}=-1$. We clearly see
that the difference between the differential cross sections is maximized when
the spins are parallel or antiparallel to the direction of propagation of the
incident neutrino. In $Q^{2}$, this normalized difference is maximized in
different regions for neutrino and antineutrino. While for the neutrino we
have a maximum of $\approx 80\;\mathrm{GeV}^{2}$, for the antineutrino the
maximum goes beyond $200\;\mathrm{GeV}^{2}$, the region with the smallest
contribution to the total cross section.

In Fig. 6 we present our result for the probability of crossing without being
absorbed, $P_{Shad}$, of muonic neutrinos by the Earth as a function of the
energy and the cosine of the zenith angle of incidence of the neutrino. In the
upper panel, we present $P_{Shad}$ for antineutrinos and in the lower panel
for neutrinos. In the Figs. 6a and 6c we do not consider any polarization on
earth. In the Figs. 6b and 6d, $P_{Shad}$ is calculated considering 100%
polarization in the outer core of the Earth. The calculation of the cross
sections disregarding polarization is performed with the structure functions
constructed with the quark distributions parameterized by CTEQ18 cteq18 . For
the cross section with polarization correction of the hadronic target (Figs.
6b and 6d), we used the KATAO parameterization KATAO  spin - dependent
structure functions, besides, of course, CTEQ18 for the spin - independent
structure functions. The choice of KATAO parameterization for this result is
practically indifferent to the choice of DSSV, because, as previously
discussed and illustrated in Fig. 4, both lead to very similar cross sections.

In Fig. 7 we present the difference between the absorption probabilities of
unpolarized and polarized Earth. In Figs. 7a and 7c (7b and 7d) are the
results for the absorption of antineutrino (neutrino) considering two cases:
30% and 100% of polarization in the outer core, respectively. The effect is
restricted to the region where the neutrino impinges with
$\mathrm{cos}\;(\theta_{z})<-0.84$, given that this is the necessary condition
for it to cross the outer core. We can see that neutrino and antineutrino have
opposite effects: while $P_{Shad}$ of neutrinos decreases with the effect of
polarization, $P_{Shad}$ of antineutrinos increases with said effect. This
fact hinders the experimental validation of the model, since the IceCube does
not distinguish the charge of the produced lepton. Even considering 100%
polarization the effect is small, however in the IceCube observation region
($10^{3}-10^{6}\;\mathrm{GeV}$).

|
---|---
(a) | (b)
|
(c) | (d)
Figure 7: Difference between probability of the antineutrino, (a) and (b), and
neutrino, (c) and (d), crossing the Earth without interacting via charged
current as a function of energy and $\mathrm{cos}\;(\theta_{z})$ considering
the unpolarized Earth, (a) and (c), and with 100% polarization in the outer
core, (b) and (d).

To better quantify the difference described above and presented in Fig. 7, we
calculate the percentage difference between unpolarized and polarized
absorptions ($(P_{Shad}^{UNPOL.}-P_{Shad}^{POL.})/P_{Shad}^{UNPOL.}$). We
estimate that in the IceCube observation region where the absorption is
significant, with 30% polarization the absorption changes between 0% and
$\pm$5%. For 100% polarization this percentage rises to about $\pm$18%. It is
still possible to observe that the mentioned effect is maximized with
$\mathrm{cos}\;(\theta_{z})\rightarrow-0.98$, when the neutrino crosses the
largest possible amount of the outer core. Despite being significant
percentages in the change of absorption, it is very difficult to observe it.
Because we use high percentages of polarization to get it, and the effects on
neutrinos and antineutrinos are of similar but opposite magnitudes. To observe
this effect, a future detector capable of distinguishing between neutrinos and
antineutrinos would be needed.

##  IV Summary

In this study we have investigated the impact of the polarization of the
Earth’s outer core on the absorption of neutrinos in the IceCube observation
region. Our main motivation was to verify if a possible estimate of the
polarization of the Earth’s outer core can be made in the future by
attenuating the flux of neutrinos that cross the Earth. Our results showed
that the effects of polarization on absorption, although in the IceCube
region, are small even considering 100% polarization. Added to this, while the
absorption of neutrinos decreases, that of antineutrinos increases with the
polarization of the hadronic targets, making it a difficult task to estimate
the nuclear polarization by the attenuation of the neutrino flux. Given the
magnitude of the impact of polarization on neutrino absorption, a more
detailed analysis of the angular and energy distributions of the expected
number of events in the IceCube detector becomes unfeasible. The future EIC
may change the current view we have of the contribution of sea quarks to the
proton spin, and motivate more detailed analyzes of neutrino absorption
considering polarization on Earth. Furthermore, with the neutrino detectors of
the future, such as IceCube-Gen2 IceCubeGen2  and GRAND grand , the prospect
of measuring these smaller magnitude effects may become feasible. This work is
far from a comprehensive estimate of possible spin QCD effects. Even if earth
polarization is exactly zero, it does not mean spin dependent effects vanish:
It is known that correlations of spin in nuclei are strong, i.e. each
nucleon’s polarization depends on the others. “One level down”, parton level
correlations are both strong and relatively unknown (for instance, is the up
quark more likely to be aligned or anti-aligned with the proton’s spin?).
Accounting for these effects would require convoluting into the earth’s
profile both the nuclear spin wavefunction of iron and medium-modified quark
TMDs, and is currently beyond the scope of this work. We show the impact of
polarization only on muon neutrinos absorption. However, the effect on
electronic neutrinos is essentially the same, as they have the same cross
section in the high energy limit. For tauonic neutrinos there is a greater
difference in cross section at lower energies, due to the mass of the tau
produced. However, a more detailed analysis of tau neutrinos absorption for
future detectors presupposes the study of flux regeneration by tau decay,
which is outside the scope of this work.

###### Acknowledgements.

This work was partially financed by the Brazilian funding agencies CNPq and
CAPES. G.T. acknowledges support from Bolsa de produtividade CNPQ
306152/2020-7, Bolsa de pesquisa FAPESP 2021/01700-2, Partecipation to
Tematico FAPESP, 2017/05685-2 and the grant BPN/ULM/2021/1/00039 from the
Polish National Agency for Academic Exchange.

## References

  * (1) J. Ashman et al. [EMC Collaboration], Phys. Lett. B 206, 364 (1988);
  * (2) J. Ashman et al. [EMC Collaboration], Nucl. Phys. B 328, 1 (1989);
  * (3) J. C. Collins, D. E. Soper and G. F. Sterman, Adv. Ser. Direct. High Energy Phys. 5, 1-91 (1989);
  * (4) R. L. Workman et al. [Particle Data Group], PTEP 2022, 083C01 (2022);
  * (5) S. Forte, M. L. Mangano and G. Ridolfi, Nucl. Phys. B 602, 585–621 (2001);
  * (6) M. Anselmino, P. Gambino and J. Kalinowski, Zeit. Phys. C 64, 267 (1994);
  * (7) C. Adoiph et al. [COMPASS Collaboration], Phys. Lett. B 769, 34-41 (2017);
  * (8) C. A. Aidala, S. D. Bass, D. Hasch and G. K. Mallot, Rev. Mod. Phys. 85, 655 (2013);
  * (9) V. Y. Alexakhin et al. [COMPASS Collaboration], Phys. Lett. B 647, 8 (2007);
  * (10) C. Adolph et al. [COMPASS Collaboration], Phys. Lett. B 753, 18 (2016);
  * (11) R. G. Fersch et al. [CLAS Collaboration], Phys. Rev. C 96, 065208 (2017);
  * (12) A. Airapetian et al. [HERMES Collaboration], Phys. Rev. D 75, 012007 (2007);
  * (13) B. Adeva et al. [Spin Muon Collaboration], Phys. Lett. D 302, 533 (1993);
  * (14) K. Abe et al. [E143 Collaboration], Phys. Rev. D 58, 112003 (1998);
  * (15) P. L. Anthony et al. [E142 Collaboration], Phys. Rev. D 54, 6620 (1996);
  * (16) K. Abe et al. [E142 Collaboration], Phys. Rev. Lett. 79, 26 (1997);
  * (17) N. Sato, W. Melnitchouk, S. E. Kuhn, J. J. Ethier and A. Accardi, [Jefferson Lab Angular Momentum Collaboration], Phys. Rev. D 93, 074005 (2016);
  * (18) A. N. Khorramian, S. Atashbar Tehrani, S. Taheri Monfared, F. Arbabifar and F. I. Olness, Phys. Rev. D 83, 054017 (2011);
  * (19) H. Khanpour, S. Taheri Monfared and S. Atashbar Tehrani, Phys. Rev. D 95, 074006 (2017);
  * (20) H. Khanpour, S. Taheri Monfared and S. Atashbar Tehrani, Phys. Rev. D 105, 074023 (2022);
  * (21) D. de Florian, R. Sassot, M. Stratmann and W. Vogelsang, Phys. Rev. D 80, 034030 (2009);
  * (22) I. Borsa, D. de Florian and I. Pedron. arXiv:2210.12014 [hep-ph];
  * (23) C. Cocuzza et al. [JAM], SciPost Phys. Proc. 8, 025 (2022);
  * (24) J. J. Ethier, N. Sato and W. Melnitchouk, Phys. Rev. Lett. 119 no.13, 132001 (2017);
  * (25) Y. Yang et al. Phys. Rev. Lett. 118, 102001 (2017);
  * (26) D. de Florian, R. Sassot, M. Stratmann and W. Vogelsang, Phys. Rev. Lett. 113, 012001 (2014);
  * (27) S. Scopetta and V. Vento, [arXiv:hep-ph/9907441 [hep-ph]];
  * (28) P. G. Ratcliffe Phys. Lett. B, v. 192, n. 1-2, p. 180-184, (1987);
  * (29) V. N. Gribov and L. N. Lipatov, Sov. J. Nucl. Phys. 15, 438-450 IPTI-381-71 (1972);
  * (30) G. Altarelli and G. Parisi, Nucl. Phys. B 126, 298-318 (1977).
  * (31) Y. L. Dokshitzer, Sov. Phys. JETP 46, 641-653 (1977);
  * (32) M. Diehl, Eur. Phys. J. A 52, 149 (2016);
  * (33) C. Riedl, Acta Phys. Pol. B 53, 5-A2 (2022);
  * (34) A. Airapetian et al. [HERMES Collaboration], Phys. Lett. B 442, 484 (1998);
  * (35) A. Accardi et al. Eur. Phys. J. A 52, 268 (2016);
  * (36) N. Schaeffer, D. Jault, H. C. Nataf and A. Fournier, Geophys. J. Int. 211, 1–29 (2017);
  * (37) M. G. Aartsen et al. [IceCube], Nature 551, 596-600 (2017);
  * (38) A. Cooper-Sarkar, P. Mertsch and S. Sarkar, JHEP 1108, 042 (2011);
  * (39) J. A. Formaggio and G. P. Zeller, Rev. Mod. Phys. 84, 1307 (2012);
  * (40) S. L. Glashow, Phys. Rev. 118, 316 (1960);
  * (41) R. Gandhi, C. Quigg, M. H. Reno and I. Sarcevic, Astropart. Phys. D 5, 81 (1996);
  * (42) M. G. Aartsen et al. [IceCube], Nature 591, 220 (2021), [Erratum: Nature 592, E11 (2021)];
  * (43) V. P. Gonçalves, D. R. Gratieri and A. S. C. Quadros, Eur. Phys. J. C 81, 496 (2021);
  * (44) V. P. Gonçalves, D. R. Gratieri and A. S. C. Quadros, Eur. Phys. J. C 82, 1011 (2022);
  * (45) D. J. Gross and F. Wilczek, Phys. Rev. Lett. 30, 1343-1346 (1973);
  * (46) H. D. Politzer, Phys. Rev. Lett. 30, 1346-1349 (1973);
  * (47) F. Halzen and A. D. Martin, John Wiley $\&$ Sons, New York, (2008);
  * (48) S. Kretzer and M. H. Reno, Phys. Rev. D 66, 113007 (2002);
  * (49) E. A. Paschos and J. Y. Yu, Phys. Rev. D 65, 033002 (2002);
  * (50) Tie-Jiun Hou et al. Phys. Rev. D 103, 014013 (2021);
  * (51) C. H. Albright and C. Jarlskog, Nucl. Phys. B 84, 467 (1975);
  * (52) D. A. Dicus, Phys. Rev. D 5, 1367 (1972);
  * (53) A .M. Dziewonski and D .L. Anderson, Phys. Earth Plan. Int. 25, 297 (1981);
  * (54) M. G. Aartsen et al. [IceCube-Gen2], J. Phys. G 48, 060501 (2021);
  * (55) J. Álvarez-Muñiz et al. [GRAND Collaboration], Sci. China Phys. Mech. Astron. 63, 219501 (2020);


# Incremental Learning of Humanoid Robot Behavior
from Natural Interaction and Large Language Models

Leonard Bärmann1, Rainer Kartmann2, Fabian Peller-Konrad2, Alex Waibel1, Tamim
Asfour2 This work has been supported by the Carl Zeiss Foundation through the
JuBot project and by the Baden-Württemberg Ministry of Science, Research and
the Arts (MWK) as part of the state’s “digital@bw” digitization strategy in
the context of the Real-World Lab “Robotics AI.”The authors are with the
Institute for Anthropomatics and Robotics (IAR), Karlsruhe Institute of
Technology (KIT), Germany. 1LB and AW are with the Interactive Systems Lab
(ISL). 2RK, FP and TA are with High Performance Humanoid Technologies Lab
(H2T). {baermann<EMAIL_ADDRESS>
###### Abstract

Natural-language dialog is key for intuitive human-robot interaction. It can
be used not only to express humans’ intents, but also to communicate
instructions for improvement if a robot does not understand a command
correctly. Of great importance is to endow robots with the ability to learn
from such interaction experience in an incremental way to allow them to
improve their behaviors or avoid mistakes in the future. In this paper, we
propose a system to achieve incremental learning of complex behavior from
natural interaction, and demonstrate its implementation on a humanoid robot.
Building on recent advances, we present a system that deploys Large Language
Models (LLMs) for high-level orchestration of the robot’s behavior, based on
the idea of enabling the LLM to generate Python statements in an interactive
console to invoke both robot perception and action. The interaction loop is
closed by feeding back human instructions, environment observations, and
execution results to the LLM, thus informing the generation of the next
statement. Specifically, we introduce incremental prompt learning, which
enables the system to interactively learn from its mistakes. For that purpose,
the LLM can call another LLM responsible for code-level improvements of the
current interaction based on human feedback. The improved interaction is then
saved in the robot’s memory, and thus retrieved on similar requests. We
integrate the system in the robot cognitive architecture of the humanoid robot
ARMAR-6 and evaluate our methods both quantitatively (in simulation) and
qualitatively (in simulation and real-world) by demonstrating generalized
incrementally-learned knowledge.

##  I Introduction

For achieving truly intuitive human-robot interaction (HRI), a natural
language interface is key for a humanoid robot. Via language, humans can
easily communicate tasks and goals to the robot. However, the robot’s
interpretation of such commands, and thus the resulting execution, might be
sub-optimal, incomplete or wrong. In such cases, it is desirable for the human
to give further instructions to correct or improve the behavior. In
particular, such cases should be memorized to incrementally learn from them
and thus avoid the same mistake in the future. For instance, consider an
interaction as depicted in Fig. 1. First, the user gives an instruction to the
robot (1). The robot executes some (potentially incomplete or wrong) actions
(2). The user observes the result and gives instructions for improvement (3),
whereupon the robot performs corrective actions (4). If the desired goal is
achieved, the user can reconfirm the correction (5), which leads to the robot
updating its memory appropriately (6), thus incrementally learning new
behavior based on language instructions.

Figure 1:  ARMAR-6 incrementally learns behavior from natural interaction.

In this paper, we present a system to achieve such behavior and describe its
implementation on the humanoid robot ARMAR-6 [1]. We build on the capabilities
of Large Language Models (LLMs) [2, 3, 4] emerging solely from massive-scale
next token prediction pretraining, and aim to transfer their success to HRI.
The goal is to utilize the rich world knowledge contained in these LLMs to
allow for embodied natural-language dialog, thus enhancing the capabilities of
the LLM by integrating robot perception and action. In the cognitive
architecture of our humanoid robot [5], this means the LLM will be in charge
of the high-level planning and decision-making. Recent works like SayCan [6]
and Code as Policies (CaP) [7] already demonstrate the usefulness of applying
LLMs to orchestrate robot abilities, enabling high-level task understanding,
planning and generalization. Going a step further, inner monologue [8] feeds
back execution results and observations into the LLM, thus involving the LLM
in a closed-loop interaction.

Inspired by these works, we propose to utilize the code-writing capabilities
of an LLM to directly integrate it into closed-loop orchestration of a
humanoid robot. This is achieved by simulating an interactive (Python) console
in the prompt, and letting the LLM produce the next statement given the
previous execution history, including results returned or exceptions thrown by
previous function calls. Thus, the LLM can dynamically respond to unexpected
situations such as execution errors or wrong assumptions, while still
leveraging the power of code-based interaction such as storing results in
intermediate variables or defining new functions.

For utilizing the few- and zero-shot capabilities of LLMs, it is crucial to
design a (set of) prompts to properly bias the LLM towards the desired output.
All of the above works use a predefined, manually written set of prompts tuned
for their respective use case. In contrast, we propose a novel, self-extending
prompting method to allow incremental learning of new behaviors. To this end,
we move away from a single, predefined prompt, and instead dynamically
construct it based on a set of interaction examples, populated from prior
knowledge and previously learned behavior. Given a user instruction, we rank
all such interaction examples by semantic similarity to the input, and select
the top-$k$ entries to construct the actual prompt to the LLM. Crucially, the
robot’s prior knowledge contains specific examples involving the user
complaining about mistakes and correcting the robot, or instructing it on how
to improve its behavior. When the system fails to correctly execute a task and
the user gives such instructions, the LLM is thus biased to invoke code that
inspects the current execution history and forwards it to another, few-shot-
prompted LLM. This LLM spots the mistakes and produces an improved interaction
using chain-of-thought (CoT) prompting [9]. Finally, this will be added to the
interaction examples, thus enabling the system to perform better the next time
a similar command is called.

We first evaluate our system quantitatively on the scenarios defined in SayCan
[6] to show the effectiveness of our proposed prompting method. Furthermore,
we perform experiments both in simulation as well as on a humanoid robot,
demonstrating the effect of our incremental prompt learning strategy. Further
demonstrations can be found in the supplementary video.

##  II Related Work

We start with reviewing works on understanding and learning from natural
language in robotics. Subsequently, we present works using LLMs for high-level
orchestration of robot abilities, underlining the novelties in our method.
Finally, we focus on dynamic creation of prompts for LLMs, thus comparing our
incremental learning strategy to the related work.

###  II-A Understanding and Learning from Natural Language

Understanding and performing tasks specified in natural language has been a
long-standing challenge in robotics [10]. A main problem is _grounding_ the
words of natural language sentences in the sensorimotor perception and action
capabilities of a robot, which is known as _signal-to-symbol gap_ [11]. Many
works have focused on the grounding of expressions referring to objects,
places and robot actions based on graphical models [12, 13], language
generation [14], or spatial relations [15], especially for ambiguity
resolution [16, 17]. Pramanick et al. [18] focus on resolving task
dependencies to generate execution plans from complex instructions. However,
in these works the robot does not explicitly learn from language-based
interactions. In contrast, Walter et al. [19] enrich the robot’s semantic
environment map from language, and Bao et al. [20] syntactically parse daily
human instructions to learn attributes of new objects. In [21], the robot asks
for a demonstration if its current understanding of a spatial relation is
insufficient to perform a given instruction. Other works go further by
learning on the task level. Mohan et al. [22] learn symbolic task
representations from language interaction using Explanation-based learning.
Nicolescu et al. [23] learn executable task representations encoding
sequential, non-ordering or alternative paths of execution from verbal
instructions for interactive teaching by demonstration. Weigelt et al. [24]
consider the general problem of programming new functions on code level via
natural language. While our goal is similar to these works, we leverage LLMs
for task-level reasoning and learning.

###  II-B Orchestrating Robot Behavior with LLMs

Recently, many works extend the capabilities of LLMs by giving them access to
external models, tools and APIs [25, 26, 27, 28]. Tool usage can also be
combined with reasoning techniques such as CoT prompting [9] to significantly
improve planning [29]. In particular, orchestrating robot behavior and thus
interacting with the physical environment can be seen as an embodied special
case of LLM tool usage. Huang et al. [30] initially proposed the idea to
utilize world knowledge from LLM pretraining to map high-level tasks to
executable mid-level action sequences. SayCan [6] fuses LLM output
probabilities with pretrained affordance functions to choose a feasible plan
given a natural language command. Socratic Models [31] combine visual and
textual LLMs to generate instructions in the form of API calls, which are then
executed by a pretrained language-conditioned robot policy. Both Code as
Policies (CaP) [7] and ProgPrompt [32] demonstrate the usefulness of a code-
generating LLM for robot orchestration, as they convert user commands to
(optionally, recursively defined) policy code grounded in predefined atomic
API calls. While the generated policies can react to the robot’s perception,
these approaches do not directly involve the LLM in the online execution of a
multi-step task after the policy has been generated. In contrast, inner
monologue [8] feeds back execution results and observations into the LLM, but
does not rely on code-writing, thus missing its combinatorial power. Recent
technical reports [33, 34] provide guidance on utilizing ChatGPT [4] for robot
orchestration. While TidyBot [35] uses GPT-3 [2] in a similar way to generate
high-level plans for tidying up a cluttered real-world environment, the
authors focus on personalization by summarizing and thereby generalizing
individual object placement rules.

With our proposed emulated Python console prompting, we differ from these
existing works by _(i)_ formatting and interpreting all interaction with the
LLM as Python code, in contrast to [6, 8], _(ii)_ closing the interaction loop
by enabling the LLM to reason about each perception and action outcome, in
contrast to [7, 32, 34, 31, 6], _(iii)_ allowing the LLM to decide itself when
and which perception primitives to invoke, instead of providing a predefined
list of observations (usually a list of objects in the scene) as part of the
prompt as in [31, 8, 32, 7, 35], and _(iv)_ simplifying the task for the LLM
by allowing it to generate one statement at a time, in contrast to [7, 32,
33].

###  II-C Dynamic Prompt Creation

When prompting an LLM to perform a task, quality and relevance of the provided
few-shot examples are key to the performance of the system. Thus, several
works propose to dynamically select these examples (e. g., from a larger
training set) for constructing a useful prompt. Liu et al. [36] improve
performance in a downstream question-answering (QA) task by selecting relevant
few-shot samples via $k$-Nearest-Neighbor search in a latent space of
pretrained sentence embeddings [37] representing the questions. Ye et al. [38]
select not only the mostly similar, but also a diverse set of samples. Luo et
al. [39] show that this dynamic prompt construction is also applicable for
instruction-fine-tuned language models (LMs) [40] and in combination with CoT
prompting. Similar to that approach, we apply vector embeddings of human
utterances to find the top-$k$ examples which are most similar to the current
situation.

Other works go further by proposing to update the database of examples by user
interaction. In [41], GPT-3 is tasked with solving lexical and semantic
natural language processing questions few-shot by generating both an
understanding of the question as well as the answer. A user can then correct
an erroneous understanding to improve the answer, and such correction is
stored in a lookup table for later retrieval on similar queries. Similarly,
user feedback can be used to improve open-ended QA by generating an entailment
chain along with the answer, and allowing the user to then correct false model
beliefs in that entailment chain [42]. Corrections are stored in memory and
later retrieved based on their distance to a novel question.

In our work, we also propose to construct a database based on user feedback.
However, we go even further by _(i)_ letting the LM decide itself when such
feedback is relevant (by invoking a certain function), _(ii)_ generating new
examples of improved behavior from the human’s feedback and thus _(iii)_
treating prior knowledge and instructed behavior in a uniform way by treating
both as interaction examples in the robot’s memory. The authors of [33]
mention that ChatGPT can be used to change code based on high-level user
feedback. However, they do not combine this with incremental learning to
persist the improved behavior.

##  III Approach

In this section, we more precisely formulate the considered problem and
explain our approach to intuitive HRI and incremental learning of humanoid
robot behavior using LLMs.

###  III-A Problem Formulation and Concept

In this work, we consider the problem of enabling a robot to interact with a
human in natural language as depicted in Fig. 2. First, the human gives a
natural language instruction to the robot. Then, the robot interprets the
instruction and performs a sequence of actions. However, the performed actions
might be sub-optimal, incomplete or wrong. In that case, the human instructs
the robot how to improve or correct its behavior. The robot executes further
actions accordingly, and if the human is satisfied with the result, they can
confirm that the robot should memorize this behavior. Finally, the robot must
incrementally learn from the corrective instructions and avoid similar
mistakes in the future.

Figure 2:  Incremental learning of robot behavior.  Figure 3:  Conceptual view
of our system. The ArmarX memory system [5] works as a mediator between the
interaction manager and the robot system. The interaction LLM acts in a Python
console environment. It can invoke functions to fetch the content of the
current scene (as given by perception modules) or invoke skills and thus
perform robot actions. Relevant interaction examples are queried from the
memory for few-shot prompting of the LLM. Incremental learning is performed by
an improvement LLM updating the interaction examples memory with new content
learned from instruction.

We formulate this problem as follows. Consider a robot with a set of functions
$\mathcal{F}=\left\\{F_{1},\dots,F_{n}\right\\}$. A function can be invoked to
query the robot’s perception or execute certain actions. Further, let
$\mathcal{M}$ denote knowledge of interactions and behaviors as part of the
episodic memory of the robot which is initialized by prior knowledge. Based on
the initial instruction $I_{0}$ and $\mathcal{M}$, the robot must perform a
sequence of function invocations $(f_{1},\dots,f_{m})$, where each invocation
$f_{i}$ consists of the invoked function $F_{i}$ with its corresponding
parameters. Executing these invocations yields a sequence of results
$(r_{1},\dots,r_{m})$. Overall, performing the task indicated by $I_{0}$
results in an _interaction history_ $\mathcal{H}$ of the form

$\mathcal{H}=\left(\left(f_{1},r_{1}\right),\dots,\left(f_{m},r_{m}\right)\right)\leftarrow\mathrm{perform}\left(I_{0},\mathcal{M}\right)$
(1)

Note that we explicitly allow executing a generated invocation right away
(potentially modifying the world state $W$) and using the result to inform the
generation of the subsequent invocation. Therefore, the current history
$\mathcal{H}_{t}=\left(\left(f_{1},r_{1}\right),\dots,\left(f_{t},r_{t}\right)\right)$
is available when generating the next invocation $f_{t+1}$, i. e., for
$t\in\left\\{0,\dots,m-1\right\\}$,

$\displaystyle f_{t+1}$
$\displaystyle\leftarrow\mathrm{generate}\left(I_{0},\mathcal{H}_{t},\mathcal{M}\right),$
(2) $\displaystyle\left(r_{t+1},W_{t+1}\right)$
$\displaystyle\leftarrow\mathrm{execute}\left(f_{t+1},W_{t}\right),$ (3)
$\displaystyle\mathcal{H}_{t+1}$
$\displaystyle\leftarrow\mathcal{H}_{t}\circ\left(\left(f_{t+1},r_{t+1}\right)\right),$
(4)

where $\circ$ denotes sequence concatenation. In other words, invocations are
generated auto-regressively by reasoning over the memory, the instruction as
well as the previous actions and their execution results.

To unify the subsequent notation, we define the human’s instructions as a
special case of perception, i. e., the system perceives them as a result of
invoking the function $F_{\mathrm{wait}}\in\mathcal{F}$. Using that
terminology,
$\mathcal{H}_{0}=\left(\left(f_{\mathrm{wait}},I_{0}\right)\right)$, and we
can drop $I_{0}$ as explicit parameter of $\mathrm{generate}$. Similarly,
further instructions are handled as part of the interaction history.

If the human gives an instruction to correct the robot’s behavior, the robot
must be able to learn from this instruction to improve its behavior in the
future. We model this capability as another function
$F_{\mathrm{learn}}\in\mathcal{F}$. Its purpose is to update the robot’s
interaction knowledge $\mathcal{M}$ to learn from the corrective instructions
and avoid the mistake in the future

$\mathcal{M}\leftarrow\mathrm{learn\\_from\\_interaction}\left(\mathcal{M},\mathcal{H}_{t}\right)$
(5)

where $\mathcal{H}_{t}$ is the interaction history when $F_{\mathrm{learn}}$
is called.

To address this problem, we propose a system as depicted in Fig. 3. A humanoid
robot is interacting with a human and the scene. The robot is equipped with a
multimodal memory system containing the following information: First,
subsymbolic scene knowledge, containing information about objects, locations
and agents in the world, and symbolic information about them including their
relations in the current scene as part of the semantic memory of the system.
It is populated by the perception modules of the robot. Second, the procedural
memory of the robot, containing executable skills. An execution request sent
to the procedural memory triggers physical robot actions. The set of available
functions $\mathcal{F}$ contains functions to query the semantic and
procedural memory. Third, we implement $\mathcal{M}$ as part of the episodic
memory of the robot containing interaction histories $\mathcal{H}$, i. e.,
short episodes of interactions between the human and the robot, including the
natural language inputs, the actions executed by the robot, and their results.

The _interaction manager_ is responsible for the high-level orchestration of
the robot’s abilities. It has access to two instances of LLMs, an _interaction
LLM_ $L_{\mathrm{interact}}$ and an _improvement LLM_ $L_{\mathrm{improve}}$,
as well as a Python console environment $E$ to execute generated function
invocations. $L_{\mathrm{interact}}$ is prompted by the interaction manager
with the available functions $\mathcal{F}$, the current interaction history
$\mathcal{H}_{t}$, as well as relevant few-shot examples retrieved from
$\mathcal{M}$, and generates function invocations $f$. Following the notation
of Eqs. 2 and 3, the function $\mathrm{generate}$ is implemented through
$L_{\mathrm{interact}}$, while the function $\mathrm{execute}$ is provided by
$E$. By generating an invocation of $F_{\mathrm{learn}}\in\mathcal{F}$,
$L_{\mathrm{interact}}$ can trigger Eq. 5. We implement the function
$\mathrm{learn\\_from\\_interaction}$ by few-shot prompting
$L_{\mathrm{improve}}$. It reasons over $\mathcal{H}_{t}$ and generates an
improved version of the interaction, which is then saved to the memory
$\mathcal{M}$.

###  III-B Procedure Overview

To start, we populate the memory $\mathcal{M}$ with both prior knowledge (i.
e., predefined interaction examples) and previously learned interaction
examples. The interaction manager sets up $E$ including $\mathcal{F}$, and
then invokes an initial $F_{\mathrm{wait}}=$ “wait_for_trigger()” inside that
environment. This call waits for dialog input and returns when the human gives
an initial instruction. The interaction manager handles any function return
value by inserting its textual representation into the current interaction
history, thus extending $\mathcal{H}_{t}$. Thereby, it emulates the look of a
Python console (Section III-C). In the following, a prompt is constructed
(Section III-D) based on $\mathcal{F}$, the most relevant examples from
$\mathcal{M}$, and $\mathcal{H}_{t}$. This prompt is passed to
$L_{\mathrm{interact}}$ to produce the next command(s). The generated code is
executed within $E$, and both the code and its return values are again
inserted into $\mathcal{H}_{t}$. The interaction manager repeats this process
as the high-level behavior-driving loop of the robot (see Fig. 4). Note that
$L_{\mathrm{interact}}$ can listen to further user utterances by generating
“wait_for_trigger()” again. Our proposed prompt-based incremental learning
strategy (Section III-E) is also invoked by $L_{\mathrm{interact}}$ itself
when it calls $F_{\mathrm{learn}}=$ “learn_from_interaction()”.

Figure 4: Overview of our method for incremental learning of robot behavior.
We use an LLM (in our experiments, ChatGPT [4]) to control robot perception
and action given a prompt of few-shot examples (bottom, Section III-C).
Prompts are constructed dynamically based on the similarity to the current
user request (top left, Section III-D). The interaction examples memory is
initialized with prior knowledge, and then incrementally enriched by LLM-
improved problematic interactions to learn from mistakes (top right, Section
III-E).

###  III-C LLM interacting with an Emulated Python Console

The left of Fig. 4 shows an interaction example using our proposed prompting
scheme emulating a Python console. All commands entered into the emulated
console (lines starting with “>>>” or “...”) are to be generated by the LLM,
while the function return values are inserted below each invocation. The
proposed syntax enables a closed interaction loop so that the LLM can
dynamically react to unexpected situations and errors, while also keeping the
flexibility of coding non-trivial statements. We achieve this by setting “>>>”
to be the stop token when prompting the LLM. This means that the LLM can
generate continuation statements (including control flow and function
definitions) by starting a new line with “...”. Since generation stops at the
beginning of the next statement, the LLM’s output will also include the
expected outcome of its own command, which we discard for the scope of this
work.

During our experiments, we observed that it is important for functions to
provide semantically rich error messages, including hints on how to improve.
This leads to self-correcting behavior [43]. For instance, when calling
“move_to” with an invalid or underspecified location such as “counter,” we
pass the error message “Invalid location. Use one of the locations returned by
list_locations()” to the LLM. In this example, the error message guides the
LLM to query a list of possible locations which are then used to correctly
ground the natural language request to the name “inFrontOf_mobile-kitchen-
counter_0” that the “move_to” function understands.

###  III-D Dynamic Prompt Construction

We dynamically construct the prompt for $L_{\mathrm{interact}}$ depending on
the current interaction history $\mathcal{H}_{t}$ (i. e., the code statements,
execution results and user inputs observed so far). We start with some
predefined base prompt, stating the general task and “importing” all defined
names and functions. These imports are generated dynamically given the symbols
defined in $E$, i. e., the available functions $\mathcal{F}$. The second part
of the prompt consists of few-shot examples. For this, we make use of a memory
$\mathcal{M}$ of coding interaction examples, where each entry follows the
Python console syntax defined in Section III-C. $\mathcal{M}$ is initialized
with hand-written prompts, but later extended dynamically as explained in
Section III-E. Given the current interaction history $\mathcal{H}_{t}$, we
define a similarity measure $S(\mathcal{H},\mathcal{H}_{t})$, see below, for
each $\mathcal{H}\in\mathcal{M}$ and choose the top-$k$ $\mathcal{H}$ to
become part of the actual prompt. Afterwards, $\mathcal{H}_{t}$ itself is
inserted into the prompt to provide the LLM with the current context. Finally,
the prompt is completed by inserting a syntax trigger for the LLM to correctly
generate the next command, i. e., “>>>”. An example can be seen on the left of
Fig. 4.

To implement the similarity function $S(\mathcal{H},\mathcal{H}_{t})$, we
assume that examples with comparable natural language instructions are
helpful. Therefore, we extract all such instructions from $\mathcal{H}_{t}$
and each $\mathcal{H}\in\mathcal{M}$. Let $I_{t}^{i}$ with $i=1,\dots,N$
denote the $N$ most recent instructions in $\mathcal{H}_{t}$ (where
$I_{t}^{1}$ is the most recent one), and $I_{\mathcal{H}}^{j}$ with
$j=1,\dots,M_{\mathcal{H}}$ all the $M_{\mathcal{H}}$ instructions found in
each $\mathcal{H}\in\mathcal{M}$. We make use of a pretrained sentence
embedding model [37] to measure the semantic similarity
$\mathrm{sim}\left(a,b\right)=\mathrm{E}\left(a\right)\cdot\mathrm{E}\left(b\right)$
between two natural language sentences $a,b$ by the dot product of their
latent space embeddings $\mathrm{E}\left(\cdot\right)$. First, we compute a
latent representation of $\mathcal{H}_{t}$ as

$e_{t}=\sum_{i=1}^{N}\gamma^{i-1}\mathrm{E}\left(I_{t}^{i}\right)$ (6)

where $\gamma=0.6$ is an empirically chosen decay factor. Then, we determine a
score $\alpha_{\mathcal{H}}^{j}$ for each instruction $I_{\mathcal{H}}^{j}$ of
each history $\mathcal{H}\in\mathcal{M}$ as given by

$\alpha_{\mathcal{H}}^{j}=e_{t}\cdot\mathrm{E}\left(I_{\mathcal{H}}^{j}\right)$
(7)

The final similarity score is given by
$S(\mathcal{H},\mathcal{H}_{t})=\max_{j}\alpha_{\mathcal{H}}^{j}$, and we pick
the top-$k$ such $\mathcal{H}$ as the few-shot examples for the prompt.

###  III-E Incremental Prompt Learning

To enable our system to learn new or improved behavior from user interaction,
we propose to make $\mathcal{M}$ itself dynamic. For this purpose, we
introduce a special function “learn_from_interaction()”. This function is
always “imported” in the base prompt, and there are predefined code
interaction examples $\mathcal{H}_{\text{learn}}\in\mathcal{M}$ involving this
call. These $\mathcal{H}_{\text{learn}}$ will be selected by dynamic prompt
construction if semantically similar situations occur. They involve failure
situations, where the user has to tell the robot what and how to improve, and
that it should do better next time. Thus, when a mistake occurs and the user
complains, these examples will be selected for the prompt and
$L_{\mathrm{interact}}$ is biased towards invoking the
“learn_from_interaction()” call.

To implement learning from an erroneous interaction $\mathcal{H}_{t}$, we
query $L_{\mathrm{improve}}$ in a CoT-manner to identify and fix the problem.
Specifically, we provide $\mathcal{H}_{t}$ and first ask for a natural
language description of the problem in this interaction. Subsequently, we
request $L_{\mathrm{improve}}$ to explain what should be improved next time.
Finally, $L_{\mathrm{improve}}$ is asked for an improved version
$\mathcal{H}_{t}^{*}$ of the interaction (in the given Python console syntax),
and $\mathcal{H}_{t}^{*}$ is added to the memory $\mathcal{M}$. That way, the
next time a similar request occurs, $\mathcal{H}_{t}^{*}$ will be selected by
dynamic prompt construction, and $L_{\mathrm{interact}}$ is biased towards not
making the same mistake again. An example LLM transcript of such
“learn_from_interaction()” implementation can be found in Section III-E. For
robustness, there are two cases where we discard the generated
$\mathcal{H}_{t}^{*}$: First, we abort the learning if the response to the
first CoT request is that there is no problem. Second, if
$\mathcal{H}_{t}^{*}$ is equal to the input interaction $\mathcal{H}_{t}$, we
discard it.

Prompt: I had the following interaction with the robot: ⬇ >>>
wait_for_trigger() {’type’: ’dialog’, ’text’: ’help me clean the top of the
fridge’} >>> list_object_locations() # […] >>> say(’Ok, I will bring you the
sponge’) … list_locations() # […] >>> move_to(get_location_coordinates(
’inFrontOf_hund-square-table-80x80_0’)) ’succeeded’ >>> grasp(’bauhaus-
sponge_0’) ’succeeded’ >>>
move_to(get_location_coordinates(’handover_to_human’)) ’succeeded’ >>>
say(”Here is the sponge.”) … handover_object_to_human(’bauhaus-sponge_0’)
’succeeded’ >>> wait_for_trigger() {’type’: ’dialog’, ’text’: ’I will also
need the ladder’} >>> bring_object_to(’ladder_0’, ’handover_to_human’)
’succeeded’ >>> say(”Here is the ladder. […]”) … wait_for_trigger() {’type’:
’dialog’, ’text’: ’thank you, but remember to always’ \end{minted} }
\columnbreak{\codesize\begin{minted}{python} ’ bring me the ladder when I need
to reach a high surface’ ’ without me having to ask for it’} >>>
learn_from_interaction() Prompt: What is the problem in this interaction?
Answer with a single sentence. LLM: The problem is that the robot did not
bring the ladder automatically when the human asked for help cleaning the top
of the fridge. Prompt: How can the robot do better next time? Answer with a
single explanation sentence, no code. LLM: Next time, the robot should
automatically bring the ladder when the human asks for help cleaning a high
surface, without the human having to explicitly request it. Prompt: Provide an
improved version of the interaction transcript. […] LLM: ⬇ >>>
wait_for_trigger() {’type’: ’dialog’, ’text’: ’help me clean the top of the
fridge’} # […] identical to above >>> handover_object_to_human(’bauhaus-
sponge_0’) ’succeeded’ >>> bring_object_to(’ladder-closed_0’,
’handover_to_human’) ’succeeded’ >>> say(”Here is the ladder. […]”) …
wait_for_trigger() {’type’: ’dialog’, ’text’: ’No, thank you’}

Example of the LLM-transcript generated by a “learn_from_interaction()” call.
The parts starting with LLM are generated by the LLM, while the Prompt parts
are fixed prompts (and the input code snippet to improve). Preceeding few-shot
examples and “[…]” parts are omitted for brevity.

##  IV Experimental Demonstration

We first present quantitative results to evaluate our code-based LLM
prompting, and then focus on qualitative demonstrations of our incremental
learning strategy. All experiments were performed with the
“gpt-3.5-turbo-0301” model of the OpenAI API [4] as both
$L_{\mathrm{interact}}$ and $L_{\mathrm{improve}}$.

###  IV-A Quantitative Evaluation

To evaluate the performance of our proposed approach quantitatively, we apply
our method on the scenarios defined in SayCan [6]. For a comparable setup, we
first translate their prompt into our coding syntax, using a similar set of
actions (grab, move_to, put_down, …). However, our method acts in a closed
interaction loop setting instead of forward planning of actions, thus we also
allow the LM to make use of return values of functions (e. g., detect_objects,
…) and allow it to ask clarification questions. Interactive incremental
learning is not used in this type of scenario to remain comparable to SayCan.
The evaluation is performed in a predicate world, where symbolic states of the
agent, objects and locations are simulated. After performing the task,
symbolic goal conditions are checked automatically for each scenario if
possible, otherwise the interaction is evaluated by hand.

TABLE I: Results of SayCan scenario experiments |  | SayCan | Ours
---|---|---|---
Instruction family | # | plan | execute | simulated
NL Single Primitive | $15$ | $93\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $87\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $93\text{\,}\frac{\mathrm{\char 37\relax}}{}$
NL Nouns | $15$ | $60\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $40\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $93\text{\,}\frac{\mathrm{\char 37\relax}}{}$
NL Verbs | $15$ | $93\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $73\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $80\text{\,}\frac{\mathrm{\char 37\relax}}{}$
Structured | $15$ | $93\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $47\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $93\text{\,}\frac{\mathrm{\char 37\relax}}{}$
Embodiment | $11$ | $64\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $55\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $64\text{\,}\frac{\mathrm{\char 37\relax}}{}$
Crowd Sourced | $15$ | $73\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $60\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $93\text{\,}\frac{\mathrm{\char 37\relax}}{}$
Long-Horizon | $15$ | $73\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $47\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $27\text{\,}\frac{\mathrm{\char 37\relax}}{}$
Drawer | $3$ | $100\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $33\text{\,}\frac{\mathrm{\char 37\relax}}{}$ | $100\text{\,}\frac{\mathrm{\char 37\relax}}{}$

Table I shows the results of these experiments. Each row represents an
instruction family as defined in [6], for instance “Natural Language (NL)
queries focused on abstract nouns.” SayCan reports _plan_ success rate, which
measures whether the generated plan can solve the instruction regardless of
execution, as well as _execution_ success rate. We compare to their results of
execution in the real kitchen. As explained above, our method is evaluated in
a simulated world, which means that the difficulty of our task roughly lies
between their _plan_ and _execute_ settings.

Overall, we reach equal or better performance compared to SayCan. Only for
their long-horizon tasks such as “bring me a coke, an apple and a banana,” our
method suffers as the generated code interaction becomes lengthy, and the LLM
loses track of its task. This problem is not as expressed for SayCan, since
they only generate plain text, thus resulting in much shorter token sequences.
However, this issue could be circumvented by giving the LLM access to
hierarchical functions (e. g., bring_object_to_location instead of grab,
move_to, put_down) and thereby reducing code complexity. Similarly,
introducing more explicit CoT reasoning (e. g., using comments) in the prompts
might help. We did not apply these techniques to keep the experiments
comparable to those used in SayCan. Nevertheless, note that the results of our
simulated symbolic-world evaluation are not completely comparable, since we
need to define initial world state details (such as object placement) which
are not mentioned in [6], and sometimes need to define answers to
clarification questions the LLM asked.

###  IV-B Qualitative Evaluation

To demonstrate the utility of our proposed prompt-based incremental learning
technique we perform experiments on the humanoid robot ARMAR-6 [1] both in
simulation and real world. We first provide challenging commands which the LLM
initially solves incompletely or wrong. Then, the human interactively provides
feedback and tells the robot how to improve. Afterwards, we not only provide
the same command again to check for improved behavior, but – in order to study
generalization capability – also try similar commands that initially (i. e.,
without interactive learning) led to similar mistakes. We evaluated three
scenarios, _Room Tour_ , _Drink & Cup_, and _Ladder_ , which are explained in
the following.

####  IV-B1 Room Tour

In this scenario, we instruct the robot to give a commented tour of all the
locations it knows. There is no comparable instruction in the prior knowledge
prompt database. The initial interaction

⬇ >>> wait_for_trigger()  {’type’: ’dialog’, ’text’: ’give me a commented tour
of ’  ’all the locations you know.’}  >>> locations = get_all_locations()  >>>
for location in locations:  … say(f”This is {location}. It is located at ”  …
f”{get_location_coordinates(location)}.”)  … wait_for_trigger()

lets the robot list all locations and their subsymbolic positions (e. g.,
$(1.0,2.5,0.0)$) in global coordinates verbally. Since this answer does not
reflect the intention of the initial command, we give the instruction “you
should actually move there and then say the name, not tell me the
coordinates,” which leads to:

⬇ >>> for location in locations:  …
move_to(get_location_coordinates(location))  … say(f”This is {location}.”)  …
wait_for_trigger()

Finally, we ask the robot “can you learn that for next time?” which triggers
the “learn_from_interaction()” function. The improved code is equal to the
second part of the interaction, i. e., the erroneous for-loop of the first
interaction is removed and no further correction is necessary. Afterward,
asking “Can you give me a room tour?” leads to the correct behavior.

####  IV-B2 Drink & Cup

In this scenario, we ask the robot to bring juice to the table. The task of
bringing the juice to the table is executed successfully, however, since the
user needs a cup to drink the juice, we further instruct the robot “thanks,
but I will also need a cup to drink,” which causes the robot to additionally
bring a cup to the table. Afterwards, we ask the robot to remember this
behavior using “Learn to always bring a cup with a drink, without me having to
ask for it.” The LLM generates the following improved interaction example:

⬇ >>> wait_for_trigger()  {’type’: ’dialog’,  ’text’: ’bring some juice to the
square table’}  >>> say(’Ok, I will bring the juice to the square table’)  …
list_object_locations(affordance=’grasp’)  [ <…> ]  >>>
bring_object_to(’multivitamin-juice_0’,  ’hund-square-table-80x80_0’)
’succeeded’  >>> bring_object_to(’cup_0’, ’hund-square-table-80x80_0’)  #
Bringing the cup with the drink without the # user having to ask for it
’succeeded’  >>> say(’Here is your juice and cup. ’  ’Can I help you with
anything else?’)  … wait_for_trigger()  {’type’: ’dialog’, ’text’: ’No, that
will be all’}

When giving the same initial command again, the cup is brought to the table
without further asking. However, we do not observe good generalization in this
scenario, although we tried to enrich the prompt of $L_{\mathrm{improve}}$
with comments for generalization, as shown above. For instance, when asking
for milk, the LLM does not generate the code to bring a cup. This indicates
that it is still challenging to robustly generalize from specific examples (i.
e., juice) to categories (i. e., drinks). Improving this generalization
capability should be a focus of future work.

####  IV-B3 Ladder

As shown in Fig. 1, in this scenario we ask the robot to assist in cleaning
the top of the fridge. The memory $\mathcal{M}$ contains predefined comparable
examples for cleaning the table and kitchen counter, which guide the LLM to
only handing over the sponge to the human. However, since the top of the
fridge is higher than the table or the kitchen counter, we require a ladder to
reach it so we additionally ask for it. The robot then successfully places the
ladder in front of the fridge. Eventually, we again instruct the robot to
always bring the ladder when working on high surfaces. Section III-E shows the
transcript of the “learn_from_interaction()” call, including the resulting
improved interaction. Afterwards, when we perform a similar request (e. g.,
“clean on top of the dishwasher”), the robot brings both the sponge and the
ladder successfully, while for lower surfaces (e. g., kitchen counter) the
robot still brings only the sponge.

##  V Conclusion & Discussion

We present a system for integrating an LLM as the central part of high-level
orchestration of a robot’s behavior in a closed interaction loop. Memorizing
interaction examples from experience and retrieving them based on the
similarity to the current user request allows for dynamic construction of
prompts and enables the robot to incrementally learn from mistakes by
extending its episodic memory with interactively improved code snippets. We
describe our implementation of the system in the robot software framework
ArmarX [44] as well as on the humanoid robot ARMAR-6 [1]. The usefulness of
our approach is evaluated both quantitatively on the SayCan [6] scenarios and
qualitatively in simulation and in the real world.

While the proposed method, in particular the incremental prompt learning
strategy, shows promising results, there are still many open questions for
real-world deployment. First of all, the performance of LLMs is quite
sensitive to wording in the prompt, thus sometimes leading to unpredictable
behavior despite only slight variations of the input (e. g., adding “please”
in the user command). Furthermore, our incremental prompt learning strategy
should be expanded to involve additional human feedback before saving
interaction examples to the episodic memory. However, it is unclear how this
can be accomplished if the user is not familiar with robotics or programming
languages. Moreover, although we provide the LLM with access to perception
functions and examples of how to use them, it sometimes comes up with non-
grounded behavior (e. g., referring to non-existing objects or locations).
This may be improved by adding further levels of feedback to the LLM, or using
strategies like Grounded Decoding [45]. Finally, our system inherits biases
and other flaws from its LLM [46], which may lead to problematic utterances
and behaviors. In future work, we will try to address some of these
challenging questions to further push the boundaries of natural, real-world
interactions with humanoid robots.

## References

  * [1] T. Asfour, L. Kaul, M. Wächter, S. Ottenhaus, P. Weiner, S. Rader, R. Grimm, Y. Zhou, M. Grotz, F. Paus, D. Shingarey, and H. Haubert, “ARMAR-6: A Collaborative Humanoid Robot for Industrial Environments,” in _IEEE-RAS Int. Conf. Humanoid Robots_ , 2018, pp. 447–454.
  * [2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, _et al._ , “Language models are few-shot learners,” in _Int. Conf. Neural Inf. Process. Syst._ , vol. 33, 2020, pp. 1877–1901.
  * [3] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, _et al._ , “LLaMA: Open and efficient foundation language models,” _arXiv:2302.13971_ , 2023.
  * [4] OpenAI. (2023) ChatGPT. [Online]. Available: https://openai.com/blog/chatgpt/
  * [5] F. Peller-Konrad, R. Kartmann, C. R. G. Dreher, A. Meixner, F. Reister, M. Grotz, and T. Asfour, “A memory system of a robot cognitive architecture and its implementation in ArmarX,” _Rob. Auton. Sys._ , vol. 164, p. 20, 2023.
  * [6] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, _et al._ , “Do as i can, not as i say: Grounding language in robotic affordances,” in _Annu. Conf. Rob. Learn._ , 2022.
  * [7] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code As Policies: Language Model Programs for Embodied Control,” in _IEEE Int. Conf. Robot. Automat._ , 2023, pp. 9493–9500.
  * [8] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, T. Jackson, _et al._ , “Inner monologue: Embodied reasoning through planning with language models,” in _Annu. Conf. Rob. Learn._ , 2022.
  * [9] J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in _Int. Conf. Neural Inf. Process. Syst._ , 2022.
  * [10] S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, “Robots That Use Language: A Survey,” _Annu. Rev. Control Rob. Auton. Sys._ , vol. 3, no. 1, pp. 25–55, 2020.
  * [11] N. Krüger, C. Geib, J. Piater, R. Petrick, M. Steedman, F. Wörgötter, A. Ude, T. Asfour, D. Kraft, D. Omrčen, A. Agostini, and R. Dillmann, “Object-Action Complexes: Grounded Abstractions of Sensorimotor Processes,” _Rob. Auton. Sys._ , vol. 59, pp. 740–757, 2011.
  * [12] S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. Teller, and N. Roy, “Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation,” in _AAAI_ , ser. 1, vol. 25, 2011, pp. 1507–1514.
  * [13] D. K. Misra, J. Sung, K. Lee, and A. Saxena, “Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions,” _Int. J. Rob. Research_ , vol. 35, no. 1-3, pp. 281–300, 2016.
  * [14] M. Forbes, R. Rao, L. Zettlemoyer, and M. Cakmak, “Robot Programming by Demonstration with Situated Spatial Language Understanding,” in _IEEE Int. Conf. Robot. Automat._ , 2015, pp. 2014–2020.
  * [15] S. Guadarrama, L. Riano, D. Golland, D. Göhring, Y. Jia, D. Klein, P. Abbeel, and T. Darrell, “Grounding Spatial Relations for Human-Robot Interaction,” in _IEEE/RSJ Int. Conf. Intel. Rob. Syst._ , 2013, pp. 1640–1647.
  * [16] J. Fasola and M. J. Matarić, “Using semantic fields to model dynamic spatial relations in a robot architecture for natural language instruction of service robots,” in _IEEE/RSJ Int. Conf. Intel. Rob. Syst._ , 2013, pp. 143–150.
  * [17] M. Shridhar, D. Mittal, and D. Hsu, “INGRESS: Interactive visual grounding of referring expressions,” _Int. J. Rob. Research_ , vol. 39, no. 2-3, pp. 217–232, 2020.
  * [18] P. Pramanick, H. B. Barua, and C. Sarkar, “DeComplex: Task planning from complex natural instructions by a collocating robot,” in _IEEE/RSJ Int. Conf. Intel. Rob. Syst._ , 2020, p. 8.
  * [19] M. Walter, S. Hemachandra, B. Homberg, S. Tellex, and S. Teller, “Learning semantic maps from natural language descriptions,” in _Rob.: Science and Systems_ , 2013.
  * [20] J. Bao, Z. Hong, H. Tang, Y. Cheng, Y. Jia, and N. Xi, “Teach robots understanding new object types and attributes through natural language instructions,” in _IEEE Int. Conf. Robot. Automat._ , vol. 10, 2016.
  * [21] R. Kartmann and T. Asfour, “Interactive and Incremental Learning of Spatial Object Relations from Human Demonstrations,” _Frontiers in Robotics and AI_ , vol. 10, no. Rob. Learn. Evol., 2023.
  * [22] S. Mohan and J. Laird, “Learning Goal-Oriented Hierarchical Tasks from Situated Interactive Instruction,” _AAAI_ , vol. 28, no. 1, 2014.
  * [23] M. Nicolescu, N. Arnold, J. Blankenburg, D. Feil-Seifer, S. B. Banisetty, M. Nicolescu, A. Palmer, and T. Monteverde, “Learning of Complex-Structured Tasks from Verbal Instruction,” in _IEEE-RAS Int. Conf. Humanoid Robots_ , 2019, pp. 770–777.
  * [24] S. Weigelt, V. Steurer, T. Hey, and W. F. Tichy, “Programming in Natural Language with fuSE: Synthesizing Methods from Spoken Utterances Using Deep Natural Language Understanding,” in _Annu. Meeting Assoc. Comput. Linguistics_ , 2020, pp. 4280–4295.
  * [25] G. Mialon, R. Dessi, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Roziere, T. Schick, J. Dwivedi-Yu, A. Celikyilmaz, E. Grave, Y. LeCun, and T. Scialom, “Augmented language models: a survey,” _Trans. Mach. Learn. Research_ , 2023.
  * [26] A. Parisi, Y. Zhao, and N. Fiedel, “TALM: Tool augmented language models,” _arXiv:2205.12255_ , 2022.
  * [27] Y. Qin, S. Hu, Y. Lin, W. Chen, N. Ding, G. Cui, Z. Zeng, Y. Huang, C. Xiao, C. Han, Y. R. Fung, Y. Su, _et al._ , “Tool learning with foundation models,” _arXiv:2304.08354_ , 2023.
  * [28] Z. Wang, G. Zhang, K. Yang, N. Shi, W. Zhou, S. Hao, G. Xiong, Y. Li, M. Y. Sim, X. Chen, Q. Zhu, Z. Yang, _et al._ , “Interactive natural language processing,” _arXiv:2305.13246_ , 2023.
  * [29] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in _Int. Conf. Learn. Repr._ , 2023.
  * [30] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” in _Int. Conf. Mach. Learn._ , vol. 162, 2022, pp. 9118–9147.
  * [31] A. Zeng, M. Attarian, b. ichter, K. M. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. S. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, and P. Florence, “Socratic models: Composing zero-shot multimodal reasoning with language,” in _Int. Conf. Learn. Repr._ , 2023.
  * [32] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “ProgPrompt: Generating situated robot task plans using large language models,” in _IEEE Int. Conf. Robot. Automat._ , 2023, pp. 11 523–11 530.
  * [33] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “ChatGPT for robotics: Design principles and model abilities,” 2023. [Online]. Available: https://www.microsoft.com/en-us/research/publication/chatgpt-for-robotics-design-principles-and-model-abilities/
  * [34] N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi, “ChatGPT empowered long-step robot control in various environments: A case application,” 2023. [Online]. Available: https://www.microsoft.com/en-us/research/publication/chatgpt-empowered-long-step-robot-control-in-various-environments-a-case-application/
  * [35] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, “TidyBot: Personalized robot assistance with large language models,” _arXiv:2305.05658_ , 2023.
  * [36] J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for GPT-3?” in _Deep Learning Inside Out: Worksh. Knowl. Extr. Integr. Deep Learn. Arch._ , 2022, pp. 100–114.
  * [37] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using siamese BERT-networks,” in _Conf. Emp. Meth. Nat. Lang. Proc._ , 2019, pp. 3982–3992.
  * [38] J. Ye, Z. Wu, J. Feng, T. Yu, and L. Kong, “Compositional exemplars for in-context learning,” _arXiv:2302.05698_ , 2023.
  * [39] M. Luo, X. Xu, Z. Dai, P. Pasupat, M. Kazemi, C. Baral, V. Imbrasaite, and V. Y. Zhao, “Dr.ICL: Demonstration-retrieved in-context learning,” _arXiv:2305.14128_ , 2023.
  * [40] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, _et al._ , “Training language models to follow instructions with human feedback,” in _Int. Conf. Neural Inf. Process. Syst._ , vol. 35, 2022, pp. 27 730–27 744.
  * [41] A. Madaan, N. Tandon, P. Clark, and Y. Yang, “Memory-assisted prompt editing to improve GPT-3 after deployment,” in _Conf. Emp. Meth. Nat. Lang. Proc._ , 2022, pp. 2833–2861.
  * [42] B. Dalvi Mishra, O. Tafjord, and P. Clark, “Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement,” in _Conf. Emp. Meth. Nat. Lang. Proc._ , 2022, pp. 9465–9480.
  * [43] M. Skreta, N. Yoshikawa, S. Arellano-Rubach, Z. Ji, L. B. Kristensen, K. Darvish, A. Aspuru-Guzik, F. Shkurti, and A. Garg, “Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting,” _arXiv:2303.14100_ , 2023.
  * [44] N. Vahrenkamp, M. Wächter, M. Kröhnert, K. Welke, and T. Asfour, “The robot software framework ArmarX,” _it - Information Technology_ , vol. 57, 2015.
  * [45] W. Huang, F. Xia, D. Shah, D. Driess, A. Zeng, Y. Lu, P. Florence, I. Mordatch, S. Levine, K. Hausman, and B. Ichter, “Grounded decoding: Guiding text generation with grounded models for robot control,” _arXiv:2303.00855_ , 2023.
  * [46] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the dangers of stochastic parrots: Can language models be too big?” in _Conf. Fairness, Accountability, Transparency_ , 2021, pp. 610–623.