# PeV Sterile Masses from D-Brane Instantons — Motivating Stringy Neutrino
Options?

Jim Talbert  Niels Bohr International Academy, Niels Bohr Institute,
University of Copenhagen, Blegdamsvej 17, DK-2100 Copenhagen, Denmark

###### Abstract

We show that when sterile Majorana neutrino masses are generated non-
perturbatively through instantonic interactions in certain classes of string
compactifications (e.g. Type IIA orientifold models with intersecting
D-branes), then the eigenvalue spectrum for the tree-level mass term $M_{p}$
can be within the preferred range of the Neutrino Option resolution to the
electroweak (EW) hierarchy and neutrino mass problems of the Standard Model,
i.e. $M_{p}\sim$ PeV. This mechanism holds without tuning for a broad range of
string scales, thereby motivating a novel class of string-completed Neutrino
Options spanning the light neutrino mass, EW, and deep ultraviolet scales.

##  I Introduction and Motivation

The Neutrino Option Brivio:2017dfq ; Brivio:2018rzm  is a paradigm for
simultaneously resolving the electroweak (EW) hierarchy and neutrino mass
problems of the Standard Model (SM) by assuming that the dominant threshold
corrections to the mass of the Higgs scalar, and therefore also the generation
of the EW scale, come from integrating out heavy Majorana masses $M_{p}$ in
the effective field theory (EFT) defined by the seesaw mechanism for light
neutrino mass generation Minkowski:1977sc ; GellMann:1980vs ; Mohapatra:1979ia
; Yanagida:1980xy . Indeed, it is inevitable that when a Type-I seesaw
Lagrangian of the form

$\mathcal{L}_{N}=\frac{1}{2}\overline{N}_{p}\left(i\not{\partial}-M_{p}\right)N_{p}-\left[\overline{l^{\beta}_{L}}\tilde{H}\omega_{\beta}^{p,\dagger}N_{p}+\overline{N}_{p}\omega_{\beta}^{p}\tilde{H}^{\dagger}l^{\beta}_{L}\right]\,,$
(1)

with $N_{p}=N_{p}^{c}$ the mass-eigenstate sterile Majorana state defined by
Bilenky:1980cx ; Broncano:2002rw

$N_{p}=e^{i\theta_{p}/2}N_{R,p}+e^{-i\theta_{p}/2}(N_{R,p})^{c}\,,$ (2)

is matched to the SMEFT after integrating out heavy $N_{p}$, a threshold
correction to the Higgs boson mass of the form

$-\frac{m_{0}^{2}}{2}\,H^{\dagger}H\longrightarrow\left(-\frac{m_{0}^{2}}{2}-\frac{M_{p}^{2}|\omega_{p}|^{2}}{16\pi^{2}}\right)H^{\dagger}H$
(3)

necessarily appears at one-loop order — see Figure 1 for Feynman diagrams
relevant to this matching, and Elgaard-Clausen:2017xkq  for a complete tree-
level analysis up to dimension seven. Here $m_{0}$ is the bare Higgs mass,
$\theta_{p}$ is an arbitrary phase, and the $p,\beta\in\\{1,2,3\\}$ indices in
(1)-(3) are respectively Majorana and SM lepton flavour indices. The Higgs
threshold correction in (3) of course occurs in addition to the seesaw
generation of the active light neutrino masses $m_{\nu}$, such that

$m_{\nu}\sim\frac{\omega_{p}^{2}\bar{v}_{T}^{2}}{M_{p}},\,\,\,\,\,\,m_{h}\sim\frac{\omega_{p}M_{p}}{4\pi},\,\,\,\,\,\,\bar{v}_{T}\sim\frac{\omega_{p}M_{p}}{4\sqrt{2}\pi\sqrt{\lambda}},$
(4)

where $\sqrt{2H^{\dagger}H}\equiv\bar{v}_{T}$ defines the EW scale and
$\lambda$ is the Higgs self coupling. As noted by Vissani Vissani:1997ys , the
correction to $m_{h}$ can represent a manifestation of the hierarchy problem
in the context of the seesaw model, if extreme fine-tuning is required to
realize the experimental value of $m_{h}\sim 125$ GeV. However, in the
Neutrino Option (with degenerate or only mildly hierarchical $M_{p}$) one
instead recognizes that with the parameter space Brivio:2017dfq ;
Brivio:2018rzm ,111This estimate was made in the simplifying limit of
$M_{1}=0$, which yields a successful seesaw, but can also be generalized to a
complete three-eigenvalue analysis, with the expectation that (assuming
minimal hierarchies in $\omega_{\beta}^{p}$) $M_{1}\lesssim M_{2,3}$ as well,
in order to avoid unnatural threshold corrections to $m_{h}$. The bounds are
also largely stable against variations of other input parameters, e.g. the top
quark mass and neutrino mixing elements.

$\displaystyle M\lesssim
10^{4}\,\,\,\text{TeV}=10\,\,\,\text{PeV},\,\,\,\,\,\text{with}\,\,\,\,\,|\omega|\simeq\frac{\text{TeV}}{M}\,,$
(5)

the threshold corrections associated to $M_{p}$ can instead _minimize_ fine-
tuning by in fact defining and generating the observed $\bar{v}_{T}$. In this
way the same mechanism explains the origins of both the EW and neutrino mass
scales, and in a manner which can also accommodate successful resonant
leptogenesis if $M\gtrsim 1$ PeV Brivio:2019hrj  (also see Brdar:2019iem ).
Hence an important theoretical question to address is how the $M_{p}\sim$ PeV
scale can appear naturally, perhaps from the deep ultraviolet (UV) typically
associated to Grand Unified (GUT) or Planckian dynamics.

Figure 1: Threshold corrections generating the EW and neutrino mass scales in
the Neutrino Option.

In BTTpert  the loop quantum corrections to $M_{p}$ permitted by (1) were
studied, with the conclusion that the Neutrino Option is perturbatively
stable; neither threshold corrections nor renormalization group flow can
generate the PeV scale from a heavier Majorana state with (e.g.) $M_{3}\gg$
PeV in the context of minimal Type-I seesaw models, and indeed in many Beyond-
the-Standard Model (BSM) extensions to the Type-I seesaw. Permitting extra
scalar fields, the mass term could appear as a result of spontaneously broken
classical scale invariance, leading to a conformal Neutrino Option as in
Brdar:2018vjq . On the other hand, non-perturbative dynamics might also
provide a route to a successful, minimal UV Neutrino Option, and in this
letter we study a particular instantiation of this non-perturbative option by
focusing on a well-known mechanism Ibanez:2006da ; Ibanez:2007rs ;
Blumenhagen:2006xt ; Cvetic:2007ku  for generating sterile Majorana neutrino
masses $M_{p}$ from instanton interactions in a large class of string
compactifications. We will show that such stringy instantons can in fact
naturally generate the preferred $M_{p}\sim$ PeV scale given a broad range of
string scales, and therefore a potentially broad range of string embeddings.

##  II Stringy Majorana Mass Matrices

In a series of papers Ibanez:2006da ; Ibanez:2007rs  (also see
Blumenhagen:2006xt ; Cvetic:2007ku ) it was shown that a Majorana mass term of
the form

$m_{s}\,e^{-U}\,\overline{N}N$ (6)

can appear as a result of instanton effects in string compactifications to
four-dimensional (4D) SM and minimally-supersymmetric SM (MSSM) spectra. Here
$U$ is a set of complex string moduli and $m_{s}$ is the string scale, and the
term is allowed under the SM gauge symmetry and, as is ubiquitous in many
classes of string compactifications, an additional U(1)B-L gauge symmetry.
This latter invariance is due to the fact that $\text{Im}\,U$ is (a linear
combination of) axion-like complex scalar fields which is also shifted by two
units under $B-L$, thereby leaving the term invariant. The $U(1)_{B-L}$ gauge
boson obtains a mass on the order of $m_{s}$ through a Stueckelberg mechanism,
leaving in the ‘infrared’ (IR) only an unbroken global $U(1)_{B-L}$ and the
gauge symmetry and light particle spectrum of the SM (or e.g. the MSSM)
Ghilencea:2002da .222Hence $M_{Z^{\prime}}\sim m_{s}$ is a disconnected scale
unrelated to a vacuum expectation value Heeck:2014zfa , and any kinetic mixing
between $U(1)_{B-L}$ and $U(1)_{Y}$ sectors typical of Stueckelberg-extended
models (see e.g. Kors:2004dx ; Kors:2004iz ; Perez:2014gta ; Feldman:2011ms ;
Salvioni:2009mt ) is suppressed by $m_{s}$, and may not even be present in the
relevant classes of D-brane models Ibanez:2001nd ; Kors:2004iz . Of course, if
in a given model the $Z^{\prime}$ couples to $B$, two-loop Higgs corrections
can put naturalness constraints on $m_{s}$ or the unknown couplings. In what
follows we assume that no 4D SUSY is present, although this implies important
constraints on the string embedding — see the discussion in Section II.1.

For example, it was shown in Ibanez:2006da  that (6) can appear in Type IIA
orientifolds where fermions appear as localized string excitations at the
intersection of D6-branes, and where two such D6-branes further intersect with
D2-branes to be identified with the string instanton, giving rise to fermionic
zero modes (Grassman variables) that must be integrated over. This integration
yields a non-vanishing contribution to the fermion ($N$) bilinear amplitude
which, upon multiplying by the classical D2 instanton action, yields a
Majorana mass matrix given by333Similar instanton effects can also generate a
Weinberg operator whose relative strength with respect to (7) is of
phenomenological relevance, but is also model-dependent. In what follows we
assume that (7) is the dominant contribution to the light neutrino spectrum.

$M_{ab}=m_{s}\left(\epsilon_{ij}\epsilon_{kl}d_{a}^{ik}d_{b}^{jl}\right)\,e^{-S}\,,$
(7)

where $d$ represent flavoured couplings to be discussed below, $\epsilon_{ij}$
is the antisymmetric unit tensor, and $S$ is the instanton action with
suppression factor $\text{Re}(S)$. However, the D2-instantons giving rise to
this also exhibit a global internal symmetry in 4D, and in Ibanez:2007rs  a
somewhat exhaustive scan of available instantons/symmetries was
performed.444While Ibanez:2007rs  was largely looking for MSSM-like spectra,
the appearance of the internal $Sp(2)$ is an independent phenomenon and may
very well appear in similar scans with no 4D SUSY. I thank Luis Ibáñez for
this clarifying comment. The class that appeared most frequently had an
internal $Sp(2)\cong SU(2)$ symmetry that leads to a form of (7) whose flavour
structure factorizes according to

$M_{ab}=2\,m_{s}\sum_{r}d^{(r)}_{a}d^{(r)}_{b}e^{-S_{r}}\,,$ (8)

such that

$M=2m_{s}\sum_{r}e^{-S_{r}}\text{diag}\left(d_{1}^{(r)},d_{2}^{(r)},d_{3}^{(r)}\right)\cdot\left(\begin{array}[]{ccc}1&1&1\\\
1&1&1\\\
1&1&1\end{array}\right)\cdot\text{diag}\left(d_{1}^{(r)},d_{2}^{(r)},d_{3}^{(r)}\right)\,.$
(9)

The sum over $r$ accounts for the fact that, in general, multiple instantons
can contribute to the bilinear amplitude, and it is clear that when $r$
instantons are present, there are $3-r$ zero eigenvalues in (9). Note however
that (7)-(9) are not in their mass-eigenstate basis, and so $r\neq p$.
Furthermore, (9) has three inputs associated to it: $m_{s}$, $S_{r}$, and
$d_{a}^{(r)}$. Following the discussion in Ibanez:2006da ; Antusch:2007jd ,
they constitute:

  * •

${\bf{m_{s}}}$: The _string scale_ is not known, and could in principle be as
low as current LHC bounds allow, although in the supersymmetric case one might
want to keep this scale high (e.g. the GUT scale) to maintain coupling
unification, etc. Regardless, 4D SUSY is not required for the mechanism to
work (see Ibanez:2006da ) and we generally assume its absence here, so when
the seesaw mechanism is the dominant source of IR neutrino masses (as opposed
to the Weinberg operator) as desired in the Neutrino Option, $m_{s}$ is
essentially unconstrained.

  * •

${\bf{S_{r}}}$: The _instanton action_ is, critically, not as large a
suppression factor as in standard Yang Mills (e.g. electroweak) instanton
effects, which typically go $\propto e^{-1/g_{2}^{2}}$. In fact, there are no
phenomenological constraints on this parameter, such that the overall
suppression could be as small as an $\mathcal{O}(1)$ effect. For the
phenomenology of the next section to work, we will find that
$|\text{Re}(S_{r})|\sim\mathcal{O}(10^{-1}-10^{1})$ is desirable, although
this depends on the string scale. Since the overall normalization of the
action is a free parameter in (at least some classes of) perturbative string
compactifications, this seems quite safe, and is in fact consistent with the
neutrino phenomenology that was studied in Antusch:2007jd .

  * •

${\bf{d_{a}^{(r)}}}$: The _flavour vectors_ are fundamentally stringy objects,
and in certain classes of compactifications they are computable, as are normal
Yukawa couplings (see e.g. Cremades:2003qj ). However, they are also model-
dependent, so the authors of Antusch:2007jd  took a phenomenological
perspective and allowed these to be constrained by data. As we want to be as
model-independent as possible, we will do the same below.

Of course, as mentioned in Ibanez:2006da , this stringy instanton mechanism is
quite general and may lead to Majorana mass matrices of the form in (6) in
multiple classes of string compactifications (not just the Type IIA models
referenced above). Hence for our phenomenological purposes we simply take
(6)-(7), with the particular factorization in (8), as the starting point of
our analysis. However, we emphasize that one does not require the special
democratic factorization in (8) for the Neutrino Option to be successful — any
sterile mass matrix with $M_{p}\sim$ PeV, including a rank-two matrix with
$M_{1}=0$, can work. However, (8) allows an intuitive understanding of the
role of each instanton in the tree-level non-perturbative mass generation, and
further allows for an easy qualitative extraction of relevant UV scalings of
$m_{s}$, $d_{a}^{(r)}$, and $S_{r}$ from IR neutrino phenomenology in Section
III.

###  II.1 A Comment on Finite String Embeddings

Before proceeding though, an important comment/caveat must be mentioned in
order for the framework to be fully consistent. As is obvious from (3), a
successful Neutrino Option relies on the fact that the threshold corrections
to $m_{h}$ from $N_{p}$ are in fact dominant (or at least not sub-dominant),
such that the origin of the EW scale is both explained and remains stable.
However, in non-SUSY string embeddings one might expect potentially
destabilizing radiative corrections to $m_{h}$ on the order of
$\alpha_{SM}\,m_{s}^{2}$, driven by Kaluza-Klein or string excitations of SM
states. Furthermore, solutions to moduli stabilization which result in a 4D
EFT with a massive SUSY spectrum could circumvent the Neutrino Option
altogether — either the sparticles are roughly TeV scale and there is no
hierarchy problem (assuming safe radiative corrections from sneutrino partners
to $N_{p}$), or they are extremely heavy and can instead _generate_ the
hierarchy problem, with or without heavy $N_{p}$ (although see Giudice:2011cg
for scenarios where this is not necessarily the case, when SUSY is broken at
an extremely high scale). The former option is increasingly challenged by the
lack of LHC SUSY signals, and the latter is not solved by the Neutrino Option.

Hence it is critical that SM excitations are stabilized in the non-SUSY string
embedding our present mechanism assumes, or in any other embedding where (3)
is maintained as dominant, such that a non-standard ‘Neutrino Option’ exists.
One option may be to simply assume a low value of the string scale, as was
done in Antoniadis:1998ig , although the requirement of large extra dimensions
in this scenario introduce additional considerations in D-brane models — see
e.g. Cremades:2002qm  — and the preferred range of $m_{s}\in\\{1-10\\}$ TeV of
Antoniadis:1998ig  likely requires ‘non-perturbative’ flavour vectors in the
Neutrino Option. Regardless, in Section IV we study low(er)-scale $m_{s}$
scenarios that may naively be more compatible with non-SUSY constructions, and
show that without further theory constraints they are also viable. Of course
other efforts to stabilize moduli in non-SUSY compactifications exist (see
e.g. Blumenhagen:2002mf  which addresses intersecting D-branes in Type 0
string theory, and Blumenhagen:2006ci  for a review of other ideas), and it
was already mentioned in Ibanez:2006da  that the introduction of suitable
fluxes in the compactification (see e.g. Marchesano:2006ns ) could lift
additional zero modes of the D2-Brane instantons, thereby leaving their
deformation moduli stable. However, ultimately we are unaware of any mechanism
that has demonstrated complete stability in the Type IIA non-SUSY intersecting
D-brane models of Ibanez:2006da , or other scenarios that have been shown
capable of explicitly generating the SM $+N_{p}$ spectrum of interest —
whether this can be done remains an open question in the string
literature.555I am grateful to Graham Ross for pointing out the general
concern of SM string excitations in the Neutrino Option, and to Angel Uranga
for helpful commentary on the present literature.

In summary, for our phenomenological purposes here, we simply assume a finite
string embedding from the outset, and leave stabilization as a potentially
limiting theoretical constraint on successful ultraviolet stringy Neutrino
Options. On the other hand, the compelling observations of the upcoming
Sections III-IV also provide a novel motivation for further pursuing such
embeddings, as we show that, if found, the resulting theory can still address
the EW hierarchy and neutrino mass problems within the Neutrino Option
paradigm, i.e. in a manner previously unconsidered.

##  III Low Energy Neutrino Phenomenology and Implied Scales

Gaining complete analytic control over the Type-I seesaw in this framework is
challenging. However, given (8)-(9), the low-energy neutrino mass matrix
$M^{L}$ is given, in the simplifying limit of a diagonal neutrino Yukawa
coupling,666Small perturbations about this special case can of course be
systematically studied, but the qualitative range of $M_{p}$ we are attempting
to identify here becomes especially manifest in this diagonal limit. Note
however that (10)-(12) assumes the dominant leptonic mixing originates from
$M$, which is not necessary for our analysis, and which we relax in Section
IV. by

$M_{ab}^{L}=\left(\sum_{r}\frac{d_{a}^{(r)}}{(h_{D}^{T})_{aa}}\frac{d_{b}^{(r)}}{(h_{D})_{bb}}\tilde{I}_{r}^{-1}\right)^{-1}\,,$
(10)

with

$h_{D}=\text{diag}\left(\omega_{e}^{(\nu)},\omega_{\mu}^{(\nu)},\omega_{\tau}^{(\nu)}\right),\,\,\,\,\,\,\,\,\,\,\text{and}\,\,\,\,\,\,\,\,\,\,\tilde{I}_{r}=\frac{\langle\overline{H}\rangle^{2}}{2m_{s}}\frac{1}{e^{-S_{r}}}\,.$
(11)

However, we also recall that in this same basis one can generically derive
that, in the further limit of three non-zero masses,

$\left(M_{ab}^{L}\right)^{-1}=U\cdot\text{diag}\left(\frac{1}{m_{1}\,e^{i\alpha_{1}}},\frac{1}{m_{2}\,e^{i\alpha_{2}}},\frac{1}{m_{3}}\right)\cdot
U^{T}\,,$ (12)

since $M_{ab}^{L}$ is a complex symmetric matrix. Assuming diagonal charged-
lepton mixing (which is always possible with an appropriate basis
transformation), $U$ is then the PMNS matrix, up to the diagonal matrix of
Majorana phases $\alpha_{i}/2$ that has simply been factored into the IR
neutrino mass eigenvalues $m_{i}$. We now see that the RHS (12) can be
expanded as

$\left(M_{ab}^{L}\right)^{-1}=\frac{e^{-i\alpha_{1}}}{m_{1}}\,U_{i1}\cdot
U_{1i}+\frac{e^{-i\alpha_{2}}}{m_{2}}\,U_{i2}\cdot
U_{2i}+\frac{1}{m_{3}}\,U_{i3}\cdot U_{3i}\,.$ (13)

Equating this form of the IR mass matrix to (10), one immediately arrives at a
solution for $\tilde{I}_{r}$ in terms of the low energy neutrino mass
eigenvalues given by

$\tilde{I}_{1}=m_{1}\,,\,\,\,\,\,\,\,\,\,\,\tilde{I}_{2}=m_{2}\,,\,\,\,\,\,\,\,\,\,\,\tilde{I}_{3}=m_{3}\,,$
(14)

and a solution for the components of the flavour vectors $d_{a}^{(r)}$ in
terms of the PMNS mixing matrix given by

$\displaystyle\left(\frac{d_{1}^{(1)}}{\omega_{e}^{(\nu)}},\frac{d_{2}^{(1)}}{\omega_{\mu}^{(\nu)}},\frac{d_{3}^{(1)}}{\omega_{\tau}^{(\nu)}}\right)$
$\displaystyle=e^{-i\alpha_{1}/2}\left(U_{11},U_{21},U_{31}\right)\,,$
$\displaystyle\left(\frac{d_{1}^{(2)}}{\omega_{e}^{(\nu)}},\frac{d_{2}^{(2)}}{\omega_{\mu}^{(\nu)}},\frac{d_{3}^{(2)}}{\omega_{\tau}^{(\nu)}}\right)$
$\displaystyle=e^{-i\alpha_{2}/2}\left(U_{12},U_{22},U_{32}\right)\,$
$\displaystyle\left(\frac{d_{1}^{(3)}}{\omega_{e}^{(\nu)}},\frac{d_{2}^{(3)}}{\omega_{\mu}^{(\nu)}},\frac{d_{3}^{(3)}}{\omega_{\tau}^{(\nu)}}\right)$
$\displaystyle=\left(U_{13},U_{23},U_{33}\right)\,.$ (15)

In (10)-(15) we have simply generalized the mixing matrix elements from the
specific texture originally studied in Antusch:2007jd . As $U_{PMNS}$ is
unitary, the matrix elements are all $|U_{ij}|\leq 1$ and the Majorana phase
factor $e^{-i\alpha_{i}/2}$ is also $\mathcal{O}(1)$. Hence we derive that

$d_{a}^{(r)}\approx\omega_{i}^{(\nu)}/10\sim\mathcal{O}(10^{-5})-\mathcal{O}(10^{-4}),$
(16)

where we generically assume non-hierarchical couplings, which is consistent
with the original Neutrino Option analysis in Brivio:2017dfq ; Brivio:2018rzm
and is also more preferable from the stringy side Antusch:2007jd , and where
the scale relationship on the RHS is the range identified in (5).

Let us now derive the expected scale of the remaining parameter that enters
(8)-(9), the instanton suppression factor $S_{r}$, given (e.g.) a string scale
$m_{s}$ assumed to lie in the deep UV. Using (14), one easily finds that

$S_{r}=-\frac{1}{2}\ln\left[\frac{\langle\overline{H}\rangle^{4}}{m_{r}^{2}\,4\,m_{s}^{2}}\right]\,\,\,\Longleftrightarrow\,\,\,m_{r}^{2}=\frac{\langle\overline{H}\rangle^{4}}{4m_{s}^{2}}\,e^{2S_{r}}\,,$
(17)

from which we also obtain the neutrino mass-squared difference formula

$\Delta m_{r_{i}r_{j}}^{2}\equiv
m_{r_{i}}^{2}-m_{r_{j}}^{2}=\frac{\langle\overline{H}\rangle^{4}}{4m_{s}^{2}}\,\left(e^{2S_{i}}-e^{2S_{j}}\right)\,.$
(18)

This quantity is of course experimentally bound by neutrino oscillation
experiments. The most recent global fit from the NuFIT collaboration
Esteban:2020cvm  puts these at

$\displaystyle\text{(NO):}\ \ \ \ \ \ \Delta m_{21}^{2}$
$\displaystyle\in\\{6.82-8.04\\}\,\,\,\cdot\,\,\,\,10^{-5}\,\,\,\,\,\text{[eV]}^{2}\,,$
$\displaystyle\ \ \ \ \ \ \,\,\,\,\,\Delta m_{31}^{2}$
$\displaystyle\in\\{2.435-2.598\\}\,\cdot 10^{-3}\,\,\,\,\,\text{[eV]}^{2}\,,$
(19)

where we have shown the $3\sigma$ range of values when the neutrino masses are
of normal ordering (NO). Inverted ordering (IO) fits are also available, as
are global bounds on the PMNS elements $U_{ij}$ in (15), although as mentioned
above these will all be $\mathcal{O}(U_{ij})\lesssim 1$ due to unitarity in
our theoretical framework. So for the order of magnitude estimate we attempt
here we will not repeat the explicit bounds. Hence, using a deep-UV $m_{s}$,

$\displaystyle
m_{s}\sim\\{10^{16}-10^{19}\\}\,\,\,\,\,\text{GeV}\,,\,\,\,\,\,\langle\overline{H}\rangle\sim
246/\sqrt{2}\,\,\,\,\,\text{GeV}\,,$ (20)

we can then use (17)-(19) to derive the rough order of magnitude estimate

$|S_{r}|\sim\mathcal{O}(10^{-1}-10^{1})\,.$ (21)

This provides all of the information necessary to then determine the expected
scale of $M$.

To that end we can insert (16), (20), and (21) into (8)-(9), and compute the
eigenvalue spectrum of $M$. As a first illustration we allow for three
instantons and choose the parameters randomly within the ranges specified
above. For example, with $m_{s}=10^{18}$ GeV one can find

$\displaystyle M_{3}$
$\displaystyle\sim\mathcal{O}(10^{6}-10^{10})\,\,\,\,\,\,\text{GeV},$
$\displaystyle M_{2}$
$\displaystyle\sim\mathcal{O}(10^{4}-10^{9})\,\,\,\,\,\,\,\,\text{GeV},$
$\displaystyle M_{1}$
$\displaystyle\sim\mathcal{O}(10^{2}-10^{8})\,\,\,\,\,\,\,\,\text{GeV}\,.$
(22)

While this is only a qualitative estimate, made within the simplified
formalism implied by (10), the obvious implication is that the PeV scales
preferred in the Neutrino Option can appear automatically in this formalism,
given GUT or Planckian string scales $m_{s}$ and values implied by low energy
neutrino phenomenology. Hence (22) and its analogues across a broader range of
$m_{s}$ represent the core observation of this note, which we now study in
more numerical detail.

##  IV Scanning Viable Parameter Spaces

Figure 2: Contours realizing (roughly) successful Neutrino Options across a
broad range of string scales, given the scalings of the free parameters
considered in the text.

We now explore the viable parameter space more comprehensively by varying the
parameters relevant to calculating the $M_{p}$ spectrum, which constitute the
string scale $m_{s}$, the instanton suppression factors $S_{r}$, and the
flavour vectors $d^{(r)}_{a}$. Since our goal is to show that successful
Neutrino Options can be realized with a broad range of string scales, we
define the following scale-setting parameters

$\mathbb{s}\equiv\log_{10}m_{s},\,\,\,\mathbb{d}\equiv\log_{10}d_{a}^{(r)},\,\,\,\mathbb{p}\equiv\log_{10}M_{p},$
(23)

with dimensionful quantities given in GeV, and note that, up to
$\mathcal{O}(1)$ coefficients and flavour effects, naive dimensional analysis
implies the following contour:

$\mathbb{s}+2\mathbb{d}-\left(\frac{1}{2}S_{r}+\mathbb{p}\right)\approx 0\,,$
(24)

where we assume non-hierarchical flavour vectors, but do _not_ assume the
strict mixing limits of Section III.777Here we have slightly abused notation,
setting $\text{Re}(S_{r})=S_{r}$. In the upcoming numerical analysis, we will
treat $S_{r}$ as a complex number with randomly varied real and imaginary
components. A successful Neutrino Option then fixes $\mathbb{p}$ as in (5),
such that (24) defines a hyperplane with coordinates $\mathbb{s}$,
$\mathbb{d}$, and $S_{r}$. For the instanton suppression factors, one notices
from the mass-difference formula of (18) (and more generally from (9)) that
$r>1$ is required to reproduce low-energy neutrino data. This can be achieved
naturally since each $S_{r}$ is associated to a different instanton, as can
hierarchies amongst the individual $S_{r}$, facts that were first noted in
Antusch:2007jd . In what follows we consider the $r=3$ case, such that there
are three non-zero eigenvalues in $M$. Continuing, the string parameter
$\mathbb{s}$ is only bound by experiment — low values are interesting in
general, but especially so due to the potential stability issues mentioned in
Section II.1 for non-SUSY string embeddings, and of course high values provide
a direct link to the deep UV without relying on large compact dimensions.
Finally, we remain agnostic as to the possible values of $\mathbb{d}$ (which
are extremely model-dependent), and allow for the following broad range of
scalings:

$-6\leq\mathbb{d}\leq 5\,.$ (25)

The lower bound is motivated by Figure 2, (16), and the simultaneous fact that
$d^{(r)}_{a}$ are computed analogously to standard Yukawa couplings in certain
types of toroidal compactifications, including those of Ibanez:2006da .
Assuming $\text{Re}(S_{r})>0$ (such that one truly has an instanton
_suppression_ factor), one immediately recognizes from (5) that the models of
Antoniadis:1998ig , with $\mathbb{s}\in\\{3,4\\}$, are not realized in this
limit. On the other hand, for $\mathbb{d}\geq 0$, low-scale models are viable.
While one might suspect that $d^{(r)}_{a}\lesssim 1$ in order to remain
perturbative (motivated by the analogy with Yukawa couplings), we recall that
(7) is a non-perturbative operator, and may not necessarily be constrained by
such arguments.

Given these inputs, we plot the naive scaling contours expected for viable
Neutrino Options in Figure 2, for string scales ranging from 1 TeV to 1018
GeV, and where we have treated $S_{r}$ in (24) as a single
$\mathcal{O}(1)$-$\mathcal{O}(10)$ parameter, which is varied over a
significant (positive) range to account for (27). The width of the colored
bands represent the bounds on $M_{p}$ from (5), and allowing for continuous
string scales between the discretized values shown, one notes that, without
further theory constraints from a specific string embedding/model, a broad
range of parameter values can yield successful Neutrino Options in the
framework.

To better illustrate this, we now account for the various $\mathcal{O}(1)$
coefficients and flavour structures in (9) by defining the components of the
flavour vectors as

$d_{a}^{(r)}\equiv\left(\rho\,e^{i\phi}\,\cdot\,10^{\mathbb{d}_{0}}\right)_{a}^{(r)}\,,$
(26)

such that $\mathbb{d}_{0}\sim\mathbb{d}$, and $\\{\rho,\phi\\}$ simply
parameterize the arbitrary $\mathcal{O}(1)$ complex coefficient. Then, in
order to avoid tuning amongst the $S_{r}$ and avoid large hierarchies in
$M_{p}$ (in accordance with the analysis in Brivio:2017dfq ; Brivio:2018rzm ),
we impose the following simple scaling relationship,

$S_{2,3}\overset{!}{=}c_{2,3}\,S_{1},\,\,\,\,\,\text{with}\,\,\,\,\,c_{2,3}\sim\mathcal{O}(1),$
(27)

where $c_{23}$ are arbitrary real coefficients, and where both the real and
imaginary components of $S_{1}$ are varied as in Table 1. Realistic
compactifications like that found in the scans of Ibanez:2007rs  can generate
larger hierarchies amongst $S_{r}$ if desired, e.g.
$c_{23}\sim\mathcal{O}(10)$. We also note that, in the simplified formalism of
Section III, one may be tempted to relate $S_{2,3}$ to $S_{1}$ via (18) since
one apparently is guaranteed to realize the phenomenologically accurate
$\Delta m^{2}_{ij}$, at least at the scale $M(m_{s})$. However, this is
redundant, as the bounds in (5), derived in Brivio:2017dfq ; Brivio:2018rzm ,
_already_ account for renormalization group evolution (RGE), threshold
correction effects, and low-energy experimental bounds. That is, achieving (5)
upon transforming to the mass-eigenstate basis of $N_{p}$ already ensures
(within the rough set of assumptions in this analysis and those of
Brivio:2017dfq ; Brivio:2018rzm ) that IR SM and neutrino phenomenology is
realized. Furthermore, the fastidious reader will notice that enforcing the
exponential relationship in (18) on $S_{2,3}$ can yield an $M_{p}$ spectrum
that is largely insensitive to orders-of-magnitude changes in $S_{1}$, and
that tuning at the third or fourth digit (or worse) between $S_{r}$ can be
required for $\mathbb{s}\lesssim 13$. Both of these undesirable effects are
unnecessary artifacts of the simplified formalism of Section III, which while
helpful in making our qualitative point analytically, we now relax. As a final
comment, we have confirmed for ourselves in BTTpert  that the RGE and
threshold corrections to $M_{p}$ down to (e.g.) $\mu\sim m_{Z}$ are extremely
small in comparison to the $\sim$ PeV scales we aim to extract below
Casas:1999tp ; Casas:1999tg ; Antusch:2005gp ; Chankowski:2001mx ;
Antusch:2003kp ; Ibarra:2018dib ; Antusch:2002ek ; Ibarra:2020eia .

| $\mathbb{d}_{0}$ | $\rho$ | $\phi$ | $c_{23}$ | $S_{1}$ | $\mathbb{s}$ | $N$
---|---|---|---|---|---|---|---
Range A | $-\left[5,3\right]$ | $\left[0,1\right]$ | $\left[0,2\pi\right]$ | $\left[\frac{1}{10},\frac{15}{10}\right]$ | $(0,10\,]$ | $3\mathbb{Z}\leq 18$ | $4$
Range B | $\left[\frac{15}{10},4\right]$ | $\left[0,1\right]$ | $\left[0,2\pi\right]$ | $\left[\frac{1}{10},\frac{15}{10}\right]$ | $(0,10\,]$ | $3$ | $5$
Range C | $-\left[6,3\right]$ | $\left[0,1\right]$ | $\left[0,2\pi\right]$ | $\left[\frac{1}{10},\frac{15}{10}\right]$ | $(0,10\,]$ | $18$ | $5$

Table 1: The scan ranges implemented for the relevant free parameters in our
formalism.

.

Figure 3: TOP: The spectrum of $|M_{2,3}|$ given scan Range A from Table 1.
BOTTOM: The same for scan Ranges B-C, and in the domain of successful Neutrino
Options.

Given (26)-(27), we then perform scans by allowing random variations of
$\mathbb{d}_{0}$, $\rho$, $\phi$, $c_{23}$ and $S_{1}$ within the ranges
identified in Table 1. Specifically, our scripts first assemble $M$ with (9),
given a string scale $m_{s}$ that we set by hand, and then we randomly draw
the other parameters within their allowed ranges, calculating the magnitudes
of the associated $M_{p}$ spectrum for each draw. We repeat this procedure
$10^{N}$ times, collect all of the inputs, and analyze the outputs.

In the upper plot of Figure 3 we show the spectrum of the heaviest $|M_{2,3}|$
for $\mathbb{s}\in\\{3-18\\}$ given the Range A values of Table 1. In this
case we use scalings of the flavour vectors motivated by (16),
$-5\leq\mathbb{d}_{0}\leq-3$, and we note that if these are to truly scale
like the neutrino Yukawa couplings $\omega^{(\nu)}$, then intermediate to high
values of the string scale, $m_{s}\sim 10^{12-18}$ GeV, are preferred given
(5). Of course, this conclusion can be changed given a different range of
suppression factors $S_{r}$, although we recall that as discussed in
Ibanez:2006da , one does not necessarily expect the D-brane instanton actions
to be highly suppressed. Keeping $S_{r}$ fixed, in the bottom plot of Figure 3
we study the more tailored values of the flavour vectors in Range B-C from
Table 1, for the extreme limits of the string scales we consider:
$\mathbb{s}=\\{3,18\\}$. Here we isolate the mass domain to the values
preferred in (5), and observe that with this particular subset of
$\mathbb{d}_{0}$, both large and small string-scale scenarios can yield
successful Neutrino Options. In all of these models we note that, by
construction, the lightest eigenvalue $M_{1}$ is also of a scale which would
maintain natural threshold corrections to the Higgs mass from (3).

Ultimately Figures 2-3 demonstrate that, in the absence of an explicit string-
derived model, or any other further theoretical constraints on the input
parameters, the implications of (9) are not terribly predictive — a range of
eigenvalues spanning most conceivable scales is achievable, including very low
values of $M_{p}$ relevant to, e.g., sterile neutrino dark matter studies. On
the other hand, they also demonstrate that successful, UV-complete Neutrino
Options can be achieved in a variety of string embeddings, and further
motivate theoretical studies to that end.

##  V Summary and Outlook

We have shown that a sterile Majorana mass matrix with eigenvalues $M_{p}\sim$
PeV can arise naturally through non-perturbative instanton effects, which are
known to be present in a large class of string compactifications, including
Type IIA non-SUSY intersecting D-brane models. Hence successful Neutrino
Options can potentially be realized across a broad range of string scales in
this framework, thereby simultaneously explaining the origins of the EW and
light neutrino mass scales in a UV-complete way, providing an elegant
resolution to (at least) two of the SM’s most pressing phenomenological
issues. Furthermore, this mechanism does so without relying on 4D SUSY,
although its potential absence puts an important constraint on the types of
finite string embeddings that give a consistent framework, free of
destabilizing string excitations of SM states. Hence we believe that this
mechanism provides an interesting new approach to these important issues in
BSM physics and motivates further theoretical exploration of the types of
finite string compactifications that can yield (7)-(9) and an otherwise stable
Higgs mass, any implied constraints on the stringy free parameters at hand
(i.e. $S_{r}$, $d_{a}$, and $m_{s}$), and eventually a more detailed
phenomenological survey of the resulting neutrino mass and mixing spectrum.

## Acknowledgements

I gratefully acknowledge support from the Villum Fund, project number
00010102, and thank Stefan Antusch, Luis Ibáñez, Graham Ross, and Angel Uranga
for helpful comments and insight. I also thank Ilaria Brivio and Michael Trott
for inspiring discussions and collaboration on related work, and both Ilaria
and Graham for their review of the manuscript. I also appreciate Ilaria
allowing me to recycle the Feynman diagrams in Fig. 1.

## References

  * (1) I. Brivio and M. Trott, Phys. Rev. Lett. 119 (2017) no.14, 141801 [arXiv:1703.10924 [hep-ph]].
  * (2) I. Brivio and M. Trott, JHEP 02 (2019), 107 [arXiv:1809.03450 [hep-ph]].
  * (3) P. Minkowski, Phys. Lett. B 67 (1977), 421-428
  * (4) M. Gell-Mann, P. Ramond and R. Slansky, Conf. Proc. C 790927 (1979), 315-321 [arXiv:1306.4669 [hep-th]].
  * (5) R. N. Mohapatra and G. Senjanovic, Phys. Rev. Lett. 44 (1980), 912
  * (6) T. Yanagida, Prog. Theor. Phys. 64 (1980), 1103
  * (7) S. M. Bilenky, J. Hosek and S. Petcov, Phys. Lett. B 94 (1980), 495-498
  * (8) A. Broncano, M. Gavela and E. E. Jenkins, Phys. Lett. B 552 (2003), 177-184 [arXiv:hep-ph/0210271 [hep-ph]].
  * (9) G. Elgaard-Clausen and M. Trott, JHEP 11 (2017), 088 [arXiv:1703.04415 [hep-ph]].
  * (10) F. Vissani, Phys. Rev. D 57 (1998), 7027-7030 [arXiv:hep-ph/9709409 [hep-ph]].
  * (11) I. Brivio, K. Moffat, S. Pascoli, S. T. Petcov and J. Turner, JHEP 10 (2019), 059 [arXiv:1905.12642 [hep-ph]].
  * (12) V. Brdar, A. J. Helmboldt, S. Iwamoto and K. Schmitz, Phys. Rev. D 100 (2019), 075029 [arXiv:1905.12634 [hep-ph]].
  * (13) I. Brivio, J. Talbert and M. Trott, _work in progress_
  * (14) V. Brdar, Y. Emonds, A. J. Helmboldt and M. Lindner, Phys. Rev. D 99 (2019) no.5, 055014 [arXiv:1807.11490 [hep-ph]].
  * (15) L. Ibanez and A. Uranga, JHEP 03 (2007), 052 [arXiv:hep-th/0609213 [hep-th]].
  * (16) L. Ibanez, A. Schellekens and A. Uranga, JHEP 06 (2007), 011 [arXiv:0704.1079 [hep-th]].
  * (17) R. Blumenhagen, M. Cvetic and T. Weigand, Nucl. Phys. B 771 (2007), 113-142 [arXiv:hep-th/0609191 [hep-th]].
  * (18) M. Cvetic, R. Richter and T. Weigand, Phys. Rev. D 76 (2007), 086002 [arXiv:hep-th/0703028 [hep-th]].
  * (19) D. M. Ghilencea, L. E. Ibanez, N. Irges and F. Quevedo, JHEP 08 (2002), 016 [arXiv:hep-ph/0205083 [hep-ph]].
  * (20) J. Heeck, Phys. Lett. B 739 (2014), 256-262 [arXiv:1408.6845 [hep-ph]].
  * (21) B. Kors and P. Nath, [arXiv:hep-ph/0411406 [hep-ph]].
  * (22) P. Fileviez Perez and S. Spinner, Phys. Rev. D 89 (2014) no.9, 095004 [arXiv:1401.7327 [hep-ph]].
  * (23) B. Kors and P. Nath, Phys. Lett. B 586 (2004), 366-372 [arXiv:hep-ph/0402047 [hep-ph]].
  * (24) D. Feldman, P. Fileviez Perez and P. Nath, JHEP 01 (2012), 038 [arXiv:1109.2901 [hep-ph]].
  * (25) E. Salvioni, G. Villadoro and F. Zwirner, JHEP 11 (2009), 068 [arXiv:0909.1320 [hep-ph]]. Ibanez:2001nd
  * (26) L. E. Ibanez, F. Marchesano and R. Rabadan, JHEP 11 (2001), 002 [arXiv:hep-th/0105155 [hep-th]].
  * (27) S. Antusch, L. Ibanez and T. Macri, JHEP 09 (2007), 087 [arXiv:0706.2132 [hep-ph]].
  * (28) D. Cremades, L. E. Ibanez and F. Marchesano, JHEP 07 (2003), 038 [arXiv:hep-th/0302105 [hep-th]].
  * (29) G. F. Giudice and A. Strumia, Nucl. Phys. B 858 (2012), 63-83 [arXiv:1108.6077 [hep-ph]].
  * (30) I. Antoniadis, N. Arkani-Hamed, S. Dimopoulos and G. R. Dvali, Phys. Lett. B 436 (1998), 257-263 [arXiv:hep-ph/9804398 [hep-ph]].
  * (31) D. Cremades, L. E. Ibanez and F. Marchesano, [arXiv:hep-ph/0212048 [hep-ph]].
  * (32) R. Blumenhagen, B. Kors and D. Lust, Phys. Lett. B 532 (2002), 141-151 [arXiv:hep-th/0202024 [hep-th]].
  * (33) R. Blumenhagen, B. Kors, D. Lust and S. Stieberger, Phys. Rept. 445 (2007), 1-193 [arXiv:hep-th/0610327 [hep-th]].
  * (34) F. Marchesano, JHEP 05 (2006), 019 [arXiv:hep-th/0603210 [hep-th]].
  * (35) I. Esteban, M. C. Gonzalez-Garcia, M. Maltoni, T. Schwetz and A. Zhou, [arXiv:2007.14792 [hep-ph]].
  * (36) J. Casas, J. Espinosa, A. Ibarra and I. Navarro, Nucl. Phys. B 556 (1999), 3-22 [arXiv:hep-ph/9904395 [hep-ph]].
  * (37) J. Casas, J. Espinosa, A. Ibarra and I. Navarro, Nucl. Phys. B 573 (2000), 652-684 [arXiv:hep-ph/9910420 [hep-ph]].
  * (38) S. Antusch, J. Kersten, M. Lindner, M. Ratz and M. A. Schmidt, JHEP 03 (2005), 024 [arXiv:hep-ph/0501272 [hep-ph]].
  * (39) S. Antusch, J. Kersten, M. Lindner and M. Ratz, Nucl. Phys. B 674 (2003), 401-433 [arXiv:hep-ph/0305273 [hep-ph]].
  * (40) P. H. Chankowski and S. Pokorski, Int. J. Mod. Phys. A 17 (2002), 575-614 [arXiv:hep-ph/0110249 [hep-ph]].
  * (41) A. Ibarra, P. Strobl and T. Toma, Phys. Rev. Lett. 122 (2019) no.8, 081803 [arXiv:1802.09997 [hep-ph]].
  * (42) S. Antusch and M. Ratz, JHEP 07 (2002), 059 [arXiv:hep-ph/0203027 [hep-ph]].
  * (43) A. Ibarra, P. Strobl and T. Toma, [arXiv:2006.13584 [hep-ph]].


# Investigating the Role of Instruction Variety and Task Difficulty in Robotic
Manipulation Tasks

Amit Parekh, Nikolas Vitsakis, Alessandro Suglia, Ioannis Konstas
Heriot-Watt University

###### Abstract

Evaluating the generalisation capabilities of multimodal models based solely
on their performance on out-of-distribution data fails to capture their true
robustness. This work introduces a comprehensive evaluation framework that
systematically examines the role of instructions and inputs in the
generalisation abilities of such models, considering architectural design,
input perturbations across language and vision modalities, and increased task
complexity. The proposed framework uncovers the resilience of multimodal
models to extreme instruction perturbations and their vulnerability to
observational changes, raising concerns about overfitting to spurious
correlations. By employing this evaluation framework on current Transformer-
based multimodal models for robotic manipulation tasks, we uncover limitations
and suggest future advancements should focus on architectural and training
innovations that better integrate multimodal inputs, enhancing a model’s
generalisation prowess by prioritising sensitivity to input content over
incidental correlations.111Code available
https://github.com/amitkparekh/CoGeLoT.

Gobbledygook Gobbledygook Words Gobbledygook Tokens Language Model (LM)[LM]
VIMA-Bench Vision+Language Large Language Models (LLMs)[LLMs]

Investigating the Role of Instruction Variety and Task Difficulty in Robotic
Manipulation Tasks

Amit Parekh, Nikolas Vitsakis, Alessandro Suglia, Ioannis Konstas Heriot-Watt
University

Figure 1:  Our evaluation framework. Each perturbation affects the instruction
or observation inputs, which can be linguistic, visual, or a combination of
both. The plausibility of a perturbation relates to a model’s expected
performance. Sensitivity to unreasonable conditions (✗) indicates that a model
should not perform the task successfully given the perturbation, while
plausible perturbations (✓) suggest that it should still perform successfully.

##  1 Introduction

Designing artificial agents to follow natural language instructions—to
understand and act within the context of their environment—is a long-term goal
of artificial intelligence (Winograd, 1972). An artificial agent should
generalise to unseen scenarios by combining concepts and skills underpinning
its training data in novel ways (Lake et al., 2017).

Previous work which proposed several language-guided tasks for tackling this
challenge, largely focused on generalising to environments with different
scenes from the training ones (e.g., ALFRED; Shridhar et al., 2020). However,
relying solely on language for embodied action execution tasks can be
inefficient, especially in collaborative settings with high ambiguity, such as
visually cluttered scenes Chiyah-Garcia et al. (2023); Li et al. (2023).
Multimodal prompts—instructions which interleave vision and language
tokens—represent a way to specify commands which can be more flexible and
specific than can be explained using text only (Ma et al., 2024; Jiang et al.,
2023; Stone et al., 2023). This capability is crucial for realistic human-
robot collaboration tasks and can be viewed as analogous to pointing at
objects within a scene (Chen et al., 2021; Islam et al., 2022). For this
reason, Jiang et al. (2023) presented , the first benchmark aimed at studying
several axes of generalisation involving novel concepts and tasks, with models
receiving instructions combining both language and visual referents.

Many other benchmarks test for generalisation solely by looking at held-out
examples (Open X-Embodiment Collaboration, 2024; Stone et al., 2023). However,
as highlighted by Hupkes et al. (2023), generalisation should be evaluated
across multiple dimensions when creating truly robust models, capable of
performing safely in varied environments. Inspired by these ideals, we assess
generalisation along important axes such as structural, compositional, and
robustness through specific covariate shifts (i.e., input perturbations) as
outlined in Figure 1. Specifically, we looked at 1) an extensive set of
linguistic perturbations on the instructions, such as paraphrasing, corrupting
the language content, and replacing visual referents with language
descriptions, 2) masking entire modalities within instructions, 3) introducing
visual perturbations by permuting object order, and 4) increasing the
difficulty of the tasks e.g., by placing distractors between source and
target. We categorise each perturbation as either “plausible” (e.g.,
paraphrases) or “unrealistic” (e.g., nonsensical instructions). We expect the
model to be robust to plausible inputs while dropping performance when faced
with unrealistic inputs.

To implement this formalisation, we use , which unlike other state-of-the-art
benchmarks such as ALFRED (Shridhar et al., 2020), CLIPort (Shridhar et al.,
2022), ARNOLD (Gong et al., 2023), and Ravens (Zeng et al., 2021), provides
several advantages: 1) it covers the majority of robotic manipulation tasks,
2) it offers more fine-grained levels for assessing the systematic
generalisation of models; 3) it represents a benchmark that allows for careful
examinations of specific architecture and training regimes. For this reason,
this paper builds on the controllability of to extensively study the impact
properties of multimodal prompts and visual representations have on model
performance.

We applied our novel evaluation setup on multiple state-of-the-art
architectures commonly used for different Embodied AI tasks and datasets
(Jiang et al., 2023; Octo Model Team, 2023; Open X-Embodiment Collaboration,
2024; Shridhar et al., 2022; Reed et al., 2022; Zhao et al., 2023).222While
feasible, we refrain from applying our evaluation framework on larger Vision
and Language Models (VLMs) such as LLaVa (Liu et al., 2023), as we focus on
models of a similar size to VIMA, which are amenable to on-device processing
when deployed on real-world robots. We uncover several deficiencies of current
“generalist agents" including 1) insensitivity to language perturbations, as
they could still perform several tasks when provided with gibberish
instructions; 2) inability to handle tasks of increasing difficulty
potentially including more distractors in the visual scene. These findings aim
to shed light on state-of-the-art model performance and call for more research
on systematically assessing model robustness with adequate tasks and settings
that are indicative of the generalisation capacities of Embodied AI models
designed to safely and effectively complete tasks in the real world in
collaboration with humans.

##  2 Related Work

#### Language-driven Embodied AI

Embodied AI focuses on designing agents that are embodied in some environment
(simulated or real) and generates actions to complete a given task, whose
objective is typically specified in natural language (Das et al., 2018). Tasks
for Embodied AI have been formulated in different ways depending on the degree
of complexity of the action space. For example, Vision+Language Navigation
(VLN; Anderson et al., 2018; Thomason et al., 2020) requires agents to
generate navigation actions to follow natural language instructions and reach
the given destination in the environment. With more sophisticated 3D simulated
environments such as AI2Thor (Kolve et al., 2017), more recent works also
define several tasks involving object interaction (e.g., ALFRED; Shridhar et
al., 2020, Alexa Arena; Shi et al., 2023).

#### Language in Robotic Manipulation Tasks

Language plays a crucial role in many Embodied AI tasks, providing an
interface for task learning (Laird et al., 2017), with many Embodied AI tasks
that require language instructions which are typically hand-crafted via
templates (e.g., , CLIPort, ARNOLD) or crowdsourced (e.g., ALFRED). However,
benchmarks often focus on evaluating generalisation using held-out episodes
(Open X-Embodiment Collaboration, 2024) and do not thoroughly evaluate the
importance of language (Stone et al., 2023; Open X-Embodiment Collaboration,
2024; Octo Model Team, 2023). For instance, models trained on datasets like
ALFRED have been shown to be insensitive to language instructions (Akula et
al., 2022), while nonsensical instructions have even improved downstream
performance on the VLN benchmark (Zhu et al., 2023).

We focus on tabletop robotic manipulation tasks with natural language
instructions to measure performance on a well-scoped action execution task.
This allows for assessment of visual grounding capabilities from grounding
instructions in the real world, while also removing the extra complexity of
sophisticated skills (e.g., SLAM) required for navigation tasks (Anderson et
al., 2018) or the need to predict fine-grained joint control by relying on
inverse-kinematics (Ma et al., 2024; Open X-Embodiment Collaboration, 2024;
Octo Model Team, 2023; Zeng et al., 2021).

#### Assessing Generalisation and Robustness

Embodied AI systems must generalise to any complex and novel tasks they might
face (Duan et al., 2022), making robustness a highly-desired characteristic in
models, illustrating how well they can ignore spurious correlations and
generalise to new domains and tasks (Akula et al., 2022; Gong et al., 2023;
Hupkes et al., 2023). Embodied AI benchmarks often assess generalisation
through seen/unseen scenes (e.g., Shridhar et al., 2020; Shi et al., 2023;
Zheng et al., 2022), assuming that all tasks the agent has to complete, and
the objects the agent must interact with are fully specified at training time.
While recent benchmarks evaluate models on unseen objects and scenes, (Gong et
al., 2023; Stone et al., 2023; Open X-Embodiment Collaboration, 2024), there
is no notion of systematic or compositional generalisation to new concepts,
affordances (Suglia et al., 2020; Pantazopoulos et al., 2022), or novel tasks
(Chung et al., 2022).

##  3 Experimental Setup

#### Evaluation Data

As it is the best placed for evaluating the role instructions play in
generalising from multimodal prompts, we use to compare how well models
perform on multiple skills and tasks, and also at different levels of
systematicity. More specifically, we explicitly check the compositional
generalisation capabilities at four distinct levels of systematicity
(Pantazopoulos et al., 2022; Hupkes et al., 2020): object pose sensitivity
(L1), combinatorial generalisation (L2), novel objects (L3), and novel tasks
(L4). See LABEL:app:environment-details for further environment details.

#### Models

We compare four model architectures: encoding visual representations with
either object-centric or image-patches; and conditioning prompts on the state
through either cross-attention or concatenation (Ma et al., 2024). Models are
trained on multimodal instructions containing interleaved visual and
linguistic features. Multimodal instructions are encoded through a frozen
pretrained T5 language model (Raffel et al., 2020), where encoded visual
features are injected into the embedding space of the language model (Driess
et al., 2023; Tsimpoukelli et al., 2021; Ma et al., 2024). Visual features are
implicitly encoded through embedding image frames per observation—a more
adaptable, more efficient, and better performing method than using explicit
symbolic representations (Song et al., 2024; Gadre et al., 2022). For each
observation, the model predicts an action that defines a linear movement
between two end-effector poses in $\textbf{SE}(3)$—each representing position
and rotation in 3D space. See LABEL:app:training for more details on the
training regime and objectives.

##  4 The Evaluation Framework

To explore the importance of both visual and linguistic information in
multimodal prompts, we systematically apply perturbations to model inputs at
test time to explore how each characteristic of the inputs directly relates to
a model’s ability to develop an understanding of their task. We report full
per-task results in LABEL:app:further-experimental-results.

###  4.1 Substitutivity in Instructions

We investigate how resilient models with multimodal prompts are to
substitutivity Hupkes et al. (2020) by comparing performance on paraphrased
instructions—a meaning-preserving operation—and expect a robust model to
perform similarly to these plausible inputs.333See LABEL:app:paraphases for
implementation details. We also substitute visual referents with natural
language descriptors to explore how models map visuals to the language
embedding space.444We only consider single-object referents, excluding
instructions with scenes/frames. See LABEL:app:textual-details for further
implementation details.

#### Baseline

LABEL:tab:perf-paraphrases.1 shows model performance for each combination of
prompt-conditioning method and visual encoder when trained and evaluated using
the default instructions from .555At this time, Jiang et al. (2023) has not
released a reproducible benchmark to use. Please see LABEL:app:cannot-
reproduce for details on how we remedied this issue. The best-performing
approach uses cross-attention to condition prompts on object-centric
observations, outperforming image patches. Performance for each model is
similar at L1–2 but worsens at L3–4, indicating their inability to generalise
to new objects and tasks.

#### Evaluating on Paraphrases

LABEL:tab:perf-paraphrases.2 shows that models trained on original
instructions are robust to substitutivity in instructions—likely due to the
frozen pretrained language model (Tsimpoukelli et al., 2021). However, models
exhibit a small performance loss with models using Cross-Attn affected the
least. The lack of syntactic or lexical diversity in the inputs, suggests that
the models might overfit to the surface form rather than learning to
generalise to new sentences.

#### Training on Paraphrases

LABEL:tab:perf-paraphrases also shows that training on linguistically diverse
instructions can improve performance for models that use cross-attention to
condition prompts or use object-centric visual features. However, performance
worsens when evaluated on paraphrased instructions. Taken together, this
suggests that training on diverse instructions helps models better connect the
semantics of a multimodal instruction over its surface form, aiding in
generalisation to novel scenes. However, poor performance on L3 shows that
models struggle more with unseen objects. Furthermore, using image patches and
concatenation performs better when trained and evaluated on diverse
instructions over any training/evaluation condition, suggesting that these
architectures are more resilient to unseen objects and unseen tasks allowing
for better generalisation in more complex settings.

#### Replacing Visual Referents With Descriptors

LABEL:tab:perf-obj-text-para shows object-centric models perform comparably
when replacing objects with natural language descriptors, suggesting that
models have learned to map visual features within the language model’s
embedding space (Tsimpoukelli et al., 2021; Driess et al., 2023). Furthermore,
cross-attention outperforms concatenation, indicating it better preserves
relationships between natural language descriptors and visual referents.
Additionally, both models that use image patches perform notably worse on
L1–3. When using image-patches, all visuals provided to the model in the
prompt—be it a single object or a frame—are encoded into a fixed number of
patches, whereas object-centric methods encode one object per token. Due to
the cardinality of this mapping, the former is a much harder task than the
latter.

###  4.2 Perturbations of Instruction Syntax

We introduce two methods to distort the natural language in a multimodal
prompt: and . As shown in LABEL:fig:lang-perturbation-example, each method
removes information from the language modality differently without affecting
the visual referents. preserves the tokenised sequence length, while maintains
the word count but increases the tokenised sequence length (see LABEL:app:gdg
for implementation details). As perturbations are unrealistic, we expect
performance to plummet to near-random chance.666We derive random chance in
LABEL:app:random-chance. Furthermore, while applying perturbations removes
signal from the linguistic channel, it does not remove language from the
instruction. While irrelevant to the task, they are still provided to, and
considered by, the language model. To further investigate the contribution of
each modality, we also compare their individual impact on the overall model
performance.

####  Perturbations

LABEL:tab:perf-gdg-on-para shows both perturbations degrade performance across
architectures, but not to random chance levels, implying that models rely on
other cues to infer tasks, despite nonsensical instructions.

When applying , object-centric features outperform image-patches, regardless
of the prompt-conditioning method used. This implies that object-centric
features provide a stronger signal for models to infer the desired task
without explicit direction. While we would expect similar performance drops
across both perturbation methods, object-centric models exhibit poorer
performance with compared to . Additionally, applying using cross-attention
helps models uncover the right task better at lower levels, whereas cross-
attention with patches struggles more with novel tasks and objects, a possible
indication of overfitting. This problem might arise because the decoder uses
absolute positional embeddings which are well-known to poorly extrapolate to
longer sequences (Press et al., 2022; Sun et al., 2022).

#### Comparing Modalities

We investigate whether the model relies equally on both modalities by masking
one modality at test time—a perturbation which should significantly decrease
performance. LABEL:tab:perf-mask-modalities-on-para shows that across all
models and levels, performance is lower when language tokens are masked versus
when visual referents are masked. We impute this to the nature of
autoregressive sequence modelling: as models are trained on multimodal
instructions that all start with a language token, the model struggles more
when the sequence does not start with a language token. However, as
performance is above random chance, the model is still able to successfully
determine the task.

###  4.3 Are Models Relying on Heuristics?

When provided with incomplete instructions, humans often combine available
information with heuristics to act rationally in the face of uncertainty
(Simon, 1955; Gigerenzer and Goldstein, 1996). Similarly, models may rely on
heuristics—combining any available information with prior knowledge and world
understanding—to infer appropriate actions and complete tasks. Furthermore,
when given the opportunity, models may attempt to recover from mistakes
through trial and error.

#### Models Try To Recover Before the Task Ends

LABEL:tab:perf-no-mistakes-on-para shows object-centric representations
outperform models encoding visuals with image-patches, showing that these
models are better at solving the tasks without any mistakes. However, the
performance across all levels and models is lower compared to the more lenient
time limit (LABEL:tab:perf-paraphrases), indicating that given additional
time, models explore alternative actions, often successfully. While
beneficial, extended time may lead to misleading conclusions when evaluating
model performance under unreasonable conditions or nonsensical instructions,
as models simply have more time to perform suboptimal action sequences that
eventually lead to success. We attribute this behaviour to recovery
demonstrations from (see discussion in LABEL:app:data-not-optimal).

#### Can Models Act Without Instructions?

LABEL:tab:perf-no-instructions-trained-para demonstrates that models can still
perform tasks even when the instruction is entirely removed. However,
concatenation with object-centric visual features exhibits worse performance,
indicating a higher sensitivity to the presence of an instruction, which is a
desirable characteristic. However, performance across each level and model is
greater when allowing the model to recover from mistakes, suggesting that if
not stopped, the model will persistently attempt to solve the task. This
weakness has important implications concerning safety: the model is acting
without a clear direction and is somehow finding the right answer.

Notably, concatenation with object-centric features performs worse than cross-
attention with object-centric features. Although both models employ the same
visual encoding method, they differ in how the instruction is conditioned on
the prompt.

###  4.4 Task Complexity

As each architecture can infer the correct task without instruction, it
implies that rely on cues from observations, as that is the only other source
of input into the model. We test this in two ways: 1) introducing distractors
with the Distracting difficulty level, or 2) increasing task difficulty with
the Extreme difficulty level. Distractors are defined as objects similar to
the target objects in either texture or shape. Additionally, “task difficulty
increases” are scene settings specific to each task. The Extreme level
assesses the extent to which a model relies on object affordances when
reasoning about actions Lohmann et al. (2020). See LABEL:fig:difficulty-
example for an example and LABEL:app:difficulty-levels for details. These new
difficulty levels are plausible, agents should be able to disregard
unnecessary details and focus on objects/aspects most important to the task.
LABEL:tab:perf-difficulty-on-para shows results on our novel evaluation set.

Models using patches likely perform poorly due to their inability to represent
objects in complex scenes. This is a known issue of Transformer-based vision
encoders (Darcet et al., 2024; Pantazopoulos et al., 2024) and recent work has
proposed several solutions to favour suitable object-centric representation
learning (Locatello et al., 2020). While increasing the resolution per patch
or image might help improve performance (Karamcheti et al., 2024; Liu et al.,
2024), it can also lead to an increase in the number of tokens in the decoder,
which can introduce other issues. The changes in the expected affordances of
objects (e.g. objects that were not trained as “containers” are now being used
to place things on top of ) in the Extreme difficulty level have a greater
impact on the patch-based models compared to the object-centric models. This
indicates that the patch-based models are weaker when objects are used in ways
that are not expected, while the object-centric models are more robust to
these adjustments. At the Extremely Distracting difficulty, patch-based models
truly struggle to maintain performance, indicating that they cannot handle
objects having different affordances from what they were trained on, nor can
they handle excessive visual clutter.

###  4.5 Order Permutations

Object-centric models outperform others, but how they succeed without
instruction remains unclear, possibly due to cues from observation encoding.
We explore whether permuting the order of object tokens when provided in the
model’s input affects model performance (see LABEL:fig:shuffle-example for
example permutations). We assume that Transformer-based models using object-
centric tokens should be invariant to order permutations (Carion et al.,
2020).

Instead, as shown in LABEL:tab:perf-eval-shuffle-on-para, we note that
permuting the order of object-centric tokens causes performance on the default
difficulty level to half. Exploring how well models perform without the
opportunity to recover from mistakes halves this result further. This
indicates that when they do not rely on spurious correlations, models try to
recover from mistakes until an episode terminates. Further proof of this is
that performance degrades as the environment becomes more complex: both with
more objects present (Distracting) and when various affordances are not as
expected (Extreme).

Similar to findings from Carion et al. (2020), Transformer-based models are
vulnerable to order permutations. When trained on these permutations, model
performance improves, however, it is not at the same level as LABEL:tab:perf-
paraphrases, suggesting that a considerable proportion of model performance
stems from learned spurious correlations.

##  5 Conclusion

We define an evaluation framework for Embodied AI grounded in the
generalisation framework from Hupkes et al. (2023). Specifically, we assess
generalisation across important axes by means of specific multimodal input
perturbations including paraphrases, replacing visual referents with
descriptors, and manipulating the instruction syntax as well as entire input
modalities. We instantiate this evaluation framework in to assess the
robustness of state-of-the-art models.

Overall, our findings indicate that while substitutivity can lead to
performance gains, language perturbations do not impact performance as
expected. To further explore this effect, we evaluate whether models rely on
heuristics to complete tasks by removing individual modalities. We show that
models perform tasks even without instructions by relying on spurious
correlations within observations, as learned during training. We further prove
this effect by showing that performance decreases when the number of objects
in an environment increases, and agents can no longer randomly perform the
correct sequence of actions.

Taken together, our findings suggest that it is important to define evaluation
frameworks like ours that can assess generalisation across multiple axes in
order to have a more reliable characterisation of the overall model
performance. In future work, we aim to apply this evaluation framework
systematically to other benchmarks as well to discover important architectural
insights that will guide the next generation of Embodied AI models.

## References

  * Akula et al. (2022) Arjun Akula, Spandana Gella, Aishwarya Padmakumar, Mahdi Namazifar, Mohit Bansal, Jesse Thomason, and Dilek Hakkani-Tur. 2022.  ALFRED-L: Investigating the Role of Language for Action Learning in Interactive Visual Environments.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9369–9378, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Anderson et al. (2018) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018.  Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3674–3683.
  * Brohan et al. (2023) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. 2023.  RT-1: Robotics Transformer for Real-World Control at Scale.  In _Robotics: Science and Systems XIX_. Robotics: Science and Systems Foundation.
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020.  End-to-End Object Detection with Transformers.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, _Computer Vision – ECCV 2020_ , volume 12346, pages 213–229. Springer International Publishing, Cham.
  * Chen et al. (2021) Yixin Chen, Qing Li, Deqian Kong, Yik Lun Kei, Song-Chun Zhu, Tao Gao, Yixin Zhu, and Siyuan Huang. 2021.  YouRefIt: Embodied Reference Understanding with Language and Gesture.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 1365–1375, Montreal, QC, Canada. IEEE.
  * Chiyah-Garcia et al. (2023) Javier Chiyah-Garcia, Alessandro Suglia, Arash Eshghi, and Helen Hastie. 2023.  ‘What are you referring to?’ Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchanges.  In _Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 175–182, Prague, Czechia. Association for Computational Linguistics.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling Instruction-Finetuned Language Models.  (arXiv:2210.11416).
  * Darcet et al. (2024) Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024.  Vision Transformers Need Registers.  In _The Twelfth International Conference on Learning Representations_.
  * Das et al. (2018) Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018.  Embodied Question Answering.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 2135–213509.
  * Deng et al. (2019) Eric Deng, Bilge Mutlu, Maja J Mataric, et al. 2019.  Embodiment in socially interactive robots.  _Foundations and Trends® in Robotics_ , 7(4):251–356.
  * Driess et al. (2023) Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. 2023.  PaLM-E: An Embodied Multimodal Language Model.
  * Duan et al. (2022) Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. 2022.  A Survey of Embodied AI: From Simulators to Research Tasks.  _IEEE Transactions on Emerging Topics in Computational Intelligence_ , 6(2):230–244.
  * Duan et al. (2017) Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. 2017.  One-Shot Imitation Learning.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Gadre et al. (2022) Samir Yitzhak Gadre, Kiana Ehsani, Shuran Song, and Roozbeh Mottaghi. 2022.  Continuous Scene Representations for Embodied AI.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 14829–14839, New Orleans, LA, USA. IEEE.
  * Gigerenzer and Goldstein (1996) G. Gigerenzer and D. G. Goldstein. 1996.  Reasoning the fast and frugal way: Models of bounded rationality.  _Psychological Review_ , 103(4):650–669.
  * Gong et al. (2023) Ran Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng, Xiaofeng Gao, Qingyang Wu, Wensi Ai, Ziheng Zhou, Demetri Terzopoulos, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. 2023.  ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes.  In _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 20426–20438.
  * Grauman et al. (2024) Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria Escobar, Cristhian Forigua, Abrham Gebreselasie, Sanjay Haresh, Jing Huang, Md Mohaiminul Islam, Suyog Jain, Rawal Khirodkar, Devansh Kukreja, Kevin J. Liang, Jia-Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh Kumar Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina Gonzalez, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anush Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Mi Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatumininu Oguntola, Xiaqing Pan, Penny Peng, Shraman Pramanick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran Somasundaram, Chenan Song, Audrey Southerland, Masatoshi Tateno, Huiyu Wang, Yuchen Wang, Takuma Yagi, Mingfei Yan, Xitong Yang, Zecheng Yu, Shengxin Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbelaez, Gedas Bertasius, David Crandall, Dima Damen, Jakob Engel, Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C. V. Jawahar, Richard Newcombe, Hyun Soo Park, James M. Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng Shou, and Michael Wray. 2024.  Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives.
  * Hupkes et al. (2020) Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020.  Compositionality Decomposed: How do Neural Networks Generalise? _Journal of Artificial Intelligence Research_ , 67:757–795.
  * Hupkes et al. (2023) Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2023.  A taxonomy and review of generalization research in NLP.  _Nature Machine Intelligence_ , 5(10):1161–1174.
  * Islam et al. (2022) Md Mofijul Islam, Reza Mirzaiee, Alexi Gladstone, Haley Green, and Tariq Iqbal. 2022.  CAESAR: An Embodied Simulator for Generating Multimodal Referring Expression Datasets.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 21001–21015.
  * Jiang et al. (2023) Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. 2023.  VIMA: General Robot Manipulation with Multimodal Prompts.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 14975–15022. PMLR.
  * Karamcheti et al. (2024) Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. 2024.  Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models.
  * Kolve et al. (2017) Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. 2017.  Ai2-thor: An interactive 3d environment for visual ai.  _arXiv preprint arXiv:1712.05474_.
  * Laird et al. (2017) John E. Laird, Kevin Gluck, John Anderson, Kenneth D. Forbus, Odest Chadwicke Jenkins, Christian Lebiere, Dario Salvucci, Matthias Scheutz, Andrea Thomaz, Greg Trafton, Robert E. Wray, Shiwali Mohan, and James R. Kirk. 2017.  Interactive Task Learning.  _IEEE Intelligent Systems_ , 32(4):6–21.
  * Lake et al. (2017) Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017.  Building machines that learn and think like people.  _Behavioral and brain sciences_ , 40:e253.
  * Li et al. (2023) Jiachen Li, Qiaozi Gao, Michael Johnston, Xiaofeng Gao, Xuehai He, Suhaila Shakiah, Hangjie Shi, Reza Ghanadan, and William Yang Wang. 2023.  Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning.  ArXiv:2310.09676 [cs].
  * Liu et al. (2024) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024.  Improved Baselines with Visual Instruction Tuning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 26296–26306.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023.  Visual Instruction Tuning.  In _Thirty-Seventh Conference on Neural Information Processing Systems_.
  * Locatello et al. (2020) Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. 2020.  Object-centric learning with slot attention.  _Advances in neural information processing systems_ , 33:11525–11538.
  * Lohmann et al. (2020) Martin Lohmann, Jordi Salvador, Aniruddha Kembhavi, and Roozbeh Mottaghi. 2020.  Learning About Objects by Learning to Interact with Them.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 3930–3941. Curran Associates, Inc.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled Weight Decay Regularization.  In _International Conference on Learning Representations_.
  * Ma et al. (2024) Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. 2024.  A Survey on Vision-Language-Action Models for Embodied AI.
  * Octo Model Team (2023) Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. 2023.  Octo: An Open-Source Generalist Robot Policy.
  * Open X-Embodiment Collaboration (2024) Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J. Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R. Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martín-Martín, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, and Zipeng Lin. 2024.  Open X-Embodiment: Robotic Learning Datasets and RT-X Models.  (arXiv:2310.08864).
  * Pantazopoulos et al. (2022) George Pantazopoulos, Alessandro Suglia, and Arash Eshghi. 2022.  Combine to Describe: Evaluating Compositional Generalization in Image Captioning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 115–131, Dublin, Ireland. Association for Computational Linguistics.
  * Pantazopoulos et al. (2023) Georgios Pantazopoulos, Malvina Nikandrou, Amit Parekh, Bhathiya Hemanthage, Arash Eshghi, Ioannis Konstas, Verena Rieser, Oliver Lemon, and Alessandro Suglia. 2023.  Multitask Multimodal Prompted Training for Interactive Embodied Task Completion.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 768–789, Singapore. Association for Computational Linguistics.
  * Pantazopoulos et al. (2024) Georgios Pantazopoulos, Alessandro Suglia, Oliver Lemon, and Arash Eshghi. 2024.  Lost in space: Probing fine-grained spatial understanding in vision and language resamplers.  _arXiv preprint arXiv:2404.13594_.
  * Press et al. (2022) Ofir Press, Noah A Smith, and Mike Lewis. 2022.  Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.  In _International Conference on Learning Representations_ , page 25.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Reed et al. (2022) Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. 2022.  A Generalist Agent.  _Transactions on Machine Learning Research_.
  * Shi et al. (2023) Hangjie Shi, Leslie Ball, Govind Thattai, Desheng Zhang, Lucy Hu, Qiaozi Gao, Suhaila Shakiah, Xiaofeng Gao, Aishwarya Padmakumar, Bofei Yang, et al. 2023.  Alexa, play with robot: Introducing the first alexa prize simbot challenge on embodied ai.  _arXiv preprint arXiv:2308.05221_.
  * Shridhar et al. (2022) Mohit Shridhar, Lucas Manuelli, and Dieter Fox. 2022.  CLIPort: What and Where Pathways for Robotic Manipulation.  In _Proceedings of the 5th Conference on Robot Learning_ , pages 894–906. PMLR.
  * Shridhar et al. (2020) Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020.  ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , Seattle, WA, USA. IEEE.
  * Simon (1955) Herbert A. Simon. 1955.  A Behavioral Model of Rational Choice.  _The Quarterly Journal of Economics_ , 69(1):99–118.
  * Song et al. (2024) Yaoxian Song, Penglei Sun, Haoyu Liu, Zhixu Li, Wei Song, Yanghua Xiao, and Xiaofang Zhou. 2024.  Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI.  _IEEE Transactions on Knowledge and Data Engineering_ , pages 1–14.
  * Stone et al. (2023) Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, Chelsea Finn, and Karol Hausman. 2023.  Open-World Object Manipulation using Pre-Trained Vision-Language Models.  In _Proceedings of The 7th Conference on Robot Learning_ , pages 3397–3417. PMLR.
  * Strohmann et al. (2023) Timo Strohmann, Dominik Siemon, Bijan Khosrawi-Rad, and Susanne Robra-Bissantz. 2023.  Toward a design theory for virtual companionship.  _Human–Computer Interaction_ , 38(3-4):194–234.
  * Suglia et al. (2020) Alessandro Suglia, Ioannis Konstas, Andrea Vanzo, Emanuele Bastianelli, Desmond Elliott, Stella Frank, and Oliver Lemon. 2020.  CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7625–7641, Online. Association for Computational Linguistics.
  * Sun et al. (2022) Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2022.  A Length-Extrapolatable Transformer.
  * Takeda et al. (2019) Mizuki Takeda, Yasuhisa Hirata, Yueh-Hsuan Weng, Takahiro Katayama, Yasuhide Mizuta, and Atsushi Koujina. 2019.  Accountable system design architecture for embodied ai: a focus on physical human support robots.  _Advanced Robotics_ , 33(23):1248–1263.
  * Thomason et al. (2020) Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020.  Vision-and-Dialog Navigation.  In _Proceedings of the Conference on Robot Learning_ , pages 394–406. PMLR.
  * Tsimpoukelli et al. (2021) Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. 2021.  Multimodal Few-Shot Learning with Frozen Language Models.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 200–212. Curran Associates, Inc.
  * Winograd (1972) Terry Winograd. 1972.  Understanding Natural Language.  _Cognitive Psychology_ , 3(1):1–191.
  * Zeng et al. (2021) Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. 2021.  Transporter Networks: Rearranging the Visual World for Robotic Manipulation.  In _Proceedings of the 2020 Conference on Robot Learning_ , pages 726–747. PMLR.  ISSN: 2640-3498.
  * Zhang et al. (2024) Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2024.  Multimodal Chain-of-Thought Reasoning in Language Models.  _Transactions on Machine Learning Research_.
  * Zhao et al. (2023) Tony Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. 2023.  Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware.  In _Robotics: Science and Systems XIX_. Robotics: Science and Systems Foundation.
  * Zheng et al. (2022) Kaizhi Zheng, Xiaotong Chen, Odest Jenkins, and Xin Eric Wang. 2022.  VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation.  In _Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Zhu et al. (2023) Wang Zhu, Ishika Singh, Yuan Huang, Robin Jia, and Jesse Thomason. 2023.  Does VLN Pretraining Work with Nonsensical or Irrelevant Instructions? In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_.

##  Appendix A Evaluation Details

##  Appendix B Extensions to VIMA-Bench

In this work, we propose multiple extensions to . In this section, we provide
further analysis and details for each.