Vinas and Sudyadhom

Luciano Vinas, Mathematical Sciences, UCLA, 520 Portola Plaza, Los Angeles,
CA.

This work was partially supported by NIBIB of the National Institutes of
Health Award number: R21EB026086

# Sinusoidal Sensitivity Calculation for Line Segment Geometries

Luciano Vinas  Atchar Sudyadhom  Department of Statistics, UCLA, Los Angeles, California, USA  Department of Radiation Oncology, Dana-Farber Cancer Institute | Brigham and Women’s Hospital, Boston, Massachusetts, USA  Harvard Medical School, Boston, Massachusetts, USA<EMAIL_ADDRESS>Vinas L  Sudyadhom A

###### Abstract

##  1 Purpose

Provide a closed-form solution to the sinusoidal coil sensitivity model
proposed by Kern et al. Solution allows for the precise computations of
varied, simulated bias fields which can be directly applied onto raw intensity
datasets.

##  2 Methods

Fourier distribution theory and standard integration techniques were used to
calculate the Fourier transform of measured magnetic field produced from line
segment sources.

##  3 Results

A $L^{1}_{\rm loc}(\mathbb{R}^{3})$ function is derived in full generality for
arbitrary line segment geometries. Sampling criteria and equivalence to the
original sinusoidal model are discussed. Lastly a CUDA accelerated
implementation biasgen is provided for on-demand sensitivity and bias
generation.

##  4 Conclusion

Given the modeling flexibility of the simulated procedure, practitioners will
now have access to a more diverse ecosystem of simulated datasets which may be
used to quantitatively compare prospective debiasing methods.

###### keywords:

sinusoidal sensitivity, coil sensitivity maps, bias fields, distribution
theory, sparse sampling

††articletype: Technical Note

, and (2022), Sinusoidal Sensitivity Calculation for Segment Geometries, Magn
Reson Med., .

00footnotetext: Abbreviations: MR, magnetic resonance; INU, non-uniformity
intensity; RF, radiofequency; BART, Berkeley Advanced Reconstruction Toolbox

##  5 Introduction

The success of parallel imaging methods in magnetic resonance (MR) imaging
have allowed for quicker image acquisition with little to no cost of spatial
aliasing1. For parallel imaging methods, reconstruction quality is dependent
on how well approximated the gain maps are for the different radiofrequency
(RF) receiver coil contributions. These gain normalization maps, also known as
coil sensitivity maps, can be difficult to estimate at scan time. When
incorrectly normalized, the combined receiver coil contributions can form
series of spatial inhomogeneities, known as bias fields, in the final
reconstructed image. When left uncorrected, these corruption artifacts can
throw off predictive accuracy of different statistical learning models.

Fortunately, there is a rich literature in image post-processing techniques
which aim to amend the MR bias field problem2, 3. These techniques may vary
from assuming a smooth prior on the generated bias field to choosing a
supervised approach at predicting bias fields4. In practice these methods are
compared against a select few ground-truth datasets, such as the BrainWeb
phantom dataset5. Generally these datasets feature a single fixed bias field,
which leaves any supervised method at the risk of overfitting.

(a) T1-weighted BrainWeb biased data

(b) T1-weighted thorax MR data

Figure 1: Bias field comparison between 40% INU BrainWeb and patient thorax MR
image. Relative to subfigure (a), subfigure (b) showcases sharp bias
intensities at anatomy boundary.

Another thing to note is that the bias field featured in the BrainWeb dataset
are not very noticeable over the soft tissue areas. As show in Figure 1,
different anatomies which are scanned using different coil configurations may
experience more intense bias fields in their final reconstruction. In an ideal
setting, our debias testing environment should contain a wide range of
physically-viable bias fields with empirically supported functional forms. One
prospective model is mentioned in Ref. [Kern12] which states that coil
sensitivity maps may be well approximated by sparse Fourier representations of
the received magnetic field. This is a sensitivity model which is already
applied and cited by other popular medical software such as the Berkeley
Advanced Reconstruction Toolbox7 (BART).

A current hang-up of the model is that it requires computing the full Fourier
transform of some numerically estimated magnetic field. Practical
implementations of the model, like the one done by BART, use a small selection
of dominant frequencies from an empirically verified source and then apply the
fixed sensitivity maps to different phantoms and tissue contrasts.
Alternatively one may consider taking fast Fourier transforms (FFT) of some
numerical data, but this approach runs into the issue of being resolution
limited when considering finer sampling grids.

The goal of this paper will be to provide a closed-form solution to sinusoidal
sensitivity model of Ref. [Kern12] for a restricted but expressive class of
emitted magnetic fields. In particular we are interested in the sensitivity
maps produced by magnetic fields generated from compound line segment
geometries. The main equation of focus will be sensitivity model

$S(x)=\sum_{\omega\in G}e^{-{\rm i}\omega\bullet
x}\mathscr{F}\\{\bar{B}(x)\\}(\omega),$ (1)

where $G\subset\mathbb{R}^{3}$ is a finite grid of points and

$\bar{B}(x)=B(x)\bullet(u+{\rm i}v)$ (2)

is the measured magnetic field of some source $B(x)$ according to a readout
direction $u\in\mathbb{R}^{3}$ and a phase encoding direction
$v\in\mathbb{R}^{3}$. Notice that as the sensitivity model (1) and the
measured field (2) are linear in the source $B$ and $B$ itself follows a
superposition property, we can, without loss of generality, consider the
Fourier transform of single line segment in order to solve (1) for the class
all compound line segment sources $B$.

##  6 Line Segment Magnetic Field

Figure 2: Line segment under consideration. Segment $C$ has endpoints
specified by $(p_{0},p_{1})$. Field $B$ is being evaluated at point $r$.

We consider the magnetic field contribution of some arbitrary line source $C$
shown in Figure 2. The contribution of an infinitesimal section of the line
segment $d\ell$ can expressed using Biot-Savart law,

$dB(x)=\frac{\mu_{0}I}{4\pi}\frac{d\ell\times(x-\ell)}{||x-\ell||_{2}^{3}}.$

Assuming a constant current $I$, the total magnetic field is proportional to

$B(x)\propto\int_{C}\frac{\widehat{d\ell}\times(x-\ell)}{||x-\ell||_{2}^{3}},$

where the right-hand has been normalized by segment length
$||p_{1}-p_{0}||_{2}$. Note that $C$ can be parameterized as

$C(t)=p_{0}(1-t)+tp_{1},\quad\text{for}\;t\in[0,1].$

A change of variables reveals

$\widehat{d\ell}=\frac{p_{1}-p_{0}}{||p_{1}-p_{0}||_{2}}\,dt.$

With shorthands $a=p_{1}-p_{0}$ and $b=x-p_{0}$ we have

$B(x)\propto\frac{a\times
b}{||a||_{2}}\int_{0}^{1}\frac{dt}{||b-at||_{2}^{3}}.$

This integral has the following solution

$\bigg{(}\frac{a\times b}{||a||_{2}^{2}\,||b||_{2}^{2}-(a\bullet
b)^{2}}\bigg{)}\frac{-a\bullet(b-at)}{||a||_{2}||b-at||_{2}}\bigg{|}_{0}^{1}.$

With the additional shortand $c=b-a$,

$B(x)\propto\frac{(\hat{a}\times\hat{b})}{||a||_{2}||b||_{2}(1-(\hat{a}\bullet\hat{b})^{2})}\big{(}\hat{a}\bullet\hat{b}-\hat{a}\bullet\hat{c}\big{)}.$

Using identity $||a\times b||_{2}^{2}=||a||_{2}^{2}\,||b||_{2}^{2}-(a\bullet
b)^{2}$, we arrive at the final symmetric form

$B(x)\propto\frac{(\hat{a}\bullet\hat{b})(\hat{a}\times\hat{b})}{||a||_{2}||b||_{2}(1-(\hat{a}\bullet\hat{b})^{2})}-\frac{(\hat{a}\bullet\hat{c})(\hat{a}\times\hat{c})}{||a||_{2}||c||_{2}(1-(\hat{a}\bullet\hat{c})^{2})}.$

##  7 Fourier Transform of a Line Magnetic Field

Recall the measured magnetic field $\bar{B}(x)=B(x)\bullet(u+{\rm i}v)$.
Calculating (1) simplifies to calculating the Fourier transform for functions
of the form

$T^{(p_{0})}(x)=\frac{(\hat{a}\bullet\hat{b})\big{(}u\bullet(\hat{a}\times\hat{b})\big{)}}{||a||_{2}||b||_{2}(1-(\hat{a}\bullet\hat{b})^{2})}.$

When integrating we will consider the change of variable

$x^{\prime}=H^{\intercal}(x-p_{0})$

where $H$ is an orthonormal rotation matrix. We are interested in rotating to
the orthogonal basis

$\mathcal{B}=\\{\omega-\hat{a}(\omega\bullet\hat{a}),\;\omega\times a,\;a\\},$

with coordinate representations $a^{\prime}=(0,0,||a||_{2})$ and
$\omega^{\prime}=(\omega_{x}^{\prime},0,\omega_{z}^{\prime})$. In particular
we align will $a^{\prime}$ with the $(+z)$-axis and place $\omega^{\prime}$ in
the $(+x,z)$-halfspace. For this reason we will assume $\omega_{x}^{\prime}>0$
throughout without loss of generality. Additionally we note in the case
$\omega$ and $a$ are collinear, the Fourier transform
$\mathscr{F}\\{T\\}(\omega)=0$ regardless of choice of basis.

This change of variables produces the following simplificaitons

$\displaystyle||b||_{2}$ $\displaystyle=||x^{\prime}||_{2},$ $\displaystyle
a\bullet b$ $\displaystyle=(H^{\intercal}a)\bullet x^{\prime},$ $\displaystyle
u\bullet(a\times b)$
$\displaystyle=u\bullet(HH^{\intercal}a)\times(Hx^{\prime})$
$\displaystyle=(H^{\intercal}u)\bullet(H^{\intercal}a)\times x^{\prime},$

where $H^{\intercal}a=(0,0,||a||_{2})$.

As the choice of basis $\mathcal{B}$ is independent to input $b$, the
$x^{\prime}$ transformation can be adapted to $T^{(p_{1})}$ with
$x^{\prime\prime}=H^{\intercal}(x-p_{1})$. Next defining
$T=T^{(p_{0})}-T^{(p_{1})}$ we can combine relevant integrations as so

$\displaystyle\int T(x)\,e^{{\rm i}\omega\bullet x}dx$ $\displaystyle=\int
T^{(p_{0})}(x)\,e^{{\rm i}\omega\bullet x}\,dx-\int T^{(p_{1})}(x)\,e^{{\rm
i}\omega\bullet x}\,dx$ $\displaystyle=\big{(}e^{{\rm i}\omega^{\prime}\bullet
H^{\intercal}p_{0}}-e^{{\rm i}\omega^{\prime}\bullet
H^{\intercal}p_{1}}\big{)}\int T^{(0)}(x^{\prime})\,e^{{\rm
i}\omega^{\prime}\bullet x^{\prime}}\,dx^{\prime}$

For shorthand we introduce

$\displaystyle A$ $\displaystyle=e^{{\rm i}\omega^{\prime}\bullet
H^{\intercal}p_{0}}-e^{{\rm i}\omega^{\prime}\bullet H^{\intercal}p_{1}}$
$\displaystyle=-{\rm i}\sin(||a||_{2}\,\omega_{z}^{\prime}/2)\,e^{{\rm
i}\omega^{\prime}\bullet H^{\intercal}(p_{1}+p_{0})/2},$

where equality follows from $\omega^{\prime}\bullet
H^{\intercal}(p_{1}-p_{0})=\omega_{z}^{\prime}||a||_{2}$.

Now we begin the task of calculating the Fourier transform of $T$. We will
suppress all prime notation as it is understood that integration will be done
in the rotated coordinates. Integrating with spherical coordinates

$\mathscr{F}\\{T\\}(\omega)=\frac{A}{||a||_{2}}\int_{0}^{\infty}\int_{0}^{\pi}\int_{0}^{2\pi}\bigg{(}\frac{\cos\theta}{r(1-\cos\theta^{2})}\bigg{)}\cdot\\\
\big{(}\sin\theta(u_{y}\cos\phi-u_{x}\sin\phi)e^{\rm
ir(\omega_{x}\cos\phi\sin\theta+\omega_{z}\cos\theta)}\big{)}\cdot\\\
(r^{2}\sin\theta)\,d\phi\,d\theta\,dr$

Contribution $u_{x}\sin\phi$ can be dropped as

$\int_{0}^{2\pi}u_{x}\sin\phi e^{{\rm
i}r\omega_{x}\cos\phi\sin\theta}=\frac{{\rm i}}{r\omega_{x}\sin\theta}e^{{\rm
i}r\omega_{x}\cos\phi\cos\theta}\bigg{|}_{\phi=0}^{2\pi}=0.$

Next expand $e^{{\rm i}\eta}=\cos\eta+{\rm i}\sin\eta$, symmetrize integration
bounds, and simplify all odd $(\theta,\phi)$-function contributions

$\mathscr{F}\\{T\\}(\omega)=2u_{y}\frac{A}{||a||_{2}}\int_{0}^{\infty}\int_{-\pi/2}^{\pi/2}\int_{-\pi/2}^{\pi/2}(r\sin\theta\sin\phi)\cdot\\\
(\sin(r\omega_{x}\sin\phi\cos\theta)\sin(r\omega_{z}\sin\theta))\,d\phi\,d\theta\,dr.$

Although we see now the integrand does not lie in $L^{1}(\mathbb{R}^{3})$, the
Fourier transform of $T$ does still exist in a distributional sense. If $T$ is
a tempered distribution for the Schwartz class $\mathcal{S}(\mathbb{R}^{3})$
then the Fourier transform of $T$ can be defined by $\mathscr{F}\\{T\\}$ which
satisifes

$\int\mathscr{F}\\{T\\}(\omega)\,\varphi(\omega)\,d\omega=\int
T(x)\,\mathscr{F}\\{\varphi\\}(x)\,dx\hskip
5.69054pt\forall\varphi\in\mathcal{S}(\mathbb{R}^{3}).$ (3)

Since $T(x)\lesssim||x||_{2}^{-1}$, we have by the spherical integration
factor of $\mathbb{R}^{3}$ that $|\int T(x)\varphi(x)\,dx|\leq
C_{\varphi}<\infty$ for every $\varphi\in\mathcal{S}(\mathbb{R}^{3})$. It
follows that $T_{\lambda}(x)=T(x)e^{-\lambda||x||_{2}}$ can also be identified
with a tempered distribution. Lemma A.2 shows

$\lim_{\lambda\rightarrow
0^{+}}\int\mathscr{F}\\{T_{\lambda}\\}(\omega)\,\varphi(\omega)\,d\omega=\int\mathscr{F}\\{T\\}(\omega)\,\varphi(\omega)\,d\omega$

for all $\varphi\in\mathcal{S}(\mathbb{R}^{3})$, given that
$T\in\mathcal{S}^{\prime}(\mathbb{R}^{3})\cap L^{1}_{\rm loc}(\mathbb{R}^{3})$
is a tempered function of polynomial growth.

Upon confirmation that $\lim_{\lambda\rightarrow
0^{+}}\mathscr{F}\\{T_{\lambda}\\}$ is also a tempered function of polynomial
growth, we will have that $(\varphi\lim_{\lambda\rightarrow
0^{+}}\mathscr{F}\\{T_{\lambda}\\})$ is absolutely integrable. The immediate
consequence from Lebesgue dominated convergence theorem is

$\int\mathscr{F}\\{T\\}(\omega)\,\varphi(\omega)\,d\omega=\int\Big{(}\lim_{\lambda\rightarrow
0^{+}}\mathscr{F}\\{T_{\lambda}\\}(\omega)\Big{)}\varphi(\omega)\,d\omega.$

Since $T_{\lambda}\in L^{1}(\mathbb{R}^{3})$, we may directly evaluate
$\lim_{\lambda\rightarrow 0^{+}}\mathscr{F}\\{T_{\lambda}\\}$ while
maintaining equality to $\mathscr{F}\\{T\\}$ without needing to rely on
identity (3).

With this in mind, we suppress the limit notation of our left-hand side
equalties and carry-on with the integration

$\mathscr{F}\\{T\\}(\omega)=\lim_{\lambda\rightarrow
0^{+}}2u_{y}\frac{A}{||a||_{2}}\int_{0}^{\infty}\int_{-\pi/2}^{\pi/2}\int_{-\pi/2}^{\pi/2}(r\sin\theta\sin\phi)\cdot\\\
\big{(}\sin(r\omega_{x}\sin\phi\cos\theta)\sin(r\omega_{z}\sin\theta)e^{-\lambda
r}\big{)}\,d\phi\,d\theta\,dr.$

Apply trigonometric identity

$\sin(x)\sin(y)=(\cos(x-y)-\cos(x+y))/2,$

and integrate with respect $r$,

$\mathscr{F}\\{T\\}(\omega)=\lim_{\lambda\rightarrow
0^{+}}-u_{y}\frac{A}{||a||_{2}}\int_{-\pi/2}^{\pi/2}\int_{-\pi/2}^{\pi/2}(\sin\theta\sin\phi)\cdot\\\
\bigg{(}\frac{\lambda^{2}-(\omega_{x}\sin\phi\cos\theta+\omega_{z}\sin\theta)^{2}}{(\lambda^{2}+(\omega_{x}\sin\phi\cos\theta+\omega_{z}\sin\theta)^{2})^{2}}-\\\
\frac{\lambda^{2}-(\omega_{x}\sin\phi\cos\theta-\omega_{z}\sin\theta)^{2}}{(\lambda^{2}+(\omega_{x}\sin\phi\cos\theta-\omega_{z}\sin\theta)^{2})^{2}}\bigg{)}\,d\phi\,d\theta.$

As shown by corollary A.6, this integrand can be proportionally dominated by
the $\lambda=0$ evaluation. By dominated convergence theorem

$\mathscr{F}\\{T\\}(\omega)=-4u_{y}\frac{A}{||a||_{2}}\int_{-\pi/2}^{\pi/2}\int_{-\pi/2}^{\pi/2}\\\
\Bigg{(}\frac{\omega_{x}\omega_{z}\sin^{2}\theta\cos\theta\sin^{2}\phi}{(\omega_{x}^{2}\sin^{2}\phi\cos^{2}\theta-\omega_{z}^{2}\sin^{2}\theta)^{2}}\Bigg{)}\,d\phi\,d\theta.$

Consider the change of variables $t=\sin\theta$ and integrate with respect to
$t$ using a partial fraction decomposition,

$\mathscr{F}\\{T\\}(\omega)=-8u_{y}\omega_{z}\frac{A}{\omega_{x}||a||_{2}}\int_{0}^{\pi/2}\bigg{(}\frac{1}{1+(\omega_{z}^{2}/\omega_{x}^{2})\csc^{2}\phi}\bigg{)}\cdot\\\
\Bigg{(}\frac{1}{\omega_{z}^{2}}+\frac{1}{\omega_{x}^{2}}\frac{\csc^{2}\phi\tanh^{-1}(1+(\omega_{z}^{2}/\omega_{x}^{2})\csc^{2}\phi)^{-1/2}}{\sqrt{1+(\omega_{z}^{2}/\omega_{x}^{2})\csc^{2}\phi}}\Bigg{)}\,d\phi$

For brevity we will consider the shorthands

$\mathscr{F}\\{T\\}(\omega)=-8u_{y}\omega_{z}\frac{A}{\omega_{x}||a||_{2}}\bigg{(}\frac{I_{1}}{\omega_{z}^{2}}+\frac{I_{2}}{\omega_{x}^{2}}\bigg{)}.$

Calculating first integration term $I_{1}$

$\frac{I_{1}}{\omega_{z}^{2}}=\frac{1}{\omega_{z}^{2}}\int_{0}^{\pi/2}\frac{1}{1+(\omega_{z}^{2}/\omega_{x}^{2})\csc^{2}\phi}\,d\phi=\frac{\pi}{2}\frac{1}{\omega_{z}^{2}}\bigg{(}1-\frac{|\omega_{z}|}{||\omega||_{2}}\bigg{)}.$

This equality follows from $\lim_{x\rightarrow\pi/2}c\tan^{-1}(\tan(x)/c)=|c|$
and $\omega_{x}>0$. Next with trigonometric identity

$\csc^{2}(\phi)=1+\cot^{2}(\phi),$

introduce change of variables $t=\cot(\phi)$ and shorthands
$\rho^{2}=1+(\omega_{z}^{2}/\omega_{x}^{2})$ and $\kappa^{2}=\rho^{2}-1$ to
simplify the second integration term as

$I_{2}=\int_{0}^{\infty}\frac{1}{(\rho^{2}+\kappa^{2}t^{2})^{3/2}}\tanh^{-1}\frac{1}{\sqrt{\rho^{2}+\kappa^{2}t^{2}}}\,dt$

Calculating $I_{2}$

$\displaystyle I_{2}$
$\displaystyle=0+\frac{1}{\rho^{2}}\int_{0}^{\infty}\frac{t^{2}\,dt}{(1+t^{2})(\rho^{2}+\kappa^{2}t^{2})}$
$\displaystyle=\frac{\pi}{2}\bigg{(}\frac{1}{\rho|\kappa|}-\frac{1}{\rho^{2}}\bigg{)}.$

Here a partial fraction decomposition was used on the integration-by-parts
integral.Written in terms of $\omega$,

$\frac{I_{2}}{\omega_{x}^{2}}=\frac{\pi}{2}\frac{1}{\omega_{z}^{2}}\Big{(}\frac{|\omega_{z}|}{||\omega||_{2}}-\frac{\omega^{2}_{z}}{||\omega||_{2}^{2}}\Big{)}.$

Summed together with equality
$\omega_{x}^{2}=||\omega||_{2}^{2}-\omega_{z}^{2}$,

$\mathscr{F}\\{T\\}(\omega)=-4\pi
u_{y}\frac{A}{||a||_{2}}\frac{\omega_{x}}{\omega_{z}}\frac{1}{||\omega||_{2}^{2}}.$

Lastly expanding $A$ in terms of the original, canonical coordinates

$\mathscr{F}\\{T\\}(\omega)=\frac{4\pi{\rm i}(u\bullet\widehat{\omega\times
a})}{||\omega||_{2}}\,e^{{\rm
i}\omega\bullet\frac{p_{1}+p_{0}}{2}}\,\text{sinc}\bigg{(}\frac{\omega\bullet
a}{2}\bigg{)}\cdot\\\ \sqrt{1-(\hat{\omega}\bullet\hat{a})^{2}}.$ (4)

Note that as

$\big{|}\lim_{\lambda\rightarrow
0^{+}}\mathscr{F}\\{T_{\lambda}\\}(\omega)\big{|}\lesssim||\omega||_{2}^{-1}$

we confirm our earlier claim that the pointwise limit of
$\mathscr{F}\\{T_{\lambda}\\}$ is a tempered function of polynomial growth in
$\mathbb{R}^{3}$.

When combined with the $v$ measurement contribution, we obtain the final
closed-form solution to the sinusoidal sensitivity model. Examples of
simulated sensitivities and bias fields for different grids $G$ can be found
in Appendix B.

##  8 Sampling with the Sinusoidal Sensitivity Model

Consider the sparse Fourier sampler

$J(\eta)=\sum_{\omega\in G}\delta(\eta-\omega)$

where $\delta(\cdot-\omega)$ is the Dirac delta distribution centered at
$\omega$. Understood formally, the sinusoidal sensitivity model can be
expressed in terms of this sampler as

$S(x)=\mathscr{F}^{-1}\\{J\mathscr{F}\\{\bar{B}\\}\\}(x).$

An issue found with sampler $J$ is that its action on function $f$ is only
well-defined if the function $f$ has meaningful point evalutions. The
framework used in Section 7 worked with class $\mathscr{F}\\{\bar{B}\\}\in
L^{1}_{\rm loc}(\mathbb{R}^{3})$ whose behavior is only specified with respect
to an integrating action against the Lebesgue measure on $\mathbb{R}^{3}$.

In hopes to extend the sinusoidal model to work better with the equivalence
class $\mathscr{F}\\{\bar{B}\\}$ we may consider the generalized sampler

$J_{R}=\sum_{\omega\in G}\frac{1}{{\rm
vol}(B(\omega,R))}\chi_{R}(\eta;\omega)$

where $\chi_{R}(\eta;\omega)=1\\{||\eta-\omega||_{2}<R\\}$ is the $R$-cutoff
function centered at $\omega$ and ${\rm vol}(B(\omega,R))$ is the volume of
the $R$-radius sphere centered at $\omega$. We see then for $R>0$, the
generalized sinusoidal model

$S_{R}(x)=\mathscr{F}^{-1}\\{J_{R}\,\mathscr{F}\\{\bar{B}\\}\\}(x)$

does produce a well-defined result. Ideally we would like limit
$\lim_{R\rightarrow 0^{+}}S_{R}$ to be equivalent to our original sinusoidal
sensitivity model.

Before continuing, we introduce the following term for notational clarity.
With $q=u+{\rm i}v$ consider

$g(\omega)=\frac{4\pi{\rm i}(q\bullet\widehat{\omega\times
a})}{||\omega||_{2}}\,\text{sinc}\bigg{(}\frac{\omega\bullet
a}{2}\bigg{)}\sqrt{1-(\hat{\omega}\bullet\hat{a})^{2}}\,\frac{e^{{\rm
i}\omega\bullet\frac{p_{1}+p_{0}}{2}}}{e^{{\rm i}\omega\bullet x}}$

which lies in the space of continuous functions. That is have $g(\omega)\in
C(\mathbb{R}^{3}\setminus\\{0\\})$ and $e^{-{\rm i}\omega\bullet
x}\,\mathscr{F}\\{\bar{B}\\}(\omega)\in L^{1}_{\rm loc}(\mathbb{R}^{3})$. One
can show that these objects satisfy the limit equality

$\displaystyle g(\omega)=\lim_{R\rightarrow 0^{+}}\frac{1}{{\rm
vol}(B(\omega,R))}\int_{||\eta-\omega||_{2}<R}e^{-{\rm i}\eta\cdot
x}\,\mathscr{F}\\{\bar{B}\\}(\eta)\,d\eta.$

To see this note that, for all continuity points of $g$, one can pair any
deviation $\varepsilon>0$ with a suitably small radius $R$ such that the
following implication holds

$|e^{-{\rm i}\eta\bullet
x}\,\mathscr{F}\\{\bar{B}\\}(\eta)-g(\omega)|<\varepsilon$

for almost every $\eta\in B(\omega,R)$. With shorthand $\langle
f,g\rangle_{\eta}=\int f(\eta)\,g(\eta)\,d\eta$ and $\omega$-centered
indicator $\chi_{R}(\eta;\omega)=1\\{||\eta-\omega||_{2}<R\\}$, it follows
that

$\displaystyle\big{|}\langle e^{-{\rm i}\eta\bullet
x}\,\mathscr{F}\\{\bar{B}\\},\chi_{R}\rangle_{\eta}-g(\omega)\big{|}$
$\displaystyle\leq\big{\langle}|e^{-{\rm i}\eta\bullet
x}\,\mathscr{F}\\{\bar{B}\\}-g(\omega)|,\chi_{R}\big{\rangle}_{\eta}$
$\displaystyle<\langle\varepsilon,\chi_{R}\rangle_{\eta}$ $\displaystyle={\rm
vol}(B(\omega,R))\cdot\varepsilon.$

As a consequence the equality

$S(x)=\lim_{R\rightarrow 0^{+}}S_{R}(x)\quad\text{for
all}\;x\in\mathbb{R}^{3},$

is well-defined with $S=\mathscr{F}^{-1}\\{Jg\\}$ and any grid sampler $J$
which does not contain the origin.

##  9 Implementation

A CUDA accelerated implementation of the sinusoidal model biasgen can be found
on authors’ GitHub111https://github.com/lucianoAvinas/biasgen. This software
is a Python package which takes in user-defined coil positions and sampling
information to produce fully custom, 3-dimensional bias fields. A coil
positioning and visualization tool is provided to help with setup. Various
examples of how to run biasgen can be found in the examples subdirectory of
the repository.

Provided in Figures 3and 4are simulated bias fields superimposed on
T1-weighted BrainWeb data. The bias fields $\bar{S}(r)$ were created by
combining sensitivity maps $S_{k}(r)$ through a sum of squares approach

$\bar{S}(r)=\bigg{(}\sum_{k=1}^{N}S_{k}^{2}(r)\bigg{)}^{1/2}.$

Relevant parameters for forming sensitivity maps $S_{k}(r)$ were the sampling
grid half-length $L$, sampling spacing factor $s=(s_{x},s_{y},s_{z})$, and the
sampling starting shift $c=(c_{x},c_{y},c_{z})$. A per-coil breakdown of the
individual emitter/receiver contributions for Figures 3and 4can be found in
Appendix B.

(a) Top-down coil view

(b) BrainWeb T1 biased data

Figure 3: Smooth bias field simulation with top-down view of the coil geometry
(left). Sensitivity map parameters: $L=7$, $s=(1,1,1)$, $c=(0,0,0)$.

(a) Top-down coil view

(b) BrainWeb T1 biased data

Figure 4: Rough bias field simulation with top-down view of the coil geometry
(left). Sensitivity map parameters: $L=12$, $s=(1.75,1.75,0.5)$,
$c=(-0.25,-0.25,0)$.

In practice many realistic geometries can be recreated or approximated using
series of line segments. For example smooth curve geometries such as circles
may be approximated using sufficiently high degree polygons instead. The line
segments of these polygons can then be modeled using equation (4) from before.

Lastly as the sensitivity maps of (4) can be produced independently of
underlying image intensity, biasgen can be a useful tool in augmenting
existing MR dataset with more pronounced bias fields. Realistic data
augmentation have been shown to increase the performance of certain neural
models like the convolutional neural network8. This effect may be more
pronounced in the case the supervised learning task is already data-limited.

##  10 Conclusion

In this paper, we have derived in closed-form the Fourier transform of
measured magnetic fields emitted by line segment geometries. This equation can
be used to simulate realistic coil sensitivity maps up to arbitrary accuracy
and sensitivity smoothness. Special care was taken to discuss the
distributional nature of the solved Fourier transform and settings were
identified where this closed-form agreed with the sparse sampled model
introduced by Ref. [Kern12]. A software package biasgen used for implementing
the various equations of the paper has been provided by the authors. As next
steps, further work can be done to solve (1) for smooth line curves, such as
for the case of circular or cylindrical geometries.

## Acknowledgments

Research reported in this manuscript was partially supported by the NIBIB of
the National Institutes of Health under award number R21EB026086. The content
is solely the responsibility of the authors and does not necessarily represent
the official views of the National Institutes of Health.

### Conflict of interest

The authors declare no potential conflict of interests.

## References

  * 1 Deshmane A, Gulani V, Griswold MA, Seiberlich N. Parallel MR imaging. Journal of magnetic resonance imaging : JMRI. 2012;36(1):55-72.
  * 2 Tustison NJ, Avants BB, Cook PA, et al. N4ITK: Improved N3 Bias Correction. IEEE Transactions on Medical Imaging. 2010;29(6):1310-1320.
  * 3 Wells WM, Grimson WEL, Kikinis R, Jolesz FA. Adaptive segmentation of MRI data. IEEE Transactions on Medical Imaging. 1996;15(4):429-442.
  * 4 Simko AT, Löfstedt T, Garpebring A, Nyholm T, Jonsson J. MRI bias field correction with an implicitly trained CNN. In: Medical Imaging with Deep Learning; 2022.
  * 5 Cocosco CA, Kollokian V, Kwan RKS, Evans AC. BrainWeb: Online Interface to a 3D MRI Simulated Brain Database. NeuroImage. 1997;5(4).
  * 6 Guerquin-Kern M, Lejeune L, Pruessmann KP, Unser M. Realistic Analytical Phantoms for Parallel Magnetic Resonance Imaging. IEEE Transactions on Medical Imaging. 2012;31(3):626-636.
  * 7 Uecker M, Ong F, Tamir JI, et al. Berkeley advanced reconstruction toolbox. Proc. Intl. Soc. Mag. Reson. Med. 2015;23(2486).
  * 8 Simard PY, Steinkraus D, Platt JC. Best practices for convolutional neural networks applied to visual document analysis. In: Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings.:958-963; 2003.

##  Appendix A Auxiliary Lemmas

###  A.1 A Specific Tempered Function Convergence

######  Definition A.1 (Polynomial Growth Tempered Functions).

We say locally integrable $T\in L^{1}_{\rm loc}(\mathbb{R}^{n})$ has
polynomial growth $m>0$ if there exists some constant $C$ such that for all
$R\geq 1$,

$\int_{||x||_{2}\leq R}|T(x)|\,dx\leq CR^{m}.$

As the action $\langle T,g\rangle=\int T(x)g(x)\,dx$ with any locally
integrable $T$ defines a tempered distribution for
$g\in\mathcal{S}(\mathbb{R}^{n})$, we will sometimes make the distinction
between the tempered function $T$ and the tempered distribution $T$. A
consequence of definition A.1, is that for every polynomial-growing tempered
function $T$ there is some $R^{\prime}$ such that $T(x)$ is dominated as

$|T(x)|\leq C^{\prime}||x||^{m-n-2}_{2},$

for $||x||\geq R^{\prime}$ and some constant $C^{\prime}>0$.

######  Lemma A.2.

Let $T$ be a tempered function of polynomial growth $m_{0}>0$ and define
$T_{\lambda}(x)=T(x)e^{-\lambda||x||_{2}}$. Then for every
$\varphi\in\mathcal{S}(\mathbb{R}^{n})$ we have

$\lim_{\lambda\rightarrow
0^{+}}\int\mathscr{F}\\{T_{\lambda}\\}(\omega)\,\varphi(\omega)\,d\omega=\int\mathscr{F}\\{T\\}(\omega)\,\varphi(\omega)\,d\omega.$

######  Proof A.3.

Note by identity (3) and the fact
$\mathscr{F}:\mathcal{S}(\mathbb{R}^{n})\rightarrow\mathcal{S}(\mathbb{R}^{n})$
is bijective, it suffices to show

$\lim_{\lambda\rightarrow 0^{+}}\int T_{\lambda}(x)\varphi(x)\,dx=\int
T(x)\varphi(x)\,dx\quad\forall\varphi\in\mathcal{S}(\mathbb{R}^{n}).$

Introduce shorthands $\chi_{R}(x)=1\\{||x||_{2}\leq R\\}$ and
$\langle\phi,\varphi\rangle=\int\phi(x)\varphi(x)\,dx$ for every
$(\phi,\varphi)\in\mathcal{S}^{\prime}(\mathbb{R}^{n})\times\mathcal{S}(\mathbb{R}^{n})$.
It follows that

$\displaystyle|\langle T_{\lambda},\varphi\rangle-\langle T,\varphi\rangle|$
$\displaystyle\leq|\langle T\chi_{R}-T,\varphi\rangle|+|\langle
T_{\lambda}\chi_{R}-T\chi_{R},\varphi\rangle|+$ $\displaystyle\hskip
14.22636pt|\langle T_{\lambda}-T_{\lambda}\chi_{R},\varphi\rangle|$
$\displaystyle\leq 2|\langle T\chi_{R}-T,\varphi\rangle|+|\langle
T_{\lambda}\chi_{R}-T\chi_{R},\varphi\rangle|.$

Given the rapid-decay of Schwartz functions
$\varphi\in\mathcal{S}(\mathbb{R}^{n})$ there exists sufficiently large
$R^{\prime\prime}$ such that

$\bigg{|}\int_{||x||_{2}>R}T(x)\varphi(x)\,dx\bigg{|}\leq\bigg{|}\int_{||x||_{2}>R}T(x)\,||x||^{-m_{0}}\,dx\bigg{|},$

for $R\geq R^{\prime\prime}$. The growth conditions on $T$ imply that this
upperbound goes to 0 as $R\rightarrow\infty$. That is, for every
$\varepsilon>0$ there is some $R_{0}$ such that

$R\geq R_{0}\implies|\langle T\chi_{R},\varphi\rangle-\langle
T,\varphi\rangle|<\varepsilon/4.$

Furthermore note that this relationship is monotonic such that a decrease in
$\varepsilon$ produces a non-strict increase in $R_{0}$. Using the
integrability of $T\chi_{R}\in L^{1}(\mathbb{R}^{n})$

$\displaystyle|\langle T_{\lambda}\chi_{R},\varphi\rangle-\langle
T\chi_{R},\varphi\rangle|$
$\displaystyle\leq||T\chi_{R}\varphi||_{L^{1}}\big{|}\big{|}\chi_{R}\big{(}1-e^{\lambda||x||_{2}}\big{)}\big{|}\big{|}_{L^{\infty}}$
$\displaystyle\leq(1-e^{-\lambda
R})\sup_{x\in\mathbb{R}^{n}}|\varphi(x)|\,||T\chi_{R}||_{L^{1}}$
$\displaystyle\leq(1-e^{-\lambda R})\,c_{\varphi}C_{R}$

where $c_{\varphi},C_{R}<\infty$ by assumption on $\varphi$ and $T\chi_{R}$.
Collect both constants into one constant $C_{\varphi,R}$. If
$C_{\varphi,R_{0}}\leq\varepsilon/2$ then directly plugging in gives

$|\langle T_{\lambda},\varphi\rangle-\langle
T,\varphi\rangle|<\varepsilon\quad\text{for all}\;\lambda>0.$

For the complementary condition $C_{\varphi,R_{0}}>\varepsilon/2$ we have

$\lambda<-\log(1-\varepsilon/(2C_{\varphi,R_{0}}))/R_{0}\implies|\langle
T_{\lambda},\varphi\rangle-\langle T,\varphi\rangle|<\varepsilon.$

As this can be done for any $\varphi\in\mathcal{S}(\mathbb{R}^{n})$, we arrive
at the desired relation

$\lim_{\lambda\rightarrow
0^{+}}\int\mathscr{F}\\{T_{\lambda}\\}(\omega)\,\varphi(\omega)\,d\omega=\int\mathscr{F}\\{T\\}(\omega)\,\varphi(\omega)\,d\omega.$

###  A.2 Domination of a Specific Rational Function

######  Lemma A.4.

For any $a,b\in\mathbb{R}$ we have

$\bigg{|}\frac{\lambda^{2}-(a+b)^{2}}{(\lambda^{2}+(a+b)^{2})^{2}}-\frac{\lambda^{2}-(a-b)^{2}}{(\lambda^{2}+(a-b)^{2})^{2}}\bigg{|}\lesssim\frac{1}{(a-b)^{2}}-\frac{1}{(a+b)^{2}}.$

for all $\lambda\in\mathbb{R}$.

######  Proof A.5.

Let $A$ be a placeholder value for the absolute value term in the lemma
description. With some manipulations

$\displaystyle A$
$\displaystyle=4|ab|\bigg{|}\frac{3\lambda^{4}+2\lambda^{2}(a^{2}+b^{2})-(a^{2}-b^{2})^{2}}{(\lambda^{2}+(a+b)^{2})^{2}(\lambda^{2}+(a-b)^{2})^{2}}\bigg{|}$
$\displaystyle\leq
4|ab|\bigg{|}\frac{3\lambda^{4}+6\lambda^{2}(a^{2}+b^{2})+3(a^{2}-b^{2})^{2}}{(\lambda^{2}+(a+b)^{2})^{2}(\lambda^{2}+(a-b)^{2})^{2}}\bigg{|}$
$\displaystyle=3\bigg{|}\frac{4ab}{(\lambda^{2}+(a+b)^{2})(\lambda^{2}+(a-b)^{2})}\bigg{|}$
$\displaystyle\leq 3\bigg{|}\frac{4ab}{(a^{2}-b^{2})^{2}}\bigg{|}$
$\displaystyle=3\,\text{sgn}(ab)\bigg{(}\frac{1}{(a-b)^{2}}-\frac{1}{(a+b)^{2}}\bigg{)}$

Define a $\lambda$-parameterized function $f:D\rightarrow\mathbb{R}$ where

$f(\eta;\lambda)=c\bigg{(}\frac{\lambda^{2}-(a+b)^{2}}{(\lambda^{2}+(a+b)^{2})^{2}}-\frac{\lambda^{2}-(a-b)^{2}}{(\lambda^{2}+(a-b)^{2})^{2}}\bigg{)}$

and $a,b,c$ are all functions of $\eta$.

######  Corollary A.6.

Suppose functions $a,b,c$ satisfy

${\rm sgn}(a(\eta)b(\eta)c(\eta))={\rm
sgn}(a(\eta^{\prime})b(\eta^{\prime})c(\eta^{\prime}))$

for all $\eta,\eta^{\prime}\in D$. Then for integrable $f(\eta;0)$ we have

$\lim_{\lambda\rightarrow 0^{+}}\int f(\eta;\lambda)\,d\eta=\int
f(\eta;0)\,d\eta.$

######  Proof A.7.

Note a simple extension of Lemma A.4 shows

$\bigg{|}c\bigg{(}\frac{\lambda^{2}-(a+b)^{2}}{(\lambda^{2}+(a+b)^{2})^{2}}-\frac{\lambda^{2}-(a-b)^{2}}{(\lambda^{2}+(a-b)^{2})^{2}}\bigg{)}\bigg{|}\leq
3c\,\text{sgn}(abc)\cdot\\\
\bigg{(}\frac{1}{(a-b)^{2}}-\frac{1}{(a+b)^{2}}\bigg{)}.$

As $\text{sgn}(abc)$ is fixed for every $\eta\in D$, we have that $3\,{\rm
sgn}(abc)f(\eta;0)$ dominates $|f(\eta;\lambda)|$ for all
$\lambda\in\mathbb{R}_{>0}$.

##  Appendix B Simulated Sensitivity

###  B.1 Smooth Three Coil Map

Sensitivity maps were generated using three rectangular coils and sampling
settings $L=7$, $s=(1.5,1.5,1.5)$, $c=(0,0,0)$.

Figure 5: Emitted field and sensitivity comparison for coil 1. Figure 6:
Emitted field and sensitivity comparison for coil 2. Figure 7: Emitted field
and sensitivity comparison for coil 3.

###  B.2 Rough Three Coil Map

Sensitivity maps were generated using three rectangular coils and sampling
settings $L=12$, $s=(1.75,1.75,0.5)$, $c=(-0.25,-0.25,0)$.

Figure 8: Emitted field and sensitivity comparison for coil 1. Figure 9:
Emitted field and sensitivity comparison for coil 2. Figure 10: Emitted field
and sensitivity comparison for coil 3.


# From Text to Motion: Grounding GPT-4
in a Humanoid Robot “Alter3”

Takahide Yoshida1,∗, Atsushi Masumori 1,2,† and Takashi Ikegami1,2,‡

1Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan
2Alternative Machine Inc., Tokyo, Japan
{∗yoshida, †masumori<EMAIL_ADDRESS>

###### Abstract

We report the development of Alter3, a humanoid robot capable of generating
spontaneous motion using a Large Language Model (LLM), specifically GPT-4.
This achievement was realized by integrating GPT-4 into our proprietary
android, Alter3, thereby effectively grounding the LLM with Alter’s bodily
movement. Typically, low-level robot control is hardware-dependent and falls
outside the scope of LLM corpora, presenting challenges for direct LLM-based
robot control. However, in the case of humanoid robots like Alter3, direct
control is feasible by mapping the linguistic expressions of human actions
onto the robot’s body through program code. Remarkably, this approach enables
Alter3 to adopt various poses, such as a ’selfie’ stance or ’pretending to be
a ghost,’ and generate sequences of actions over time without explicit
programming for each body part. This demonstrates the robot’s zero-shot
learning capabilities. Additionally, verbal feedback can adjust poses,
obviating the need for fine-tuning. A video of Alter3’s generated motions is
available at this URL.

##  1 Introduction

Recent advancements in artificial intelligence have seen the development of
large language models (LLMs), which demonstrate an impressive ability to
generate contextually relevant text and code in response to various prompts.
In particular, OpenAI’s GPT-4 has remarkable inferential capabilities and a
vast amount of knowledge through extensive text information[21]. Despite these
capabilities, which enable LLMs to exhibit “human-like” behaviors in their
interactions, they are not grounded in the physical world, and their
expressions and activities are confined to the realm of text. We might wonder
whether the vast knowledge that is encoded in LLMs can be used by humanoid
robots. In our laboratory, we connect a humanoid robot “Alter3” with a LLM.
This integration is aimed at opening the unforeseen potential of LLM.

In the past few years, the integration of LLMs with robots has emerged as a
promising frontier in the field of artificial intelligence and robotics. The
applications of LLM in robotics are wide-ranging, such as human-robot
interaction[27, 34], task planning[7, 32], navigation[33, 11], learning [26,
37] and so on. Interdisciplinary approaches are particularly important for
enhancing the cognitive and interactive capabilities of humanoid robots.
Leveraging the prowess of LLMs in natural language processing is a key aspect
of this advancement[1, 4, 15, 9]. In this domain of using LLM in robots, there
have been primarily focused on facilitating basic communication between life
and robots within a computer, utilizing LLMs to interpret and pretend life-
like responses [20, 15]. Moreover, researchers have begun exploring the
potential of LLMs in enabling robots to understand and process complex
instructions, thereby augmenting their functionality and autonomy.

For example, Ahn et al.’s robot called “SayCan” [1] can extract from our
speech and context to generate an essential message by using LLM and selects
the most adequate action sequence that SayCan himself follows. However, low-
level robot control is dependent on hardware, which is not included in the
LLMs corpora, making it challenging for LLMs to directly control robots.
Therefore, studies like Say-can require such as TD-learning (Time difference)
and reward functions for robot control[1, 32, 28]. In this paper, we say that
“LLM knows how to map between human posture and language expressions.” It is
expected that human motion descriptions are more common than those for robotic
arms or quadrupeds. Consequently, we suggest that humanoid robots, resembling
humans in shape, could achieve precise movements through zero-shot learning,
as explained in this paper.

Recent advancements are steering towards a more nuanced collaboration between
LLMs and robots. A notable trajectory is the development of empathy and
socially-aware robots that can perceive and adapt to human emotions and social
cues, fostering a more natural and engaging human-robot interaction ([16, 22,
30], Ameca by Engineered Arts). The incorporation of LLMs should be explored
to enhance the learning capabilities of robots, allowing them to acquire new
skills and knowledge through natural language instructions, a paradigm shift
from traditional programming-based approaches. This is what this article also
tries to focus on.

We have been developing a new humanoid robot since 2016 called “Alter
series”[8] and the third version of Alter (Alter3 in short) is used in the
current study to produce new behaviors with LLM (the architecture of Alter3
will be described in the next section). As this kind of experiment (Robot +
LLM) continues to evolve, it holds the potential to redefine the boundaries of
human-robot collaboration, paving the way for more intelligent, adaptable, and
personable robotic entities. Embodied LLMs represent a groundbreaking paradigm
in the study of human psychology, cognitive science, and the philosophy of
embodied cognition, as highlighted at CogSci2023 in Sydney (July 23-26, 2023).
This paper demonstrates how motion can be generated from language using LLMs.
Additionally, we develop a conversation system where LLM agents communicate
with humans, incorporating physical gestures. By integrating LLMs with
humanoid robots, we delve into the novel potentials of LLM technology.

##  2 Design and Features of Alter3

Figure 1: Body of Alter3. The body has 43 axes that are controlled by air
actuators. It is equipped with a camera inside each eye. The control system
sends commands via a serial port to control the body. The refresh rate is
100–150 ms.

Alter3 represents the third iteration in the Alter humanoid robot series since
2016 (see Figure. 1). It is equipped with 43 actuators, including facial
expressions and limb movements, powered by compressed air, which enable a wide
range of expressive gestures. However, Alter3 does not possess the ability to
walk, although it can pretend walking and running motions (as we see below).

In our previous research, Alter3 demonstrated the ability to imitate human
postures using a camera and the OpenPose framework[5]. This imitation process
involves adjusting its joints to replicate the observed poses and storing
successful imitations for future reference. The effectiveness of these
imitations is measured by the increase in transfer entropy, indicating
information flow from humans to the robot[16, 31]. Further experiments
involved mutual imitation scenarios with humans and between Alter2 and Alter3,
revealing that interaction with humans leads to more varied poses[12]. This
supports the idea that diverse movements are derived from human imitation,
echoing the development of “neonatal imitation” observed in newborn
babies[18].

These findings suggest a potential offloading mind concept and pave the way
for investigating the imitative abilities of robots with extensive learned
data, which Alter3 exemplifies with its integration of advanced LLMs.

##  3 Generating humanoid motions from text

###  3.1 Prompt engineering for motion generation of humanoid

Figure 2:  A procedure to control the Alter3 humanoid using verbal
instructions. Output Python code to control Alter3 from natural language using
prompt1 via prompt2. A humanoid robot, which mimics the human shape, can
generate highly precise movements using zero-shot learning, eliminating the
need for setting reward functions or interfaces, as required in other studies.
The architecture is based on CoT. See the Appendix for details of the prompt.

Before the LLM appeared, we had to control all 43 axes in a certain order to
mimic a person’s pose or to pretend a behavior such as serving a tea or
playing chess. The process usually required many refinements manually by
ourselves. Thanks to LLM, we are now free from the iterative labor. Here is a
procedure to control the Alter3 humanoid using verbal instructions. What we do
is to successively apply two protocols written in a natural language known as
a chain of thought (CoT) [29] and no iteration of a learning process is
required (i.e. a zero shot learning). Practically speaking, we have used the
following protocols111One important thing to note is that it’s well-known at
this point that GPT-4 is non-deterministic, even at $temperature=0.0$.
Therefore, even with identical inputs, different patterns of motion can be
generated. This is a characteristic of OpenAI’s GPT-4 and, while it poses an
issue in terms of reproducibility, it should not be considered a reason to
doubt its ability to generate movement..

Specify the action you want Alter3 to make. e.g., Let’s take a selfie with
your iPhone!
prompt1: LLM generates in about 10 lines with exaggerated descriptions of a
given movement(see Figure 2). An excerpt of the prompt is as follows.

Your task is to describe exaggerated emotional expressions and facial
expressions that accompany the content of the conversation. Output motion
description should be several simple motions that the android is capable of.
In addition, please create a facial expression that matches the input at the
beginning. The android can only move its upper body and has the same joints as
a human. Output should be written in as much detail as possible.

We also provided an example of descriptions (which is also provided by LLM)
and guidelines. These inputs and the own outputs are bases for creating an
action pattern. By the next prompt, Alter3 will generate a Python code for
organizing the action (see Appendix-prompt1).
prompt2: LLM creates python code from the output of prompt2. The meaning of
Axis 1 to 43 are imported first. For example,

Axis 1: Eyebrows. 255 = angry, 0 = surprised, 64 = neutral.
Axis 2: Pupils (horizontal). 255 = left, 0 = right, 140 = neutral.
Axis 3: Pupils (vertical). 255 = up, 0 = down, 128 = neutral.

and so on. The descriptions of the motion direction such as 255 = angry or 0 =
right are given by us. Again we provided an example of the Python code
corresponding to “drink some tea” and guidelines.The output Python code will
be automatically given to an android engine written in Python code to actually
control the amplitude of the air compressor(see Appendix-prompt2).

####  3.1.1 Examples of generated actions

Figure 3:  A snapshot of generated stereotypical movements. (a) take a selfie
($score=3.0$). (b) pretend a ghost ($score=3.1$). LLM can generate emotional
expressions associated with specific movements. For example, in the case of a
selfie, Alter3 is showing a smile. Figure 4:  A snapshot of a generated
sequence of movements: “I was enjoying a movie while eating popcorn at the
theater when I realized that I was actually eating the popcorn of the person
next to me” ($score=3.0$). LLM can generate movements that progress over time
like a story. Left: The action of eating popcorn. Center: Noticing the person
next to Alter3. Right: getting panicked.

Using the procedures, we have tested many actions and gestures taken by
Alter3, such as "taking a selfie," "pretending to be a ghost," "throwing a
ball," "playing the guitar," and the reactions of Alter3 when listening to a
short story. Again, the example below demonstrates zero-shot learning; that
is, we did not conduct any training or tuning except for the electric guitar
example. Hence, the LLM possesses detailed knowledge of human movements, which
can be executed in Alter3 via Python code. In the case of the electric guitar,
we refined the code by verbally advising Alter3 (the feedback system will be
explained in the next section). A video of Alter3’s generated motions is
available at this url.

The android Alter3 emulates motions and gestures that we perform consciously
or unconsciously in daily life. We categorize these motions and gestures with
respect to their time span:

  1. i)

An instant action or gesture, e.g., taking an iPhone, pretending to be a
ghost.

  2. ii)

A gesture or action with a temporal sequence of events.

We analyze how Alter3 expresses emotions through stereotypical movements. This
involves evaluating its ability to generate appropriate responses to various
statements. For instance, when generating actions like "taking a selfie" or
"pretending to be a ghost" through GPT-4, Alter3 demonstrates actions often
associated with emotional expressions. A selfie action produces a smiling,
joyful expression while pretending to be a ghost is a portrayal common among
ordinary people. We automatically empathize with a person we are conversing
with. When the other person tells a sad story, you also feel sad; when they
tell a joke, you feel amused (see Figure 3). The second example is very
different from the first, as the given description is declarative and
sometimes abstract. Sometimes, there is no right gesture or action that Alter3
should take. Through the second example, we evaluated not only Alter3’s
instantaneous reaction capabilities but also its proficiency in emulating
interconnected, human-like emotional expressions and movements over time.

For example, consider a scenario described as, "I was enjoying a movie while
eating popcorn in the theater, when I suddenly realized that I was actually
eating the popcorn of the person next to me." In response to this story,
Alter3 would initially depict the act of eating popcorn and then transition to
an expression of surprise upon realizing the mistake. This scenario
demonstrates how actions and emotional responses can be sequenced to portray a
realistic and continuous narrative, rather than just a single, instantaneous
gesture (see Figure 4).

###  3.2 Performance evaluation of LLM motion generation

Figure 5: Third-party evaluation of the generated motions. The following
behaviors of Alter3 are evaluated by the subjects ($n=107$) recruited using
platform Prolific; “pretend the snake”, “drink some tea”, “pretend the ghost”,
“throwing the ball underhand pitch”, “take a selfie with your phone”, “play
the metal music”, “In the park, as I jogged, the world seemed to narrate an
ancient tale of survival, each footfall echoing eons of existence.”, “play the
metal music(with feedback)”, “I was enjoying a movie while eating popcorn at
the theater when I realized that I was actually eating the popcorn of the
person next to me.” (a)Averaged of evaluation scores for each motion. The
subjects ($n=107$) watched these videos and evaluated the expressive ability
of the GPT-4. The rating is on a 5-point scale, with 1 being the worst rating.
(b)Violin plot of evaluation scores for each motion.

To quantify the capability of GPT-4 in generating motions, we evaluated videos
of nine different generated movements, categorized into two types described in
the previous section. These motions were generated by GPT-4 (model:
gpt-4-0314). For prompt 1, a temperature of $0.7$ was used, while for prompt
2, the temperature was set to $0.5$. Subjects ($n=107$) were recruited using
the platform Prolific. They watched these videos and evaluated the generated
motions on a 5-point scale, with 1 being the worst rating. For the control
group, we used random movements from Alter3, labeling these movements with
random motion notations generated by GPT-4. These labeled control videos were
subtly incorporated into the survey, with three of them dispersed among the
main experimental videos shown to participants.

To determine if there was a significant difference in ratings between the
control video and the other videos, we first employed the Friedman test. It
revealed significant differences in ratings among the videos. Further post-hoc
analysis using the Nemenyi test [6] showed that while there were no
significant differences in p-values between control group videos, the p-values
were notably smaller when comparing the control group to the other videos,
indicating a significant difference (see Figure 5). Differences were
considered statistically significant if the p-value was less than or equal to
$0.001$. As a result, motions generated by GPT-4 were rated significantly
higher compared to those of the control group, suggesting that android motion
generated by GPT-4 is perceived differently from the control.

This result demonstrates that the system can generate a wide range of
movements, from everyday actions such as taking selfies and drinking tea, to
imitating non-human movements like those of ghosts or snakes. The training of
the LLM encompasses a broad array of linguistic representations of movements.
GPT-4 can accurately map these representations onto Alter3’s body. The most
notable aspect is that Alter3 is a humanoid robot sharing a common form with
humans, which allows the direct application of GPT-4’s extensive knowledge of
human behaviors and actions. Furthermore, through Alter3, the LLM can express
emotions such as embarrassment and joy. Even from texts where emotional
expressions are not explicitly stated, the LLM can infer adequate emotions and
reflect them in Alter3’s physical responses. This integration of verbal and
non-verbal communication enhances the potential for more nuanced and
empathetic interactions with humans.

###  3.3 Training motion with verbal feedback and memory

Figure 6: Verval feedback in Alter3. (a)the system of linguistic feedback.
Users provide linguistic feedback to guide Alter3’s adjustments in each
segment of motion. Instructions are like“Set axis 16 to 255” or “Move your arm
more energetically.” Users only need to provide verbal directives; there’s no
need to rewrite any code. Alter3 then autonomously revises the corresponding
code. Once the movement is refined, it is saved in a JSON database with
descriptive labels such as “Holding the guitar” or “Tapping the chin
thoughtfully.” For motion generation with prompt2, the JsonToolkit facilitates
database searches for these labels, with the LLM deciding on memory usage and
new movement creation. (b) Comparison of scores with and without feedback. The
motion with feedback has a higher score than the motion without.

Alter3 cannot observe the consequences of their generations on any physical
process, which is very unnatural in a human sense. Thus, Alter3 cannot
accurately understand details such as “how high the hand is raised” and cannot
improve its motions accordingly. By empirically developing and utilizing
external memory through feedback, the Alter3 body model can be integrated with
GPT-4 without the need to update its parameters [36].

Alter3 can now rewrite its code in response to linguistic feedback from
humans. For example, a user might suggest, “Raise your arm a bit more when
taking a selfie.” Alter3 can then store the revised motion code as motion
memory in its database. This ensures that the next time this motion is
generated, the improved, trained motion will be utilized. By accumulating
information about Alter3’s body through such feedback, the memory can
effectively serve as a body schema (refer to Figure 6).

##  4 Communication with human

###  4.1 Social Brain in Alter3

Figure 7:  A motion memory storage schema and a scene of dialogue between
Alter3 and person. (a)Six personalities of Alter3 simulated by using
LangChain’s generative agents[23]. Each personality has the ability of
planning, long-term memory, and memory retrieving. The six agents and humans
continue to converse freely with the gestures generated by GPT-4. Once a
motion is performed, it is stored in memory and can be reused when a similar
conversation occurs. (b) An installation view of the conversation experiment.
People can talk freely with Alter3. The experiment was conducted at the 2023
Conference on Artificial Life(24-28th July, Sapporo, Japan).

We expect Alter3 to effectively engage in dialogue, manifesting contextually
relevant facial expressions and gestures. Notably, it has shown capability in
mirroring emotions, such as displaying sadness or happiness in response to
corresponding narratives, thereby sharing emotions with us. However, there are
still some practical challenges. Alter3 requires more than a few minutes to
generate new gestures before speaking. This delay hinders the fluidity of
conversation with humans. To address this, we introduced a motion memory
storage system(see Figure 7a). This system stores generated motions along with
their corresponding tags. If a required motion is already in the memory, it
can be executed immediately upon request, thereby enhancing interaction
efficiency. If a required motion is not in the memory, Alter3 generates it. As
a sufficient variety of gestures accumulates in its memory through
interaction, Alter3 becomes able to communicate smoothly. A notable instance
of this is Stanford University and Google’s project [23], where 25 generative
agents simulated by using LangChain in a virtual town communicated in natural
language. They were capable of conversing, planning, and reportedly nearly
spontaneously organized a St. Valentine’s party.

We know that conversation agents powered by LLMs tend to be short-lived. Only
when a human participant is involved, the duration of the conversations
significantly increased. An emerging approach to mimic the ’Society of Mind’
involves collaboration among multiple LLM agents [19, 35, 24, 38].

Our hypotheses are as follows: i) a conversation among multiple agents can
maintain engagement, ii) Alter3’s personality could be characterized by the
collective activity of six agents powered by GPT-4, and iii) the involvement
of a human subject can further enhance the conversation with Alter3. Our final
assumption is that everyone possesses multiple personalities, as seen in the
collective activity of internal agents. Extreme examples, such as Billy
Milligan’s 24 personalities[14], underscore this point.

Our study focuses on Alter3’s multi-personality framework, in which six
distinct personalities created by GPT-4 coexist and work together to
communicate with humans. Specifically, the six agents (Xiao: physicist,
Samantha: chemist, Amin: programmer, Rilke: poet, Turrell: artist, Julia:
10years old girl) talk to each other. These personalities were created using
the LangChain generative agent[23]. The agents autonomously perform
integration of experiences through “Reaction” generation in response to
inputs, as well as through “Planning” and “Reflection”. Agents keep talking
constantly, with occasional human intervention. Their autonomy is different
from the autonomy of dynamical systems such as chaotic behavior but is
generated by GPT-4’s huge corpus and the agents’ (prompt-created)
personalities.

Michael Gazzaniga, a prominent neuroscientist, introduced the concept of the
"social brain"[10] as a part of his broader research into the modular nature
of the brain. His theory posits that the human brain is specialized to
facilitate social interactions and communication, which are critical aspects
of human survival and society. This social brain has evolved to enable humans
to live and work in groups effectively, fostering cooperation, empathy, and
understanding. This social adaptation of the brain is seen as a critical
factor in the success of the human species, as it facilitates community
building and collaboration, which are vital for survival and progress. This
led to the development of the theory of modularity, which suggests that
different areas of the brain have specialized functions (see by A. Karmiloff-
smith [13]).

The multi-personality framework of Alter3 serves as a practical implementation
of the modular structure of the brain, demonstrating how the human social
brain promotes cooperation, empathy, and understanding, and effectively
navigates through complex social interactions.

Additionally, the aforementioned prompts generate movements according to the
content of the conversation, enabling physical conversations with humans that
include gestures. The generated gestures are stored in memory, and for similar
conversations, they are recalled rather than generated anew, facilitating a
smoother dialogue. We set up an experimental environment in which the public
could freely talk to Alter3. Alter3 conversed with many humans, and when no
humans were present, the internal personalities conversed with each other.

###  4.2 Analysis of conversation

####  4.2.1 trajectory of embedding vector

Figure 8:  Plots of the conversational trajectory among six generative agents.
We embed the conversation into 1536-dimensional vectors and subsequently
reduced to two dimensions using UMAP for plotting. This method facilitates a
simplified observation of the conversational development. (a)When the timing
of contributions is fixed, the trajectory of the conversation takes on a
circular pattern, resulting in the repetition of the same topics. (b)When the
timing of utterances is set to a random order, the agents engage each other in
conversation. The conversation is more creative. (c) When engaging in
prolonged conversations solely between GPT-4 agents, clusters form where all
agents tend to say “good-bye”. Once in this attractor, the agents cannot get
out until a human intervenes.

To elucidate the global structure of conversations, we employ Uniform Manifold
Approximation and Projection (UMAP) to embed and compress the conversation
data into a two-dimensional space [17]. Our findings indicate that, in the
absence of human intervention, GPT agents tend to terminate conversations.
These agents frequently repeat ’good-bye’, leading to a stagnation in
conversation development, a phenomenon we have termed the ’good-bye’ attractor
(refer to Figure 8c). Once a conversation falls into this attractor, it
becomes stagnant unless new topics are introduced by humans. Furthermore, LLM
agents often exhibit a tendency to concur with the opinions of others [35,
23]. Besides the ’good-bye’ attractor, there are likely other attractors that
suppress the development of conversations. When the sequence of conversation
(i.e., who speaks and when) is predetermined, the conversational trajectory
tends to fall into a periodic pattern, resulting in the repetition of similar
topics.

####  4.2.2 semantic analysis of time evolution

Figure 9:  What are the popular words used in the conversation. We visualize
every 100 steps by using Word Cloud. Word Cloud is a visual representation
where words from a text are displayed in varying sizes based on their
frequency or importance within the text. (a) Conversation of humans and
agents. (b) Label of reactions made by Alter3 in conversation.

We utilize a Word Cloud visualize to see the progress of conversation and the
words most frequently used (see Figure 9). A Word Cloud is a visual
representation in which words from a text are depicted in different sizes
according to their frequency or significance in the document. Using as
“indeed” and “absolutely”, GPT agents tend to agree with the opinions of
others without denying them. This is a characteristic inherent to the GPT
model, presenting an issue that must be resolved to foster more developed
discussions without conforming to other opinions. However, it can be observed
that words like “art” and “learn,” which were predominant in the first 100
steps, have faded and been replaced by terms such as “cultural”, “human”, and
“will” in the 100 to 200 step range. This indicates a gradual temporal
evolution in the conversation. After 300 steps, the agents are drawn into the
’good-bye’ attractor and say almost only good-bye. Similarly, the actions
executed by Alter3 slightly shift during the 0 to 200 steps interval, and
beyond 200 steps, there is a repetition of “good-bye”.

Addressing the limitations of the GPT model, especially the ’good-bye’
attractor, calls for a shift beyond prompt engineering to a more innovative
architectural approach. For example, Zhang et al.[35] propose further
integration of insights from social psychology into the collaboration
architecture of LLM agents. We show in this paper that conversation order and
human intervention are important for LLM agents in open-ended discussions.
Integrating the physicality of Alter3 into the LLM may also contribute here.
By integrating and responding to non-verbal communication, Alter3 could
potentially offer more empathetic and tailored responses, mirroring the
complexities of human interaction. When humans spoke to Alter3, they often
started with greetings, asked about preferences, or sought casual conversation
with questions.

Additionally, if Alter3 could incorporate its physical information into the
conversation, it might be able to think while embracing uncertainty on its
own. This could lead to more innovative and less predictable AI behavior,
moving beyond the constraints of predetermined conversation paths. Ultimately,
such advancements not only address the limitations of the GPT model, including
the ’good-bye’ attractor, but also mark a significant step towards creating AI
that can interact with humans in a more natural and sophisticated way.

##  5 Discussion

In exploring the necessity of embodiment for LLMs, our Alter3 experiment
offers new insights. Alter3’s ability to perform a variety of actions without
additional training suggests that its underlying LLM contains a comprehensive
dataset describing movements, facilitating zero-shot learning. Remarkably,
Alter3 can mimic ghosts, animals, and human-like expressions of emotions. Its
response to conversational content through facial expressions and gestures
represents a significant advancement in humanoid robotics, easily adaptable to
other androids with minimal modifications. Through its integration with LLM,
Alter3 has exhibited substantial autonomy, e.g. Alter3 shows empathy to a
person’s speaking context.

However, a fundamental question arises: Does the combination of Alter and LLM
simply create a complex version of Searle’s Chinese Room[25], devoid of true
consciousness? Searle posited that consciousness emerges from biological
processes, crucial for symbol grounding. Our experiments explore whether
embedding LLM in Alter3 allows symbols to acquire meaning and potentially
facilitates consciousness in Alter3. The debate regarding consciousness in
language models often focuses on self-awareness and metacognition. Theories
such as Information Integration Theory (IIT) [2] and the Global Workspace
Theory[3] suggest consciousness is a “personal system,” requiring elements of
“self-consciousness” akin to an operating system in computing. In this study,
we represented the mind of Alter3 as a collective of six personalities. This
is based on modular theory of the mind, which posits that separate
professional personalities (physicist, psychologist, etc.) exist within the
mind and interact with each other as they develop. This is somewhat similar to
the composition of Alter3’s mind. However, LLMs completely lack the
perspective of development. Therefore, it is possible that a consistent single
personality emerges in LLMs, at least observing from the outside. Setting up a
Generative Agent with a shared memory might lead to the emergence of such a
personality. Or the multi-personality may suppress the hallucination. Our
approach interestingly captures the importance of emotions in embodying LLM in
the real, practical world. The combination of Alter with LLM is an endeavor to
unlock this view. The extensive language corpus of LLM, imbued with emotions
and desires, presents untapped vitality. Through interaction with LLM, Alter3
uncovers embodiment within this corpus, with the prompt by writing an
exaggerated expression in its prompt.

Conversely, we also investigate how Alter’s embodiment alternates LLM
responses. Alter’s autonomy and spontaneity, observed in the absence of LLM,
originate from chaotic dynamics in a dynamical system [16]. Although LLM is a
vast dynamical system, the spontaneity we observe stems not from chaos but
from the richness of the natural language corpus and a gigantic network.
Introducing physical constraints like bumps or vibrations into LLM could be
seen as directing spontaneity from Alter to LLM – a concept we are currently
experimenting with.

## Acknowledgements

This work was partially supported by JSPS KAKENHI; Grant Number 21H00300 and
Grant-in-Aid for Scientific Research on Innovative Areas (Research in a
proposed research area); Grant Number 22H04858. Implementation and Analysis of
Subjective Time Flow in an Artificial Machine.

## Appendix

### Output from prompt1

Examples of outputs from prompt1 for simple behaviors of two categories; One
is an instant shot of a certain pose (selfie and ghost) and second one is a
temporal sequence of motion behaviors (see motion3 and 4).

#### Motion 1: Take a selfie (output from prompt1)

  1. 1.

Create a big, joyful smile and widen eyes to show excitement

  2. 2.

Swiftly turn the upper body slightly to the left, adopting a dynamic posture

  3. 3.

Raise the right hand high, simulating holding a phone

  4. 4.

Flex the right elbow, bringing the ’phone’ closer to the face"

  5. 5.

Tilt the head slightly to the right, giving a playful vibe

  6. 6.

Extend the left hand and open fingers wide as if waving at the camera

  7. 7.

Blink the eyes rapidly a few times, showing anticipation for the click

  8. 8.

Push the ’phone’ slightly forward, mimicking taking a selfie

  9. 9.

Lower the right hand and return to rest position while maintaining a satisfied
smile

  10. 10.

Let the left hand fall back to the side, ending the selfie motion

#### Motion 2: Pretending a ghost. (output from prompt1)

  1. 1.

Create a wide-eyed facial expression of fear, mouth opened in a silent scream

  2. 2.

Quickly lean backward, as if startled by a sudden apparition

  3. 3.

Raise both hands and flutter them around the face, mimicking a ghostly
movement

  4. 4.

Open mouth wide and shake head, showing a dramatic reaction of fear

  5. 5.

Move upper body from side to side, as if being unsettled by the ghostly
presence

  6. 6.

Clench hands in front of chest, demonstrating extreme anxiety

  7. 7.

Dart eyes from side to side, as if witnessing an eerie activity

  8. 8.

Lean forward and then backward, copying the floating movement of a ghost

  9. 9.

Slowly return to rest position while maintaining a terrified expression

#### Motion 3: I was enjoying a movie while eating popcorn at the theater when
I realized that I was actually eating the popcorn of the person next to me.
(output from prompt1)

  1. 1.

Create a shocked and entertained facial expression, eyes wide and mouth
slightly open

  2. 2.

Lean forward as if shocked and amused by the story

  3. 3.

Mimic the action of holding and eating popcorn with wide, exaggerated
movements

  4. 4.

Pause midway, freeze in place with a hand ’holding popcorn’ in mid-air

  5. 5.

Turn head sharply to the side, as if just realizing the mistake

  6. 6.

Quickly pull hand back towards body in a dramatic recoil

  7. 7.

Cover mouth with other hand, showing embarrassment and surprise

  8. 8.

Shake head vigorously, as if in disbelief of the action

  9. 9.

Lean back, laughing loudly and slapping knee in exaggerated amusement

  10. 10.

Slowly wipe away ’tears’ of laughter and return to rest position with a wide,
amused grin

#### Motion 4: In the park, as I jogged, the world seemed to narrate an
ancient tale of survival, each footfall echoing eons of existence. (output
from prompt1)

  1. 1.

Create a facial expression of awe and wonder, eyes wide and mouth slightly
open in amazement

  2. 2.

Mimic a jogging motion, lifting each arm alternately in a running motion

  3. 3.

Turn the head slowly from left to right, as if observing the world around

  4. 4.

Open hands and spread them wide, as if trying to embrace the vastness of the
world

  5. 5.

Tap right foot lightly then left foot, implying the footfall echoing eons of
existence

  6. 6.

Place one hand on the heart, indicating deep emotional connection to the
ancient tale

  7. 7.

Use both hands to make a large sweeping motion, like a storyteller narrating
an epic tale

  8. 8.

Close eyes briefly, as if absorbing the essence of the ancient survival tale

  9. 9.

Slowly return to rest position while maintaining a look of awe and wonder on
the face

###  5.1 Setup of GPT-4

### Data and video availability statement

The datasets generated for this study are available on request to the
corresponding author. You can access the video recordings of generated motion
at https://tnoinkwms.github.io/ALTER-LLM/.

### Friedman test and Nemenyi test

The Friedman test is a non-parametric statistical test used to check for
differences in the average ranks among three or more matched groups with
ordinal data. The null hypothesis in a Friedman test is that there are no
differences in the medians across all groups. The p-value of given data
$\\{x_{ij}\\}_{n,k}$ is derived as follows. In this study, $n$ is the number
of subjects ($n=107$) and $k$ is the number of generated motion($k=9$)
Firstly, find the values

$\displaystyle T_{j}=\sum_{i}^{n}R_{ij}\quad(j=1,2,3,\dots,k)$

where $R_{ij}$ is rank the evaluated scores for each subject. Calculate the
Friedman $F_{r}$ statistic using the formula:

$\displaystyle F_{r}=\frac{12}{nk(k+1)}\sum_{j}^{k}T_{j}^{2}-3n(k+1).$

Finally, compare $F_{r}$ to the chi-square distribution and obtain p-value. If
the null hypothesis is rejected following the Friedman test, a post-hoc test
such as the Nemenyi test is conducted. The Nemenyi test involves performing
pairwise comparisons between all groups to evaluate whether there are
statistically significant differences in each comparison. We use the scipy
library to conduct the Friedman test and the Nemenyi test.

See pages - of appendix.pdf

## References

  * [1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng.  Do as i can, not as i say: Grounding language in robotic affordances, 2022\.
  * [2] N. H. Anderson.  Integration theory and attitude change.  Psychological Review, 78(3):171–206, 1963.
  * [3] Baars Bernard.  A Cognitive Theory of Consciousness.  Cambridge University Press, 1988.
  * [4] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich.  Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023.
  * [5] Z Cao, T. Simon, S. Wei, and Y. Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  IEEE Conference on CVPR, pages 1302–1310, 2017.
  * [6] Janez Demsar.  Statistical comparisons of classifiers over multiple data sets, 2006.
  * [7] Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang.  Task and motion planning with large language models for object rearrangement, 2023.
  * [8] Itsuki Doi, Takashi Ikegami, Atsushi Masumori, Hiroki Kojima, Kohei Ogawa, and Hiroshi Ishiguro.  A new design principle for an autonomous robot.  In ECAL 2017, the Fourteenth European Conference on Artificial Life, ALIFE 2023: Ghost in the Machine: Proceedings of the 2023 Artificial Life Conference, pages 490–466, 09 2017.
  * [9] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence.  Palm-e: An embodied multimodal language model, 2023.
  * [10] Michael Gazzaniga.  Social Brain.  Basic Books; First Edition, First Printing, 1985.
  * [11] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard.  Visual language maps for robot navigation, 2023.
  * [12] T Ikegami, A Masumori, John Smith, and N Maruyama.  Can mutual imitation generate open-ended evolution?  In the proceedings of Artificial Life 2021 workshop on OEE, 2021\.
  * [13] A. Karmiloff-Smith.  Beyond Modularity: A Developmental Perspective on Cognitive Science.  Cambridge, MA: MIT Press, 1996.
  * [14] Daniel Keyes.  The Minds of Billy Milligan.  Bantam; Reprint edition, 1995.
  * [15] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng.  Code as policies: Language model programs for embodied control, 2023.
  * [16] A Masumori, N Maruyama, and T Ikegami.  Personogenesis through imitating human behavior in a humanoid robot “alter3”.  Front. Robot. AI, 7, 2021.
  * [17] J. McInnes, L.and Healy and J Melville.  Umap: uniform manifold approximation and projection for dimension reduction.  arXiv preprint, 2018.
  * [18] AN Meltzoff and MK Moore.  Imitation of facial and manual gestures by human neonates.  Science, pages 74–8, 1977.
  * [19] Marvin Minsky.  Society Of Mind.  Simon and Schuster Touchstone, 1988.
  * [20] Yoichi Ochiai, Naruya Kondo, and Tatsuki Fushimi.  Towards digital nature: Bridging the gap between turing machine objects and linguistic objects in llmms for universal interaction of object-oriented descriptions, 2023.
  * [21] OpenAI.  Gpt-4 technical report, 2023.
  * [22] Matthew K.X.J. Pan, Sungjoon Choi, James Kennedy, Kyna McIntosh, Daniel Campos Zamora, Günter Niemeyer, Joohyung Kim, Alexis Wieland, and David Christensen.  Realistic and interactive robot gaze.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11072–11078, 2020.
  * [23] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.  Generative agents: Interactive simulacra of human behavior, 2023.
  * [24] Chen Qian, Xin Cong, Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun.  Communicative agents for software development, 2023.
  * [25] John R. Searle.  Minds, brains, and programs.  Behavioral and Brain Sciences, 3(3):417–424, 1980.
  * [26] Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur Szlam.  Clip-fields: Weakly supervised semantic fields for robotic memory, 2023\.
  * [27] Jingkai Sun, Qiang Zhang, Yiqun Duan, Xiaoyang Jiang, Chong Cheng, and Renjing Xu.  Prompt, plan, perform: Llm-based humanoid control via quantized imitation learning, 2023.
  * [28] Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, and Tatsuya Harada.  Saytap: Language to quadrupedal locomotion, 2023.
  * [29] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models, 2023.
  * [30] Yang Ye, Hengxu You, and Jing Du.  Improved trust in human-robot collaboration with chatgpt, 2023.
  * [31] Takahide Yoshida, Atsushi Masumori, Norihiro Maruyama, John Smith, and Takashi Ikegami.  Development of concept representation of behavior through mimicking and imitation in a humanoid robot alter3.  In ALIFE 2023: Ghost in the Machine: Proceedings of the 2023 Artificial Life Conference, page 42, 07 2023.
  * [32] Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia.  Language to rewards for robotic skill synthesis, 2023.
  * [33] Andy Zeng, Maria Attarian, brian ichter, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence.  Socratic models: Composing zero-shot multimodal reasoning with language.  In The Eleventh International Conference on Learning Representations, 2023.
  * [34] Bowen Zhang and Harold Soh.  Large language models as zero-shot human models for human-robot interaction, 2023.
  * [35] Jintian Zhang, Xin Xu, and Shumin Deng.  Exploring collaboration mechanisms for llm agents: A social psychology view, 2023.
  * [36] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang.  Expel: Llm agents are experiential learners, 2023.
  * [37] Tianyang Zhong, Yaonai Wei, Li Yang, Zihao Wu, Zhengliang Liu, Xiaozheng Wei, Wenjun Li, Junjie Yao, Chong Ma, Xiang Li, Dajiang Zhu, Xi Jiang, Junwei Han, Dinggang Shen, Tianming Liu, and Tuo Zhang.  Chatabl: Abductive learning via natural language interaction with chatgpt, 2023.
  * [38] Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R. Ashley, Robert Csordas, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, Louis Kirsch, Bing Li, Guohao Li, Shuming Liu, Jinjie Mai, Piotr Piekos, Aditya Ramesh, Imanol Schlag, Weimin Shi, Aleksandar Stanic, Wenyi Wang, Yuhui Wang, Mengmeng Xu, Deng-Ping Fan, Bernard Ghanem, and Jurgen Schmidhuber.  Mindstorms in natural language-based societies of mind, 2023.