# Comparative analysis of the Compton ionization of Hydrogen and Positronium.

Ivan S. Stepantsov1<EMAIL_ADDRESS>(corresponding author) Igor P.
Volobuev2, Yuri V. Popov2,3 1 Physics Faculty, Lomonosov Moscow State
University, Moscow, Russia
2Skobeltsyn Institute of Nuclear Physics, Lomonosov Moscow State University,
Moscow, Russia
3Joint Institute for Nuclear Research, Dubna, Russia

###### Abstract

The paper deals with the Compton disintegration of positronium and a
comparison of the differential cross sections of this process with the similar
cross sections in the case of the Compton ionization of the hydrogen atom.
Special attention is paid to the resonances arising, when the electron and
positron move parallel to each other in continuum states with the same
velocities and zero relative momentum. It is likely that a manifestation of
this effect in the double differential cross section is found as an additional
peak, which grows with a decrease in the photon energy.

## 1\. INTRODUCTION

Traditionally, we believe that the hydrogen atom is the simplest quantum
system, about which everything is known and which is used as a basic target
for the study of various atomic reactions: excitation, ionization, capture,
photoprocesses, etc. It is really so, but there is one more simple quantum
object with the same properties, this is the positronium (the so-called exotic
atom). The singlet parapositronium state lives 0.12 ns, the triplet
orthopositronium state lives 138.6 ns. These values can be calculated
theoretically AB . A hydrogen atom lives on average 0.1 - 10 ns in an excited
state, although highly excited states can exist in vacuum for up to several
seconds. In this sense the lifetime of different forms of positronium is quite
comparable with the lifetime of an excited hydrogen atom, which means that
these atoms can be experimentally studied with the same devices. Beyond the
lifetime, the main difference between hydrogen and positronium is the mass of
the positively charged particles: the proton mass is $M=1836.4$ a.u., whereas
the mass of the positron (equal to that of the electron) is $m=1$ a.u., and
this difference results in significant differences in the scattering cross
sections.

In paper EPJD20 , the Compton ionization of a hydrogen atom by a high-energy
photon of several keV was discussed in detail, and interesting features of
various differential cross sections were noted. This paper was inspired by the
latest experiments on the Compton ionization of helium atoms using the
COLTRIMS detector Nature20 . A characteristic feature of these experiments is
the measurement in coincidence of the momenta of an electron and the residual
ion in the range of angles practically equal to $4\pi$, whereas the angle and
energy of the scattered photons are calculated from the conservation laws. The
motivation behind the paper was the possibility to use such a reaction for the
purpose of direct studies of the momentum distribution of an active electron
in an atom JQSRT .

The experiments on Compton scattering at atoms without measuring the final
photon are pioneering. Back in the early 1920s, following the discovery of the
Compton effect, W. Bothe proposed to use the Compton ionization of a bound
electron by a photon to study its momentum distribution in the target Bothe .
For this purpose, he developed and implemented a scheme for measuring in
coincidence the momenta of the final photon and electron. This scheme was
imperfect, the accuracy was low, and mainly the photons scattered forward were
measured. Therefore, predominantly solid targets were used. Quite different
possibilities for working with atoms and molecules were opened by the COLTRIMS
detector, which makes the work with a cold positronium beam not a fantasy.

In a sense, this paper is a continuation of paper EPJD20 , and here we will
compare some cross sections of the Compton ionization of the hydrogen atom (H)
and positronium (Ps). In particular, we will discuss in detail an interesting
effect that occurs, when, after the positronium decay, the electron-positron
pair moves with zero relative momentum.

We will use the atomic system of units: $m_{e}=\hbar=|e|=1$. In these units,
the speed of light is $c=137$ and the fine structure constant $\alpha=1/c$.

## 2\. THEORY

The non-relativistic Hamiltonian of a positronium atom in the electromagnetic
field of a laser has the following form (particle 1 is an electron, particle 2
is a positron):

$H=\frac{1}{2}\left(-i\vec{\nabla}_{r_{1}}-\frac{1}{c}\vec{A}(\vec{r}_{1},t)\right)^{2}+\frac{1}{2}\left(-i\vec{\nabla}_{r_{2}}+\frac{1}{c}\vec{A}(\vec{r}_{2},t)\right)^{2}+V(\vec{r}_{1}-\vec{r}_{2})$
$None$

In (1) $V=-1/|\vec{r}_{1}-\vec{r}_{2}|$ is the Coulomb potential of the
interaction between an electron and a positron, $\vec{A}$ is the vector
potential of the electromagnetic field. The Coulomb gauge
$\vec{\nabla}_{r}\cdot\vec{A}(\vec{r},t)=0$ is used

We put

$\frac{1}{c}\vec{A}(\vec{r},t)=\sqrt{\frac{2\pi}{\omega_{i}}}\ \vec{e}_{i}\
e^{i(\vec{k}_{i}\vec{r}-\omega_{i}t)}+\sqrt{\frac{2\pi}{\omega_{f}}}\
\vec{e}_{f}\ e^{-i(\vec{k}_{f}\vec{r}-\omega_{f}t)}+(c.c.).$ $None$

In Eq. (2) $\vec{e}_{i}(\vec{e}_{f})$ are the linear polarizations of the
initial (final) photons, $\vec{k}_{i}(\vec{k}_{f})$ denote their momenta, and
the frequency (energy) of the photon is $\omega=kc$. Since
$(\vec{k}_{i}\cdot\vec{e}_{i})=(\vec{k}_{f}\cdot\vec{e}_{f})=0$, the chosen
Coulomb gauge is obviously fulfilled. This choice of the vector potential
corresponds to one absorbed and one emitted photon. Let us write down the
potential of interaction of an electron and a photon

$V_{int}^{el}=i\frac{1}{c}\
(\vec{A}(\vec{r}_{1},t)\cdot\vec{\nabla}_{1})+\frac{1}{2c^{2}}A^{2}(\vec{r}_{1},t)\
=$ $(c.c.)+i\left(\sqrt{\frac{2\pi}{\omega_{i}}}\
e^{i(\vec{k}_{i}\vec{r}_{1}-\omega_{i}t)}(\vec{e}_{i}\cdot\vec{\nabla}_{1})+\sqrt{\frac{2\pi}{\omega_{f}}}\
e^{-i(\vec{k}_{f}\vec{r}_{1}-\omega_{f}t)}(\vec{e}_{f}\cdot\vec{\nabla}_{1})\right)\
+$ $\left(\frac{\pi}{\omega_{i}}\
e^{2i(\vec{k}_{i}\vec{r}_{1}-\omega_{i}t)}+\frac{\pi}{\omega_{f}}\
e^{-2i(\vec{k}_{f}\vec{r}_{1}-\omega_{f}t)}+\frac{2\pi}{\sqrt{\omega_{i}\omega_{f}}}\
(\vec{e}_{i}\cdot\vec{e}_{f})\
e^{i[(\vec{k}_{i}-\vec{k}_{f})\vec{r}_{1}-(\omega_{i}-\omega_{f})t]}\right),$
$None$

and the same for a positron

$V_{int}^{pos}=-i\frac{1}{c}\
(\vec{A}(\vec{r}_{2},t)\cdot\vec{\nabla}_{2})+\frac{1}{2c^{2}}A^{2}(\vec{r}_{2},t)\
=$ $(c.c.)-i\left(\sqrt{\frac{2\pi}{\omega_{i}}}\
e^{i(\vec{k}_{i}\vec{r}_{2}-\omega_{i}t)}(\vec{e}_{i}\cdot\vec{\nabla}_{2})+\sqrt{\frac{2\pi}{\omega_{f}}}\
e^{-i(\vec{k}_{f}\vec{r}_{2}-\omega_{f}t)}(\vec{e}_{f}\cdot\vec{\nabla}_{2})\right)\
+$ $\left(\frac{\pi}{\omega_{i}}\
e^{2i(\vec{k}_{i}\vec{r}_{2}-\omega_{i}t)}+\frac{\pi}{\omega_{f}}\
e^{-2i(\vec{k}_{f}\vec{r}_{2}-\omega_{f}t)}+\frac{2\pi}{\sqrt{\omega_{i}\omega_{f}}}\
(\vec{e}_{i}\cdot\vec{e}_{f})\
e^{i[(\vec{k}_{i}-\vec{k}_{f})\vec{r}_{2}-(\omega_{i}-\omega_{f})t]}\right).$
$None$

Let us consider first the so-called $A^{2}$ model and select from (3) the
terms corresponding to Compton scattering, when we have one incoming photon
and one outgoing photon. In this case, it follows from the second brackets in
(3)

${\tilde{V}}_{int}(t)=\frac{2\pi}{\sqrt{\omega_{i}\omega_{f}}}(\vec{e}_{i}\cdot\vec{e}_{f})\
\left[e^{i[(\vec{k}_{i}-\vec{k}_{f})\vec{r}_{1}-(\omega_{i}-\omega_{f})t]}+e^{i[(\vec{k}_{i}-\vec{k}_{f})\vec{r}_{2}-(\omega_{i}-\omega_{f})t]}\right].$
$None$

### 2.1 Schrödinger equation

The evolution of the system can be described by the time-dependent Schrödinger
equation:

$\left[i\frac{\partial}{\partial
t}-H_{0}-(V_{int}(t)-{\tilde{V}}_{int}(t))\right]|\Psi(t)>={\tilde{V}}_{int}(t)|\Psi(t)>$
$None$

where the total potential of the interaction of positronium with a photon
$V_{int}(t)$ is given by the sum of expressions (3.1) and (3.2), and the
Coulomb potential is included in $H_{0}$. Let us express the variables
$\vec{r}_{1}$ and $\vec{r}_{2}$ in terms of the relative coordinate
$\vec{\rho}=\vec{r}_{1}-\vec{r}_{2}$ and the coordinate of the positronium
center of mass $\vec{R}=(\vec{r}_{1}+\vec{r}_{2})/2$:

$\vec{r}_{1}=\vec{R}+\frac{\vec{\rho}}{2},\quad\vec{r}_{2}=\vec{R}-\frac{\vec{\rho}}{2}.$
$None$

Then the conjugate momenta look like

$\vec{p}_{1}=\frac{1}{2}\vec{p}_{R}+\vec{p}_{\rho},\quad\vec{p}_{2}=\frac{1}{2}\vec{p}_{R}-\vec{p}_{\rho}.$
$None$

This makes it possible to represent all the quantities in $H$ in terms of the
variables $\vec{\rho}$ and $\vec{R}$ in particular:

$H_{0}=\frac{1}{4}p^{2}_{R}+p^{2}_{\rho}+V(\vec{\rho}),$ $None$

and

${\tilde{V}_{int}}(\vec{R},\vec{\rho};t)=\frac{2\pi}{\sqrt{\omega_{i}\omega_{f}}}(\vec{e}_{i}\cdot\vec{e}_{f})\
e^{i[(\vec{k}_{i}-\vec{k}_{f})\vec{R}-(\omega_{i}-\omega_{f})t]}\left[e^{i(\vec{k}_{i}-\vec{k}_{f})\vec{\rho}/2}+e^{-i(\vec{k}_{i}-\vec{k}_{f})\vec{\rho}/2}\right].$
$None$

We denote by $\vec{Q}=\vec{k}_{i}-\vec{k}_{f}$ the momentum transferred from a
photon to positronium. In what follows, we need to calculate the matrix
element

${\mathcal{M}}=\int
d^{3}R\int_{-\infty}^{\infty}dte^{-i\varepsilon_{0}t}<\Psi^{-}(\vec{p}_{1},\vec{p}_{2},t)|{\tilde{V}}_{int}(t)|\phi_{0}>,$
$None$

which describes the positronium disintegration under the action of the field.
Here $\varepsilon_{0}=-1/8$ denotes the ground energy of positronium.

We need to solve Eq. (5) to find the exact wave function for substituting it
in (10), but now we consider the simplest case, where we neglect the term
$(V_{int}-{\tilde{V}}_{int})$ in this equation. This term is really small
because of the presence of $\omega$ in denominators. Then

$<\Psi^{-}(\vec{p}_{1},\vec{p}_{2},t)|\vec{r}_{1},\vec{r}_{2}>=e^{i\vec{p}_{R}\vec{R}-i(E_{1}+E_{2})t}\phi^{-}(\vec{p}_{\rho},\vec{\rho}),$
$None$

and the function $|\phi^{-}>$ satisfies the hydrogen-like equation and
describes A Coulomb function of the positronium continuum spectrum satisfies
the equation:

$[p^{2}_{\rho}+\triangle_{\rho}-V(\vec{\rho})]\phi^{-}(\vec{p}_{\rho},\vec{\rho})=0,$
$None$

In this case, Eq. (10) can be easily integrated, and these integrals give the
laws of conservation of energy and momentum.

### 2.2 Differential cross section

The fully differential cross section (FDCS) for the decay of positronium under
the action of the EM field of the laser is written as

$d^{3}\sigma=\frac{(2\pi)^{2}\alpha}{\omega_{i}\omega_{f}}\sum_{e_{i},e_{f}}|M|^{2}\
\times$
$(2\pi)^{4}\delta(\omega_{i}+\varepsilon_{0}-\omega_{f}-p^{2}_{1}/2-p^{2}_{2}/2)\delta^{3}(\vec{k}_{i}-\vec{k}_{f}-\vec{p}_{1}-\vec{p}_{2})\
\frac{d^{3}k_{f}}{(2\pi)^{3}}\frac{d^{3}p_{1}}{(2\pi)^{3}}\frac{d^{3}p_{2}}{(2\pi)^{3}},$
$None$

where the matrix element is given by the expression

$M=(\vec{e}_{i}\cdot\vec{e}_{f})<\phi(\vec{p}_{\rho})|e^{i\vec{Q}\cdot\vec{\rho}/2}+e^{-i\vec{Q}\cdot\vec{\rho}/2}|\phi_{0}>,$
$None$

Here $<\vec{\rho}|\phi_{0}>$ and $<\phi(\vec{p}_{\rho})|\vec{\rho}>$ are the
initial (ground) and final (disintegrated) states of positronium. Also
$\alpha^{2}=r_{0}$ denotes the classical radius of the electron in atomic
units. The sum denotes averaging over the initial photon polarizations and
summing over the final polarizations. In this case the result does not depend
on the choice of the polarisation type, either linear or circular.
Approximation (14) will be called the first Born approximation (FBA) by
analogy with the scattering of ordinary particles.

Further simplification of formula (13) depends on what we actually measure. In
the COLTRIMS detector only the momenta of charged particles are measured, but
the photon momentum can be easily calculated from those of the electron and
positron. So, let us assume that we do not measure the momentum of the
positron. Then, after integrating (13) with respect to $d^{3}p_{2}$, we obtain

$d^{3}\sigma=\frac{(2\pi)^{2}\alpha}{\omega_{i}\omega_{f}}\sum_{e_{i},e_{f}}|M|^{2}\
(2\pi)\delta(\omega_{i}+\varepsilon_{0}-\omega_{f}-p^{2}_{1}/2-(\vec{k}_{i}-\vec{k}_{f}-\vec{p}_{1})^{2}/2)\
\frac{d^{3}p_{1}}{(2\pi)^{3}}\frac{d^{3}k_{f}}{(2\pi)^{3}}.$ $None$

The phase volume of the photon can be transformed as follows
$d^{3}k_{f}=\omega_{f}^{2}d\omega_{f}d\Omega_{f}/c^{3}$ with the subsequent
integration with respect to $\omega_{f}$. In so doing, we assume that the
initial photon frequency is not larger than 5 keV, i.e. $k_{i}\sim 1$, which
gives some simplification of the integration results. Finally, we get

$\frac{d^{3}\sigma}{dE_{e}d\Omega_{1}d\Omega_{f}}=FDCS=\frac{\alpha^{4}}{(2\pi)^{3}}\
p_{1}\
\left(1-\frac{p^{2}_{1}/2-\varepsilon_{0}+(\vec{k}_{i}-\vec{p}_{1})^{2}/2}{\omega_{i}}\right)\sum_{e_{i},e_{f}}|M|^{2}.$
$None$

For the chosen photon energy, the second term in the brackets plays the role
of a correction. In what follows, we denote

$t=\frac{\omega_{f}}{\omega_{i}}=\left(1-\frac{p^{2}_{1}/2-\varepsilon_{0}+(\vec{k}_{i}-\vec{p}_{1})^{2}/2}{\omega_{i}}\right).$

In these notations

$Q=\frac{\omega_{i}}{c}\sqrt{1-2t\cos\theta+t^{2}},$

and $\theta$ is the scattering angle between the vectors $\vec{k}_{i}$ and
$\vec{k}_{f}$.

We note that

$\sum_{e_{i},e_{f}}|M|^{2}=\frac{1}{2}(1+\cos^{2}\theta)\left|<\phi^{-}(\vec{p}_{\rho})|e^{i\vec{Q}\cdot\vec{\rho}/2}+e^{-i\vec{Q}\cdot\vec{\rho}/2}|\phi_{0}>\right|^{2},$
$None$

We also note that, after calculating the matrix element, we must put
$\vec{p}_{\rho}=1/2[\vec{p}_{1}-(\vec{Q}-\vec{p}_{1})]=\vec{p}_{1}-\vec{Q}/2$
in it, which follows from the law of momentum conservation.

Each term in (17) is analogous to the matrix element for hydrogen, where
$Z=1/2$ should be set. Then (see EPJD20 )

$J_{0}(\vec{Q}/2,\vec{p}_{\rho})=\langle\phi^{-}(\vec{p}_{\rho})|e^{i\vec{Q}\cdot\vec{\rho}/2}|\varphi_{0}\rangle=$
$-16\pi\sqrt{\frac{Z^{5}}{\pi}}e^{-\pi\zeta/2}\Gamma(1+i\zeta)\frac{[Q^{2}/4-(p_{\rho}+iZ)^{2}]^{-1+i\zeta}}{[(\vec{Q}/2-\vec{p}_{\rho})^{2}+Z^{2}]^{2+i\zeta}}\
\times$
$\frac{Q}{2}\left[\frac{Q}{2}-(p_{\rho}+iZ)\frac{(\vec{Q}\cdot\vec{p}_{\rho})}{Qp_{\rho}}\right],\quad\zeta=-\frac{1}{2p_{\rho}},\
Z=\frac{1}{2}.$ $None$

and

$<\phi^{-}(\vec{p}_{\rho})|e^{i\vec{Q}\cdot\vec{\rho}/2}+e^{-i\vec{Q}\cdot\vec{\rho}/2}|\phi_{0}>=J_{0}(\vec{Q}/2,\vec{p}_{\rho})+J_{0}(-\vec{Q}/2,\vec{p}_{\rho}).$

## 3\. NUMERICAL CALCULATIONS AND DISCUSSION

.

Thus, we evaluate:

$FDCS=\frac{\alpha^{4}}{(2\pi)^{3}}p_{1}t\sum_{e_{i},e_{f}}|M|^{2},$ $None$

where

$\sum_{e,e_{1}}|M|^{2}=\frac{1}{2}(1+\cos^{2}\theta)\times$
$\left|-16\pi\sqrt{\frac{Z^{5}}{\pi}}\ e^{-\pi\zeta/2}\Gamma(1+i\zeta)\
\frac{Q}{2}\ [(Q^{2}/4-(|\vec{p}_{1}-\vec{Q}/2|+iZ)^{2}]^{-1+i\zeta}\
\times\right.$
$\left.\left[\frac{[Q/2-(|\vec{p}_{1}-\vec{Q}/2|+iZ)\cos\gamma]}{[(\vec{p}_{1}-\vec{Q})^{2}+Z^{2}]^{2+i\zeta}}+\frac{[Q/2+(|\vec{p}_{1}-\vec{Q}/2|+iZ)\cos\gamma]}{[p_{1}^{2}+Z^{2}]^{2+i\zeta}}\right]\right|^{2},$
$None$

When calculating the cross section (19), it is possible to use various systems
of units. The cross section is now written in the atomic units. To get the
FDCS in cm ${}^{2}/{\rm eV}\cdot{\rm Sr}^{2}$, one needs to multiply Eq. (19)
by the factor $1.03\cdot 10^{-18}$. Thus, $\alpha^{4}=0.28\cdot 10^{-8}a.u.\to
0.29\cdot 10^{-26}\ {\rm cm}^{2}/{\rm eV}\cdot{\rm Sr}^{2}=0.29\cdot 10^{-2}\
{\rm barn}/{\rm eV}\cdot{\rm Sr}^{2}$.

In (20), the effective charge of positronium is
$Z=1/2,\quad\vec{p}_{\rho}=\vec{p}_{1}-\vec{Q}/2$, and also,

$\cos\gamma=(\vec{Q}\cdot\vec{p}_{\rho})/Qp_{\rho}=\frac{(\vec{p}_{1}\cdot\vec{Q})-Q^{2}/2}{Q\sqrt{p^{2}_{1}-(\vec{p}_{1}\cdot\vec{Q})+Q^{2}/4}},$
$(\vec{Q}\cdot\vec{p}_{1})=k_{i}p_{1}[\cos\varphi_{1}-t\cos\chi]=Qp_{1}\cos\beta.$
$None$

Here

$Q=k_{i}\sqrt{1-2t\cos\theta+t^{2}},\quad\cos\chi=\cos\theta\cos\varphi_{1}+\sin\theta\sin\varphi_{1}\cos\Phi,$

$\varphi_{1}$ is the angle between $\vec{p}_{1}$ and $\vec{k}_{i}$, $\Phi$ is
the angle between the planes, in which the vectors $(\vec{k}_{i},\vec{k}_{f})$
and $(\vec{k}_{i},\vec{p}_{1})$ lie, and $\theta$ is the scattering angle
between the vectors $\vec{k}_{i}$ and $\vec{k}_{f}$. We also repeat that

$\zeta=-Z/p_{\rho}=-1/|2\vec{p}_{1}-\vec{Q}|,\quad
t=1-\frac{p^{2}_{1}/2-\varepsilon_{0}+(\vec{k}_{i}-\vec{p}_{1})^{2}/2}{\omega}.$

For reference we present similar expressions for hydrogen from paper EPJD20 :

$FDCS_{H}=\frac{\alpha^{4}}{(2\pi)^{3}}p_{1}t\ \sum_{e_{i},e_{f}}|M_{H}|^{2},$
$None$ $\sum_{e_{i},e_{f}}|M_{H}|^{2}=\frac{1}{2}(1+\cos^{2}\theta)\times$
$\left|-16\pi\sqrt{\frac{Z^{5}}{\pi}}\ e^{-\pi\zeta/2}\Gamma(1+i\zeta)\
Q[Q^{2}-(p_{1}+iZ)^{2}]^{-1+i\zeta}\left[\frac{Q-(p_{1}+iZ)\cos\chi]}{[(\vec{p}_{1}-\vec{Q})^{2}+Z^{2}]^{2+i\zeta}}\right]\right|^{2}.$
$None$

Here in the case of hydrogen

$\zeta=-Z/p_{1},\quad t=1-\frac{p^{2}_{1}/2-\varepsilon_{0}}{\omega_{i}},\quad
Z=1.$

Figure 1:  FDCS (a.u.) as a function of the electron scattering angle
$\varphi_{1}$ in the case of ionization of the hydrogen atom H and positronium
Ps. $\omega_{i}=5$ keV, $E_{e}=27.2$ eV, $\theta=\pi/3$.  Figure 2:  The same
as in Fig. 4 for $\theta=\pi/2$. Figure 3:  The same as in Fig. 4 for
$\theta=\pi$.

Let us analyze Figs. 1 - 3, which show the total differential cross sections
of the Compton ionization of the hydrogen atom and positronium. We should say,
that in all calculations we found that the dependence on $t$ is quite weak, so
we put everywhere $t=1$. First of all, we see the dominant peaks, when the
angle $\beta$ between the vectors $\vec{p}_{1}$ and $\vec{Q}$ is equal to
zero. In the theory of particle scattering, this peak is called the binary one
and corresponds to the minimum of $|\vec{p}_{1}-\vec{Q}|$. This peak reflects
a mechanism, when the entire transferred momentum is transferred to the
electron, and the positron remains a ”spectator”. Sufficiently flat second
peak of the curve Ps is called the recoil peak in the theory of particle
scattering. In this case, the value of $|\vec{p}_{1}-\vec{Q}|$ reaches its
maximum, and the behavior of the cross section is determined mainly by the
second term in (20), whose denominator does not depend on the angle $\beta$.
In this region, the positron absorbs a $\gamma$-quantum, and the electron
emits it during the disintegration. The transfer of energy from the positron
to the electron occurs due to the Coulomb interaction of the particles in
positronium. Such a process is possible, but unlikely, it is reflected in a
very large ratio of the magnitudes of these peaks. In addition, the described
process is secondary, and the electron escapes from the positronium
isotropically, which is reflected in the ”flat” character of the backward
peak. Thus, Figs. 1 - 3 present the main mechanisms of the Compton
disintegration of positronium. It is interesting that, at a sufficiently high
photon energy of 5 keV, the backward peak in hydrogen is not only absent, but
turns into a minimum at the same place, where the Ps curve has the recoil
peak. However, the structural differences between the curves are visible only
on a logarithmic scale.

It should also be noted that the magnitude of the binary peak of the Ps curve
slightly decreases, while the recoil peak slightly increases with increasing
momentum transfer $Q$. This is due to an increase in the value of
$|\vec{p}_{1}-\vec{Q}|$ for a fixed electron energy.

### 3.1 System of Coulomb resonances

Let us now consider the behavior of the cross section at the resonance, where
the electron and positron fly parallel to each other with the same velocities.
This happens, when the Coulomb number goes to infinity, $\zeta\to\infty$, or
$\mu=|\vec{p}_{1}-\vec{Q}/2|\to 0.$ This is possible, when
$\vec{p}_{1}\parallel\vec{Q}$, i.e. in the binary peak region, and for
$p_{1}/k_{i}=\sin(\theta/2)<1$, if we set $t=1$ without loss of generality. In
this case, it follows from Eq. (20)

$\sum_{e_{i},e_{f}}|M|^{2}\approx\frac{1}{\mu}\
(1+\cos^{2}\theta)\frac{32\pi^{2}Z^{5}Q^{4}}{(Q^{2}/4+Z^{2})^{6}}\exp\left(-\
{\frac{2Z}{Q^{2}/4+Z^{2}}}\right).$ $None$

As can be seen from (22), a pole appears in the cross section under the given
kinematic conditions, which is characteristic of infinitely narrow dynamic
resonances. However, this pole is not dangerous when integrating with respect
to $d^{3}p_{1}$, since we make a change of variables and compensate this pole
with the phase volume of the integration. However, let us assume that we need
to integrate the FDCS over the photon scattering angle, for example, to
calculate

$DDCS_{\varphi_{1}}=\frac{d^{2}\sigma}{dE_{e}d\varphi_{1}}=2\pi\int_{0}^{\pi}\
\sin\theta\ d\theta\int_{0}^{2\pi}d\Phi\ FDCS.$ $None$

This double differential cross section describes the angular and energy
distribution of the electron for any direction of photon scattering. Let us
ask ourselves a question, whether this integral will converge? To answer this
question, we use expression (21) and write down the expression for $\mu^{2}$:

$\mu^{2}=p^{2}_{1}-p_{1}Q\left(\cos\varphi_{1}\sin\frac{\theta}{2}-\sin\varphi_{1}\cos\frac{\theta}{2}\cos\Phi\right)+\frac{Q^{2}}{4}.$
$None$

Recall that for $t=1$, $Q=2k_{i}\sin(\theta/2)$, and denote $\gamma=p_{1}/k$.
Omitting intermediate calculations, we rewrite (24.1) in the form

$\mu^{2}=k_{i}^{2}\left[\left(\gamma-\sin\frac{\theta}{2}\right)^{2}+2\gamma\sin\frac{\theta}{2}\left(1-\cos(\frac{\pi}{2}-\varphi_{1}-\frac{\theta}{2})\right)\right.\
+$ $\left.\gamma\sin\varphi_{1}\sin\theta(1+\cos\Phi)\frac{}{}\right].$ $None$

All terms in (24.2) are positive, therefore $\mu=0$ if the conditions
$\gamma=\sin(\theta_{0}/2),\ (\pi/2-\varphi_{1}-\theta_{0}/2)=0,\
(1+\cos\Phi_{0})=0$ are fulfilled simultaneously. Hence, in particular, it
follows that $\gamma=\cos\varphi_{1}<1$. This is the condition, where one can
expect a singularity in the FDCS. If $p_{1}>k_{i}$, or in our case
$E_{e}>24.4$ eV, the singularities are absent.

Let us put $\theta=\theta_{0}+\delta$ and $\Phi=\pi-\lambda$ in (24.2), where
the deviations $\lambda$ and $\delta$ are small. Then

$\mu\approx k\sqrt{\frac{1}{4}\delta^{2}+\lambda^{2}\
\gamma^{2}(1-\gamma^{2})}.$ $None$

and the FDCS is completely integrable in the neighborhood of the points
$\theta=\theta_{0}$ and $\Phi=\pi$, if only $\gamma\neq 1$. However, in the
phase volume of integral (23) in the vicinity of the point $\theta=\theta_{0}$
there is a factor $\sin\theta_{0}=2\gamma\sqrt{1-\gamma^{2}}$, which
compensates this divergence. Thus, integral (23) converges, although numerical
integration requires a certain accuracy in the vicinity of the zeros of
expression (24.2).

Figure 4:  FDCS as a function of the photon scattering angle $\theta$ and the
electron energy $E_{e}$ under the condition $\vec{p}_{1}\parallel\vec{Q}$. The
value of $\mu$ is smoothed: $\mu\to\sqrt{\mu^{2}+\varepsilon^{2}},\
\varepsilon=0.01$. The line of resonances $\mu\sim 0$ is clearly visible.

Fig. 4 shows a 3D plot of the FDCS versus the electron energy and photon
scattering angle in the binary peak region. It should be said right away that
the angle $\beta$ itself depends on the scattering angles of the photon and
the electron according to (21). Its fixation imposes a constraint on these
observed angles. However, the resonance line is easily seen in the figure. The
calculations were carried out while smoothing the singularity
$\mu\to\sqrt{\mu^{2}+\varepsilon^{2}},\ \varepsilon=0.01$.

Let us discuss the above effect. In principle, a similar situation arises,
when a hydrogen atom is ionized by a photon, if the final state is described
by a Coulomb wave. Due to the difference between the masses of the proton and
the electron, the singularity in the FDCS arises, when the electron energy
tends to zero. This is precisely the energy boundary between the continuum and
the spectrum of bound Coulomb states, which is sometimes called ”the Keldysh
swamp”. In this case, the size of the system tends to infinity. Formally, we
are faced with an infinite number of thresholds of excitation reactions. As we
can see, the matrix element (20.1) for the hydrogen ionization has a
singularity at $p_{1}\to 0$. However, in the case of hydrogen, this pole is
compensated by the factor $p_{1}$ in cross section (19.1), and the cross
section tends to a constant as $p_{1}\to 0$.

The same situation takes place in positronium, only now the singularity
depends on the angles. And there is a very simple condition for its occurrence
$E_{e}=0.5(\omega_{i}/c)^{2}\sin^{2}(\theta/2)$. In this case, the electron
and positron move parallel to each other with the same velocity, and they can
be separated by any distance, including a rather large one. Formally, such a
state can exist for an arbitrarily long time within the framework of non-
relativistic physics, until a second photon is emitted (even with an energy
close to zero), and the electron-positron pair again falls into a bound state.
Thus, a resonance arises. An interesting question is: how to maintain such a
state for as long as you like?

The described picture resembles the Thomas effect in the classical description
of the capture of an electron by an ion from a target Thomas . That time there
was no possibility to describe the bound state of the final atom in this
effect, and it was described as the parallel motion of an electron and an ion
with the same velocity. From this condition, the trajectory of the electron
motion between heavy fragments was calculated, which gave a peak in the
capture cross section (a second-order process in a quantum description) .

### 3.3 Double differential cross section

Now we consider the double differential cross sections, the measurements of
which are easier. For example, in the case of cross section (23), we measure
only the electron momentum. A 3D picture of this cross section for positronium
is shown in Fig. 5. Here the photon energy is $\omega_{i}=3$ keV. First of
all, it is striking that the cross section is noticeably different from zero
at small angles of electron emission (forward scattering cone of the photon)
and its low energies. The cross section goes to zero at $E_{e}\to 0$ in
contrast to hydrogen, since there is no compensating pole here, which we wrote
about above. This results in the sharp peak at low energies. Further, at small
angles, two more peaks of different heights and widths stand out.

Let us consider these peaks in more detail in Figs. 6 – 8, where the same
cross section is presented for the angle $\varphi_{1}=0$, but at different
photon energies. These figures also show a similar slice of the cross section
for the hydrogen atom.

Let us pay attention to the middle peak, which becomes discernible at the
photon energy $\omega_{i}=5$ keV, but noticeably grows up as the photon energy
decreases. It is easy to see that this peak appears at $p_{1}=k_{i}$, or
$E_{e}=0.5(\omega_{i}/c)^{2}$. The origin of this peak has not been fully
understood yet. It is not excluded that it is somehow related to the
resonances discussed in the previous section, although it is difficult to
reliably say about this after integration over the photon scattering angle.
Hydrogen has nothing of this kind.

Let us discuss this peculiarity in the behavior of the cross section in more
detail. In the theory of electron capture from a target and single ionization
of the target by an ion at relatively low projectile and electron speeds,
singularities are observed in the integral cross sections, when the electron
and projectile velocities coincide. In the theory of capture, this is the
above mentioned Thomas effect, while in the theory of ionization, this is the
so-called ”shoulder” in the single differential cross section Schulz . It is
observed quite well at proton energies of 75 keV and 100 keV. In our case,
this is the coincidence of the electron and positron velocities after the
disintegration of positronium by a photon. Thus, the assumption about the
nature of the discussed peak is quite acceptable.

A wide peak at larger energies is observed for both positronium and hydrogen.
The position of these peaks is also close to each other. Earlier in the study
of hydrogen, it was found that the FDCSH has a peak, when the electron takes
over the entire transferred momentum, which corresponds to Compton scattering
at a free electron. It seems that even after the integration over the angle of
the scattered photon, this trend persists. Indeed, the peak is seen most
clearly, when the photon is scattered into the backward cone, which leads to
forward scattering of the electron. The integration over all angles smoothes
the peak somehow, but does not override its physics. The presence of the light
positron in positronium slightly changes the position and magnitude of this
peak, but the physical effect remains the same.

Figure 5:  DDCS${}_{\varphi_{1}}$ (23) depending on the electron emission
angle $\varphi_{1}$ and its energy $E_{e}$. The photon energy is
$\omega_{i}=3\ keV$ Figure 6:  DDCS${}_{\varphi_{1}}$ (23) with the fixed
angle $\varphi_{1}=0$. The photon energy is $\omega_{i}=5\ keV$. The red curve
corresponds to positronium, the black curve corresponds to hydrogen.  Figure
7:  The same as in Fig. 6 for the photon energy $\omega_{i}=3.75\ keV$.
Figure 8: The same as in Fig. 6 for the photon energy $\omega_{i}=3\ keV$ .

## 4\. CONCLUSION

We have considered the Compton disintegration of positronium in the non-
relativistic approximation at a photon energy of several keV. A number of
features are noted in the differential cross sections that distinguish the
ionization (disintegration) of positronium from the ionization of the hydrogen
atom. In particular, an interesting feature of the process are the resonances
that arise in the parallel motion of the electron and positron with equal
velocities after the disintegration. In hydrogen, only one such resonance
state is observed in the total differential cross section at the energy of the
emitted electron equal to zero. This is due to the use of the Coulomb function
as the final state. For positronium, this point unfolds into a line of
resonances at the electron energies
$E_{e}=0.5(\omega_{i}/c)^{2}\sin^{2}(\theta/2)$. For the photon
backscattering, the energy of the electron (and the positron), which is
required to provide resonances, reaches its maximum value, after which the
parallel motion of the electron and positron is no longer possible.

## Acknowledgements

The authors are grateful to the experimental team headed by Prof. R. Dörner
(Institut für Kernphysik, J. W. Goethe Universität, Frankfurt/Main, Germany)
whose experiments inspired us for writing this paper, and for long-time
collaboration. The work supported in part by the Heisenberg-Landau program.
Yu. P. is grateful to Russian Foundation for Basic Research (RFBR) for the
financial support under grant No.19-02-00014-a.

## Author contribution statement

All authors contributed equally to the paper.

## References

  * (1) A.I.Akhiezer,V.B.Berestetskii, QuantumElectrodynamics (John Wiley and Sons, 1965), §30
  * (2) S. Houamer, O. Chuluunbaatar, I.P. Volobuev, and Yu.V. Popov. ”Compton ionization of hydrogen atom near threshold by photons in the energy range of a few keV: Nonrelativistic approach” Eur.Phys.Journ.D 74 (2020), 81(9pp). https://doi.org/10.1140/epjd/e2020-100572-1
  * (3) M. Kircher, F. Trinter, S. Grundmann, I. Vela-Perez, S. Brennecke, N. Eicke, J. Rist, S. Eckart, S. Houamer, O. Chuluunbaatar, Yu.V.Popov, I. P.Volobuev, Kai Bagschick, M.N. Piancastelli, M. Lein, T. Jahnke, M.S.Schöffler, and R. Dörner. ”Kinematically complete experimental study of Compton scattering at helium atoms near the threshold”, Nature Physics 16 (2020), 756 -760. https://doi.org/10.1038/s41567-020-0880-2
  * (4) O. Chuluunbaatar, S. Houamer, Yu.V. Popov, I.P. Volobuev, M. Kircher, and R. Doerner. ”Compton ionization of atoms as a method of dynamical spectroscopy”, Journal of Quantitative Spectroscopy and Radiative Transfer (JQSRT) 272 (2021), 107820 (8pp) https://doi.org/10.1016/j.jqsrt.2021.107820
  * (5) W. Bothe and H.Geiger, ”Uber das Wesen des Comptoneffekts; ein experimenteller Beitrag zur Theorie der Strahlung”. Z. Phys. 32 (1925), 639.
  * (6) L.H. Thomas, ”On the capture of electrons by swiftly moving electrified particles”. Proc. R. Soc. Lond. 114 (1927), 561-578. https://www.jstor.org/stable/94828
  * (7) M. Schulz et al, ”Absolute doubly differential single-ionizationcross sections in p+He collisions”, Phys. Rev.A 54 (1996), 2951. https://doi.org/10.1103/PhysRevA.54.2951


# R+X: Retrieval and Execution from
Everyday Human Videos

Georgios Papagiannis∗ &Norman Di Palo∗ &Pietro Vitiello &Edward Johns The
Robot Learning Lab at Imperial College London

###### Abstract

We present R+X, a framework which enables robots to learn skills from long,
unlabelled, first-person videos of humans performing everyday tasks. Given a
language command from a human, R+X first retrieves short video clips
containing relevant behaviour, and then executes the skill by conditioning an
in-context imitation learning method on this behaviour. By leveraging a Vision
Language Model (VLM) for retrieval, R+X does not require any manual annotation
of the videos, and by leveraging in-context learning for execution, robots can
perform commanded skills immediately, without requiring a period of training
on the retrieved videos. Experiments studying a range of everyday household
tasks show that R+X succeeds at translating unlabelled human videos into
robust robot skills, and that R+X outperforms several recent alternative
methods. Videos are available at https://www.robot-learning.uk/r-plus-x.

**footnotetext: Equal contribution, random order.**footnotetext: Contact at:
{g.papagiannis21<EMAIL_ADDRESS>Figure 1: Given a language
prompt, R+X first retrieves short relevant video clips extracted from a long
unlabelled video of a human performing everyday tasks, recorded with a
wearable camera. With the use of the retrieved video clips and a VLM, R+X
performs in-context imitation learning allowing it to immediately generate and
execute the desired behaviour on the robot.

##  1 Introduction

Robot learning of diverse, everyday tasks in natural environments, is a
significant challenge. Imitation learning is a promising solution which has
been widely adopted in recent years [1, 2, 3], but it remains difficult to
scale up due to the cost of hardware and human demonstration time. If robots
could instead learn from data which does not require any robot hardware or
dedicated human time, we may be able to scale up data collection and learning
substantially.

In this work, we study the problem of learning robot skills from long,
unlabelled, first-person videos of humans performing everyday tasks in
everyday environments. "Long, unlabelled videos" means that a human simply
goes about their everyday life and passively records videos of themselves,
without the need to specify which behaviour is being performed. "Everyday
environments" means that videos contain diversity in tasks, scenarios,
objects, distractors, illumination, and camera viewpoints and motion. It is
likely that such videos will become abundant through the adoption of wearable
devices such as AR headsets and glasses [4, 5, 6, 7], and thus this offers a
significant opportunity for future scalability if robots could learn from such
videos.

Previous approaches to learning from videos of humans have often relied on a
set of strong constraints and assumptions, such as human videos manually
aligned with robot videos, human videos manually labelled with language
descriptions, or demonstrations on robot or mocap hardware [8, 9, 1, 10, 11,
12, 13]. In this work, we remove all of these constraints, and present a
framework that requires only an unlabelled first-person video of a human
performing tasks.

Our framework, R+X, is a two-stage pipeline of Retrieval and EXecution (see
Figure 1), which uses Foundation Models for both stages. Upon receiving a
language command from a human, a Vision Language Model retrieves all clips
where the human executes the specified task. By extracting the trajectories of
the human’s hand in each clip, we then employ a few-shot in-context imitation
learning method to condition on these trajectories, which enables a robot to
ingest, learn from, and replicate the retrieved behaviours, whilst adapting to
previously unseen settings and objects. Recent literature has demonstrated
that, by finetuning large VLM on robotics data, they can transfer their common
knowledge and ability to reason and plan to robotics settings [14, 15]. This,
however, requires very expensive finetuning of often intractably large models.
With our proposed framework, we can equally leverage these abilities, but now
via retrieval and video understanding, thus without the need for any
finetuning. Rather than requiring explicit policy training, the retrieved
skills are executed immediately following the language command. We later
demonstrate the benefits that this brings over spatial and language
generalisation, compared to large monolithic policy networks.

In summary, the two main properties of R+X are: 1) it enables robots to
execute everyday tasks from language commands, given long, unlabelled, first-
person videos, collected naturally by recording a human’s daily activities;
and 2) it achieves this without the need for any training or finetuning of
models, allowing a robot to learn and execute tasks immediately after the
language command.

##  2 Related Work

Figure 2: The main assumptions and constraints of many recent Learning from
Observation methods.

Collecting Robotics Data. Scaling data collection has proven to be a
successful path towards increasingly more general machine learning models [16,
17, 18, 19]. To collect robotics data, a popular paradigm is through
kinesthetic teaching or teleoperation, collecting datasets of paired
observations and actions [3]. This, however, needs dedicated robot hardware
and optionally teleoperation hardware, and needs human operators to allocate
their time to proactively teach a robot new tasks. In our framework, humans
interact with their environments much more naturally, collecting data
passively during their daily activity, resulting in a more scalable and time
efficient paradigm.

Learning from Human Videos. Many recent works have proposed solutions to teach
new skills to robots by observing a human executing such tasks, known as
Learning from Observation. However, as we illustrate in Fig. 2, they often
rely on a set of assumptions and constraints that limit their use "in-the-
wild". Bahl et al. [9], Wang et al. [20] require a combination of human data
and either robot exploration or teleoperation, therefore needing active
assistance by humans in teaching the robot, and videos are recorded from a
fixed, third-person camera. R+X relies entirely on the videos recorded by the
human with a mobile camera in their natural environments. Heppert et al. [10],
Zhu et al. [12], Bahety et al. [11] can learn robot skills entirely from human
videos. Heppert et al. [10], Zhu et al. [12], however, they focus on
replicating the object trajectory from the demonstration, and therefore cannot
perform non prehensile tasks that do not involve grasping and moving, such as
pushing or pressing, which R+X can execute. Bahety et al. [11] can learn a
large repertoire of skills, but still relies on learning single tasks in
isolation from a fixed camera, while R+X can replicate tasks from a given
language command after receiving a single, long, unlabelled video depicting
tens of tasks, without the need to specify which clip demonstrated which
behaviour. Methods like Wang et al. [13] allow a human to naturally interact
with their environment while collecting data that can teach a robot new
skills. However, they require additional hardware to wear, like specialised
MoCap gloves. R+X only needs a single RGB-D camera, and does not require any
extrinsics calibration. This means that data could be recorded using a
wearable camera, smart glasses, AR visors, and more.

Language-Conditioned Policy Learning. Robots should learn a wide repertoire of
skills, and be able to execute many different tasks when deployed. Language is
considered a viable way to instruct robots and guide their task execution [15,
14, 21]. The common approach is to train a large, language-conditioned and
image-conditioned policy that can, at test time, receive human commands and
camera observations [3]. This, however, presents some problems: training such
models can require enormous computational effort [14], and yet robots should
ideally continually learn new skills over time, where re-training or
finetuning such networks becomes problematic. R+X does not train or finetune
any networks, but takes advantage of pre-trained models able to retrieve
examples of relevant tasks [17], enabling robots to learn and execute new
behaviour directly at deployment via in-context learning [22].

Figure 3: Upon receiving a language command, R+X retrieves all the relevant
clips from the human video. Each retrieved clip is transformed from pixels to
a sparse 3D points representations of the hand joints movement and salient
parts of the visual observation.

##  3 R+X: Retrieval and Execution

Figure 4: The visual 3D keypoints of the first frame of each of the Z videos
obtained from the retrieval phase along with each extracted hand joint
trajectory are used as context for KAT [22], a recent in-context imitation
learning method. To execute a skill, visual 3D keypoints are extracted from
the live observation and used as input to an imitation learning model which
generates a sequence of hand joints. By mapping this sequence to gripper poses
the robot executes the desired task.

We now describe R+X, our proposed method to learn robot skills from long,
unlabelled videos of humans interacting with their environments. We assume R+X
has access to a long, unlabelled video of a human performing a multitude of
tasks in many different locations. We call this long, first-person video of
everyday tasks the "human video" $\mathcal{H}$ in the rest of the paper. The
goal of our method is to learn to emulate the behaviours recorded in the human
video upon receiving a language command from the human.

Overall, from a high-level perspective, our pipeline takes three inputs during
deployment: 1) a single, long, unlabelled video of all the interactions
recorded by the humans, $\mathcal{H}$; 2) a task to execute in language form,
$\mathcal{L}$; and 3) the robot’s current observation as an RGB-D image,
$\mathcal{O}_{live}$. It then outputs a trajectory of 6-DoF gripper poses,
which are executed by the robot to perform the commanded task.

There are however a set of non-trivial challenges. While the robot receives a
language command at deployment, it receives no language information before
deployment: it only has access to the unlabelled video. Additionally, the
robot receives no action information: unlike the case of teleoperation, no
joints or end-effector position/velocity/acceleration data are recorded, and
the correct movements need to be inferred by the videos alone; this problem is
additionally complex due to the cross-embodiment between human videos and
robot actions. Furthermore, as the human is interacting with their natural
environment, the visual observations can be filled with distractor objects
typical of everyday environments, which are unrelated to the task the human is
performing. To tackle all these aforementioned challenges, we leverage the
abilities of Foundation Models.

Specifically, at the first phase of R+X, which we call the retrieval phase, we
use a VLM to extract from the human video all the examples of the desired
behaviour described in the prompt $\mathcal{L}$, as a list of $Z$ shorter
video clips $[\mathcal{V}_{1},\mathcal{V}_{2},\dots,\mathcal{V}_{Z}]$. We map
the $Z$ videos into a lower dimensional 3D representation, extracting for each
video: (1) a list of $K$ visual 3D keypoints that describe the scene,
$[k_{1},k_{2},\dots,k_{K}]$ where $k=(x_{k},y_{k},z_{k})$, and (2) the
movement of the human’s hand as a trajectory of length $T$ of 21 hand joints
that parametrise the MANO hand model [23],
$\mathcal{J}=[j_{1},j_{2},\dots,j_{T}]$, where
$j=[[x_{0},y_{0},z_{0}],\dots,[x_{21},y_{21},z_{21}]]$. Finally, at the second
stage of R+X, which we call the execution phase, to emulate the behaviours
observed in the retrieved video clips, we condition a few-shot in-context
imitation learning model [22] on this data which, given a live observation of
the environment, generates a trajectory of 3D hand joints to execute the
desired behaviour described in the prompt. To map such joints to gripper
poses, we designed a heuristic that we describe in the Supplementary Material.

###  3.1 Retrieval: Extracting visual examples from a long, unlabelled video

The first main phase of R+X is retrieval, shown in Figure 3. Upon receiving a
language command $\mathcal{L}$, and given the human video $\mathcal{H}$, the
goal is to retrieve all video clips from the human video that depict the
execution of the requested task. This is accomplished using a recent VLM,
Gemini Pro 1.5 Flash [24]. Gemini, which we denote $\mathcal{G}$, is natively
multi-modal and can take as input images, videos, and text, and outputs text.
We prompt the model with the human video, and ask it to retrieve the starting
and ending seconds at which the received task happens. The inputs of this
phase are therefore a language command and the human video, and the output is
a list of $Z$ shorter video clips demonstrating the desired task,
$\mathcal{G}(\mathcal{H},\mathcal{L})\rightarrow[\mathcal{V}_{1},\dots,\mathcal{V}_{Z}]$.
Each clip comprises $T$ RGB-D frames, where $T$ can vary across different
clips.

####  3.1.1 Preprocessing videos into a sparse 3D representation

Given the $Z$ extracted video clips,
$[\mathcal{V}_{1},\dots,\mathcal{V}_{Z}]$, we apply a preprocessing step that
converts each video clip from a list of RGB-D frames to a set of 3D points
describing the visual scene and the trajectory of the human’s hand, a
representation that we will then feed to our few-shot in-context imitation
learning model.

Visual 3D keypoints. To transform the complex RGB-D observations into a lower-
dimensional, easier to interpret input, we harness the powerful vision
representations generated by DINO [25], a recent Vision Foundation Model. As
proposed in [26, 22], given the $Z$ clips retrieved as described before, we
find a set of $K$ common 3D visual keypoints that capture interesting semantic
or geometrical aspects of the scene. We extract these keypoints from the first
frame of each of the $Z$ videos only.

Figure 5: Visual keypoints are extracted autonomously from RGB via a matching
algorithm [26].

We first compute the DINO descriptors for the first frame of each of the $Z$
videos, obtaining a list of $Z$ different $N\times 384$ outputs, where $N$ is
the number of patches of each image [27, 28, 25]. Via a clustering and
matching algorithm [26], we select from the $N\times 384$ descriptors of the
first frame of the first video, $\mathcal{O}_{\mathcal{V}_{1},1}$, a list of
the $K$ descriptors that are the most common in all the remaining $Z-1$
frames, as shown in Figure 5. We denote these descriptors as
$\mathcal{D}\in\mathbb{R}^{K\times 384}$. Therefore, this way, we autonomously
extract descriptors that focus on the object of interest, as its appearance is
common among videos, while distractors and overall scene will vary [26, 22].

Finally, for each of the $K$ descriptors in $\mathcal{D}$, we extract
keypoints by finding the $K$ nearest neighbours between the $N\times 384$
descriptors of each of the $Z$ frames, and compute their 2D coordinates. We
then project these in 3D using each frame’s depth image and known camera
intrinsics. As such, given the retrieved clips, we obtain and store a list
$\Lambda=[\mathcal{K}_{V_{1}},\dots,\mathcal{K}_{V_{z}}]\in\mathbb{R}^{Z\times
K\times 3}$ of visual 3D keypoints, $K$ for each clip, where each
$\mathcal{K}_{V_{i}}=[k_{1},...,k_{K}]$ and $k_{j}=(x_{j},y_{j},z_{j})$. For a
more detailed description please refer to Amir et al. [26].

Hand Joint Actions. To extract human actions from each retrieved video clip we
use the HaMeR model [29], a recent technique for 3D hand pose estimation from
images based on DINO. In particular, using HaMeR we extract the 3D hand pose
from each video frame $\mathcal{O}_{\mathcal{V}_{z},t}$ (where $1\leq t\leq
T$) of each of the $Z$ clips, represented as a set of 21 3D points,
$j_{\mathcal{V}_{z},t}$, describing the hand joints that parameterise the MANO
hand model, as it is commonly done in the literature [23]. As the camera moves
between frames, we design a stabilisation technique to compute transformations
between camera poses that is robust to dynamic scenes (for more details please
see our Supplementary Material). This enables us to express the extracted hand
joints relative to a single reference frame, that of the first camera frame of
each clip. For each video clip $\mathcal{V}_{z}$, this process results in a
sequence of 3D hand joint actions
$\mathcal{J}_{\mathcal{V}_{z}}=[j_{\mathcal{V}_{z},1},\dots,j_{\mathcal{V}_{z},T}]$,
expressed relative to the first frame of each video
$\mathcal{O}_{\mathcal{V}_{z},1}$ where the visual 3D keypoints are also
expressed in. As such, at the end of this process we are left with a list
$\mathcal{M}=[\mathcal{J}_{\mathcal{V}_{1}},...\mathcal{J}_{\mathcal{V}_{Z}}]$
of $Z$ hand joint action sequences.

In summary, from the $Z$ retrieved video clips, we extract the list of hand
joints actions $\mathcal{M}$, along with the list of visual 3D keypoints
$\Lambda$, that will be used as context for our in-context imitation learning
model, as $Z$ input-output pairs.

###  3.2 Execution: Few-Shot, In-Context Imitation from Video Examples

The second main phase of R+X is execution. Our framework is based on the use
of a model capable of performing few-shot, in-context imitation learning,
receiving a few examples of desired inputs and outputs pairs describing a
desired behaviour, and able to replicate such behaviour immediately upon
receiving a new input. We use Keypoint Action Tokens (KAT) [22] to achieve
this, a recently proposed technique that takes 3D visual keypoints as input
and outputs a trajectory of 3D points describing the gripper movement. Instead
of explicitly training a model on robot data, KAT demonstrates that recent,
off-the-shelf Large Language Models are able to extract such numerical
patterns, and behave as few-shot, in-context imitation learning machines,
without the need for any further finetuning.

Given the output of the retrieval phase $[\Lambda,\mathcal{M}]$ and a new,
live RGB-D observation collected by the robot $\mathcal{O}_{live}$, we extract
its visual 3D keypoints representation $\mathcal{K}_{live}$ by first
extracting its $N\times 384$ DINO descriptors, and then finding the $K$
nearest neighbours to each of the $K$ descriptors in $\mathcal{D}$, that we
obtained in the retrieval phase. This results in $K$ 2D coordinates, that we
project in 3D. We then input to KAT as context $[\Lambda,\mathcal{M}]$, as $Z$
examples of the desired input-output mapping, and the new visual 3D keypoints,
and generate a new trajectory of desired hand joint actions,
$\text{KAT}([\Lambda,\mathcal{M}],\mathcal{K}_{live})\rightarrow\mathcal{J}_{live}$
with $\mathcal{J}_{live}=[j_{1},\dots,j_{T}]$, as shown in Figure 4. We then
map the predicted trajectories of hand joints actions into gripper poses as
described in our Supplementary Material, and execute them.

To summarize, given a language command $\mathcal{L}$ and a human video
$\mathcal{H}$, R+X first retrieves $Z$ videos depicting the described
behaviour using a VLM. From the retrieved videos, a list of
$[\Lambda,\mathcal{M}]$ visual 3D points and hand joint actions are extracted.
Then, to execute the desired behaviour described in the prompt $\mathcal{L}$,
$[\Lambda,\mathcal{M}]$ along with the visual 3D keypoints of the live
observation $\mathcal{K}_{live}$ are used as context for KAT, which performs
few-shot in-context imitation learning to generate a trajectory of hand joints
actions, which is then mapped to gripper poses and executed on the robot.

Figure 6: We test R+X on 12 everyday tasks, executed by a human in different
rooms and with different distractors.

##  4 Experiments

Human Video. We collect the human video $\mathcal{H}$ using an Intel RealSense
455, worn by the human on their chest as shown in Figure 1. To reduce
downstream computational time, we filter out each frame in which human hands
are not visible right after recording. As our robot is single-armed, we limit
ourselves to single-hand tasks. However, our method could be applied to
bimanual settings and dexterous manipulators.

Robot Setup. At execution, we use a Sawyer robot equipped with a RealSense 415
head-camera and a two-fingered parallel gripper, the Robotiq 2F-85. As the
robot is not mobile, we set up different scenes in front of it with variations
of the tasks recorded by the human, placing several different distractors for
each task, while the human video was recorded in many different rooms.
Although we have a wrist-camera mounted to the robot as shown in some of our
figures, we do not use it in our work.

Tasks. To evaluate our proposed framework, we use a set of 12 everyday tasks,
where the human interacts with a series of common household objects, listed in
Fig. 6. We include movements like grasping, opening, inserting, pushing,
pressing, and wiping.

Baselines. We compare R+X, and its retrieval and execution design, to training
a single, language-conditioned policy. To obtain language captions from the
human video, we use Gemini to autonomously caption snippets of the video,
obtaining a (observation, actions, language) dataset. We finetune R3M
(ResNet-50 version [30]) [31] and Octo [32] on this data. We extend R3M to
also encode language via SentenceBERT and use a Diffusion Policy [33] head to
predict actions from intermediate representations. We denote this version as
R3M-DiffLang.

More details on all aspects are provided in the Supplementary Material.

Method / Task | Plate | Push | Wipe | Beer | Wash | Box | Kettle | Micro. | Basket | Phone | Can | Light | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
R3M-DiffLang | 0.5 | 0.7 | 0.4 | 0.7 | 0.5 | 0.5 | 0.4 | 0.8 | 0.7 | 0.4 | 0.7 | 0.3 | 0.55
Octo | 0.5 | 0.8 | 0.5 | 0.6 | 0.5 | 0.5 | 0.4 | 0.7 | 0.6 | 0.4 | 0.6 | 0.3 | 0.53
R+X | 0.6 | 0.8 | 0.7 | 0.8 | 0.6 | 0.7 | 0.6 | 0.8 | 0.7 | 0.7 | 0.8 | 0.6 | 0.7
Table 1: Success rates of the various methods on the 12 proposed tasks.

###  4.1 Results

Can R+X learn robot skills from long, unlabelled videos? How does it perform
with respect to a monolithic language-conditioned policy? In these
experiments, we evaluate the performance of R+X in learning a set of everyday
skills. At deployment, we place the object to be interacted with in front of
the robot, and issue a language command. We then run the retrieval and
execution pipeline using the human video, the current observation and the
language command. We run 10 episodes per task, for each episode randomising 1)
the object pose, 2) the type and number of distractors, 3) for tasks where it
is possible, we swap the object to interact with, with another one from the
same class to test for generalisation (e.g. a different can, a different piece
of clothing, a different telephone). More details are provided in the
Supplementary Material.

Figure 7: Examples of hard spatial and language generalisation. Gripper moves
from red to blue.

In Table 1, we report the performance of R+X on these tasks, together with the
baselines. As we demonstrate, the framework is able to tackle a wide range of
everyday tasks, surpassing the monolithic policy baselines. These results
prove the benefit of modeling language-conditioned learning from observation
as distinct retrieval and execution phases, to fully leverage the abilities of
recent Foundation Models.

What are the main sources of difference in performance between R+X and a
monolithic policy? In these experiments, we investigate more in detail what
changes in the inputs lead to the most noticeable difference in performance
between R+X and the baselines, R3M-DiffLang and Octo. We explored two aspects,
related to two properties of R+X:

Hard Spatial Generalisation: R+X, by retrieving videos of the desired task,
can also extract a series of relevant keypoints $\mathcal{K}_{live}$ from the
current observation $\mathcal{O}_{live}$, something not possible when using a
single policy network. Going from RGB-D to a list of 3D points for inputs and
outputs allows us to apply simple geometric data augmentation techniques for
KAT, such as normalisation or random translations and rotations. This leads to
a stronger spatial generalisation: in the "grasp a can" and "grasp a beer"
tasks, we test performance of each method when the objects are on the table,
or when they are positioned on top of other objects (a box, a microwave), a
situation never encountered in the training videos. By running 5 test episodes
for each case, we demonstrate how R+X retains strong performance, while the
performance of the baselines drop. Results and an example of the predicted
actions can be seen in Fig. 7.

Hard Language Generalisation: By leveraging the language and video
understanding abilities of recent large Vision Language Models, such as
Gemini, R+X can interpret and execute nuanced commands. To evaluate this, we
set up a scene with many objects, and ask the robot to perform three tasks:
"give me something to call my mom", "give me a non-alcoholic drink", and "make
the room less dark". We run 5 test episodes for each of these commands,
modifying the positions of the objects and the distractors. The language-
conditioned policies struggle to interpret these commands, since they are
strongly out of distribution. R+X, on the other hand, leverages Gemini’s
ability to understand the meaning of these commands, as it is able to retrieve
useful video clips from the human video (respectively, picking up the phone,
grasping a Fanta can, and turning on the light). Results and an example of
output gripper trajectories can be seen in Fig. 7.

Figure 8: Left (barplots 1-3): We compare R+X and the baselines’ ability to
learn tasks in succession, and the time needed to learn such new tasks.
Rightmost barplot: Gemini’s retrieval performance.

Can R+X learn task sequentially over time? In these experiments, we
demonstrate how R+X can learn tasks continually, with no need for any
additional training or finetuning, while obtaining strong performance both on
the new task and on the old ones, a desirable ability for a robot learning
from an ever increasing dataset of human experience. To measure this ability,
and highlight the different behaviour with respect to a single language-
conditioned policy, we first collect 10 demos for 3 tasks: grasping a can,
cloth in basket, and picking up the phone. We train the baselines on these
tasks (after extracting captions via Gemini as described for the experiments
of Table 1) and evaluate them and R+X. Then, we add 10 demos of 3 new tasks
(kettle on stove, open box, plate in rack), finetune the baselines, then
measure performance on the 3 new tasks, and on the 3 old tasks, for all
methods. In Fig. 8, Left, we see how the performance of R3M-DiffLang and Octo
deteriorates on the old tasks, due to the well known effect of catastrophic
forgetting [34]. On the other hand, R+X performance does not deteriorate, due
to the ability to retrieve from an ever-growing video of tasks and adapting
the behaviour model at test time on the retrieved data.

If instead we train the baselines on all the data each time, the performance
does not deteriorate: however, this leads to a substantial growth in time
needed to train. As the dataset grows, those networks would need an increasing
amount of time per each new added task in order to be retrained, while R+X
does not need any training or finetuning as further tasks are added to the
video.

How does the length of the videos affect the retrieval performance? In this
experiment, we first collect a video with 10 demos of 5 tasks (kettle on
stove, cloth in basket, pick up phone, cloth in washing machine, close
microwave), and measure Gemini’s ability to retrieve clips for each task. We
then add 10 demos for 5 additional tasks, and compute again the precision and
recall of retrieval of the original 5 tasks, to study how length of video and
number of tasks affects it. The original video is close to 4 minutes in
length, while the second is close to 10 minutes. In Fig. 8, Right, we show the
mean over all tasks of the retrieval’s precision and recall. Results
demonstrate how Gemini’s performance remains high while increasing the length
of the video, allowing to scale R+X as the human records more and more tasks.

##  5 Discussion and Limitations

We have presented R+X, a framework to learn robot skills from long, unlabelled
videos of humans interacting naturally in everyday environments, and
demonstrated the clear benefit of using retrieval over training monolithic
language-conditioned policy networks. There are, however, limitations which we
now describe, opening up possible avenues for future work. The method is
currently bottlenecked by the performance of Vision Language Model; however,
such models have seen dramatic improvement in recent times and this will
likely continue. While we can learn a large series of everyday tasks, the
errors arising from the hand pose prediction currently inhibits us from
learning precise tasks. Furthermore, while the keypoints extraction leads to
strong spatial generalisation properties, it can become a bottleneck in the
presence of many similar objects; methods that find a dynamic number of
keypoints might tackle this issue. Finally, due to the use of Foundation
Models for retrieval and execution, our method has a few seconds of delay
between the issuing of the command and execution; however, this has reduced
dramatically in recent months with the release of faster models such as GPT-4o
[35] or Gemini Flash [24].

## References

  * Jang et al. [2021] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn.  BC-z: Zero-shot task generalization with robotic imitation learning.  In _5th Annual Conference on Robot Learning_ , 2021.
  * Johns [2021] E. Johns.  Coarse-to-fine imitation learning: Robot manipulation from a single demonstration.  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2021.
  * Collaboration [2023] O. X.-E. Collaboration.  Open x-embodiment: Robotic learning datasets and rt-x models, 2023.  arXiv:2310.08864.
  * [4] Apple.  Apple vision pro.  URL https://www.apple.com/apple-vision-pro/.
  * [5] Meta.  Meta quest 3.  URL https://www.meta.com/gb/quest/quest-3/.
  * [6] M. Ray-Ban.  Ray-ban smart glasses.  URL https://www.ray-ban.com/uk/ray-ban-meta-smart-glasses.
  * [7] MagicLeap.  Magicleap.  URL https://www.magicleap.com/.
  * Jain et al. [2024] V. Jain, M. Attarian, N. J. Joshi, A. Wahid, D. Driess, Q. Vuong, P. R. Sanketi, P. Sermanet, S. Welker, C. Chan, et al.  Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers.  _arXiv e-prints_ , pages arXiv–2403, 2024.
  * Bahl et al. [2022] S. Bahl, A. Gupta, and D. Pathak.  Human-to-robot imitation in the wild.  _arXiv preprint arXiv:2207.09450_ , 2022.
  * Heppert et al. [2024] N. Heppert, M. Argus, T. Welschehold, T. Brox, and A. Valada.  Ditto: Demonstration imitation by trajectory transformation.  _arXiv preprint arXiv:2403.15203_ , 2024.
  * Bahety et al. [2024] A. Bahety, P. Mandikal, B. Abbatematteo, and R. Martín-Martín.  Screwmimic: Bimanual imitation from human videos with screw space projection.  _arXiv preprint arXiv:2405.03666_ , 2024.
  * Zhu et al. [2024] Y. Zhu, A. Lim, P. Stone, and Y. Zhu.  Vision-based manipulation from single human video with open-world object graphs.  _arXiv preprint arXiv:2405.20321_ , 2024.
  * Wang et al. [2024] C. Wang, H. Shi, W. Wang, R. Zhang, L. Fei-Fei, and C. K. Liu.  Dexcap: Scalable and portable mocap data collection system for dexterous manipulation.  _arXiv preprint arXiv:2403.07788_ , 2024.
  * Brohan et al. [2023] A. Brohan et al.  RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.  _arXiv e-prints_ , art. arXiv:2307.15818, July 2023.  doi:10.48550/arXiv.2307.15818.
  * Driess et al. [2023] D. Driess et al.  PaLM-e: An embodied multimodal language model.  In A. Krause et al., editors, _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 8469–8488. PMLR, 23–29 Jul 2023.  URL https://proceedings.mlr.press/v202/driess23a.html.
  * Brown et al. [2020] T. Brown et al.  Language models are few-shot learners.  In H. Larochelle et al., editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Gemini-Team [2023] Gemini-Team.  Gemini: A family of highly capable multimodal models.  _arXiv_ , 2023.
  * Mirchandani et al. [2023] S. Mirchandani et al.  Large Language Models as General Pattern Machines.  _arXiv e-prints_ , art. arXiv:2307.04721, July 2023.  doi:10.48550/arXiv.2307.04721.
  * Hoffmann et al. [2022] J. Hoffmann et al.  Training Compute-Optimal Large Language Models.  _arXiv e-prints_ , art. arXiv:2203.15556, Mar. 2022.  doi:10.48550/arXiv.2203.15556.
  * Wang et al. [2023] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar.  Mimicplay: Long-horizon imitation learning by watching human play.  _arXiv preprint arXiv:2302.12422_ , 2023.
  * Kwon et al. [2023] T. Kwon, N. D. Palo, and E. Johns.  Language models as zero-shot trajectory generators, 2023.  arXiv:2310.11604.
  * Di Palo and Johns [2024] N. Di Palo and E. Johns.  Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics.  _arXiv e-prints_ , art. arXiv:2403.19578, Mar. 2024.  doi:10.48550/arXiv.2403.19578.
  * Romero et al. [2017] J. Romero, D. Tzionas, and M. J. Black.  Embodied hands: modeling and capturing hands and bodies together.  _ACM Transactions on Graphics_ , 36(6):1–17, Nov. 2017.  ISSN 1557-7368.  doi:10.1145/3130800.3130883.  URL http://dx.doi.org/10.1145/3130800.3130883.
  * Team [2024] G. Team.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv_ , 2024.
  * Caron et al. [2021] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  _arXiv_ , 2021.
  * Amir et al. [2022] S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel.  Deep vit features as dense visual descriptors, 2022.  arXiv:2112.05814.
  * Dosovitskiy et al. [2021] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv_ , 2021.
  * OpenAI [2023] OpenAI.  GPT-4 Technical Report.  _arXiv e-prints_ , art. arXiv:2303.08774, Mar. 2023.  doi:10.48550/arXiv.2303.08774.
  * Pavlakos et al. [2024] G. Pavlakos, D. Shan, I. Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik.  Reconstructing hands in 3D with transformers.  In _CVPR_ , 2024.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016.  doi:10.1109/CVPR.2016.90.
  * Nair et al. [2022] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta.  R3m: A universal visual representation for robot manipulation.  _arXiv_ , 2022.
  * Octo Model Team et al. [2024] Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, L. Y. Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine.  Octo: An open-source generalist robot policy.  In _Proceedings of Robotics: Science and Systems_ , Delft, Netherlands, 2024.
  * Chi et al. [2023] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song.  Diffusion policy: Visuomotor policy learning via action diffusion.  In _Proceedings of Robotics: Science and Systems_ , 2023.
  * Wang et al. [2024] L. Wang, X. Zhang, H. Su, and J. Zhu.  A comprehensive survey of continual learning: Theory, method and application.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 46(8):5362–5383, 2024.  doi:10.1109/TPAMI.2024.3367329.
  * [35] OpenAI.  Gpt-4o.  URL https://openai.com/index/hello-gpt-4o/.
  * Zhu et al. [2024] Y. Zhu, Z. Ou, X. Mou, and J. Tang.  Retrieval-augmented embodied agents.  _arXiv preprint arXiv:2404.11699_ , 2024.
  * Du et al. [2023] M. Du, S. Nair, D. Sadigh, and C. Finn.  Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets.  _arXiv preprint arXiv:2304.08742_ , 2023.
  * Di Palo and Johns [2024] N. Di Palo and E. Johns.  DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models.  _arXiv e-prints_ , art. arXiv:2402.13181, Feb. 2024.  doi:10.48550/arXiv.2402.13181.
  * Palo and Johns [2024] N. D. Palo and E. Johns.  On the effectiveness of retrieval, alignment, and replay in manipulation.  _RA-Letters_ , 2024.
  * Borgeaud et al. [2022] S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. De Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. Rae, E. Elsen, and L. Sifre.  Improving language models by retrieving from trillions of tokens.  In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 2206–2240. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/borgeaud22a.html.
  * Lüddecke and Ecker [2021] T. Lüddecke and A. S. Ecker.  Prompt-based multi-modal image segmentation.  _arXiv_ , abs/2112.10003, 2021.  URL https://arxiv.org/abs/2112.10003.
  * Matsuki et al. [2024] H. Matsuki, R. Murai, P. H. J. Kelly, and A. J. Davison.  Gaussian splatting slam.  _arXiv_ , 2024.  URL https://arxiv.org/abs/2312.06741.
  * Mur-Artal et al. [2015] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos.  Orb-slam: A versatile and accurate monocular slam system.  _IEEE Transactions on Robotics_ , 31(5):1147–1163, Oct. 2015.  ISSN 1941-0468.  doi:10.1109/tro.2015.2463671.  URL http://dx.doi.org/10.1109/TRO.2015.2463671.
  * Doersch et al. [2023] C. Doersch, Y. Yang, M. Vecerik, D. Gokay, A. Gupta, Y. Aytar, J. Carreira, and A. Zisserman.  Tapir: Tracking any point with per-frame initialization and temporal refinement.  _arXiv_ , 2023.
  * Reimers and Gurevych [2019] N. Reimers and I. Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2019.  URL https://arxiv.org/abs/1908.10084.
  * Ronneberger et al. [2015] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  _arXiv_ , 2015.

##  6 Supplementary Material

###  6.1 Additional Related Work

In this section, we discuss some additional related papers from the
literature.

Retrieval in Robotics. Other papers have proposed the use of retrieval in
robotics. Zhu et al. [36] retrieves directly from a bank of policies trained
on robotics data. Du et al. [37] retrieves from an unlabelled data of robotics
experience, not human videos, and needs an additional example at test time to
guide the retrieval phase, while we leverage a language command alone. Di Palo
and Johns [38, 39] retrieves a goal observation to then perform visual
servoing and replay a pre-recorded demo. R+X leverages large Vision Language
Models to retrieve directly from long videos given a language query, without
requiring pre-trained policies or new demonstrations as queries.

Retrieval Augmented Generation in Language Modelling: Large Language Models,
through extensive pre-training and finetuning on web-scale datasets, are able
to implicitly record countless facts and concepts in their weights [28, 35,
17]. However, it has been proved useful to add to their implicit knowledge the
ability to explicitly access datasets of information, either offline or
through web search [40]. This allows models to be more factually correct. In
this work, we do not deal with factuality or information retrieval, but we
take inspiration from these ideas to form a sort of "retrieval augmented
execution", which is the main idea behind R+X.

###  6.2 Processing the Long, Unlabelled Human Video

After recording the video of a human performing everyday tasks, we process it
to remove unnecessary frames, i.e., frames that do not contain human hands. We
achieve this automatically by leveraging HaMeR [29] to detect the frames where
hands are present. Consequently, we are left with one long, unlabelled video
of smaller video clips, concatenated together, each containing only frames
where a human interacts with various objects with their hands. This video is
then passed to Gemini to proceed with the retrieval phase of R+X, as discussed
in section 3.1. Fig. 9 shows a short segment of the uncut video we recorded.
The frames marked with red correspond to frames where no human hands were
detected. The frames where human hands were detected are marked with green.
After processing the video with HaMeR [29] we remove the frames marked with
red and are left with one long video of the green frames concatenated
together.

Figure 9: A short segment of our recorded long, unlabelled video of a human
performing everyday tasks. The frames marked with red correspond to frames
where no human hands were detected, while the frames marked with green
correspond to frames where HaMeR [29] detected human hands. In practice, we
discard the frames where no human hands are present and only retain a long
video of frames concatenated together depicting only the human interacting
with various objects.

###  6.3 Stabilisation of First Person Videos

Figure 10: An example of keypoints being tracked in a retrieved video clip.
Notice how, thanks to the segmentation model [41], we can sample keypoints
that stick to static parts of the scene, like the table.

As we record videos using a wearable camera, the point of view of the camera
changes substantially frame by frame. However, our method is based on
representing the visual inputs as fixed, 3D keypoints, and the actions as a
series of 3D hand joints actions expressed in the same frame of the keypoints.
Therefore, we need to compute the relative movement of the camera at each step
in order to express everything in the original frame of reference, the first
of the recorded video clip.

The computer vision literature has proposed several methods to address this
problem [42, 43]. However, most of these techniques assume that the scene in
front of the camera is fixed, and only the camera is moving. This is in sharp
contrast with our setting, where the arms and hands of the human move in front
of the camera, together with some of the objects. Classic techniques, in our
experiments, failed to achieve a robust estimation of the movement of the
camera, a task we hereby refer has the stabilisation of the camera. In the
following section, we describe how we stabilise each video clip we retrieve
from $\mathcal{H}$ while using R+X.

In order to tackle the aforementioned problem, we leverage the use of a series
of recent computer vision models. The first one, CLIPSeg [41], is an open-
vocabulary image segmentation model. The model receives an image and a series
of language queries as an input, and outputs a segmentation mask localising
the pixels of the desired objects,
$\text{CLIPSeg}(\mathcal{O}_{live},[\mathcal{L}_{obj,0},\dots,\mathcal{L}_{obj,n}])\rightarrow
M_{live}\in\mathbb{R}^{H\times W\times 1}$, where $H$ and $W$ are the height
and width of the observation. We query CLIPSeg with some static parts of any
scene, namely "floor", "wall", "table". In addition, we query it with "arm",
"hand", "person" in order to obtain segmentation masks of parts of the human
arms, and remove it from the segmentation mask of the static objects.

Given the segmentation of such static objects, we sample a set of random 2D
pixel keypoints in that area. We then track the movement of such keypoints
frame by frame using TAPIR [44], a recent keypoint-tracking network. The
position of these keypoints in 2D at each frame allows us to then project them
in 3D using the depth-channel from the RGB-D camera we use. Given these sparse
point-clouds, we can compute the relative $SE(3)$ rigid transformation between
each new frame and the first one. We perform this process separately for each
video clip where human hands where detected as discussed in section 6.2.

This pipeline, in our experiments, has proven to be better than classic
techniques [43], as it can handle the presence of the moving hands and arms of
a person, together with the movement of some objects the human interacts with,
as it learns to only focus on static parts of the environment, ignore the
human, and track the selected keypoints over time.

###  6.4 Extracting Gripper Actions from Human Hands

To map the human hand joints to gripper actions we deploy different heuristics
based on the type of interaction the human performs in the retrieved video
clips. In scenarios where a robot is equipped with an end-effector with human-
like fingers, such as a dexterous 5-fingered hand, such heuristics are not
needed as the hand joints would map directly to the robotic, human-like
fingers.

Fig. 11 shows instances of extracted hand joints using HaMeR [29] mapped to
different gripper actions for grasping, pressing, and pushing tasks. For ease
of illustration, this figure demonstrates how human hand joints are mapped to
gripper poses from images captured in the human video. In practice, we deploy
the following heuristics on the predictions made by each method. Our
heuristics are applied to the Robotiq Gripper 2F85, but can be adapted
accordingly to any other parallel jaw gripper.

Grasping.  (Fig. 11, top) For tasks that require grasping we align spatially
the tips of the gripper’s fingers with the index and thumb tips of the hand.
The vector joining the two tips defines an axis of rotation around which we
rotate the gripper such that the bottom of the gripper is as close as possible
to the middle point between the index mcp and the thumb dip. Aligning the
gripper to these points enables us to obtain the gripper’s pose. Consequently,
for grasping tasks each method predicts, the index tip, thumb tip, index mcp
and thumb dip joints. To compute the gripper close/open action, we employ an
heuristic that measures the distance of the index and thumb tips and compares
it to the robot gripper width.

Pressing.  (Fig. 11, middle) For pressing tasks, we assume that the gripper is
closed. To obtain the pose of the gripper we use three points at the point of
contact of the two gripper fingers when the gripper is closed. Specifically,
one point is at the tip of the fingers, the other in the middle, and the last
at the bottom of the gripper’s fingers. We align these three points to the
index tip, index pip, index mcp, and index dip of the human hand respectively.
Consequently, each method predicts the index tip, index pip, index mcp, and
index dip.

Pushing. (Fig. 11, bottom) Pushing tasks, are similar to pressing tasks, where
we use the same three points on the gripper but to obtain the gripper’s pose,
we instead align these points with the joints on the hand halfway between the
index and middle finger tip, pip, mcp and dip.

For all heuristics, we match the pose of the gripper with the joints on the
hand using singular value decomposition.

Detecting which heuristic to use is done automatically by Gemini based on the
human’s language command during task execution. As Gemini exhibits strong
semantic language understanding, it can determine whether the requested task
from the human involves grasping, pressing, or pushing.

Figure 11: Examples of different human hand interactions and extracted gripper
actions based on three different heuristics for grasping, pressing, and
pushing. The blue dots in the third column correspond to the 21 hand joints
extracted using HaMeR [29]. Which heuristic to use is determined by Gemini at
the Retrieval phase based on Gemini’s understanding of the language command
provided by the human. Note that for the pressing and pushing task the gripper
is rendered as open, but in practice the gripper is closed as if it grasping
something.

###  6.5 Tasks Details and Success Criteria

We here list more details about the tasks we use in R+X, the generalisation
abilities we study, and the success criteria.

  * •

Plate in Rack: We randomise the position of the rack on the table in a 40cm by
30cm (width, height) area and its orientation in a $[-30,30]$ degrees range.
The robot starts with a plate grasped in its gripper. We place random
distractors in half the runs, while half are distractors free. The task is
successful if the plate can stand in one of the slots of the rack without
falling over.

  * •

Push Rack in Dishwasher: Due to the kinematics of the robot, we always put the
dishwasher on the right side of the table, and randomise its position in a
35cm by 20cm area, and its orientation in a $[-20,20]$ degrees range. As the
object is considerably large, we cannot randomise excessively its starting
point or it will end out of view/out of the robot’s working area. We place
random distractors in half the runs, while half are distractors free. The task
is successful if the rack is no more than 5cm outside of the dishwasher, and
starts being around 15cm outside of it.

  * •

Wipe Marker from Table: We draw a random spot with a marker on an horizontally
placed whiteboard, that emulates a mark on a table, some dirt, or something
spilled. We randomise the shape of the spot, that is generally $15cm^{2}$ in
area, and its position in a 40cm by 30cm area. The robot starts with a
cleaning tool grasped. We place random distractors in half the runs, while
half are distractors free. The task is successful if more than 80$\%$ of the
spot it erased and cleaned.

  * •

Grasp Beer: We place a beer in a random spot on the table in a 40cm by 30cm
area. In this case, we also place it 2 out of 10 times on top of other objects
to test for hard spatial generalisation. This setting is better studied in the
"Hard Spatial Generalisation" subsection of the main paper. We place random
distractors in half the runs, while half are distractors free. The task is
solved if the robot has grasped and lifted the beer.

  * •

Cloth in Washing Machine: We place the washing machine on the right side of
the table, randomising its position in a 20cm by 20cm area, and its
orientation in a $[-20,20]$ degrees range. We then place a random piece of
clothing (generally clothing for kids/infants) on the left side of the table,
and randomise its position in a 25cm by 25cm area, therefore testing for both
spatial and object generalisation. We place random distractors in half the
runs, while half are distractors free. The task is completed if the robot can
place the cloth inside the hole of the washing machine, also if part is
outside: as our toy washing machine is very small and its receptacle is
shallow (around 15cm), it is hard also for a human to fit a cloth entirely in
there.

  * •

Open Box: We place one of two possible boxes of different sizes in a 30cm by
30cm area on the table, and randomise its orientation in a $[-20,20]$ degrees
range. We place random distractors in half the runs, while half are
distractors free. The task is successful if, during the episode, the robot has
partially opened the box. Opening it completely is often challenging due to
kinematics constraints.

  * •

Kettle on Stove: We place a toy stove on the table, randomising its position
in a 30cm by 20cm area, and its orientation in a $[-20,20]$ degrees range. We
then randomly place a random kettle in a 20cm by 20cm area and randomise its
orientation in a $[-30,30]$ degrees range. By placing seen and unseen kettles,
we test both for spatial and object generalisation. We place random
distractors in half the runs, while half are distractors free. The task is
successful if the kettle is on top of the stove, which we consider true if the
intersection of the stove area and the kettle base is more than 50$\%$ than
the stove area.

  * •

Close Microwave: We place a toy microwave on the right side of the table,
randomising its position in a 20cm by 20cm area and its orientation in a
$[-20,20]$ degrees range. Additionally, we open its door in a range between
$[10,45]$ degrees, We place random distractors in half the runs, while half
are distractors free. We consider the task successful if the microwave door is
completely closed (it has a magnetic system that keeps it closed once it is
completely pushed in position).

  * •

Cloth in Basket: We place a basket on the right side of the table in a 20cm by
20cm area, and randomise its orientation in a $[-20,20]$ degrees range. We
place a random cloth (same as for the washing machine task) on the left side
of the table, randomising its position in a 25cm by 25cm area.We place random
distractors in half the runs, while half are distractors free. We consider the
task successful if the cloth is contained in the basket, i.e. if lifting the
basket, the cloth is lifted as well without falling on the table.

  * •

Pick Up Phone: We randomly place one out of two possible phones on the table,
randomising its position in a 30cm by 30cm area, and its orientation in a
$[-30,30]$ degrees range. By using different phones, one of which unseen at
training, we also test for object generalisation. We place random distractors
in half the runs, while half are distractors free. We consider the task
successful if the robot has grasped and lifted the phone from its base.

  * •

Grasp Can: Same setting as for Grasp Beer. In this case, we also use other
unseen cans at test time to test for object generalisation.

  * •

Turn On Light: We place a light bulb and a socket on the table, randomising
their positions independently in a 30cm by 30cm area, and the socket
orientation in a $[-20,20]$ degrees range. We place random distractors in half
the runs, while half are distractors free. The task is successful if the robot
can press the light bulb’s plug entirely in the socket, turning on the light.

Figure 12: Examples of generalisation to distractors. Gripper moves from red
to blue.

Further Investigation: What are the main sources of difference in performance
between R+X and a monolithic policy? Here we further investigate sources of
differences in performance between R+X and the baselines, extending the
experimental investigation of the main paper. In particular, here we focus on
the presence of distractors.

Distractors Generalisation: One of the main differences between R+X and the
baselines is that the former, after retrieving a set of video clips from
$\mathcal{H}$ depicting the requested task, can extract a set of keypoints
that are semantically and geometrically meaningful for the objects to interact
with, as described in the main paper. These keypoints are generally robust to
distractors, as they focus only on the DINO features that are common in all
images, whereas distractors generally through out the human video
$\mathcal{H}$ and the execution. The baselines, being monolithic policies,
receive as input the observation as it is, and must learn to generalise to the
presence of distractors during the long training phase.

To evaluate the effect of distractors on the scene, we test R+X and the
baselines on two tasks, "grasp a can" and "put the cloth in the basket",
emulating the experimental scenario of the "Hard Generalisations" sections. We
run 10 runs without distractors, and 10 with, explicitly measuring the
performance in the two cases. We can see in the results of Fig. 12 how R+X is
more robust to the presence of random distractor objects, highlighting the
advantage of extracting semantically and geometrically meaningful keypoints
after having received a language command and having retrieved the
corresponding video clips.

###  6.6 Further Details on Baselines:

####  6.6.1 Octo

One of baselines that we have compared against is a fine tuned version of Octo
[32]. We have used the original code provided by the authors and adapted it
for the specific input and output relevant to this work. More specifically, we
designed the input of the model to be the RGB image of size $256\times 256$ of
the live observation, $\mathcal{O}_{live}$ and a language description of the
task $\mathcal{L}$. The output, instead, is $\mathcal{J}_{live}$, the
trajectory of desired hand joints 3D points to be translated to gripper poses
as described before. Unlike KAT, these methods require a fixed length of the
output trajectory. We therefore pad it to 40 times steps. The output is
ultimately a tensor of dimension 480, which corresponds to a flattened tensor
of dimension $(40,4,3)$. The 4 predicted hand joints are described in section
6.4 along with how they are mapped to robot actions.

Having set the input and output dimensions, as well as modalities, the model
has been fine tuned starting from the provided checkpoint "octo-small-1.5" and
it was trained until convergence.

####  6.6.2 R3M-DiffLang

The other baseline we have compared against has been the combination of R3M
[31] and [33], with the inclusion of language conditioning. The resulting
model leveraged the ResNet-50 variant of R3M to encode the $256\times 256$
live observation RGB image $\mathcal{O}_{live}$ and Sentence-BERT [45] to
encode the language description of the task $\mathcal{L}$. The resulting
feature vectors for the image and language description was then concatenated
and passed to a UNet [46] trained via diffusion following the original code of
[33]. The output of the diffusion head has the same size as the output of
Octo, as we pad the trajectory $\mathcal{J}_{live}$ to 40 timesteps. In order
to take advantage of the pre-trained vision knowledge of R3M, the encoder has
been initialised with the provided weights and kept frozen, as it has been
done for the language encoder. The diffusion policy head has been trained from
scratch using our own dataset of trajectories until convergence was reached.