# Observation of oscillating $g$-factor anisotropy arising from strong crystal
lattice anisotropy in GaAs spin-3/2 hole quantum point contacts

K. L. Hudson<EMAIL_ADDRESS>School of Physics, University of New South
Wales  ARC Centre of Excellence for Future Low-Energy Electronics Technologies
(FLEET)  A. Srinivasan  School of Physics, University of New South Wales  ARC
Centre of Excellence for Future Low-Energy Electronics Technologies (FLEET)
D. S. Miserev  University of Basel, Switzerland  Q. Wang  School of Physics,
University of New South Wales  ARC Centre of Excellence for Future Low-Energy
Electronics Technologies (FLEET)  O. Klochan  School of Physics, University of
New South Wales  ARC Centre of Excellence for Future Low-Energy Electronics
Technologies (FLEET)  O. P. Sushkov  School of Physics, University of New
South Wales  ARC Centre of Excellence for Future Low-Energy Electronics
Technologies (FLEET)  I. Farrer  University of Sheffield, United Kingdom  D.
A. Ritchie  Cavendish Laboratory, Cambridge, United Kingdom  A. R. Hamilton
School of Physics, University of New South Wales  ARC Centre of Excellence for
Future Low-Energy Electronics Technologies (FLEET)

###### Abstract

Many modern spin-based devices rely on the spin-orbit interaction, which is
highly sensitive to the host semiconductor and depends on the crystal
orientation, crystal asymmetry, and quantum confinement asymmetry. One-
dimensional quantum point contacts are a powerful tool to probe both energy
and orientation dependence of the spin-orbit interaction through the effect on
the hole $g$-factor. Here we investigate the role of cubic crystal asymmetry
in GaAs when the quantum point contact is rotated with respect to the crystal
axes. Unexpectedly the in-plane $g$-factor is found to be extremely sensitive
to point contact orientation, changing by a factor of $5$ when rotated by
$45^{\circ}$. This exceptionally strong crystal lattice anisotropy of the in-
plane Zeeman splitting cannot be explained within axially symmetric
theoretical models. Theoretical modelling based on the combined Luttinger,
Rashba and Dresselhaus Hamiltonians reveals new spin-orbit contributions to
the in-plane hole $g$-factor and provides excellent agreement with
experimental data.

Usage

PACS numbers

###### pacs:

Valid PACS appear here

Quantum confined semiconductor hole systems are attracting significant
research interest due to the strong spin-orbit interaction in the valence band
[1]. Strong, tunable and remarkably diverse spin-orbit interaction in 2DHG
spawned a new generation of quantum devices with all-electrical control of the
hole spin, that already found applications in spin-based transistors, spin
filters, spin-flip tunnelling, and spin qubits and read-out of spin qubits via
Pauli spin blockade [2, 3, 4, 5, 6], helical spin states [7, 8, 9, 10] and
topological states in one-dimensional wires [11, 12, 13, 14]. These
applications require a detailed understanding of the nature of the spin-orbit
interaction, which is much more complex for spin-3/2 holes than for spin-1/2
electrons.

To date most experimental studies have focussed on spin-orbit terms due to
crystal inversion asymmetry (Dresselhaus) and structural inversion asymmetry
(Rashba). Here we go further, and examine the effects of cubic crystal lattice
anisotropy on the spin-orbit interaction. However disentangling different
contributions to the spin-orbit interaction (SOI) in experimental data is a
challenging task. Quantum point contacts (QPCs) are a powerful platform for
probing the spin properties of a crystalline semiconductor systems, as the 1D
$g$-factor can be directly extracted from transport measurements and related
to SOI terms in the Hamiltonian. We use QPCs fabricated on (100) zinc blende
GaAs/AlGaAs heterostructures to quantitatively study the crystal orientation
dependence of the spin-orbit interaction through its effect on the Zeeman
splitting of the 1D subbands.

In a quantum confined system without SOI spin and momentum are not coupled,
therefore the $g$-factor is isotropic in a 2D quantum well, a 1D quantum wire
or a 0D quantum dot. In the presence of SOI, confining the particles to 2D
leads to a large out-of-plane $g$-factor and a small in-plane $g$-factor due
to the quantization of $k_{z}$ [15, 16, 17, 18]. Further confinement to a 1D
quantum wire gives rise to an anisotropy in the in-plane $g$-factors, which
has been intensely studied in GaAs spin-3/2 hole QPCs [19, 20, 21, 22]. In
these studies the relative size of the $g$-factor with the magnetic field
applied in-plane and parallel to the current ($g_{\parallel}$) is up to an
order of magnitude larger than when the magnetic field is applied in-plane and
perpendicular to the current ($g_{\perp}$). This anisotropy, which cannot be
explained with purely structural inversion asymmetry, has recently been
explained using a Luttinger based 2D planar model described in [23, 24] that
does not take into account the crystal lattice anisotropy.

In this paper we go further and investigate the ratio
$g_{\perp}/g_{\parallel}$ for different orientations of hole QPCs in $(100)$
GaAs/AlGaAs heterostructure. We use three QPCs fabricated along $[011]$,
$[010]$ and $[001]$ crystal directions, and find extremely large variation of
this ratio ranging from $g_{\perp}/g_{\parallel}\approx 0.15$ in $[011]$ QPC
to $g_{\perp}/g_{\parallel}\approx 2$ in $[010]$ and $[001]$ QPCs. To
understand such dramatic dependence of $g_{\perp}/g_{\parallel}$ on the QPC
orientation, we perform analytical and numerical calculations based on the
combined Luttinger, Dresselhaus and Rashba Hamiltonians with realistic wafer
parameters. We find new strongly anisotropic spin-orbital contributions to the
in-plane Zeeman splitting (going beyond Refs [15, 25]) that provide excellent
agreement with our data, see Fig. 4b.

Experimental details Three lithographically identical QPCs with dimensions
$$300\text{\,}\mathrm{nm}$\times$300\text{\,}\mathrm{nm}$$ are fabricated
along crystal directions $[001]$, $[011]$ and $[010]$ on a single chip as
shown in Fig. 1a. The QPCs are defined by electron beam lithography in the
form of split gates deposited on the surface of the wafer. The QPCs are
fabricated on an undoped $(100)$ GaAs/AlGaAs heterostructure (wafer W917)
operated in accumulation mode. A negative voltage bias on the metal top gate
forms a two-dimensional hole gas (2DHG) at the GaAs/AlGaAs interface
$60\text{\,}\mathrm{nm}$ below the surface. The 2DHG was operated at a density
$p=2\times 10^{11}$\text{\,}{\mathrm{cm}}^{-2}$$, mobility $\mu=5.5\times
10^{5}$\text{\,}{\mathrm{cm}}^{2}\text{\,}{\mathrm{V}}^{-1}\text{\,}{\mathrm{s}}^{-1}$$,
and mean free path length
$l_{mfp}\approx$4.1\text{\,}\mathrm{\SIUnitSymbolMicro m}$$ (assuming
$m^{*}=0.2m_{e}$).

Measurements were performed in a dilution refrigerator with a superconducting
magnet at a temperature of $130\text{\,}\mathrm{mK}$. An in-situ rotation
system [26] allowed all measurements to be performed during a single cooldown.

Fig. 1b shows conductance measurements as a function of split gate voltage for
all three QPCs. Clear quantized conductance plateaus are observed, indicating
ballistic 1D transport.

Figure 1: (a) SEM image of the device showing three quantum point contacts
(QPCs) with current orientated along the [001], [011] and [010] directions on
the $(100)$ crystal plane of GaAs. The lighter regions are the split gates.
(b) QPC conductance as a function of split-gate voltage $V_{g}$, showing
quantised plateaus in integer multiples of $2e^{2}/h$ with respect to gate-
controlled confinement $V_{g}$, indicating 1D ballistic transport in each QPC.
Figure 2: Greyscale maps of the transconductance $dG/dV_{g}$ showing Zeeman
spin-splitting of the 1D hole subbands in applied magnetic field for the three
QPCs (shown in Fig. 1a). Dark regions correspond to conductance plateaus and
light regions correspond to risers or subband edges. (a-b) QPC $[011]$ with
magnetic field parallel $B_{\parallel}$ and perpendicular $B_{\perp}$ to the
direction of current, respectively. Zeeman spin-splitting is pronounced for
$B_{\parallel}$ and negligible for $B_{\perp}$. (c-d) QPC $[001]$ with
magnetic field parallel to the direction of current $B_{\parallel}$ and
perpendicular $B_{\perp}$ respectively. Zeeman spin-splitting is pronounced
for $B_{\perp}$ and small for $B_{\parallel}$. (e-f) QPC $[010]$ with magnetic
field parallel to the direction of current $B_{\parallel}$ and perpendicular
$B_{\perp}$ respectively. Zeeman spin-splitting is pronounced for $B_{\perp}$
and small for $B_{\parallel}$.

Results The SOI can be studied by measurements of the $g$-factor, which is
extracted from Zeeman spin-splitting measurements. Zeeman spin splitting is
studied for in-plane magnetic fields applied either parallel or perpendicular
to the current flow in the QPC, and then extracting $g_{\parallel}$ and
$g_{\perp}$ from the transconductance $dG/dV_{g}$. Greyscale maps of the
transconductance, showing the 1D subbands evolving in magnetic field, are
presented for each QPC in Figure 2. Dark regions correspond to conductance
plateaus and light regions correspond to risers which mark the 1D subband
edges. For each QPC the size of the Zeeman spin splitting is dependent on the
orientation of the magnetic field with respect to QPC direction. To
characterise the SOI we measure the in-plane $g$-factor anisotropy via the
ratio $g_{\perp}/g_{\parallel}$. This ratio is independent of any
multiplicative renormalizations of the $g$-factors such as the voltage to
energy conversion individual to each device (lever arm) and Coulomb
interaction effects in the QPC channel [27]. This makes the ratio
$g_{\perp}/g_{\parallel}$ a reliable probe of the SOI that can easily be
compared with theory.

The $g$-factors are extracted for the second subbands and higher as the
apparent Zeeman spin-splitting is suppressed due to the interplay of SOI and
interaction effects in the first subband [10]. For QPC [011] linear Zeeman
spin-splitting of the 1D subbands is observed with the magnetic field
$\bm{B}\parallel\bm{I}$ ($g_{\parallel}>0$) in panel Fig. 2a. In contrast
nearly no Zeeman spin splitting is observed for $\bm{B}\perp\bm{I}$
($g_{\perp}\approx 0$) in Fig. 2b. Our observation that
$g_{\parallel}>g_{\perp}$ is consistent with previous studies of Zeeman spin-
splitting in hole QPCs oriented along the $[011]$ crystal axis [20, 28, 22,
24], and is a direct consequence of the strong SOI in hole systems [23].

Surprisingly, we observe the reverse situation in $[001]$ and $[010]$ QPCs
oriented $45\text{\,}\mathrm{\SIUnitSymbolDegree}$ away from the $[011]$ QPC
in either direction. For both $[001]$ QPC, see Fig. 2c,d, and $[010]$ QPC, see
Fig. 2e,f, the Zeeman splitting in $\bm{B}\parallel\bm{I}$ is substantially
smaller than in $\bm{B}\perp\bm{I}$ with the in-plane $g$-factor ratio
$g_{\perp}/g_{\parallel}\approx 2$. Strong anisotropy of the ratio
$g_{\perp}/g_{\parallel}$ ranging over one order of magnitude from
$g_{\perp}/g_{\parallel}\approx 0.15$ for the $[011]$ QPC to
$g_{\perp}/g_{\parallel}\approx 2$ for the $[001]$ and $[010]$ QPCs cannot be
understood within the previous theoretical models where the crystal lattice
anisotropy effects were omitted [23, 24, 29, 18].

Theory For bulk semiconductors, hole dynamics is usually described using the
Luttinger Hamiltonian in the spherical approximation [30]. The Hamiltonian for
holes takes the form

$H_{L}=\left(\gamma_{1}+\frac{5}{2}\bar{\gamma}_{2}\right)\frac{\mathbf{p}^{2}}{2m}-\frac{\bar{\gamma}_{2}}{m}\left(\mathbf{p}\cdot\bm{S}\right)^{2}$
(1)

where $\mathbf{p}$ is the 3D hole momentum, $\bm{S}$ is the spin $S=3/2$,
$\gamma_{1},\bar{\gamma}_{2}=\left(2\gamma_{2}+3\gamma_{3}\right)/5$ are
Luttinger parameters, $m$ is the free electron mass [31]. For a system where
holes are confined to a 2D plane, it is convenient to assume radial symmetry
with an axis along the direction of confinement, and therefore use an axial
approximation of the Luttinger Hamiltonian. In our QPCs, valence band holes
are confined along the $z$-axis by the single heterojunction potential (width
$\approx$15\text{\,}\mathrm{nm}$$), and in the lateral $y$-direction by split
QPC gates of width $300\text{\,}\mathrm{nm}$. 1D ballistic conductance is
along the unconfined $x$-direction. The separation of the in-plane $x$-,$y$\-
and out-of-plane $z$-direction scales allows us to treat a QPC as a planar
system that can be described with the effective Hamiltonian approach developed
in Ref. [32]. Quantum confinement of holes in 2D quantum wells results in the
heavy hole-light hole splitting separating the heavy holes with spin
projections $S_{z}=\pm 3/2$ along the $z$ axis orthogonal to the 2DHG plane
and light holes with $S_{z}=\pm 1/2$. The lowest 2D subband of 2DHG is
comprised of the heavy holes and can be described by an effective 2D
Hamiltonian that only depends on the in-plane momentum $\bm{k}=(k_{x},k_{y})$
[32]. Two heavy hole spin states with $S_{z}=\pm 3/2$ form the up-down basis
for the effective heavy hole pseudospin $1/2$ $\bm{\Sigma}=\bm{\sigma}/2$,
$\bm{\sigma}$ are the Pauli matrices. The most important difference of the
heavy hole pseudospin $1/2$ from the electron spin $1/2$ is that a flip of the
heavy hole pseudospin requires three quanta of angular momentum (to promote
$S_{z}=-3/2$ to $S_{z}=+3/2$) instead of a single quantum for electron spin.
Thus, the raising and lowering heavy hole spin operators
$\Sigma_{\pm}=\Sigma_{x}\pm i\Sigma_{y}$ carry three quanta of angular
momentum. Other vector operators, e.g. $k_{\pm}=k_{x}\pm ik_{y}$ and
$B_{\pm}=B_{x}\pm iB_{y}$, carry a single quantum of angular momentum. These
angular momentum selection rules allow for two different spin-orbital terms
describing the in-plane Zeeman splitting in 2DHG in absence of the crystal
lattice anisotropy [32]:

$H_{0}=-\frac{g_{1}(k)}{2k^{2}}\Sigma_{+}B_{-}k^{2}_{-}-\frac{g_{2}(k)}{2k^{4}}\Sigma_{+}B_{+}k^{4}_{-}+h.c.$
(2)

where $k=|\bm{k}|$, $k_{\pm}=k_{x}\pm ik_{y}$, $B_{\pm}=B_{x}\pm iB_{y}$,
$\Sigma_{\pm}=\Sigma_{x}\pm i\Sigma_{y}$, $h.c.$ stands for hermitian
conjugate. The Hamiltonian $H_{0}$ valid at arbitrary $\mathbf{k}$ indicated
by the $k$-dependence of the dimensionless $g$-factors $g_{1,2}(k)$ that have
to be calculated numerically for a given heterostructure confinement. The
$1/k^{2}$ and $1/k^{4}$ normalisation of the $g_{1,2}(k)$ in Eq. (2) is chosen
for convenience. The effective Hamiltonian $H_{0}$ in Eq. (2) explains the
different values for $g_{\perp}$ and $g_{\parallel}$ observed in $[011]$ and
$[01\bar{1}]$ QPCs (references) yet it is not enough to explain the strong
dependence of the ratio $g_{\perp}/g_{\parallel}$ on the QPC orientation, see
Fig. 2, 4b.

We now step beyond the axial approximation and take into account the cubic
symmetry of the lattice (for now neglecting the more subtle contribution of
bulk inversion asymmetry, known as Dresselhaus SOI [33]). Zinc-blende $(100)$
heterostructures have square symmetry i.e. the effective Hamiltonian is
invariant under $\pi/2$ rotations. The square symmetry only allows terms in
the effective Hamiltonian that carry multiples of the angular momenta $\pm 4$
e.g. $j_{z}=0,\pm 4,\pm 8..$. However high angular momentum harmonics
$j_{z}=\pm 4N$ are suppressed by the factor $\eta^{N}$ where
$\eta=(\gamma_{3}-\gamma_{2})/(\gamma_{3}+\gamma_{2})$, and $\gamma_{2,3}$ are
Luttinger parameters. In GaAs $\eta\approx 0.17$ which allows us to take into
account only the $N=0,1$ harmonics. The following are all the possible terms
in the effective Zeeman Hamiltonian linear with respect to the in-plane
magnetic field $\bm{B}$ and the heavy hole pseudospin $\bm{\sigma}$ that carry
angular momenta $\pm 4$:

$\begin{split}H_{1}=&-\frac{g_{3}(k)}{2}\left(S_{+}B_{+}+h.c.\right)-\frac{g_{4}(k)}{2k^{2}}\left(S_{+}B_{-}k^{2}_{+}+h.c.\right)\\\
&-\frac{g_{5}(k)}{2k^{6}}\left(S_{+}B_{-}k^{6}_{-}+h.c.\right)-\frac{g_{6}(k)}{2k^{8}}\left(S_{+}B_{+}k^{8}_{-}+h.c.\right)\end{split}$
(3)

where $g_{i}(k)$ are dimensionless couplings that depend only on $k=|\bm{k}|$.
Terms $g_{3,4}$ have been previously calculated in the limit of small $\bm{k}$
[22]. Here we do not imply that $\bm{k}$ is small. Eq. (3) is written in cubic
crystal coordinates with $X=[010]$, $Y=[001]$. The terms in $H_{1}$ will pick
up additional phases $e^{\pm 4i\phi}$ in a coordinate system rotated by angle
$\phi$ (see Supplementary Material).

For numerical calculations of $g_{i}(k)$, we use a triangular potential
profile (approximating a single GaAs/AlGaAs heterojunction as in our
experimental device) self-consistently screened by the finite hole density
$p=2\times 10^{11}$\text{\,}{\mathrm{cm}}^{-2}$$, see Fig. S4 in the
Supplementary Material. At the Fermi momentum $k_{F}=(2\pi p)^{1/2}=1.12\times
10^{-2}$\text{\,}{\mathrm{\SIUnitSymbolAngstrom}}^{-1}$$ corresponding to the
2D hole density $p=2\times 10^{11}$\text{\,}{\mathrm{cm}}^{-2}$$ the
$g$-factors $g_{i}(k)$ cannot be reliably approximated by the small-$k$
expansion. In our experimental case of a more asymmetric quantum well with
strong Rashba SOI, $g_{1}$ is the dominant contribution. $g_{2}\approx-0.05$
is strongly suppressed by the Rashba SOI. However, $g_{2}\sim g_{1}$ in square
quantum wells where the Rashba SOI is much smaller [23]. $g_{3}\approx-0.25$
and $g_{4}\approx-0.18$ are responsible for the dependence of the overall
$g$-factor on QPC orientation with respect to the crystal axes.
$g_{5}\approx-0.05$ and $g_{6}\approx-0.03$ are still small but they become
significant, $|g_{5,6}|>0.1$, at high densities $p>3\times
10^{11}$\text{\,}{\mathrm{cm}}^{-2}$$ which is beyond what can be presently
achieved in our QPC devices (see Supplementary Material).

From eqs. (2) and (3), we write an expression for the overall in-plane
$g$-factor as the sum of each contributing $g_{i}$ term. The $g_{1}$ and
$g_{2}$ terms derived in the axial approximation depend on the magnetic field
orientation $\theta$ with respect to the QPC. The $g_{3}$, $g_{4}$, $g_{5}$
and $g_{6}$ terms arise from the angular momenta $\pm 4$ permitted by the
square symmetry of the $(100)$ zinc-blende crystal, and therefore depend on
the magnetic field orientation with respect to the QPC $\theta$ _and_ the QPC
orientation with respect to the crystal axes $\phi$. According to Ref. [23],
$\langle n|k_{\perp}^{2}|n\rangle\approx k_{F}^{2}$ and $\langle
n|k_{\parallel}^{2}|n\rangle\ll k_{F}^{2}$ at the edge $|n\rangle$ of a high
$n\geq 2$ QPC subband due to strong electrostatic flattening of a QPC channel.
Here $k_{\perp}$ and $k_{\parallel}$ are the hole momentum operators
perpendicular and parallel to the QPC channel, allowing for the quasiclassical
treatment of the hole momentum at the edge of $n^{\rm th}$, $n\geq 2$, QPC
subband i.e. $\langle n|g_{i}(k)|n\rangle\approx g_{i}(k_{F})$. Thus, we can
extract the in-plane hole $g$-factor of the $n\geq 2$ QPC subbands by applying
the quasiclassical approximation to the 2D Zeeman Hamiltonian $H=H_{0}+H_{1}$:

$|g\left(\theta,\phi\right)|=|-g_{1}e^{-i\theta}+g_{2}e^{i\theta}+g_{3}e^{i(\theta+4\phi)}\\\
-g_{4}e^{-i(\theta-4\phi)}-g_{5}e^{-i(\theta+4\phi)}+g_{6}e^{i(\theta-4\phi)}|$
(4)

where $g_{i}=g_{i}(k_{F})$. In the coordinate system which is rotated by angle
$\phi$ with respect to the crystal axes $X=[010]$, $Y=[001]$, the $k^{2}$
operator has only $k_{y}^{2}$ average at the subband edge where $k_{x}^{2}$ is
vanishing, hence $k_{+}^{2}=k_{-}^{2}=-k_{y}^{2}\approx-k_{F}^{2}$.

Equation (4) allows us to plot the $g$-factor parallel to the QPC current
$g_{\parallel}$, and in-plane perpendicular $g_{\perp}$, as a function of QPC
orientation with respect to crystal axes $\phi$. The $g$-factor parallel to
current $g_{\parallel}$ corresponds to $\theta=0$,
$g_{\parallel}(\phi)=|g(0,\phi)|$, and perpendicular to current $g_{\perp}$
corresponds to $\theta=\pi/2$, $g_{\perp}(\phi)=|g(\pi/2,\phi)|$.
$|g_{\parallel}(\phi)|$ and $|g_{\perp}(\phi)|$ are plotted in Figure 3 for
$p=2\times 10^{11}$\text{\,}{\mathrm{cm}}^{-2}$$. A very strong angular
dependence of $g_{\perp}$ is observed, varying from $g_{\perp}\approx 0.1$ for
$[011]$ and $[01\bar{1}]$ QPCs to $g_{\perp}\approx 1.2$ for $[001]$ and
$[010]$ QPCs, and nearly isotropic $g_{\parallel}\approx 0.6$ for any QPC
orientation. The anisotropy of the in-plane $g$-factors for the $[011]$ and
$[01\bar{1}]$ crystal directions (corresponding to $\phi=\pi/4,3\pi/4$) is
$g_{\parallel}>g_{\perp}$. In contrast, for the $[001]$ and $[010]$ crystal
directions (corresponding to $\phi=0,\pi/2$) $g_{\perp}$ becomes large, while
$g_{\parallel}$ is only slightly suppressed and the anistropy of the in-plane
$g$-factors is reversed $g_{\parallel}<g_{\perp}$.

Experimental $g$-factors extracted from Fig. 2 for all three QPCs for
different subbands $n\geq 2$ show nearly no subband dependence (within the
error bars), see Fig. S1 in the Supplementary Material. This is consistent
with electrostatic calculations performed in Ref. [23] where the QPC
$g$-factors saturate to (nearly) same value already at $n\geq 2$. Therefore,
we average the $g$-factors over the subband index $n\geq 2$ to increase
experimental accuracy. Averaged $g$-factors for each QPC are shown in Figure
4a. We see that there is a substantial difference between the $[010]$ and
$[001]$ QPCs which likely comes from the multiplicative renormalizations of
the $g$-factor originating from the voltage-to-energy conversion and the
details of the Coulomb interaction [23, 27]. These renormalizations are
cancelled out if we consider the ratio $g_{\perp}/g_{\parallel}$ shown in Fig.
4b where we find that this ratio is indeed the same for $[010]$ and $[001]$
QPCs as expected. The dashed line in Fig. 4b shows the theoretical dependence
of $g_{\perp}(\phi)/g_{\parallel}(\phi)$, where $g_{\perp}(\phi)$ and
$g_{\parallel}(\phi)$ are plotted in Fig. 3. The theoretical prediction for
$g_{\perp}/g_{\parallel}$ is thus fully consistent with observed ratios
$g_{\perp}/g_{\parallel}\approx 0.15$ in $[011]$ QPC and
$g_{\perp}/g_{\parallel}\approx 2$ in $[010]$ and $[001]$ QPCs, see Fig. 4b

Figure 3: Numerically calculated $g_{\parallel}$\- and $g_{\perp}$-factors as
a function of QPC orientation with respect to crystal axes ($\phi$).
$\phi=0,\pi/2$ correspond to crystal directions $[001]$ and $[010]$
respectively. $\phi=\pi/4$ corresponds to crystal direction $[011]$. Figure 4:
(a) Experimentally obtained $g$-factors calculated from the Zeeman spin-
splitting measurements in Fig. 2. The $g$-factor for each 1D subband is
averaged to obtain a single $g_{\parallel}$\- and $g_{\perp}$-factor for each
value of $\phi$. (b) $g$-factor anisotropy $g_{\perp}/g_{\parallel}$ plotted
as a function of QPC orientation with respect to crystal axes $\phi$. Dashed
line is numerically calculated anisotropy of the in-plane $g$-factors. Green
squares are experimental data points calculated from the data in panel (a).
$\phi=0,\pi/2$ correspond to crystal directions $[001]$ and $[010]$
respectively. $\phi=\pi/4$ corresponds to crystal direction $[011]$.

Conclusion We observed for the first time very strong crystal lattice
anisotropy of the in-plane Zeeman splitting in $(100)$ GaAs hole QPCs. The
ratio $g_{\perp}/g_{\parallel}$ of the in-plane $g$-factors measured
perpendicular, $g_{\perp}$, and parallel, $g_{\parallel}$, to the QPC channel
varies by an order of magnitude from $g_{\perp}/g_{\parallel}\approx 0.15$ in
$[011]$ QPC to $g_{\perp}/g_{\parallel}\approx 2$ in $[010]$ and $[001]$ QPCs.
The dramatic dependence of $g_{\perp}/g_{\parallel}$ on the QPC orientation
signals strong crystal lattice anisotropy of the in-plane Zeeman splitting. To
understand this effect, we derived the effective 2D Hamiltonian describing the
in-plane Zeeman splitting in 2DHG. Additionally to the axially symmetric terms
$g_{1}$ and $g_{2}$ ($k^{2}$ and $k^{4}$ in Eq. (2)), there is the crystal
lattice correction, $g_{3}$, $g_{4}$, $g_{5}$, and $g_{6}$ ($k^{0}$, $k^{2}$,
$k^{6}$, $k^{8}$ in Eq. (3)), which results in the angular dependence of the
hole $g$-factors on the QPC orientation shown in Fig. 3. Performing numerical
calculations for realistic parameters of the wafer, we find excellent
agreement between theoretical and experimental values of
$g_{\perp}/g_{\parallel}$ shown in Fig. 4b. Our work shows that the crystal
lattice anisotropy is very strong in GaAs 2D hole systems which provides
additional tunability for hole-based spin devices.

###### Acknowledgements.

This work was supported by the Australian Research Council under the Discovery
Projects scheme, and was performed in part using facilities of the NSW Node of
the Australian National Fabrication Facility. D.A.R and I.F. acknowledge
support from the Engineering and Physical Sciences Research Council, United
Kingdom. K.L.H. acknowledges support from Sydney Quantum Academy. D.S.M
acknowledges the support by the Georg H. Endress Foundation and NCCR SPIN.

## References

  * Winkler [2003] R. Winkler,  _Spin-orbit coupling effects in two-dimensional electron and hole systems_ , Springer tracts in modern physics (Springer, Berlin, 2003).
  * Li  _et al._ [2015] R. Li, F. E. Hudson, A. S. Dzurak, and A. R. Hamilton, Pauli spin blockade of heavy holes in a silicon double quantum dot, Nano Letters 15, 7314 (2015).
  * Wang  _et al._ [2016] D. Q. Wang, O. Klochan, J.-T. Hung, D. Culcer, I. Farrer, D. A. Ritchie, and A. R. Hamilton, Anisotropic pauli spin blockade of holes in a GaAs double quantum dot, Nano Letters 16, 7685 (2016).
  * Maurand  _et al._ [2016] R. Maurand, X. Jehl, D. Kotekar-Patil, A. Corna, H. Bohuslavskyi, R. Laviéville, L. Hutin, S. Barraud, M. Vinet, M. Sanquer, and S. De Franceschi, A CMOS silicon spin qubit, Nature Communications 7, 13575 (2016).
  * Liu  _et al._ [2018] H. Liu, E. Marcellina, A. R. Hamilton, and D. Culcer, Strong spin-orbit contribution to the hall coefficient of two-dimensional hole systems, Phys. Rev. Lett. 121, 087701 (2018).
  * Hendrickx  _et al._ [2018] N. W. Hendrickx, D. P. Franke, A. Sammak, M. Kouwenhoven, D. Sabbagh, L. Yeoh, R. Li, M. L. V. Tagliaferri, M. Virgilio, G. Capellini, G. Scappucci, and M. Veldhorst, Gate-controlled quantum dots and superconductivity in planar germanium, Nature Communications 9, 2835 (2018).
  * Sau  _et al._ [2010] J. D. Sau, R. M. Lutchyn, S. Tewari, and S. Das Sarma, Generic new platform for topological quantum computation using semiconductor heterostructures, Phys. Rev. Lett. 104, 040502 (2010).
  * Oreg  _et al._ [2010] Y. Oreg, G. Refael, and F. von Oppen, Helical liquids and majorana bound states in quantum wires, Phys. Rev. Lett. 105, 177002 (2010).
  * Goulko  _et al._ [2014] O. Goulko, F. Bauer, J. Heyder, and J. von Delft, Effect of spin-orbit interactions on the 0.7 anomaly in quantum point contacts, Phys. Rev. Lett. 113, 266402 (2014).
  * Hudson  _et al._ [2021] K. L. Hudson, A. Srinivasan, O. Goulko, J. Adam, Q. Wang, L. A. Yeoh, O. Klochan, I. Farrer, D. A. Ritchie, A. Ludwig, A. D. Wieck, J. von Delft, and A. R. Hamilton, New signatures of the spin gap in quantum point contacts, Nature Communications 12, 5 (2021).
  * Datta and Das [1990] S. Datta and B. Das, Electronic analog of the electro‐optic modulator, Applied Physics Letters 56, 665 (1990).
  * Loss and DiVincenzo [1998] D. Loss and D. P. DiVincenzo, Quantum computation with quantum dots, Phys. Rev. A 57, 120 (1998).
  * Wolf  _et al._ [2001] S. A. Wolf, D. D. Awschalom, R. A. Buhrman, J. M. Daughton, S. von Molnár, M. L. Roukes, A. Y. Chtchelkanova, and D. M. Treger, Spintronics: A spin-based electronics vision for the future, Science 294, 1488 (2001).
  * Awschalom  _et al._ [2002] D. Awschalom, D. Loss, and N. Samarth,  _Semiconductor Spintronics and Quantum Computation_ (Springer-Verlag, Berlin, 2002).
  * Winkler  _et al._ [2000] R. Winkler, S. J. Papadakis, E. P. De Poortere, and M. Shayegan, Highly anisotropic $\mathit{g}$-factor of two-dimensional hole systems, Phys. Rev. Lett. 85, 4574 (2000).
  * Martin  _et al._ [2008] T. P. Martin, A. Szorkovszky, A. P. Micolich, A. R. Hamilton, C. A. Marlow, H. Linke, R. P. Taylor, and L. Samuelson, Enhanced zeeman splitting in Ga0.25In0.75As quantum point contacts, Applied Physics Letters 93, 012105 (2008).
  * Martin  _et al._ [2010] T. P. Martin, A. Szorkovszky, A. P. Micolich, A. R. Hamilton, C. A. Marlow, R. P. Taylor, H. Linke, and H. Q. Xu, Field-orientation dependence of the Zeeman spin splitting in (In,Ga)As quantum point contacts, Phys. Rev. B 81, 041303 (2010).
  * Nichele  _et al._ [2014] F. Nichele, S. Chesi, S. Hennel, A. Wittmann, C. Gerl, W. Wegscheider, D. Loss, T. Ihn, and K. Ensslin, Characterization of spin-orbit interactions of GaAs heavy holes using a quantum point contact, Phys. Rev. Lett. 113, 046801 (2014).
  * Danneau  _et al._ [2006] R. Danneau, O. Klochan, W. R. Clarke, L. H. Ho, A. P. Micolich, M. Y. Simmons, A. R. Hamilton, M. Pepper, D. A. Ritchie, and U. Zülicke, Zeeman splitting in ballistic hole quantum wires, Phys. Rev. Lett. 97, 026403 (2006).
  * Klochan  _et al._ [2009] O. Klochan, A. P. Micolich, L. H. Ho, A. R. Hamilton, K. Muraki, and Y. Hirayama, The interplay between one-dimensional confinement and two-dimensional crystallographic anisotropy effects in ballistic hole quantum wires, New Journal of Physics 11, 043018 (2009).
  * Koduvayur  _et al._ [2008] S. P. Koduvayur, L. P. Rokhinson, D. C. Tsui, L. N. Pfeiffer, and K. W. West, Anisotropic modification of the effective hole $g$ factor by electrostatic confinement, Phys. Rev. Lett. 100, 126401 (2008).
  * Komijani  _et al._ [2013] Y. Komijani, M. Csontos, I. Shorubalko, U. Zülicke, T. Ihn, K. Ensslin, D. Reuter, and A. D. Wieck, Anisotropic Zeeman shift in p-type GaAs quantum point contacts, EPL (Europhysics Letters) 102, 37002 (2013).
  * Miserev  _et al._ [2017] D. S. Miserev, A. Srinivasan, O. A. Tkachenko, V. A. Tkachenko, I. Farrer, D. A. Ritchie, A. R. Hamilton, and O. P. Sushkov, Mechanisms for strong anisotropy of in-plane $g$-factors in hole based quantum point contacts, Phys. Rev. Lett. 119, 116803 (2017).
  * Srinivasan  _et al._ [2017] A. Srinivasan, D. S. Miserev, K. L. Hudson, O. Klochan, K. Muraki, Y. Hirayama, D. Reuter, A. D. Wieck, O. P. Sushkov, and A. R. Hamilton, Detection and control of spin-orbit interactions in a GaAs hole quantum point contact, Phys. Rev. Lett. 118, 146801 (2017).
  * Lu  _et al._ [1998] J. P. Lu, J. B. Yau, S. P. Shukla, M. Shayegan, L. Wissinger, U. Rössler, and R. Winkler, Tunable spin-splitting and spin-resolved ballistic transport in gaas/algaas two-dimensional holes, Phys. Rev. Lett. 81, 1282 (1998).
  * Yeoh  _et al._ [2010] L. A. Yeoh, A. Srinivasan, T. P. Martin, O. Klochan, A. P. Micolich, and A. R. Hamilton, Piezoelectric rotator for studying quantum effects in semiconductor nanostructures at high magnetic fields and low temperatures, Review of Scientific Instruments 81, 113905 (2010).
  * Vionnet and Sushkov [2016] G. Vionnet and O. P. Sushkov, Enhancement mechanism of the electron $g$ factor in quantum point contacts, Phys. Rev. Lett. 116, 126801 (2016).
  * Chen  _et al._ [2010] J. C. H. Chen, O. Klochan, A. P. Micolich, A. R. Hamilton, T. P. Martin, L. H. Ho, U. Zülicke, D. Reuter, and A. D. Wieck, Observation of orientation- and k -dependent Zeeman spin-splitting in hole quantum wires on (100)-oriented AlGaAs/GaAs heterostructures, New Journal of Physics 12, 033043 (2010).
  * Zülicke [2006] U. Zülicke, Electronic and spin properties of hole point contacts, physica status solidi c 3, 4354 (2006), NoStop
  * Luttinger [1956] J. M. Luttinger, Quantum theory of cyclotron resonance in semiconductors: General theory, Phys. Rev. 102, 1030 (1956).
  * Baldereschi and Lipari [1973] A. Baldereschi and N. O. Lipari, Spherical model of shallow acceptor states in semiconductors, Phys. Rev. B 8, 2697 (1973).
  * Miserev and Sushkov [2017] D. S. Miserev and O. P. Sushkov, Dimensional reduction of the Luttinger Hamiltonian and $g$-factors of holes in symmetric two-dimensional semiconductor heterostructures, Phys. Rev. B 95, 085431 (2017).
  * Dresselhaus [1955] G. Dresselhaus, Spin-orbit coupling effects in zinc blende structures, Phys. Rev. 100, 580 (1955).


# Triple Regression for Camera Agnostic Sim2Real Robot Grasping and
Manipulation Tasks

Yuanhong Zeng∗, Yizhou Zhao∗, and Ying Nian Wu∗ ∗UCLA Department of Statistics
& Data Science, 8125 Math Sciences Bldg, Box 951554, Los Angeles, CA,
90024-1554, United States<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Sim2Real (Simulation to Reality) techniques have gained prominence in robotic
manipulation and motion planning due to their ability to enhance success rates
by enabling agents to test and evaluate various policies and trajectories. In
this paper, we investigate the advantages of integrating Sim2Real into robotic
frameworks. We introduce the Triple Regression Sim2Real framework, which
constructs a real-time digital twin. This twin serves as a replica of reality
to simulate and evaluate multiple plans before their execution in real-world
scenarios. Our triple regression approach addresses the reality gap by: (1)
mitigating projection errors between real and simulated camera perspectives
through the first two regression models, and (2) detecting discrepancies in
robot control using the third regression model. Experiments on 6-DoF grasp and
manipulation tasks (where the gripper can approach from any direction)
highlight the effectiveness of our framework. Remarkably, with only RGB input
images, our method achieves state-of-the-art success rates. This research
advances efficient robot training methods and sets the stage for rapid
advancements in robotics and automation.

##  I INTRODUCTION

Simulation plays a pivotal role in the development and validation of robotic
systems by providing a virtual platform that closely mirrors real-world
scenarios. In industrial applications, the reliability of robotic systems is
often ascertained through multiple simulation tests. This process encompasses
(1) the creation of a digital twin representing the actual manufacturing
environment; (2) the planning and execution of varied trajectories within the
simulation; (3) a thorough assessment of each trajectory’s outcomes; and (4)
the deployment of the most effective trajectory in a real-world setting. The
focus of this research is to enhance efficiency by automating the
aforementioned stages.

Navigating simulation-to-reality (Sim2Real) applications presents two pivotal
challenges, often referred to as the reality gap. First, how to build an
accurate enough digital twin that captures the real-world scenes in a
simulation using sensor (RGB or RGBD camera) data? Second, how to seamlessly
apply the simulation result in real-world situations? The first challenge is a
fundamental problem in multi-view geometry. While recent advancements have
tackled this using 6D pose estimation [1, 2] or 3D scene reconstruction [3,
4], there are limitations. The former method struggles to generalize to
unfamiliar objects. While the latter can intricately map complex scenes, it
often obscures specific object details, making it hard to discern between
interactive and stationary objects. The second challenge arises from the
unreal nature of simulated physics and dynamics. Today, even state-of-the-art
robot simulators fall short of accurately modeling physical phenomena, such as
friction, impact, and deformation.

To tackle the challenges above in Sim2Real and digital twin robotic
applications, we introduce the triple regression framework for cameras with
undetermined intrinsic properties (eg. focal length). The initial regression
aligns the cameras in the real world and those in the simulated environment.
Leveraging image segmentation, we can estimate the target object’s location,
and subsequently place the approximated object into the simulation space. This
transformation is derived from the second regression. Finally, the third
regression is employed to compensate for errors originating from the simulated
robotic trajectory and dynamics.

Figure 1: Conceptual text-to-action framework: Tasks described language are
converted into a joint representation which records spatial, and temporal
information of the scene and task. A digital twin is created based on semantic
grounding and camera observation. Motions are planned in the simulator. VQA is
used to ascertain the success of the performed action

Employing the triple regression framework, we propose a conceptual framework
for text-to-action tasks as shown in Fig 1. The process begins with using
semantic grounding to generate a joint task representation, breaking down the
task into multiple grab-and-place actions. Using the segmentation modules [5],
the dimensions and positions of these 3D objects are determined and integrated
into a simulation engine to create a real-time digital twin. Within the
simulation, various trajectories are created. A Visual Question Answering
(VQA) model ascertains task success. Subsequently, successful simulations are
implemented in the real world. This methodology offers an automated Sim2Real
workflow, notably requiring only a standard RGB camera and rudimentary site
measurements.

We conducted an experiment of picking up a jar at a random position and
pouring its water to a cup at another random position. This experiment
involves the robot’s control of rigid bodies and fluids, as well as planning
to overcome random factors. Experimental results show that our methodology is
able to provide an offline and zero-shot solution (no training for the robot
in real space) and reduce the robot’s time to complete the task by at least
$50\%$ compared to the state-of-the-art approach. Moreover, our methodology is
able to improve the success rate by $75\%$ under conditions of randomized
object placement.

Our contribution can be summarized as (1) we present a systematic framework to
tackle the reality gap stemming from constructing real-time digital twins and
migrating simulated trajectory to reality; (2) we propose a Sim2Real solution
for the text-guided robot manipulation tasks; (3) we demonstrate its
effectiveness through physical experiments on a Kinova Gen 3 robot when
comparing our method with other state-of-the-art solutions.

##  II Related works

Sim2Real. Recent breakthroughs in simulation-to-reality (Sim2Real) transitions
now allow robots to train on intricate tasks within simulations and then
replicate those abilities in the actual world. For example, recent works have
employed Sim2Real techniques to enhance a robot’s proficiency in grasping
objects [6], and maneuvering autonomous vehicles [7]. These strides forward
are attributed to the application of machine learning methods that initially
train robots within simulations, which are later refined in real-world
settings. Consequently, the Sim2Real methodology is rapidly gaining traction
as a favored strategy for crafting agile and adaptable robots suited for
dynamic, real-world scenarios.

Robot motion planning. The motion planning problem can involve geometrically
generating the path (path planning), or more broadly, outputting interactive
trajectories that take into account kinetics, control features, robot poses,
and movements of nearby objects. Classical motion planning algorithms use
graph search (eg. $A^{*}$), random sampling (eg. rapidly exploring random tree
and probabilistic roadmap method), and curve interpolation methods. Recent
advancements purposes planners based on supervised learning [8] and
reinforcement learning [9, 10]. These developments emphasize enhancing the
reliability of plans amidst uncertainties and safeguarding human interactions
within co-working spaces [10]. The overarching aim of these initiatives is to
empower robots to operate autonomously and adeptly in fluctuating and
unpredictable settings.

Digital-twin. Most Sim2Real approach learns a policy by exploring the
environment and then transferring the learned outcome to reality. These
applications use a simulator as a playground and do not benefit from up-to-
date information. In comparison, digital twin applications attempt to create a
true-to-life replicate in the simulator based on up-to-date data. Most
applications [11] of digital twins involve predicting the problems and errors
for the future states of its physical twin, providing us an opportunity to
plan the systems accordingly.

Grasp and manipulation. Modern approaches to implementing learning-based
robotic agents that are capable of performing grasp and manipulation tasks
typically fall into these three categories. (1) The classical robotic grasping
agent [12, 13] relies on object pose estimation and grasping dataset matching
[14, 15]. While these methodologies demonstrate effectiveness, the method
cannot generalize to grasping previously unseen objects. (2) Exploration-based
learning agents[16] have demonstrated success in robotic grasp and
manipulation tasks in simulated environments and generalize well to unseen
objects. However, the reality gap can lead to overfitting to the simulated
environment [17], inaccurate results, and unstable behaviors. (3) More recent
discoveries in end-to-end models [18, 19] that map scene images to action
sequences are capable of performing complex tabletop robotic manipulation.
These methods use an action-centric approach, meaning that they do not use any
explicit representations of object poses, but they are known to require
copious amounts of data and are mostly task-specific.

##  III Method

###  III-A Joint task planning module

Figure 2: Representation of pouring water from a jar to a cup with STC-AOG.
The data structure reduces a sentence into subgoals for task planning.

The spatial-temporal-causal And-Or graph (STC-AOG) is used as the task
planning model to represent the goal as well as the current cognitive state of
the robot. This structure allows us to associate spatial, temporal, and causal
relationships within the task description. The task of pouring water from a
jar at a random position to a cup at another random position, is showcased in
Figure 2.

In general, an And-Or Graph consists of nodes and edges. The set of nodes
includes Or node, And node, and Terminal node. Each Or node specifies the Or
relation: only one of its children nodes would be performed at a given time.
An And node represents the And relation and is composed of several children
nodes. Each Terminal node represents a set of entities that cannot be further
decomposed. The edge represents the top-down sampling process from a parent
node to its children nodes. The root node of the And-Or tree is always an And
node connected to a set of And/Or nodes. Each And-node represents a sub-task
which can be further decomposed into a series of sub-tasks or atomic actions.

Causal relation. Causal knowledge represents the pre-conditions and the post-
effects of atomic actions. We define this knowledge as fluent changes caused
by an action. Fluent can be viewed as some essential properties in a state
that can change over time, e.g., the temperature in a room and the status of a
heater. For each atomic action, there are pre-conditions characterized by
certain fluents of the states. For example, the water in the experiment should
initially be in the jar and then transferred to the cup.

Temporal relation. Temporal knowledge encodes the schedule for an agent to
finish each sub-task. It also contains the temporal relations between atomic
actions in a low-level sub-task. The sub-task of preparing in this study, for
example, consists of picking up jar and pouring water from the jar to the cup.

Spatial relation. Spatial knowledge represents the physical configuration of
the environment that is necessary to finish the task. In our case, to pick up
the jar, the robot needs to know the location and the rotation jar. This is
determined by the following triple regression framework.

###  III-B Triple Regression for Matching Simulation and Reality

Figure 3: Workflow of triple regression framework. The framework creates
digital twins by 1) camera shots, 2) Semantic-based segmentation and contour
extraction, 3) key point identification, and 4) object placement in
simulation. Plans are generated and simulated. Coordinates of successful plans
are corrected and executed in reality.

The triple regression framework is designed to align the virtual environment
with the real world. The workflow is shown in Fig 3. The framework addresses
reality gaps that are caused by (1) intrinsic discrepancies between the real
and simulated camera, (2) errors while comprehending scenes and determining
object locations, and (3) differences in executing planned trajectories
between simulated robot and real robot. Equation 1 gives a formal definition
of aligning the parse graph in simulation, $pg^{s}$, with the parse graph in
reality $pg^{r}$, by aligning the camera, scene, and robot control separately.

$P(pg^{s}|pg^{r})=\prod_{i\in\\{\text{camera},\text{scene},\text{robot}\\}}P(pg^{s}_{i}|pg^{r}_{i})$
(1)

We define our symbols as follows. for a point $\textbf{p}=(x,y,z)$ and its
homogenous representation $\tilde{\textbf{p}}=(x,y,z,1)$, we define
$\textbf{u}=(x_{r},y_{r})$ to be the point in image captured by the real
camera and $\textbf{v}=(x_{s},y_{s})$ to be the point in image captured by
simulated camera. Their representations in homogeneous coordinate is
$\tilde{\textbf{u}}=(x_{r},y_{r},1)$ and $\tilde{\textbf{v}}=(x_{s},y_{s},1)$.

Camera alignment. The goal of the first regression is to align real and
simulated cameras. In other words, we aim to find a projection from v to u. In
linear cameras, this projection can be modeled as in equation 2, where $M$ is
defined in equation 3.

$\tilde{\textbf{u}}=M\tilde{\textbf{v}}$ (2)

$M=\begin{bmatrix}A_{2\times 2}&b_{2\times 1}\\\ c_{1\times 2}&1\\\
\end{bmatrix}$ (3)

To estimate $M$, we construct a reference scene (table and camera) to mirror
reality. 2D locations of multiple reference points (eg. table corner and table
center) captured in both simulated and real cameras are recorded. $M$ can then
be derived with constrained multivariate regression.

Besides projective transformation, we also consider the 2D scaling and affine
transformation. Table I provides the final comparative analysis of the three
regression methods.

Camera calibration and object allocation The goal of this phase is to estimate
the intrinsics of the simulated camera. Using this we can determine the 3D
location of a point with 2D pixel locations. Camera in the simulation
environment can be modeled as a matrix $C$ formed by multiplying intrinsic $I$
and extrinsic $E$ defined in equation 4 and 5.

$I=\begin{bmatrix}f_{x}&0&c_{x}&0\\\ 0&f_{y}&c_{y}&0\\\ 0&0&1&0\\\
\end{bmatrix}$ (4)

$E=\begin{bmatrix}R_{3\times 3}&t\\\ 0_{1\times 3}&1\end{bmatrix}$ (5)

With this model, capturing an image can be modeled as transforming point world
position $\tilde{\textbf{p}}$ to pixel position $\tilde{\textbf{v}}$ as

$\tilde{\textbf{v}}=IE\tilde{\textbf{p}}=C\tilde{\textbf{p}}$ (6)

The rotation matrix $R$ and 3D translation $t$ in equation 5 can be obtained
in the simulation environment. By sampling multiple points in the simulation
and camera, $I$ can be estimated by solving constrained least squares
problems.

In our setup, all points were placed on planes (eg. table) with known normal
$\textbf{n}=(a,b,c)$. Combining this constraint, we can solve for
$\tilde{\textbf{p}}$ given $\tilde{\textbf{v}}$ in equation 6 to determine the
3D location of a point.

Segmentation modules are used to place the object. Anchor points and bounding
boxes are generated with pattern matching or text-to-image grounding models
[20]. Contour lines of the objects are extracted using [5] and key points are
subsequently determined for cuboid and cylindrical objects. We use these key
points to align objects in simulation and reality.

Regression to accommodate miscellaneous errors. Another source of error is
control discrepancy and inaccuracy in constructing reference objects in the
simulation. We employ another regression model (Equation 7) to correct for
robot movements. We collect points $\textbf{p}_{s}$ and manually adjust their
should-be position $\textbf{p}_{r}$ during robot execution. Coefficient $D$
can be obtained by solving regression problems. Note that
$\tilde{\textbf{p}}_{s}$ and $\tilde{\textbf{p}}_{r}$ are the homogeneous
representation of $\textbf{p}_{s}$ and $\textbf{p}_{r}$

$\tilde{\textbf{p}}_{s}=D\tilde{\textbf{p}}_{r}$ (7)

###  III-C Validating Sim2Real matching

Our approach to aligning simulation with reality during robot execution
involves a multi-step strategy.

Firstly, we emphasize the dynamic nature of conditions, termed as fluents,
which can evolve over time. To capture these evolving states, we continuously
track changes in fluent at multiple time points to ensure synchronization
between the simulated and real-world environments.

Secondly, we introduce a structured questionnaire that prompts users to
provide binary true-false responses pertaining to the observed fluents. This
questionnaire serves as a reliable ground truth reference, enabling us to
quantitatively assess the correctness of simulation outputs. The questionnaire
consists of four questions. (1) Is the robot ready to pick up the jar? (2) Has
the robot already picked up the jar? (3) Is the jar above the cup? (4) Is the
robot pouring water into the cup?

Lastly, we leverage advanced Visual Question Answering (VQA) models to analyze
and interpret the questionnaire responses. This model takes in an image of
each fluent and answers the aforementioned prompts. An example of fluent
matching is shown in 4

Figure 4: Query fluent matching in robot execution. We select four crucial
moments during the execution of the robot in both simulation and reality and
report GPT-V’s VQA query answers for the questionnaire: (1) Is the robot ready
to pick up the jar? (2) Has the robot already picked up the jar? (3) Is the
jar above the cup? (4) Is the robot pouring water to the cup? Output from VQA
models and ground truths are shown

$P(pg^{r}|pg^{s})=\exp(-\sum_{i}|F^{r}_{i}-F^{s}_{i}|)$ (8)

Given the parse graph in simulation $pg^{s}$, its discrepancy with the parse
graph, in reality, $pg^{r}$ is estimated quantitatively by the hamming
distance between fluents in reality and fluent in simulation as shown in
equation 8. $F^{r}_{i}$ (fluent in reality) and $F^{s}_{i}$ (fluent in
simulation) are the VQA query results (true/false) for each question $i$ in
the questionnaire above.

##  IV Experiment

The experiment section is organized into three parts: (1) ablation studies on
the triple regression framework, (2) grasp-and-pour experiments on a physical
robot, and (3) comparisons of VQA models.

###  IV-A Ablation study on the triple regression framework

In this experiment, we validate the necessity of using the projective
transformation model perspective discrepancy between real and simulated
cameras. A consumer-grade Microsoft RGB camera is mounted in some known
location and direction above the table. The table is modeled in Nvidia
Omniverse corresponds to the real dimension, and the camera is placed with
similar extrinsic that mirror the reality. We align the cameras using three
types of regression: full projective transformation, 2D scaling ($A=0$ and
$c=0$ in Equation 3), and affine ($c=0$ in Equation 3) A cube with known
dimension is placed at known locations both in reality and in simulation. We
then compare the pixel location of the 6 corners of the cube captured both in
simulation and reality. Figure I summarizes the correlation between the degree
of freedom and the achieved Mean Square Error (MSE), signifying the necessity
of applying the projective transformation. Using this approach, errors from
Sim2Real point position mapping can be controlled as low as 2.7 pixels on
average.

TABLE I: Comparison of regression models used in camera alignment. Regression method | Degree of Freedom | MSE
---|---|---
Simple linear | 2 | $8.2\pm 4.3$
Affine | 6 | $5.1\pm 3.2$
Projective | 8 | $2.7\pm 2.8$

###  IV-B Grasp and pour experiment

In this section, we first introduce our experiment settings. Subsequently, we
explore multiple baseline models for robot planning and control. We then
compare our method with the baseline and human performance, and finally, we
summarize the results and failure cases.

(1) Setup: We conducted pick-and-pour experiments using a Kinova Gen3 7-dof
robot, equipped with a Robotiq 2f-85 gripper (jaw length 8.5cm), mounted on a
table. The setup included a transparent jar (side length 6.5cm) and a blue cup
(radius 8cm). For simulation purposes, we employed the Isaac Sim platform
powered by the PhysX engine, where we replicated the table and robotic arm. To
align the cameras, we compared their relative position in reality to that in
Isaac Sim. Lula implementation of RRT (Rapidly-exploring random tree) [21] was
used to generate multiple collision-free paths. RMPflow was employed [22] to
generate smooth policies to control the robot both in the real world and in
simulation. Following this, we input the command ”Pour water from the
transparent jar into the blue cup” into our Isaac Sim extension and monitored
the outcome.

(2) Baseline models: We introduce two state-of-the-art models that serve as
the robot motion planning module. The 6D-CLIPort [19], which deals with multi-
view observations and language input and output a sequence of 6 degrees of
freedom (DoF) actions, infers the position and rotation of the target object
given the vision and language inputs. Another reinforcement learning-based
model planning (RL) [23], predicts the position and pose of the robot gripper
(end-effector) given the vision inputs and the target object name. We also
consider the heuristic results, i.e. to control the robot motion with the aid
of human experts. This human-guided heuristic not only serves as a benchmark
of discernment but also offers a benchmark to compare human mastery and our
automated Sim2Real process.

(3) Result: The analysis of our experiment’s results reveals a compelling
advancement in both the realms of task execution speed and task success rate.

Figure 5: Task execution speed comparison. We compare our method with
6D-CLIPort, RL-based robot motion planning, and human-guided heuristic
control, and we measure the average execution time for the whole task.

As shown in Figure 5, our method is remarkably time-saving, reducing the
temporal requirements by a significant margin. Specifically, compared by
state-of-the-art RL [23] and 6D Cliport models [19], our methods exhibit a
time reduction of $54.5\%$. Even compared with the human-guided heuristic
robot control, our methods exhibit a noteworthy $43.4\%$ reduction in time
requirements. This substantial efficiency gain proves the efficacy of our
approach in robotic decision-making and manipulation.

In terms of task success rate (see Table II), the 6D-CLIPort model exhibits a
success rate of $25\%$ for both Pick up and Pour water tasks. Similarly, the
RL model showcases slightly improved results, with success rates of $35\%$ for
picking up and $40\%$ for pouring water. In contrast, our proposed method
showcases a notable leap in performance, achieving a remarkable $75\%$ success
rate for picking up and $70\%$ for pouring water. This heightened performance
substantiates the robustness and efficacy of our approach in executing these
intricate maneuvers.

TABLE II: Comparative analysis of four distinct methods’ performance of the pick-and-pour task. Method | Pick up | Pour water |  | Success
---
(soft)
| Success
---
(hard)
CLIPort | 5/20 | 5/20 | 5/20 | 4/20
RL | 7/20 | 8/20 | 8/20 | 7/20
Ours | 15/20 | 14/20 | 14/20 | 12/20
Heuristic | 19/20 | 17/20 | 17/20 | 16/20
TABLE III: VQA comparison: precision, recall, and F1 score by comparing the model prediction and ground truth | Real scene | Simulation | Overall
---|---|---|---
| Precision | Recall | F1 Score | Precision | Recall | F1 Score | Precision | Recall | F1 Score | Consistency
ViLT | $50.0\%$ | $46.2\%$ | $48.0\%$ | $42.9\%$ | $54.6\%$ | $48.0\%$ | $46.2\%$ | $50.0\%$ | $48.0\%$ | $87.5\%$
MiniGPT4 | $37.5\%$ | $46.2\%$ | $41.2\%$ | $50.0\%$ | $30.8\%$ | $38.1\%$ | $41.7\%$ | $38.5\%$ | $40.0\%$ | $75.0\%$
GPT-V | $\mathbf{85.7\%}$ | $\mathbf{92.3\%}$ | $\mathbf{88.9\%}$ | $\mathbf{85.7\%}$ | $\mathbf{85.7\%}$ | $\mathbf{85.7\%}$ | $\mathbf{85.7\%}$ | $\mathbf{88.9\%}$ | $\mathbf{87.3\%}$ | $\mathbf{96.9\%}$

Furthermore, focusing on the Success (soft) (allowing a little bit of water
leak) and Success (hard) (not allowing water leak) metrics, our method again
achieves a success rate of $70\%$ and $60\%$, respectively. While the CLIPort
and RL models exhibit success rates ranging from $20\%$ to $40\%$ in these
categories, our approach consistently demonstrates a superior ability to
accomplish the task’s objectives, even under more stringent conditions.

However, it’s crucial to acknowledge the human-guided heuristic method’s
exceptional performance, which stands out as a benchmark in this evaluation.
With success rates of $95\%$ for Pick up and $85\%$ for Pour water, as well as
$85\%$ and $80\%$ success rates for Success (soft) and Success (hard).

(4) Failure cases: The major failure case (5 of 8) is when the cup or jar lies
beyond the robot’s configuration space. In such case, no plan generated inside
the simulation environment reaches the goal and thus no actions will be
performed in the real world. Another failure case (3 of 8) is when the
segmentation module cannot output smooth contours for subsequent determination
of key points, causing our key point identification algorithm to generate
inaccurate results.

In summation, the experiment results underscore our proposed method’s
substantial advancements in achieving successful task execution, positioning
it as a promising contender against established models. While the human-guided
heuristic approach remains the gold standard, our approach showcases
remarkable potential.

###  IV-C Comparison of VQA models

We delve into three VQA models to get query results from the questionnaire.
The Vision-and-Language Transformer (ViLT) [24] stands at the intersection of
visual and linguistic understanding, exhibiting a remarkable capability to
seamlessly process and comprehend both visual and textual information. On
another model MiniGPT-4 [25] emerges as a compact yet potent architecture
consisting of the vision transform and large language models, showcasing
impressive text generation prowess within a more resource-efficient framework.
Meanwhile, GPT-4 [26], the latest iteration of the renowned GPT series,
continues to push the boundaries of large-scale language modeling. Boasting an
extensive knowledge base, GPT-4 demonstrates an unparalleled aptitude for
natural language understanding, generation, and manipulation, thus serving as
a pivotal milestone in the progression of AI-driven language processing and
generation. We apply the visual input API (GPT-V) of the GPT-4 for inference
tasks.

Table III presents a comprehensive comparison of precision, recall, and F1
scores for query fluent results across real scenes, simulation, and overall
performance. Notably, GPT-V exhibits the highest performance across multiple
metrics, boasting an impressive precision, recall, and F1 score in both real
scene and simulation scenarios. ViLT and MiniGPT-4 also showcase varying
degrees of performance, with GPT-V demonstrating superior consistency in
matching simulation with reality, as evidenced by its remarkable 96.9%
consistency score. These results underline the efficacy of GPT-V in achieving
a harmonious alignment between reality and simulation during robot execution.

##  V Conclusion

To tackle the enduring challenges of connecting simulation to reality in
robotic tasks, we introduce the triple regression framework. This framework is
designed to rectify the discrepancies arising from differences in camera
parameters, projections, and control dynamics between simulated and real-world
settings. Utilizing this framework, we propose an innovative Sim2Real
technique for managing robotic grasping and manipulation. The robustness of
our approach is demonstrated through our grasp and pour experiments.
Additionally, our integration of large vision models, And-Or graphs for task
planning, and vision question and answering opens up new doors for advancing
future robotic applications.

## References

  * [1] W. Kehl, F. Manhardt, F. Tombari, S. Ilic, and N. Navab, “Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1521–1529.
  * [2] M. Tian, L. Pan, M. H. Ang, and G. H. Lee, “Robust 6d object pose estimation by learning rgb-d features,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 6218–6224.
  * [3] M. Denninger and R. Triebel, “3d scene reconstruction from a single viewport,” in _European Conference on Computer Vision_. Springer, 2020, pp. 51–67.
  * [4] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” _Communications of the ACM_ , vol. 65, no. 1, pp. 99–106, 2021\.
  * [5] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo _et al._ , “Segment anything,” _arXiv preprint arXiv:2304.02643_ , 2023.
  * [6] C. M. Kim, M. Danielczuk, I. Huang, and K. Goldberg, “Ipc-graspsim: Reducing the sim2real gap for parallel-jaw grasping with the incremental potential contact model,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 6180–6187.
  * [7] D. Li and O. Okhrin, “A platform-agnostic deep reinforcement learning framework for effective sim2real transfer in autonomous driving,” _arXiv preprint arXiv:2304.08235_ , 2023.
  * [8] Z. Pan and D. Manocha, “Feedback motion planning for liquid pouring using supervised learning,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 1252–1259.
  * [9] M. Everett, Y. F. Chen, and J. P. How, “Motion planning among dynamic, decision-making agents with deep reinforcement learning,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 3052–3059.
  * [10] Y. F. Chen, M. Everett, M. Liu, and J. P. How, “Socially aware motion planning with deep reinforcement learning,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 1343–1350.
  * [11] M. Schluse and J. Rossmann, “From simulation to experimentable digital twins: Simulation-based development and operation of complex technical systems,” in _2016 IEEE international symposium on systems engineering (ISSE)_. IEEE, 2016, pp. 1–6.
  * [12] R. Pelossof, A. Miller, P. Allen, and T. Jebara, “An svm learning approach to robotic grasping,” in _IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA’04. 2004_ , vol. 4. IEEE, 2004, pp. 3512–3518.
  * [13] Y. Xu, L. Wang, A. Yang, and L. Chen, “Graspcnn: Real-time grasp detection using a new oriented diameter circle representation,” _IEEE Access_ , vol. 7, pp. 159 322–159 331, 2019.
  * [14] H.-S. Fang, C. Wang, M. Gou, and C. Lu, “Graspnet-1billion: A large-scale benchmark for general object grasping,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 11 444–11 453.
  * [15] A. Depierre, E. Dellandréa, and L. Chen, “Jacquard: A large scale dataset for robotic grasp detection,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 3511–3516.
  * [16] P. Mandikal and K. Grauman, “Learning dexterous grasping with object-centric visual affordances,” in _2021 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2021, pp. 6169–6176.
  * [17] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2017, pp. 23–30.
  * [18] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani _et al._ , “Transporter networks: Rearranging the visual world for robotic manipulation,” in _Conference on Robot Learning_. PMLR, 2021, pp. 726–747.
  * [19] M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in _Conference on Robot Learning_. PMLR, 2022, pp. 894–906.
  * [20] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu _et al._ , “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” _arXiv preprint arXiv:2303.05499_ , 2023.
  * [21] J. J. Kuffner and S. M. LaValle, “Rrt-connect: An efficient approach to single-query path planning,” in _Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065)_ , vol. 2. IEEE, 2000, pp. 995–1001.
  * [22] N. D. Ratliff, J. Issac, D. Kappler, S. Birchfield, and D. Fox, “Riemannian motion policies,” _arXiv preprint arXiv:1801.02854_ , 2018.
  * [23] T. Mu, Z. Ling, F. Xiang, D. C. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su, “Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations,” in _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021\.
  * [24] W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 5583–5594.
  * [25] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” _arXiv preprint arXiv:2304.10592_ , 2023.
  * [26] OpenAI, “Gpt-4 technical report,” 2023.