# Superconducting triangular islands as a platform for manipulating Majorana
zero modes

Aidan Winblad  Department of Physics, Colorado State University, Fort Collins,
CO 80523, USA  Hua Chen  Department of Physics, Colorado State University,
Fort Collins, CO 80523, USA  School of Advanced Materials Discovery, Colorado
State University, Fort Collins, CO 80523, USA

###### Abstract

Current proposals for topological quantum computation (TQC) based on Majorana
zero modes (MZM) have mostly been focused on coupled-wire architecture which
can be challenging to implement experimentally. To explore alternative
building blocks of TQC, in this work we study the possibility of obtaining
robust MZM at the corners of triangular superconducting islands, which often
appear spontaneously in epitaxial growth. We first show that a minimal three-
site triangle model of spinless $p$-wave superconductor allows MZM to appear
at different pairs of vertices controlled by a staggered vector potential,
which may be realized using coupled quantum dots and can already demonstrate
braiding. For systems with less fine-tuned parameters, we suggest an
alternative structure of a “hollow” triangle subject to uniform supercurrents
or vector potentials, in which MZM generally appear when two of the edges are
in a different topological phase from the third. We also discuss the
feasibility of constructing the triangles using existing candidate MZM systems
and of braiding more MZM in networks of such triangles.

_Introduction.—_ For more than twenty years, Majorana zero modes (MZM) in
condensed matter systems have been highly sought after due to their potential
for serving as building blocks of topological quantum computation, thanks to
their inherent robustness against decoherence and non-Abelian exchange
statistics [1, 2, 3, 4, 5]. MZM were originally proposed to be found in half-
quantum vortices of two-dimensional (2D) topological p-wave superconductors
and at the ends of 1D spinless p-wave superconductors [6, 7]. Whether a
pristine p-wave superconductor [8] has been found is still under debate.
However, innovative heterostructures proximate to ordinary $s$-wave
superconductors have been proposed to behave as effective topological
superconductors in both 1D and 2D. These include, for example, semiconductor
nanowires subject to magnetic fields [9, 10, 11], ferromagnetic atomic spin
chains [12, 13, 14, 15, 16, 17], 3D topological insulators [18, 19, 20, 21],
quantum anomalous Hall insulators [22, 23, 24], quasi-2D spin-orbit-coupled
superconductors with a perpendicular Zeeman field [25, 26, 27, 28, 29, 30],
and planar Josephson junctions [31, 32, 33, 34, 35, 36, 37], etc. It has been
a challenging task to decisively confirm the existence of MZM in the various
experimental systems due to other competing mechanisms that can potentially
result in similar features as MZM do in different probes [38, 39, 40, 41, 42,
34, 35, 43]. Other proposals for constructing Kitaev chains through a bottom-
up approach, based on, e.g. magnetic tunnel junctions proximate to spin-orbit-
coupled superconductors [44], and quantum dots coupled through superconducting
links [45, 46, 47] are therefore promising. In particular, the recent
experiment [47] of a designer minimal Kitaev chain based on two quantum dots
coupled through tunable crossed Andreev reflections (CAR) offers a compelling
route towards MZM platforms based on exactly solvable building blocks.

In parallel with the above efforts of realizing MZM in different materials
systems, scalable architectures for quantum logic circuits based on MZM have
also been intensely studied over the past decades. A major proposal among
these studies is to build networks of T-junctions, which are minimal units for
swapping a pair of MZM hosted at different ends of a junction, that allow
braiding-based TQC [5]. Alternatively, networks based on coupled wires forming
the so-called tetrons and hexons, aiming at measurement-based logic gate
operations [48], have also been extensively investigated. To counter the
technical challenges of engineering networks with physical wires or atomic
chains, various ideas based on effective Kitaev chains, such as quasi-1D
systems in thin films [49], cross Josephson junctions [37], scissor cuts on a
quantum anomalous Hall insulator [24], and rings of magnetic atoms [50], etc.
have been proposed. However, due to the same difficulty of obtaining or
identifying genuine MZM in quasi-1D systems mentioned above, it remains
unclear how practical these strategies are in the near future.

In this Letter, we propose an alternative structural unit for manipulating
MZM, triangular superconducting islands, motivated by the above challenges
associated with wire geometries and by the fact that triangular islands
routinely appear spontaneously in epitaxial growth [51] on close-packed atomic
surfaces. We first show that a minimal “Kitaev triangle” consisting of three
sites hosts MZM at different pairs of vertices controlled by Peierls phases on
the three edges [Fig. 1 (a)], which can be readily realized using quantum
dots. To generalize the minimal model to triangular structures involving more
degrees of freedom, we study the topological phase transitions of quasi-1D
ribbons driven by Peierls phases, which can be created by magnetic fields or
supercurrents [52, 53], and use the resulting phase diagram as a guide to
construct finite-size triangles with a hollow interior that host MZM [Fig. 1
(b)]. In the end we discuss possible experimental systems that can realize our
proposals and scaled-up networks of triangles for implementing braiding
operations of MZM.

(a)  (b)

Figure 1: Schematics of two triangle structures proposed in this work. (a)
Three-site Kitaev triangle with bond-dependent Peierls phases. The bottom-
left, bottom-right, and top sites are labeled by 1, 2, and 3, respectively.
(b) Hollow triangular island with a uniform vector potential.

_Kitaev triangle.—_ In this section we present an exactly solvable minimal
model with three sites forming a “Kitaev triangle” that can host MZM at
different pairs of vertices controlled by Peierls phases on the edges. The
Bogoliubov-de Gennes (BdG) Hamiltonian includes complex hopping and $p$-wave
pairing between three spinless fermions forming an equilateral triangle [Fig.
1 (a)]:

$\mathcal{H}=\sum_{\langle
jl\rangle}(-te^{i\phi_{jl}}c^{\dagger}_{j}c_{l}+\Delta
e^{i\theta_{jl}}c_{j}c_{l}+{\rm h.c.})-\sum_{j}\mu c^{\dagger}_{j}c_{j},$ (1)

where $t$ is the hopping amplitude, $\Delta$ is the amplitude of the (2D)
$p$-wave pairing, $\mu$ is the chemical potential, $\theta_{jl}$ is the polar
angle of $\mathbf{r}_{jl}=\mathbf{r}_{l}-\mathbf{r}_{j}$ (the $x$ axis is
chosen to be along $\mathbf{r}_{12}$), consistent with
$\\{c^{\dagger}_{l},c^{\dagger}_{j}\\}=0$. $\phi_{jl}$ is the Peierls phase
due to a bond-dependent vector potential $\mathbf{A}$ to be specified below
(the nearest neighbor distance $a$ is chosen to be the length unit
hereinbelow):

$\displaystyle\phi_{jl}=\dfrac{e}{\hbar}\int_{\mathbf{r}_{j}}^{\mathbf{r}_{l}}\mathbf{A}\cdot
d\mathbf{l}=-\phi_{lj}$ (2)

where $e>0$ is the absolute value of the electron charge. Below we use the
natural units $e=\hbar=1$. To get the conditions for having MZM in this model
we rewrite $\mathcal{H}$ in the Majorana fermion basis
$a_{j}=c_{j}+c^{\dagger}_{j}$, $b_{j}=\frac{1}{i}(c_{j}-c^{\dagger}_{j})$:

$\displaystyle\mathcal{H}=-\dfrac{i}{2}\sum_{\langle jl\rangle}\Big{[}$
$\displaystyle\left(t\sin\phi_{jl}-\Delta\sin\theta_{jl}\right)a_{j}a_{l}$ (3)
$\displaystyle+$
$\displaystyle\left(t\sin\phi_{jl}+\Delta\sin\theta_{jl}\right)b_{j}b_{l}$
$\displaystyle+$
$\displaystyle\left(t\cos\phi_{jl}-\Delta\cos\theta_{jl}\right)a_{j}b_{l}$
$\displaystyle-$
$\displaystyle\left(t\cos\phi_{jl}+\Delta\cos\theta_{jl}\right)b_{j}a_{l}\Big{]}-\dfrac{i\mu}{2}\sum_{j}a_{j}b_{j}$

For concreteness we consider the Kitaev limit $t=\Delta$, $\mu=0$, and choose
$\phi_{12}=0$ so that sites 1 and 2 alone form a minimal Kitaev chain with
$\mathcal{H}_{12}=itb_{1}a_{2}$ and hosting MZM $a_{1}$ and $b_{2}$. In order
for the MZM to persist in the presence of site 3, one can choose $\phi_{23}$
and $\phi_{31}$ so that all terms involving these Majorana operators cancel
out. For example, consider the $2-3$ bond, for which $\theta_{23}=2\pi/3$, we
require

$\displaystyle\sin\phi_{23}+\sin\frac{2\pi}{3}=\cos\phi_{23}+\cos\frac{2\pi}{3}=0$
(4)

which means $\phi_{23}=-\pi/3$. Similarly one can find
$\phi_{31}=-\phi_{13}=-\pi/3$. The three Peierls phases can be realized by the
following staggered vector potential

$\mathbf{A}=\left[1-2\Theta(x)\right]\frac{2\pi}{3\sqrt{3}}\hat{\mathbf{y}}$
(5)

where $\Theta(x)$ is the Heavisde step function. In fact, using a uniform
$\mathbf{A}=\frac{2\pi}{3\sqrt{3}}\hat{\mathbf{y}}$, which corresponds to
$\phi_{23}=-\pi/3=-\phi_{31}$ also works, since the existence of $a_{1}$ is
unaffected by $\phi_{23}$. However, in this case the counterpart of $b_{2}$ is
not localized on a single site. For the same reason, the above condition for
MZM localized at triangle corners can be generalized to Kitaev chains forming
a triangular loop, as well as to finite-size triangles of 2D spinless $p$-wave
superconductors in the Kitaev limit, as the existence of $a_{1}$ and $b_{2}$
are only dictated by the vector potential near the corresponding corners. It
should be noted that in the latter case, 1D Majorana edge states will arise
when the triangle becomes larger, and effectively diminish the gap that
protects the corner MZM. On the other hand, for the longer Kitaev chain, due
to the potential practical difficulty of controlling further-neighbor hopping
and pairing amplitudes, it is better to resort to the approach of controlling
the individual topological phases of the three edges which will be detailed in
the next section.

We next show that the minimal Kitaev triangle suffices to demonstrate braiding
of MZM. To this end we consider a closed parameter path linearly interpolating
between the following sets of values of $\phi_{jl}$:

$\displaystyle(\phi_{12},\phi_{23},\phi_{31})$ $\displaystyle=$
$\displaystyle\left(0,-\frac{\pi}{3},-\frac{\pi}{3}\right)\equiv\bm{\phi}_{1}$
$\displaystyle\rightarrow$
$\displaystyle\left(-\frac{\pi}{3},-\frac{\pi}{3},0\right)\equiv\bm{\phi}_{2}$
$\displaystyle\rightarrow$
$\displaystyle\left(-\frac{\pi}{3},0,-\frac{\pi}{3}\right)\equiv\bm{\phi}_{3}$
$\displaystyle\rightarrow$ $\displaystyle\bm{\phi}_{1}$

It is straightforward to show that at $\bm{\phi}_{2}$ and $\bm{\phi}_{3}$
there are MZM located at sites $3,1$ and $2,3$, respectively. Therefore the
two original MZM at sites $1,2$ should switch their positions at the end of
the adiabatic evolution.

(a)

(b)

Figure 2: (a) Evolution of the eigenvalues of the 3-site Kitaev triangle along
the closed parameter path for $\phi$ on the three edges. (b) MZM wavefunctions
at different points of the parameter path. Clockwise from the upper left
panel:
$\bm{\phi}_{1}\rightarrow\frac{1}{2}(\bm{\phi}_{1}+\bm{\phi}_{2})\rightarrow\bm{\phi}_{2}\rightarrow\bm{\phi}_{3}$.

Indeed, Fig. 2 shows that the MZM stays at zero energy throughout the
parameter path that interchanges their positions. To show that such an
operation indeed realizes braiding, we explicitly calculated the many-body
Berry phase of the evolution [54, 4, 50] and found the two degenerate many-
body ground states acquire a $\frac{\pi}{2}$ difference in their Berry phases
as expected [4]. Compared to the minimum T-junction model with four sites [4],
our Kitaev triangle model only requires three sites to achieve braiding
between two MZM, and is potentially also easier to engineer experimentally. In
the next section we will show that a more mesoscopic hollow-triangle structure
can achieve similar results and may be preferred in other materials platforms.

_Hollow triangles.—_ For systems with less fine-tuned Hamiltonians than the
minimal model in the previous section, it is more instructive to search for
MZM based on topological arguments. In this section we show that MZM generally
appear at the corners of a hollow triangle, which can be approximated by
joining three finite-width chains or ribbons whose bulk topology is
individually tuned by the same uniform vector potential.

To this end, we first show that topological phase transitions can be induced
by a vector potential in a spinless $p$-wave superconductor ribbon. In
comparison with similar previous proposals that mostly focused on vector
potentials or supercurrents flowing along the chain [52, 53], we consider in
particular the tunability by varying the direction of the vector potential
relative to the length direction of the ribbon, which will become instrumental
in a triangular structure.

Consider Eq. (1) on a triangular lattice defined by unit-length lattice
vectors
$(\mathbf{a}_{1},\mathbf{a}_{2})=(\hat{\mathbf{x}},\frac{1}{2}\hat{\mathbf{x}}+\frac{\sqrt{3}}{2}\hat{\mathbf{y}})$
with $W$ unit cells along $\mathbf{a}_{2}$ but infinite unit cells along
$\mathbf{a}_{1}$, and assume the Peierls phases are due to a uniform vector
potential $\mathbf{A}$ so that $\phi_{jl}=\mathbf{A}\cdot\mathbf{r}_{jl}$. We
also introduce $\mathbf{a}_{3}\equiv-\mathbf{a}_{1}+\mathbf{a}_{2}$ for later
convenience. The Hamiltonian is periodic along $x$ and can be Fourier
transformed through
$c^{\dagger}_{m,n}=\dfrac{1}{\sqrt{N}}\sum_{k}c^{\dagger}_{k,n}e^{-ikm}$,
where $m,n$ label the lattice sites as
$\mathbf{r}_{m,n}=m\mathbf{a}_{1}+n\mathbf{a}_{2}$. The resulting momentum
space Hamiltonian can be written as the following block form up to a constant

$\displaystyle\mathcal{H}$ $\displaystyle=$
$\displaystyle\dfrac{1}{2}\sum_{k}\Psi_{k}^{\dagger}\left(\begin{matrix}h_{t}(k)&h_{\Delta}(k)\\\
h_{\Delta}^{\dagger}(k)&-h_{t}^{*}(-k)\end{matrix}\right)\Psi_{k}$
$\displaystyle\equiv$
$\displaystyle\dfrac{1}{2}\sum_{k}\Psi_{k}^{\dagger}H(k)\Psi_{k}$

where $\Psi_{k}\equiv(c_{k,1},\dots,c_{k,W},c^{\dagger}_{-k,1},\dots
c_{-k,W}^{\dagger})^{T}$. $h_{t}(k)$ is a $W\times W$ Hermitian tridiagonal
matrix with $(h_{t})_{n,n}=-2t\cos(k+\mathbf{A}\cdot\mathbf{a}_{1})-\mu$ and
$(h_{t})_{n,n+1}=-t\left(e^{i(-k+\mathbf{A}\cdot\mathbf{a}_{3})}+e^{i\mathbf{A}\cdot\mathbf{a}_{2}}\right)$.
$h_{\Delta}(k)$ is a $W\times W$ tridiagonal matrix with
$(h_{\Delta})_{n,n}=-2i\Delta\sin k$ and $(h_{\Delta})_{n,n\pm
1}=\mp\Delta\left[e^{-i(\pm k+\frac{2\pi}{3})}+e^{-i\frac{\pi}{3}}\right]$.

By transforming Eq. (Superconducting triangular islands as a platform for
manipulating Majorana zero modes) to the Majorana basis using the unitary
transformation:

$\displaystyle U\equiv\dfrac{1}{\sqrt{2}}\left(\begin{matrix}1&1\\\
-i&i\end{matrix}\right)\otimes\mathbb{I}$ (8)

where $\mathbb{I}$ is a ${W\times W}$ identity matrix, and defining
$A_{k}\equiv-iUH(k)U^{\dagger}$, not to be confused with the vector potential,
one can calculate the Majorana number [7] $\mathcal{M}$ of the 1D ribbon as
[55]

$\displaystyle\mathcal{M}={\rm sgn}\left[{{\rm Pf}(A_{k=0}){\rm
Pf}(A_{k=\pi})}\right]$ (9)

where Pf stands for the Pfaffian of a skew-symmetric matrix [7]. When
$\mathcal{M}=-1$, the 1D system is in a nontrivial topological phase with MZM
appearing at open ends of semi-infinite ribbons, and otherwise for
$\mathcal{M}=1$.

(a)

(b)

Figure 3: (a) Topological phase diagram for a $W=1$ triangular chain with the
Hamiltonian Eq. (Superconducting triangular islands as a platform for
manipulating Majorana zero modes) obtained by superimposing the
$\mathcal{M}(A,\mu)$ plots of 1D chains with $\mathbf{A}=A\hat{\mathbf{y}}$
and
$\mathbf{A}=A(\frac{\sqrt{3}}{2}\hat{\mathbf{x}}+\frac{1}{2}\hat{\mathbf{y}})$.
Color scheme: white—$\mathcal{M}=1$, dark blue—$\mathcal{M}=-1$, light
blue—$\mathcal{M}=0$ (b) Near-gap BdG eigen-energies vs $A$ for a finite
triangle with edge length $L=50$, $W=1$, and $\mu=1.6$.

In Fig. 3 (a) we show the topological phase diagrams for a 1D ribbon with
width $W=1$, $\mathbf{A}=A\hat{\mathbf{y}}$ and
$\mathbf{A}=A(\frac{\sqrt{3}}{2}\hat{\mathbf{x}}+\frac{1}{2}\hat{\mathbf{y}})$
superimposed (see below). We found that the vector potential component normal
to the ribbon length direction has no effect on the Majorana number, nor does
the sign of its component along the ribbon length direction. However,
topological phase transitions can be induced by varying the size of the vector
potential component along the ribbon, consistent with previous results [52,
53]. These properties motivate us to consider the structure of a hollow
triangle formed by three finite-width ribbons subject to a uniform vector
potential $\mathbf{A}=A\hat{\mathbf{y}}$ as illustrated in Fig. 1 (b). The
light blue color on the phase diagram Fig. 3 (a) therefore means that the
bottom edge and the two upper edges of the hollow triangle have different
$\mathcal{M}$, which should give rise to MZM localized at the two bottom
corners if the triangle is large enough so that bulk-edge correspondence
holds, and gap closing does not occur at other places along its edges.

To show that corner MZM indeed appear when the conditions given by the phase
diagram Fig. 3 (a) are met, we directly diagonalize the BdG Hamiltonian of a
finite hollow triangle with edge length $L=50$ and width $W=1$. Fig. 3 (b)
shows the spectral flow (BdG eigen-energies evolving with increasing vector
potential $A$) close to zero energy at chemical potential $\mu=1.6$. Indeed,
zero-energy modes appear in the regions of $\mu$ and $A$ consistent with the
phase diagram (except when the bulk band gap is too small; see [54] for some
examples.). Hollow triangles with larger larger $W$ also have qualitatively
similar behavior, although the phase diagrams are more complex [54]. The
eigenfunctions for the zero-energy modes at $A=2.75$ and $\mu=1.6$ in Fig. 4
(b) also confirm their spatial localization at the bottom corners of the
triangle.

(a)  (b)  (c)  (d)

Figure 4: (a) Spectral flow of a hollow triangle with $W=1$, $L=50$,
$\mu=1.6$, and $A=2.75$ with increasing rotation angle $\varphi$, defined
through
$\mathbf{A}=A(-\sin\varphi\hat{\mathbf{x}}+\cos\varphi\hat{\mathbf{y}})$.
(b-d) BdG eigenfunction $|\Psi|^{2}$ summed over the two zero modes at
$\varphi=0,\frac{\pi}{6}$, and $\frac{\pi}{3}$, respectively.

We finally show that rotating the uniform vector potential in-plane can
manipulate the positions of the MZM without hybridizing them with bulk states
for certain ranges of $\mu$ and $A$. Fig. 4 (a) plots the spectral flow versus
the in-plane azimuthal angle of $\mathbf{A}$, which clearly shows that the
zero-energy modes persist throughout the rotation and the bulk gap never
closes. Figs. 4 (b-d) plot the BdG wavefunctions of the MZM at special values
of $\varphi$. One can see that the two MZM appear to cycle through the three
vertices by following the rotation of $\mathbf{A}$. The robustness of the MZM
therefore requires the condition of two edges being in a different topological
phase from the third one to be satisfied throughout the rotation. Such a
criterion combined with the individual phase diagrams of the edges can help
isolate the desired parameter regions of $\mu$ and $A$. We also note that the
positions of the MZM do not interchange after $\varphi$ increases from 0 to
$\pi$, different from the situation of the minimal Kitaev triangle in Fig. 2.
The reason is that the MZM in the latter case are not due to bulk-boundary
correspondence [the values of $A=\frac{2\pi}{3\sqrt{3}}$ and $\mu=0$ are a
critical point in the phase diagram Fig. 3 (a)]. While the positions of the
MZM at special points along the parameter path in the hollow triangle case
have to be additionally constrained by the bulk topological phases of the
three edges, that for the Kitaev triangle have more flexibility and are also
protected by the finite size of the system.

_Discussion.—_ The hollow interior of the triangles considered in this work is
needed for two reasons: (1) $W\ll L$ is required for bulk-edge correspondence
based on 1D topology to hold; (2) A finite $W$ is needed to gap out the chiral
edge states of a 2D spinless $p$-wave superconductor based on which Eq.
(Superconducting triangular islands as a platform for manipulating Majorana
zero modes) is written. The latter is not essential if one does not start with
a spinless $p$-wave supercondutor but a more realistic model such as the
Rashba+Zeeman+$s$-wave pairing model. On the other hand, the former constraint
may also be removed if one uses the Kitaev triangle. Nonetheless, an effective
3-site Kitaev triangle may emerge as the effective theory of triangular
structures if a three-orbital low-energy Wannier basis can be isolated,
similar to the continuum theory of moiré structures. We also note in passing
that the corner MZM in our triangles appear due to different reasons from that
in higher-order topological superconductors [41, 56].

(a)(b)(c)(d)$\gamma_{1}$$\gamma_{2}$$\gamma_{3}$$\gamma_{4}$$\gamma_{1}$$\gamma_{2}$$\gamma_{3}$$\gamma_{4}$$\gamma_{1}$$\gamma_{3}$$\gamma_{2}$$\gamma_{4}$$\gamma_{1}$$\gamma_{3}$$\gamma_{2}$$\gamma_{4}$
Figure 5: Representative steps for braiding four MZM in four triangles sharing
corners. (a) Initialization of four MZM
$\gamma_{1},\gamma_{2},\gamma_{3},\gamma_{4}$. All three edges of the bottom-
middle and the top triangles are in the trivial phase by e.g. controlling the
chemical potential. The bottom-left and bottom-right triangles have
$\varphi=0$ so that their bottom edges are nontrivial. (b) Moving $\gamma_{3}$
by “switching on” the middle triangle by changing the chemical potential under
a fixed vector potential at $\varphi=\frac{\pi}{6}$, and then turning on the
top triangle with similar means except $\varphi=0$. (c) Transporting
$\gamma_{2}$ to the right triangle through rotating the vector potential in
the middle triangle counterclockwise by $\pi/6$. (d) Moving $\gamma_{3}$ to
the left triangle by “switching off” the top triangle followed by the middle
triangle.

For possible physical realizations of our triangles, immediate choices are
quantum dots forming a Kitaev triangle [47], planar Josephson junctions or
cuts on quantum anomalous Hall insulator/superconductor heterostructures [24]
that form a hollow triangle, and triangular atomic chains assembled by an STM
tip [17] on a close-packed surface. The quantum-dot platform may be
advantageous in the convenience of implementing parity readout by turning the
third vertex temporarily into a normal quantum dot [57, 58, 59]. Looking into
the future, it is more intriguing to utilize the spontaneously formed
triangular islands in epitaxial growth [51] with the center region removed
either physically by lithography/ablation, or electrically by gating. To
create a staggered vector potential or supercurrent profile for the Kitaev
triangle, one can use a uniform magnetic field, corresponding to a constant
vector potential gradient, plus a uniform supercurrent that controls the
position of the zero. It is also possible to use two parallel superconducting
wires with counter-propagating supercurrents proximate to the triangle.

A tentative design for braiding more than two MZM, illustrated in Fig. 5,
consists of four triangles sharing corners with their neighbors. The critical
step of transporting $\gamma_{2}$ to the left vertex of the rightmost
triangle, corresponding to Figs. 5 (b,c), can be achieved by rotating the
vector potential of the bottom-middle triangle counterclockwisely from
$\varphi=\frac{\pi}{6}$ to $\frac{\pi}{3}$, which swaps the topological phases
of the two side edges as shown in Fig. 4. In [54] we show this operation does
not involve gap closing at least for certain parameter regions. Our work
provides a versatile platform for manipulating MZM based on currently
available candidate MZM systems and for potentially demonstrating the non-
Abelian nature of MZM in near-term devices.

###### Acknowledgements.

This work was supported by the start-up funding of CSU and partially by NSF
CAREER grant DMR-1945023.

## References

  * Ivanov [2001] D. A. Ivanov, Non-Abelian Statistics of Half-Quantum Vortices in p -Wave Superconductors, Phys. Rev. Lett. 86, 268 (2001).
  * Kitaev [2003] A. Yu. Kitaev, Fault-tolerant quantum computation by anyons, Annals of Physics 303, 2 (2003).
  * Nayak  _et al._ [2008] C. Nayak, S. H. Simon, A. Stern, M. Freedman, and S. Das Sarma, Non-Abelian anyons and topological quantum computation, Rev. Mod. Phys. 80, 1083 (2008).
  * Alicea  _et al._ [2011] J. Alicea, Y. Oreg, G. Refael, F. von Oppen, and M. P. A. Fisher, Non-Abelian statistics and topological quantum information processing in 1D wire networks, Nature Phys 7, 412 (2011).
  * Aasen  _et al._ [2016] D. Aasen, M. Hell, R. V. Mishmash, A. Higginbotham, J. Danon, M. Leijnse, T. S. Jespersen, J. A. Folk, C. M. Marcus, K. Flensberg, and J. Alicea, Milestones Toward Majorana-Based Quantum Computing, Phys. Rev. X 6, 031016 (2016).
  * Read and Green [2000] N. Read and D. Green, Paired states of fermions in two dimensions with breaking of parity and time-reversal symmetries and the fractional quantum Hall effect, Phys. Rev. B 61, 10267 (2000).
  * Kitaev [2001] A. Y. Kitaev, Unpaired Majorana fermions in quantum wires, Phys.-Usp. 44, 131 (2001).
  * Brison [2021] J.-P. Brison, P-Wave Superconductivity and d-Vector Representation, in  _Magnetism and Accelerator-Based Light Sources_, Springer Proceedings in Physics, edited by H. Bulou, L. Joly, J.-M. Mariot, and F. Scheurer (Springer International Publishing, Cham, 2021) pp. 165–204.
  * Mourik  _et al._ [2012] V. Mourik, K. Zuo, S. M. Frolov, S. R. Plissard, E. P. A. M. Bakkers, and L. P. Kouwenhoven, Signatures of Majorana Fermions in Hybrid Superconductor-Semiconductor Nanowire Devices, Science 336, 1003 (2012).
  * Rokhinson  _et al._ [2012] L. P. Rokhinson, X. Liu, and J. K. Furdyna, The fractional a.c. Josephson effect in a semiconductor–superconductor nanowire as a signature of Majorana particles, Nature Phys 8, 795 (2012).
  * Deng  _et al._ [2012] M. T. Deng, C. L. Yu, G. Y. Huang, M. Larsson, P. Caroff, and H. Q. Xu, Anomalous Zero-Bias Conductance Peak in a Nb–InSb Nanowire–Nb Hybrid Device, Nano Lett. 12, 6414 (2012).
  * Choy  _et al._ [2011] T.-P. Choy, J. M. Edge, A. R. Akhmerov, and C. W. J. Beenakker, Majorana fermions emerging from magnetic nanoparticles on a superconductor without spin-orbit coupling, Phys. Rev. B 84, 195442 (2011).
  * Braunecker and Simon [2013] B. Braunecker and P. Simon, Interplay between Classical Magnetic Moments and Superconductivity in Quantum One-Dimensional Conductors: Toward a Self-Sustained Topological Majorana Phase, Phys. Rev. Lett. 111, 147202 (2013).
  * Klinovaja  _et al._ [2013] J. Klinovaja, P. Stano, A. Yazdani, and D. Loss, Topological Superconductivity and Majorana Fermions in RKKY Systems, Phys. Rev. Lett. 111, 186805 (2013).
  * Nadj-Perge  _et al._ [2013] S. Nadj-Perge, I. K. Drozdov, B. A. Bernevig, and A. Yazdani, Proposal for realizing Majorana fermions in chains of magnetic atoms on a superconductor, Phys. Rev. B 88, 020407 (2013).
  * Nadj-Perge  _et al._ [2014] S. Nadj-Perge, I. K. Drozdov, J. Li, H. Chen, S. Jeon, J. Seo, A. H. MacDonald, B. A. Bernevig, and A. Yazdani, Observation of Majorana fermions in ferromagnetic atomic chains on a superconductor, Science 346, 602 (2014).
  * Schneider  _et al._ [2022] L. Schneider, P. Beck, J. Neuhaus-Steinmetz, L. Rózsa, T. Posske, J. Wiebe, and R. Wiesendanger, Precursors of Majorana modes and their length-dependent energy oscillations probed at both ends of atomic Shiba chains, Nat. Nanotechnol. 17, 384 (2022).
  * Fu and Kane [2008] L. Fu and C. L. Kane, Superconducting proximity effect and Majorana fermions at the surface of a topological insulator, Phys. Rev. Lett. 100, 096407 (2008), arxiv:0707.1692 [cond-mat] .
  * Hosur  _et al._ [2011] P. Hosur, P. Ghaemi, R. S. K. Mong, and A. Vishwanath, Majorana Modes at the Ends of Superconductor Vortices in Doped Topological Insulators, Phys. Rev. Lett. 107, 097001 (2011).
  * Potter and Lee [2011] A. C. Potter and P. A. Lee, Engineering a $p+\mathit{ip}$ superconductor: Comparison of topological insulator and Rashba spin-orbit-coupled materials, Phys. Rev. B 83, 184520 (2011).
  * Veldhorst  _et al._ [2013] M. Veldhorst, M. Snelder, M. Hoek, C. G. Molenaar, D. P. Leusink, A. A. Golubov, H. Hilgenkamp, and A. Brinkman, Magnetotransport and induced superconductivity in Bi based three-dimensional topological insulators, physica status solidi (RRL) – Rapid Research Letters 7, 26 (2013).
  * Chen  _et al._ [2018] C.-Z. Chen, Y.-M. Xie, J. Liu, P. A. Lee, and K. T. Law, Quasi-one-dimensional quantum anomalous Hall systems as new platforms for scalable topological quantum computation, Phys. Rev. B 97, 104504 (2018).
  * Zeng  _et al._ [2018] Y. Zeng, C. Lei, G. Chaudhary, and A. H. MacDonald, Quantum anomalous Hall Majorana platform, Phys. Rev. B 97, 081102 (2018).
  * Xie  _et al._ [2021] Y.-M. Xie, X.-J. Gao, T.-K. Ng, and K. T. Law, Creating Localized Majorana Zero Modes in Quantum Anomalous Hall Insulator/Superconductor Heterostructures with a Scissor (2021), arxiv:2012.15523 [cond-mat] .
  * Oreg  _et al._ [2010] Y. Oreg, G. Refael, and F. von Oppen, Helical Liquids and Majorana Bound States in Quantum Wires, Phys. Rev. Lett. 105, 177002 (2010).
  * Sau  _et al._ [2010] J. D. Sau, R. M. Lutchyn, S. Tewari, and S. Das Sarma, Generic New Platform for Topological Quantum Computation Using Semiconductor Heterostructures, Phys. Rev. Lett. 104, 040502 (2010).
  * Lutchyn  _et al._ [2011] R. M. Lutchyn, T. D. Stanescu, and S. Das Sarma, Search for Majorana Fermions in Multiband Semiconducting Nanowires, Phys. Rev. Lett. 106, 127001 (2011).
  * Potter and Lee [2012] A. C. Potter and P. A. Lee, Topological superconductivity and Majorana fermions in metallic surface states, Phys. Rev. B 85, 094516 (2012).
  * Li  _et al._ [2016a] J. Li, T. Neupert, Z. Wang, A. H. MacDonald, A. Yazdani, and B. A. Bernevig, Two-dimensional chiral topological superconductivity in Shiba lattices, Nat Commun 7, 12297 (2016a).
  * Lei  _et al._ [2018] C. Lei, H. Chen, and A. H. MacDonald, Ultrathin Films of Superconducting Metals as a Platform for Topological Superconductivity, Phys. Rev. Lett. 121, 227701 (2018).
  * Black-Schaffer and Linder [2011] A. M. Black-Schaffer and J. Linder, Majorana fermions in spin-orbit-coupled ferromagnetic Josephson junctions, Phys. Rev. B 84, 180509 (2011).
  * Pientka  _et al._ [2013] F. Pientka, A. Romito, M. Duckheim, Y. Oreg, and F. von Oppen, Signatures of topological phase transitions in mesoscopic superconducting rings, New J. Phys. 15, 025001 (2013).
  * Hell  _et al._ [2017] M. Hell, M. Leijnse, and K. Flensberg, Two-Dimensional Platform for Networks of Majorana Bound States, Phys. Rev. Lett. 118, 107701 (2017).
  * Fornieri  _et al._ [2019] A. Fornieri, A. M. Whiticar, F. Setiawan, E. Portolés, A. C. C. Drachmann, A. Keselman, S. Gronin, C. Thomas, T. Wang, R. Kallaher, G. C. Gardner, E. Berg, M. J. Manfra, A. Stern, C. M. Marcus, and F. Nichele, Evidence of topological superconductivity in planar Josephson junctions, Nature 569, 89 (2019).
  * Ren  _et al._ [2019] H. Ren, F. Pientka, S. Hart, A. T. Pierce, M. Kosowsky, L. Lunczer, R. Schlereth, B. Scharf, E. M. Hankiewicz, L. W. Molenkamp, B. I. Halperin, and A. Yacoby, Topological superconductivity in a phase-controlled Josephson junction, Nature 569, 93 (2019).
  * Scharf  _et al._ [2019] B. Scharf, F. Pientka, H. Ren, A. Yacoby, and E. M. Hankiewicz, Tuning topological superconductivity in phase-controlled Josephson junctions with Rashba and Dresselhaus spin-orbit coupling, Phys. Rev. B 99, 214503 (2019).
  * Zhou  _et al._ [2020] T. Zhou, M. C. Dartiailh, W. Mayer, J. E. Han, A. Matos-Abiague, J. Shabani, and I. Žutić, Phase Control of Majorana Bound States in a Topological $\mathsf{}X{}$ Junction, Phys. Rev. Lett. 124, 137001 (2020).
  * Xu  _et al._ [2015] J.-P. Xu, M.-X. Wang, Z. L. Liu, J.-F. Ge, X. Yang, C. Liu, Z. A. Xu, D. Guan, C. L. Gao, D. Qian, Y. Liu, Q.-H. Wang, F.-C. Zhang, Q.-K. Xue, and J.-F. Jia, Experimental Detection of a Majorana Mode in the core of a Magnetic Vortex inside a Topological Insulator-Superconductor Bi 2 Te 3 / NbSe 2 Heterostructure, Phys. Rev. Lett. 114, 017001 (2015).
  * Albrecht  _et al._ [2016] S. M. Albrecht, A. P. Higginbotham, M. Madsen, F. Kuemmeth, T. S. Jespersen, J. Nygård, P. Krogstrup, and C. M. Marcus, Exponential protection of zero modes in Majorana islands, Nature 531, 206 (2016).
  * Sun  _et al._ [2016] H.-H. Sun, K.-W. Zhang, L.-H. Hu, C. Li, G.-Y. Wang, H.-Y. Ma, Z.-A. Xu, C.-L. Gao, D.-D. Guan, Y.-Y. Li, C. Liu, D. Qian, Y. Zhou, L. Fu, S.-C. Li, F.-C. Zhang, and J.-F. Jia, Majorana Zero Mode Detected with Spin Selective Andreev Reflection in the Vortex of a Topological Superconductor, Phys. Rev. Lett. 116, 257003 (2016).
  * Wang  _et al._ [2018] D. Wang, L. Kong, P. Fan, H. Chen, S. Zhu, W. Liu, L. Cao, Y. Sun, S. Du, J. Schneeloch, R. Zhong, G. Gu, L. Fu, H. Ding, and H.-J. Gao, Evidence for Majorana bound states in an iron-based superconductor, Science 362, 333 (2018).
  * Jäck  _et al._ [2019] B. Jäck, Y. Xie, J. Li, S. Jeon, B. A. Bernevig, and A. Yazdani, Observation of a Majorana zero mode in a topologically protected edge channel, Science 364, 1255 (2019).
  * Manna  _et al._ [2020] S. Manna, P. Wei, Y. Xie, K. T. Law, P. A. Lee, and J. S. Moodera, Signature of a pair of Majorana zero modes in superconducting gold surface states, Proceedings of the National Academy of Sciences 117, 8775 (2020).
  * Fatin  _et al._ [2016] G. L. Fatin, A. Matos-Abiague, B. Scharf, and I. Žutić, Wireless Majorana Bound States: From Magnetic Tunability to Braiding, Phys. Rev. Lett. 117, 077002 (2016).
  * Sau and Sarma [2012] J. D. Sau and S. D. Sarma, Realizing a robust practical Majorana chain in a quantum-dot-superconductor linear array, Nat Commun 3, 964 (2012).
  * Leijnse and Flensberg [2012] M. Leijnse and K. Flensberg, Parity qubits and poor man’s Majorana bound states in double quantum dots, Phys. Rev. B 86, 134528 (2012).
  * Dvir  _et al._ [2023] T. Dvir, G. Wang, N. van Loo, C.-X. Liu, G. P. Mazur, A. Bordin, S. L. D. ten Haaf, J.-Y. Wang, D. van Driel, F. Zatelli, X. Li, F. K. Malinowski, S. Gazibegovic, G. Badawy, E. P. A. M. Bakkers, M. Wimmer, and L. P. Kouwenhoven, Realization of a minimal Kitaev chain in coupled quantum dots, Nature 614, 445 (2023).
  * Karzig  _et al._ [2017] T. Karzig, C. Knapp, R. M. Lutchyn, P. Bonderson, M. B. Hastings, C. Nayak, J. Alicea, K. Flensberg, S. Plugge, Y. Oreg, C. M. Marcus, and M. H. Freedman, Scalable designs for quasiparticle-poisoning-protected topological quantum computation with Majorana zero modes, Phys. Rev. B 95, 235305 (2017).
  * Potter and Lee [2010] A. C. Potter and P. A. Lee, Multichannel Generalization of Kitaev’s Majorana End States and a Practical Route to Realize Them in Thin Films, Phys. Rev. Lett. 105, 227003 (2010).
  * Li  _et al._ [2016b] J. Li, T. Neupert, B. A. Bernevig, and A. Yazdani, Manipulating Majorana zero modes on atomic rings with an external magnetic field, Nat Commun 7, 10395 (2016b).
  * Pietzsch  _et al._ [2006] O. Pietzsch, S. Okatov, A. Kubetzka, M. Bode, S. Heinze, A. Lichtenstein, and R. Wiesendanger, Spin-Resolved Electronic Structure of Nanoscale Cobalt Islands on Cu(111), Phys. Rev. Lett. 96, 237203 (2006).
  * Romito  _et al._ [2012] A. Romito, J. Alicea, G. Refael, and F. von Oppen, Manipulating Majorana fermions using supercurrents, Phys. Rev. B 85, 020502 (2012).
  * Takasan  _et al._ [2022] K. Takasan, S. Sumita, and Y. Yanase, Supercurrent-induced topological phase transitions, Phys. Rev. B 106, 014508 (2022).
  * [54] See Supplemental Material at [url] .
  * Li  _et al._ [2014] J. Li, H. Chen, I. K. Drozdov, A. Yazdani, B. A. Bernevig, and A. H. MacDonald, Topological superconductivity induced by ferromagnetic metal chains, Phys. Rev. B 90, 235433 (2014).
  * Pahomi  _et al._ [2020] T. E. Pahomi, M. Sigrist, and A. A. Soluyanov, Braiding Majorana corner modes in a second-order topological superconductor, Phys. Rev. Res. 2, 032068 (2020).
  * Mishmash  _et al._ [2020] R. V. Mishmash, B. Bauer, F. von Oppen, and J. Alicea, Dephasing and leakage dynamics of noisy Majorana-based qubits: Topological versus Andreev, Phys. Rev. B 101, 075404 (2020).
  * Széchenyi and Pályi [2020] G. Széchenyi and A. Pályi, Parity-to-charge conversion for readout of topological majorana qubits, Phys. Rev. B 101, 235441 (2020).
  * Feng and Zhang [2022] G.-H. Feng and H.-H. Zhang, Probing robust Majorana signatures by crossed Andreev reflection with a quantum dot, Phys. Rev. B 105, 035148 (2022).


llmshort=LLM, long=Large Language Model kgshort=KG, long=Knowledge Graph
reshort=RE, long=Relation Extraction cotshort=CoT, long=Chain-of-Thought

# Construction of Hyper-Relational Knowledge Graphs Using Pre-Trained Large
Language Models

Preetha Datta
Department of Computer Science, Aalto University
Finland
<EMAIL_ADDRESS>Fedor Vitiugin
Department of Computer Science, Aalto University
Finland
<EMAIL_ADDRESS>Anastasiia Chizhikova
Department of Computer Science, Aalto University
Finland
<EMAIL_ADDRESS>Nitin Sawhney
Department of Computer Science, Aalto University
Finland
<EMAIL_ADDRESS>
###### Abstract

Extracting hyper-relations is crucial for constructing comprehensive knowledge
graphs, but there are limited supervised methods available for this task. To
address this gap, we introduce a zero-shot prompt-based method using OpenAI’s
GPT-3.5 model for extracting hyper-relational knowledge from text. Comparing
our model with a baseline, we achieved promising results, with a recall of
0.77. Although our precision is currently lower, a detailed analysis of the
model outputs has uncovered potential pathways for future research in this
area.

##  1 Introduction

kg have captured widespread interest in both industry and academia,
particularly in situations demanding the utilization of vast, constantly
changing collections of data [12]. kg use a graph-based data model to capture
knowledge in application scenarios that involve integrating, managing and
extracting value from diverse sources of data at large scale [18]. Graphs
offer a concise and intuitive representation for numerous domains, where
connections and routes delineate diverse and sometimes intricate relationships
among domain entities [14]. Moreover, kg enable maintainers to postpone the
definition of a schema, allowing the data to evolve in a more flexible manner
[1].

re task represents an opportunity for automated creation of extensive kg by
mining facts from natural language text. In general, these methods concentrate
on binary relations [10], linking two entities to form a relation triple: the
starting entity, the relationship, and the ending entity. However, real re
often feature complex hyper-relational facts [9].

These facts include additional qualifier attributes – like time, quantity, or
location – for each relationship triple, as illustrated in the provided Figure
1. Therefore, simply extracting these relation triples might oversimplify the
intricate structure of kg, failing to capture its richness and complexity.

Figure 1: Example of extraction entities, relations and qualifiers from the
text using our zero-shot prompting technique.

Hyper-relational facts differ from standard relation triples because their
qualifiers apply to the entire triple rather than a specific entity within it.
Despite this complexity, hyper-relational facts offer practical advantages,
enhancing fact verification [24] and refining how kg are represented and
learned [8]. Consequently, extracting relation triples with their associated
qualifiers becomes essential for constructing these hyper-relational facts.
Unfortunately, there’s a scarcity of models dedicated to hyper-relational
extraction. For instance, the CubeRE model [3], a supervised learning
approach, which requires a training dataset and presents uncertainties
regarding its adaptability to different domains.

The key motivation behind our work is the promise of significant saving of
resources requested for building training data. We found that leveraging llm
for information extraction could have noteworthy implications, especially in
scenarios where adapting to specific domains or tasks is critical. The use of
pre-trained llm has the potential to expedite the deployment of information
extraction techniques across diverse domains, offering an efficient and
resource-friendly alternative to traditional supervised approaches.

One of the key contributions of our work is a prompt-based method for hyper-
relational extraction from unstructured text data, which is used for
construction of rich and complex kg.

##  2 Related Work

The term “Knowledge Graph” was originally introduced by Google in a seminal
blog post in 2012, signifying a “critical first step towards the next
generation of search” [21]. While graph-based textual documents have existed
for decades, it was this blog post that largely popularized and highlighted
the flexibility of the kg architecture.

###  2.1 Relation Extraction

While kg could be constructed based on semi-structured websites [20], tables
[6] or link prediction [19], extracting facts from unstructured data is still
the most challenging task due to the vast amount of text on the web [31].

re is one of the main parts of kg construction and a well-established task
which could be done with supervised [26] or semi-supervised [11] learning
methods. While the Zero-Shot Relation Extraction approach [26] could adapt and
extrapolates knowledge across diverse domains and applications, it faces
limitations when labeled training data is scarce or expensive. In such cases,
unsupervised techniques offer compelling alternatives by combining Pre-Trained
Language Models such as RoBERTa, with Open Information Extraction [23].
Prompt-Based Zero-Shot Relation Extraction Method is a prominent technique to
maximize the efficiency of data exploitation by prompt-tuning technique in
pre-trained language models [15].

The use of carefully crafted instructions, known as prompts, allows to better
uncover connections and details in unstructured textual data [13]. A recent
approach called ChatIE transforms the zero-shot information extraction method
into a multi-turn question-answering framework by using prompt-like
conversation with the llm to construct a robust information extraction model
[29]

###  2.2 Prompt Engineering in llm

Before discussing prompt engineering in detail, let’s examine llm, and more
specifically, the GPT series of models released by OpenAI. The specific
technical details, encompassing aspects such as architecture, hardware,
dataset construction, and training methodology, remain undisclosed for the
OpenAI’s GPT models [30]. Overall, GPT models are autoregressive language
models – meaning it utilizes a decoder-only approach to language modeling, and
employs a one-way language encoding-decoding method and token-by-token
prediction of words.

Prompt engineering serves as an effective paradigm that closes the divides
between pre-training tasks and their corresponding downstream applications in
case of llm [16]. There have been several prompting techniques specifically
explored for information extraction. One such technique is cot prompting. cot
prompting [28] is a technique that involves presenting a coherent series of
intermediate reasoning steps as exemplars in the prompting process. Research
indicates that employing cot prompting enhances performance across various
tasks, including arithmetic, commonsense, and symbolic reasoning.

However, when working with llm’s, one must always be cognizant of erroneous
predictions or what is popularly referred to as "hallucinations". Within the
NLP community, the term "hallucination" while problematic in many ways (since
it gives agency to the AI system rather than its creators), has been
extensively embraced; it commonly denotes the generation of information that
deviates from logical or faithful representation of the given source content.
This consideration is pivotal when navigating the intricacies of llm, as the
potential for generating nonsensical or inaccurate information necessitates a
nuanced approach to model evaluation and refinement. There could be many
reasons attributed to so-called hallucinations, whether it is incompleteness
in the prompting technique, bias in the training data corpus, or under
informativeness of the llm if the question is particularly niche [34].

###  2.3 Hyper-Relational Knowledge Graphs

While kg are a great way to encapsulate simple, factual information, real data
tends to differ with specific nuances, conditionals and additional information
for extracted fact. A hyper-relational kg, such as Wikidata, is a kg that
enable the inclusion of supplementary key:value pairs alongside the primary
triple, offering a means to clarify or limit the applicability of a given
piece of information [8]. Hyper-relational kg can be understood as follows –
consider the sentence, “Barack Obama graduated from Harvard University in
1991.”. In the context of conventional triple-based kg it becomes evident that
the richness of the information in the sentence is not fully encapsulated by
mere triples. However, Figure 2, which includes qualifiers as part of its
hyper-relational kg structure, is easily capable in showcasing the nuances
embedded in the given sentence.

Figure 2: Example of Hyper-Relational Knowledge Graph.

Our paper is providing the novel zero-shot prompt-based approach for hyper-
relational extraction from the HyperRED dataset [3].

##  3 Methodology

This section provides an in-depth description of our prompting techniques in
constructing kg using llm, along with details of method’s implementation.

###  3.1 Prompting Technique

Prompt engineering is the process of designing and creating prompts that
generate desired responses from language models. These prompts can be
classified into various types based on their structure, function, and
complexity. Each prompt comprises a natural language query crafted to elicit a
particular response from the pre-trained language model.

In our prompts to llm, we incorporate an ontology presented in the HyperRED
dataset, coupled with a cot example to illustrate the desired outcome for the
relation and entity extraction task. Additionally, we define an expected
output format that we anticipate the model to follow. More specifically, our
prompts instruct the model to extract relations, entities, and qualifiers from
the at the time. We detail a description of entities, describing it as a
person, country, type of document etc. We also give a lengthy description of
the 62 relations according to HyperRED ontology. For example, we define the
relation “educated at” as “educational institution attended by subject”.
Finally, we describe what a qualifier is, and supply to the model individual
definitions of the 44 qualifiers available in the dataset’s ontology. While
the qualifiers themselves are expected in a key:value format, we supply the
model with a detailed description of the set of qualifier descriptions in
order for there to be better contextual understanding. For example, “end time”
is defined as “time an event ends, an item stops existing, or a statement
becomes invalid”. The key:value format qualifier pair can be seen in Figure 2.
For example, a correct qualifier output would be as follows – “end time:
1991”. The motivation behind this structure was to prevent as much erroneous
prediction of information by the llm as possible. The final prompt used for
our model is available in a public repository 111https://github.com/preetha-
sellforte/CoT-GRex.

Finally, to apply cot prompting, we give the model one example of a context
and clear reasoning as to why we choose to extract the specific entities,
relations and qualifiers from that sentence. We also provide an output format
that we expect the model to adhere to.

###  3.2 OpenAI’s GPT-3.5

We use the pre-trained OpenAI’s GPT-3.5 (GPT-3.5) model and access it via
Azure API calls. OpenAI’s GPT-3 (Generative Pre-trained Transformer) model was
first introduced by OpenAI [2] company in 2020. OpenAI’s GPT series of models
are transformer-based architectures [27], as the name might suggest.

OpenAI’s GPT-3 model underwent fine-tuning through reinforcement learning from
human feedback [4]. This method leverages rewards to enhance the quality of
generation and ensure alignment with human preferences. Several studies have
been conducted researching the mathematical, reasoning, text generative
properties of GPT-3.5 [7], which is why we have decided to use the model due
to its widely regarded state-of-the-art status [32].

##  4 Experiment Setup

This section contains a detailed description of the dataset, baseline model
and evaluation metrics used for our experiment.

###  4.1 Dataset

We utilized a hyper-relational dataset HyperRED spanning diverse domains such
as business, sports, and politics. The dataset creation involved primarily
collecting data from Wikipedia introduction sections, followed by entity
linking and fusion using DBpedia Spotlight. As Wikipedia articles often tend
to use pronouns when referring to a subject or entity, the authors also used
the Stanford CoreNLP tool [17] to resolve coreferences. The dataset’s
annotation by two professional annotators resulted in 39,840 training samples,
1,000 development samples, and 4,000 test samples. HyperRED comprises a set of
62 unique relations and 44 unique qualifiers. The dataset’s authors declared
it as the first-ever dataset with hyper-relational facts.

###  4.2 Baseline Model

Authors of the HyperRED dataset also present CubeRE, a supervised learning
method aimed at extracting hyper-relational information. CubeRE employs a two-
step process for hyper-relational extraction. Firstly, input sentences are
encoded using the BERT [5] language model, generating contextualized
representations for each word. The model then captures interactions between
potential head and tail entities by concatenating word representations and
using a dimension-reducing feed-forward network. This results in a cubic
(3-dimensional) matrix of probabilities for entity and relation labels.
Secondly, to predict qualifiers for hyper-relational facts, interactions are
considered between possible relation triples and value entities. To manage
computational costs, CubeRE introduces a cube-pruning method, focusing on
interactions with the top-k words with the highest entity scores. Entity
scores are obtained from the diagonal of the 3D matrix, and the top-k indices
are ranked for each dimension of the cube representation.

###  4.3 Evaluation Metrics

In addition to classic precision, recall and f1 evaluation metrics used for
relation extraction task, we additionally utilized BERTScore metric. BERTScore
[33] serves as an automated evaluation metric designed for the evaluation of
text generation quality. We use BERTScore in our evaluations since it employs
contextual learnings that can capture the semantic similarities of the
information extracted by the llm. This is useful in our case since we expect
the model to capture more relations and qualifiers.

##  5 Results

Table 1 presents results for exact-match scores between the CubeRE method and
the proposed approach. Initially, it seemed our model performed very poorly
comparing to the baseline according to the exact match metric. However, during
the prompt engineering step, we observed very promising results. To understand
the reason of such unsatisfactory performance, we carried out an analysis of
the generated relations.

Table 1: Table encompassing the precision, recall and F1 scores on exact-match evaluations. Model | Precision | Recall | F1
---|---|---|---
GPT-3.5 | 0.01 | 0.02 | 0.01
CubeRE (Baseline) | 0.62 | 0.66 | 0.64

Further exploration presented in Figure 3 illustrates how the proposed method
extracts entities and relations from source texts. The first hyper-relation is
very close to the gold standard. The second hyper-relation is erroneous, by
providing a mix of person (entity – Conrad IV of Germany, qualifier –
position: King of the Romans) and place (entity – Palermo, relation – capital
of). The third relation is a correct triple describing the capital relation
between city and country. The last relation is an incorrect triple. In
conclusion, we can see the proposed method extracts a correct hyper-relation,
but it also extracts additional relations which could be incorrect. Therefore,
we decided to explore non-exact match evaluation.

Figure 3: Comparing gold standard annotation, with our results. The model
extracts information from the given sentence in high granularity, but often
not in an “exact” format.

Results of the semantic similarities’ evaluation made with BERTScore are
presented in the Table 2. The promising recall score obtained during the
experiment indicates the ability of the proposed approach to capture high
proportion of hyper-relations. The results could be substantially enhanced
with more complex prompting techniques, as well as fine-tuned model settings
specific to the task at hand.Additionally, some filtering and ranking
mechanisms applied to the model’s output could also improve the precision.

We also measured reproducibility of our approach with the use of Levenshtein
Distance in several experimental outputs of the same data samples. The average
reproducibility score during experiments with GPT-3.5 equals 36%, while the
reproducibility score during experiments with OpenAI’s GPT-4 equals 98%. Such
a crucial increase in reproducibility became possible because of the support
of predefined low randomization of OpenAI’s GPT-4 model outputs (temperature =
0).

Table 2: Table encompassing the precision, recall and F1 scores on BERTScore evaluations. Model | Precision | Recall | F1
---|---|---|---
GPT-3.5 | 0.46 | 0.77 | 0.58
CubeRE (Baseline) | 0.93 | 0.88 | 0.91

##  6 Conclusions & Future Work

In this paper, we present a zero-shot prompting technique for hyper-relational
kg construction using llm. We have utilized the HyperRED dataset and compared
our results against the cubeRE method.

Despite not surpassing the baseline, further exploration of sophisticated
prompting techniques should yield better outputs from the model. Additionally,
given the promising recall, attention should be directed toward enhancing
precision for continued improvement. We should also look at fine-tuned
settings in GPT models, as well as look for alternatives in open-source
domains, such as Meta’s Llama 2 [25]. These measures could significantly
improve our results.

In our future work, we will explore different techniques of prompting, as
recently outlined [22]. We also plan to create an approach to filter out
irrelevant information in the given context, to be able to improve the
performance. Finally, we intend to test our approach on a close-domain dataset
that contains detailed instructions and valuable information about migrants’
employment in Finland. This experiment aims to demonstrate the flexibility and
effectiveness of our method across different domains.

## Acknowledgments

This work is supported by the Trust-M research project, a partnership between
Aalto University, University of Helsinki, Tampere University, and the City of
Espoo, funded in-part by a grant from the Strategic Research Council (SRC) in
Finland.

## References

  * [1] Renzo Angles, Marcelo Arenas, Pablo Barceló, Aidan Hogan, Juan Reutter, and Domagoj Vrgoč.  Foundations of modern query languages for graph databases.  ACM CSUR, 50(5):1–40, 2017.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In Proc. of NeurIPS, 33:1877–1901, 2020.
  * [3] Yew Ken Chia, Lidong Bing, Sharifah Mahani Aljunied, Luo Si, and Soujanya Poria.  A dataset for hyper-relational extraction and a cube-filling approach.  In In Proc. of EMNLP, pages 10114–10133, 2022.
  * [4] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  In Proc. of NeurIPS, 30, 2017.
  * [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In In Proc. of NAACL, pages 4171–4186. Association for Computational Linguistics, 2019.
  * [6] Xin Luna Dong, Hannaneh Hajishirzi, Colin Lockard, and Prashant Shiralkar.  Multi-modal information extraction from text, semi-structured, and tabular data on the web.  In In Proc. of ACM SIGKDD, pages 3543–3544, 2020.
  * [7] Jessica López Espejel, El Hassane Ettifouri, Mahaman Sanoussi Yahaya Alassan, El Mehdi Chouham, and Walid Dahhane.  Gpt-3.5, gpt-4, or bard? evaluating llms reasoning ability in zero-shot setting and performance boosting through prompts.  Natural Language Processing Journal, 5:100032, 2023.
  * [8] Mikhail Galkin, Priyansh Trivedi, Gaurav Maheshwari, Ricardo Usbeck, and Jens Lehmann.  Message passing for hyper-relational knowledge graphs.  In In Proc. of EMNLP, pages 7346–7359, 2020.
  * [9] Saiping Guan, Xiaolong Jin, Yuanzhuo Wang, and Xueqi Cheng.  Link prediction on n-ary relational data.  In In Proc. of WWW, pages 583–593, 2019.
  * [10] Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Jie Zhou, and Maosong Sun.  More data, more relations, more context and more openness: A review and outlook for relation extraction.  In In Proc. of AACL-IJCNLP, pages 745–758, 2020.
  * [11] Xu Han, Tianyu Gao, Yuan Yao, Deming Ye, Zhiyuan Liu, and Maosong Sun.  Opennre: An open and extensible toolkit for neural relation extraction.  In In Proc. of EMNLP-IJCNLP, pages 169–174, 2019.
  * [12] Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al.  Knowledge graphs.  ACM CSUR, 54(4):1–37, 2021.
  * [13] Yan Hu, Iqra Ameer, Xu Zuo, Xueqing Peng, Yujia Zhou, Zehan Li, Yiming Li, Jianfu Li, Xiaoqian Jiang, and Hua Xu.  Zero-shot clinical entity recognition using chatgpt.  arXiv preprint arXiv:2303.16416, 2023.
  * [14] Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip.  A survey on knowledge graphs: Representation, acquisition, and applications.  IEEE transactions on neural networks and learning systems, 33(2):494–514, 2021.
  * [15] Zhao Jun, Hu Yuan, Xu Nuo, Gui Tao, Zhang Qi, Chen Yunwen, and Gao Xiang.  An exploration of prompt-based zero-shot relation extraction method.  In In Proc. of CCL, pages 786–797, 2022.
  * [16] Zhigang Kan, Linhui Feng, Zhangyue Yin, Linbo Qiao, Xipeng Qiu, and Dongsheng Li.  A composable generative framework based on prompt learning for various information extraction tasks.  IEEE Transactions on Big Data, 2023.
  * [17] Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky.  The stanford corenlp natural language processing toolkit.  In In Proc. of ACL, pages 55–60, 2014.
  * [18] Ikujiro Nonaka and Hirotaka Takeuchi.  The knowledge creating company: Oxford university press.  New York, 995, 1995.
  * [19] Andrea Rossi, Denilson Barbosa, Donatella Firmani, Antonio Matinata, and Paolo Merialdo.  Knowledge graph embedding for link prediction: A comparative analysis.  ACM TKDD, 15(2):1–49, 2021.
  * [20] Ritesh Sarkhel, Binxuan Huang, Colin Lockard, and Prashant Shiralkar.  Self-training for label-efficient information extraction from semi-structured web-pages.  In Proc. of VLDB, 16(11):3098–3110, 2023.
  * [21] Amit Singhal.  Introducing the knowledge graph: Things, not strings.  https://blog.google/products/search/introducing-knowledge-graph-things-not/, 2012.  [Accessed 15-11-2023].
  * [22] Sonish Sivarajkumar, Mark Kelley, Alyssa Samolyk-Mazzanti, Shyam Visweswaran, and Yanshan Wang.  An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing.  arXiv preprint arXiv:2309.08008, 2023.
  * [23] Alessandro Temperoni, Maria Biryukov, and Martin Theobald.  Enriching relation extraction with openie.  arXiv preprint arXiv:2212.09376, 2022.
  * [24] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal.  Fever: a large-scale dataset for fact extraction and verification.  In In Proc. of NAACL, pages 809–819, 2018.
  * [25] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [26] Van-Hien Tran, Hiroki Ouchi, Taro Watanabe, and Yuji Matsumoto.  Improving discriminative learning for zero-shot relation extraction.  In In Proc. of SpaNLP, pages 1–6, 2022.
  * [27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proc. of NeurIPS, 30, 2017.
  * [28] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  In Proc. of NeurIPS, 35:24824–24837, 2022.
  * [29] Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al.  Zero-shot information extraction via chatting with chatgpt.  arXiv preprint arXiv:2302.10205, 2023.
  * [30] Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang.  A brief overview of chatgpt: The history, status quo and potential future development.  IEEE/CAA Journal of Automatica Sinica, 10(5):1122–1136, 2023.
  * [31] Hongbin Ye, Ningyu Zhang, Hui Chen, and Huajun Chen.  Generative knowledge graph construction: A review.  In In Proc. of EMNLP, pages 1–17, 2022.
  * [32] Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al.  A comprehensive capability analysis of gpt-3 and gpt-3.5 series models.  arXiv preprint arXiv:2303.10420, 2023.
  * [33] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  arXiv preprint arXiv:1904.09675, 2019.
  * [34] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al.  Siren’s song in the ai ocean: a survey on hallucination in large language models.  arXiv preprint arXiv:2309.01219, 2023.


# Whole-Body Control on Non-holonomic Mobile Manipulation for Grapevine Winter
Pruning Automation*

Tao Teng1,3, Miguel Fernandes2, Matteo Gatti3, Stefano Poni3, Claudio
Semini${}^{1},$ Senior Member, IEEE,
Darwin Caldwell2, Senior Member, IEEE, Fei Chen†4, Senior Member, IEEE *This
study was supported by the Doctoral School on the Agro-Food System
(Agrisystem) of Università Cattolica del Sacro Cuore (Italy), and was
supported in part by the project “Grape Vine Perception and Winter Pruning
Automation” funded by joint lab of Istituto Italiano di Tecnologia and
Università Cattolica del Sacro Cuore, and the project “Improving
Reproducibility in Learning Physical Manipulation Skills with Simulators Using
Realistic Variations” funded by EU H2020 ERA-Net Chist-Era program.1Dynamic
Legged Systems (DLS) Lab, Istituto Italiano di Tecnologia, Via S. Quirico 19D,
16163 Genoa, Italy (e-mail: name.surname@iit.it)2Department of Advanced
Robotics, Istituto Italiano di Tecnologia, Via S. Quirico 19D, 16163 Genoa,
Italy (e-mail:name.surname@iit.it)3Department of Sustainable Crop Production,
Università Cattolica del Sacro Cuore, Via Emilia Parmense 84, 29122 Piacenza,
Italy (e-mail: name.surname@unicatt.it)4Department of Mechanical and
Automation Engineering, T-Stone Robotics Institute, The Chinese University of
Hong Kong, Hong Kong (e-mail: f.chen@ieee.org)$\dagger$Corresponding author

###### Abstract

Mobile manipulators that combine mobility and manipulability, are increasingly
being used for various unstructured application scenarios in the field, e.g.
vineyards. Therefore, coordinated motion of the mobile base and manipulator is
an essential feature of the overall performance. In this paper, we explore a
whole-body motion controller of a robot which is composed of a 2-DoFs non-
holonomic wheeled mobile base with a 7-DoFs manipulator (non-holonomic wheeled
mobile manipulator, NWMM) This robotic platform is designed to efficiently
undertake complex grapevine pruning tasks. In the control framework, a task
priority coordinated motion of the NWMM is guaranteed. Lower-priority tasks
are projected into the null space of the top-priority tasks so that higher-
priority tasks are completed without interruption from lower-priority tasks.
The proposed controller was evaluated in a grapevine spur pruning experiment
scenario.

##  I Introduction

Technology is one of the key driving forces in the development of precision
agriculture [1, 2, 3]. Robots are incredibly effective in combating the
pressures of population growth [4]. Agricultural robots are not only able to
help farmers solve labor shortages, but they also help to mitigate
environmental impact [5]. In the drive to increase yields and maximize
resources through the utilization of new technologies, grapevine pruning
automation is a typical application scenario in precision agriculture [6, 7].
The main purpose of grapevine winter pruning is to determine the number and
location of the nodes remaining over the winter. These remaining nodes will
grow into new canes in the next harvest season, and grapes will grow on these
new canes. As a consequence, it fundamentally determines the final yield. Such
tasks can be accomplished with a mobile manipulator extends the manipulability
and operational space by combining the strengths of the mobile base and
manipulator.

Figure 1:  An example of a robot grapevine pruning system Rolling Panda. To
achieve this scenario, the following temporal sequences are followed:
grapevine spur detection, whole-body motion to approach pruning point, and
cutting the spur.

This combined, more complex structure creates motion planning and control
issues due to the high redundancy and versatility of mobile platform [8].
Previously, the mobile base and manipulator could be considered to be two
separate subsystems [9, 10, 11, 12]. Wrock et al. created an automatic
switching scheme that used teleoperation methods to achieve decoupled mobile
manipulator motion. Decoupled motion refers to motion in which the mobile base
remains fixed while the manipulator is operated, or vice versa. This provides
high tracking accuracy at the end effector, but it takes more time to complete
the task because tracking must be interrupted during motion. Coordinated
movement of the mobile base and manipulator [13, 14] is required to fully
utilize the strengths of mobile base and manipulator and boost efficiency.

De Luca et al. [15] proposed a comprehensive theory to deal with modeling and
redundancy resolution for non-holonomic mobile manipulators. A multi-tasks
whole-body regulating strategy based on velocity control was proposed by Li et
al [16] for a highly redundant mobile manipulator. The mobile manipulator
follows the predefined end-effector trajectory while avoiding low-priority
control primitives by using null-space projection. To obtain the inverse
kinematics, Roberto et al. [17] provided a systematic mobile manipulator
solution that included a selection of redundancy parameters. The solution was
capable of managing obstacle avoidance, mobile base motion restriction, and
dexterity enhancement.

In this paper, a whole-body motion controller enhanced by hierarchical tasks
that can regulate a non-holonomic mobile manipulator for grapevine winter
pruning is proposed. The trajectory of the end effector, which is treated as
the top-priority task, is created by using quintic polynomial programming.
Conflicts between the end-effector tasks and the constraint tasks are handled
within the stack-of-tasks (SoTs) framework[18, 19, 20] by correctly assigning
an order of priorities to the given tasks and then ensuring that the lower
priority tasks are projected into the null space of the higher-priority tasks.

The manuscript is organized in the following way: Firstly, Sec. II introduces
the overall non-holonomic wheeled mobile manipulator system platform Rolling
Panda. Sec. III presents the kinematics and dynamics models of this NWMM
system. Whole-body controller for the non-holonomic mobile manipulator is
designed in Sec. IV. Subsequently, Sec. V. describes the experimental
validations of the proposed control scheme. Finally, the paper is concluded in
Sec. VI.

##  II Overview of Non-holonomic Mobile Manipulator System

The designed grapevine winter pruning system is called Rolling Panda. It
primarily consists of a non-holonomic wheeled mobile base, a manipulator, an
RGB-D (Intel Realsense D435i) eye-in-hand camera, and pruning clippers.

###  II-A Hardware of the Mobile Manipulator System

As shown in Fig. 1, Rolling Panda consists of two parts: the velocity
controlled two-wheel non-holonomic mobile robot, MP-500 (Neobotix GmbH. Co.)
and 7-DoFs robot arm manipulator, Panda (Franka Emika. Co.). Both the mobile
base and manipulator have their own controller interfaces which are simple and
user-friendly programming interfaces. On this mobile base, there is a ROS
interface for low-level, real-time velocity controller and localization
algorithms using wheel 111 https://robots.ros.org/neobotix-mp-500/ . The
localization algorithm returns the direction and position and velocity of the
mobile base’s central frame in relation to its global frame. The Franka ROS
Interface provides utilities for controlling and managing the Franka Emika
Panda 222 https://frankaemika.github.io/docs/ . The control frequencies of the
manipulator and the base are 1 kHz and 50 Hz, respectively. The ROS master
laptop, used for the controller, is a core-i7 processor 1.8 GHz with 16 GB
RAM. The RGB-D camera (Intel Realse D435i) is mounted on the end effector of
the Rolling Panda.

##  III Non-holonomic Mobile Manipulator Modeling

In this section, we explored a system kinematic and dynamic modeling for the
non-holonomic mobile manipulator Polling Panda.

###  III-A Kinematic Modeling and Kinematic Constraints

Considering the non-holonomic mobile manipulator with $2$ wheels and $7$-DoFs
manipulator. The moving base of the Rolling Panda contains 3-DoFs of rigid
body motion, hence a generalized coordinate system can be defined as
following:

Figure 2: Schematic drawing of Rolling Panda. $x_{m}$, $y_{m}$, and $\phi$
denote the center position and heading angle of the mobile platform with
respect to the global coordinate.

$q=\left[\begin{array}[]{lll}q_{m}^{T}&q_{w}^{T}&q_{n}^{T}\end{array}\right]^{T}$
(1)

where
$q_{m}=\left[\begin{array}[]{lll}x_{m}&y_{m}&\phi\end{array}\right]^{T}\in\mathbb{R}^{3}$
is the coordinate of the rotation central frame of the mobile base,
$q_{w}=\left[\begin{array}[]{ll}\theta_{l}&\theta_{r}\end{array}\right]^{T}\in\mathbb{R}^{2}$
is spinning of the wheel joints and $q_{n}\in\mathbb{R}^{n}$ is the joint
vector for the manipulator. $\mu$ is the distance between the driving wheels
and the mobile platform geometric center , $\rho$ is the distance from the
mobile platform rotation center to the center of mass of mobile platform, and
$R$ is the radius of the wheels.

Due to its inherent properties a differential mobile platform can not move
sideways [21]. Hence, the velocity of the mobile base in the lateral
directions should be zero.

$-\dot{x}_{m}\sin\phi+\dot{y}_{m}\cos\phi-\rho\dot{\phi}=0$ (2)

The other two constraints are a pure rolling constraint, relating the base
velocities $\dot{x}_{m},\dot{y}_{m},\dot{\phi}$ while the wheel velocities
$\dot{\theta}_{l},\dot{\theta}_{r},$ ensure the no-slip condition at each
rolling wheel in the forward directions.

$\displaystyle\dot{x}_{m}\cos\phi+\dot{y}_{m}\sin\phi-\mu\dot{\phi}$
$\displaystyle=R\dot{\theta}_{l}$ (3)
$\displaystyle\dot{x}_{m}\cos\phi+\dot{y}_{m}\sin\phi+\mu\dot{\phi}$
$\displaystyle=R\dot{\theta}_{r}$

We can set the constraint matrix between rigid body motion of the mobile base
and the generalized coordinates to satisfy the following equation.

$A(q)\dot{q}=0$ (4)

where $A(q)\in\mathbb{R}^{3\times\left(5+n\right)}$ is the full-ranked
constraint matrix.

$A=\left[\begin{array}[]{ccccccc}-\sin\phi&\cos\phi&-\rho&0&0&\cdots&0\\\
-\cos\phi&-\sin\phi&-\mu&R&0&\cdots&0\\\
-\cos\phi&-\sin\phi&\mu&0&R&\cdots&0\end{array}\right]$ (5)

Using the null-space of $A(q),$ we can obtain the following transform
equation,

$\dot{q}=S(q)\dot{\xi}$ (6)

where $S(q)$ satisfies $A(q)S(q)=0$, and

$\begin{split}\begin{array}[]{c}S={\left[\begin{array}[]{ccccc}c(\mu\cos\phi-\rho\sin\phi)&c(\mu\cos\phi+\rho\sin\phi)&0&\cdots&0\\\
c(\mu\sin\phi+\rho\cos\phi)&c(\mu\sin\phi-\rho\cos\phi)&0&\cdots&0\\\
c&-c&0&\cdots&0\\\ 1&0&0&\cdots&0\\\ 0&1&0&\cdots&0\\\ 0&0&1&\cdots&0\\\
\vdots&\vdots&\vdots&\ddots&\vdots\\\
0&0&0&\cdots&1\end{array}\right]}\end{array}\end{split}$ (7)

where $c=R/\left(2\mu\right)$. The set of feasible velocities may be expressed
in terms of a suitable vector,
$\dot{\xi}=\left[\begin{array}[]{cc}\dot{q}_{w}^{T}&\dot{q}_{n}^{T}\end{array}\right]^{T}\in\mathbb{R}^{2+n}$
is the joint velocity for actuators of the robot.

Jacobian matrix [22] between Cartesian velocity space and actual joint
velocity space, $J_{\xi}\in\mathbb{R}^{6\times(2+n)},$ can be derived as

$J_{\xi}(q)=J_{q}(q)S(q)$ (8)

where $J_{q}(q)$ is the Jacobian matrix for $q$.

Figure 3: Whole-body kinematic model of Rolling Panda

###  III-B Dynamic model

The following is the unconstrained equation of motion for a non-holonomic
handheld manipulator [13]:

$M(q)\ddot{q}+V(q,\dot{q})\dot{q}+G(q)=B(q)\tau+\tau_{dis}+\Lambda^{T}(q){\lambda}$
(9)

where $M(q)$ is an $n\times n$ symmetric positive definite inertia matrix,
$V(q,\dot{q})$ is the centripetal and coriolis matrix, $G(q)$ is the
gravitational vector, $\tau_{dis}$ is the vector of bounded unknown
disturbances including unstructured unmodeled dynamics, $B(q)$ is the input
matrix, $\tau$ is the input torque vector, $\Lambda^{T}(q)$ is the matrix
associated with the kinematic constraints, and $\lambda$ is the Lagrange
multipliers vector.

Finally, the dynamic motion of no-holonomic mobile manipulator with respect to
$\xi$ and $\dot{\xi}$ can be reformulated by removing the generalized
constrains, $\Lambda(q)^{T}\lambda,$ in (9) by using (6) and combining (9) and
time derivative of (6), as follow:

${M}_{\xi}(q)\dot{\xi}+{V}_{\xi}(q,\dot{q})\xi+{G}_{\xi}(q)=u+S(q)^{T}\tau_{dis}$
(10)

where

$\begin{array}[]{l}{M}_{\xi}(q)=S(q)^{T}M(q)S(q)\\\
{V}_{\xi}(q,\dot{q})=S(q)^{T}[M(q)\dot{S}(q)+C(q,\dot{q})S(q)]\\\
{G}_{\xi}=S(q)^{T}G(q)\\\ u=S(q)^{T}B(q)\tau.\end{array}$

##  IV PROPOSED CONTROLLER DESIGN

To automate the grapevine winter pruning, a hierarchical control formulation
based on multi-tasks scheduling has been designed.

###  IV-A Null-Space Dynamic Control Strategy

Robots with redundancy (particularly high redundancy) can deal very
effectively with constrained tasks in the Cartesian space. The redundant self
motion can can simultaneously satisfy both, the higher-priority end-effector
task and the additional low-priority constrained tasks.

The task-space augmentation principle incorporates a constraint task that must
be performed simultaneously with the end effector task. In this case, an
augmented Jacobian matrix is constructed, the inverse of which yields the
required joint velocity solution [23].

The relation between the $i$-th configuration coordinate vector $\xi_{i}$ and
the $i$-th Cartesian space task vector ${x}_{\mathrm{i}}$ can be considered as
a direct kinematics equation:

$\dot{{x}}_{\mathrm{c}}={J}_{\xi}\dot{\xi}$ (11)

The well-known generalized Moore-Penrose pseudo inverse $J^{+}(q)$ is used
since the inverse of the nonsquare (analytical) Jacobian $J_{\xi}$ does not
exist in the redundant case. This proposed strategy often employs a special
solution of equation (2).

Optimization criteria for the redundant self motion can be supplemented by,
for example, null-space projection, which leads to the relation:

$\dot{\xi}={J}_{\xi}^{+}\dot{{x}}_{\xi}+\left({I}-{J}_{\xi}^{+}{J}_{\xi}\right)\dot{\xi}_{0}$
(12)

The expression $({I}-{J}_{\xi}^{+}{J}_{\xi})$ represents the orthogonal
projection matrix in the null space of $J_{\xi}$, and $\dot{\xi}_{0}$ is an
arbitrary joint-space velocity satisfying argumented constraint tasks; hence,
the second part of the solution is therefore a null-space velocity.

##  V GRAPEVINE WINTER PRUNING EXPERIMENTS

Grapevine winter pruning is a key agricultural activity involving cutting of
the canes of each vine that grew during the previous growth cycle. The pruning
leaves a certain number of nodes that are able to guide the ideal direction of
future growth of the vine towards a desired vine balance [24, 25]. The aim of
this work is that pruning which currently requires experienced and skilled
technicians, can eventually be conducted by intelligent and autonomous robotic
systems [7, 26].

The proposed hierarchical control framework was validated through grapevine
winter pruning experiments. The subsections below describe the details of our
system specification and experimental results with the Rolling Panda.

Figure 4: Overall paradigm of grapevine pruning experiment

###  V-A Grapevine Pruning Task

The definition and determination of the “correct” grapevine pruning point is
based on grapevine modeling and physiological response [27]. Fig. 5 (a) gives
a intuitive and graphical illustration of the potential optimal pruning point.

Figure 5: Effective pruning point illustration: (a) diagrammatic
representation of pruning points from an agricultural and physiological
perspective, (b) potential pruning points generated by deep learning

In this paper, we use the visual perception methodology mentioned in [28] to
determine potential pruning points, first creating a representative model of a
grapevine plant using object segmentation on grapevine images and , second,
generating a set of potential pruning points.

Figure 6: Experimental results for grapevine pruning point approach: snapshots

After the perception system finds the position of the pruning point in the
global frame as showed in Fig. 5 (b). The whole-body controller generates a
trajectory to approach the target pruning point using quintic polynomial
interpolation programming [29]. The planned trajectory is defined as follows:

$\begin{array}[]{l}P_{x}(t)=c_{0}+c_{1}t+c_{2}t^{2}+c_{3}t^{3}+c_{4}t^{4}+c_{5}t^{5}\\\
P_{v}(t)=c_{1}t+2c_{2}t+3c_{3}t^{2}+4c_{4}t^{3}+5c_{5}t^{4}\\\
P_{a}(t)=2c_{2}+6c_{3}t+12c_{4}t^{2}+20c_{5}t^{3}\end{array}$ (13)

where $P_{x}$, $P_{v}$ and $P_{a}$ respectively correspond to the position,
velocity, and acceleration in Cartesian space. Therefore, Eq. 13 can be
rewritten as:

$\left[\begin{array}[]{cccccc}1&t_{s}&t_{s}^{2}&t_{s}^{3}&t_{s}^{4}&t_{s}^{5}\\\
1&t_{e}&t_{s}^{2}&t_{s}^{3}&t_{s}^{4}&t_{s}^{5}\\\
0&1&2ts&3t_{s}^{2}&4t_{s}^{3}&5t_{s}^{4}\\\
0&1&2t_{e}&3t_{e}^{2}&4t_{e}^{3}&5t_{e}^{4}\\\
0&0&2&6t_{s}&12t_{s}^{2}&20t_{s}^{3}\\\
0&0&2&6t_{e}&12t_{e}^{2}&20t_{e}^{3}\end{array}\right]\left[\begin{array}[]{l}c_{0}\\\
c_{1}\\\ c_{2}\\\ c_{3}\\\ c_{4}\\\
c_{5}\end{array}\right]=\left[\begin{array}[]{l}x_{s}\\\ x_{e}\\\ v_{s}\\\
v_{e}\\\ a_{s}\\\ a_{e}\end{array}\right]$ (14)

where $x_{s}$, $v_{s}$, and $a_{s}$ correspond to the position, velocity, and
acceleration of the interpolation initial point respectively, and $x_{e}$,
$v_{e}$ and $a_{e}$ correspond to the position, velocity, and acceleration of
the interpolation end point respectively.

Constraint task 1: Singularities are configurations in which a robot loses
control in one or more directions, and they should be avoided when planning
and controlling robot motion. The manipulability calculation has been widely
used and proven to be an effective way to keep robots away from singular
configurations among several methods. Since singularities have no effect on
the location of the mobile base, only upper manipulator configurations will be
considered in this hierarchical control formulation. The term ”manipulation”
refers to the ability to manipulate something. The manipulability measurement
is defined as

$\omega\left(\xi_{\mathrm{c1}}\right)=\sqrt{\operatorname{det}\left({J}_{\xi}\left(\xi_{\mathrm{c1}}\right){J}_{\xi}^{\mathrm{T}}\left(\xi_{\mathrm{c1}}\right)\right)}$
(15)

When the manipulability measurement is increased, the manipulator will move
away from singularities. The following equation can be used to determine the
corresponding joint inputs:

$\dot{\xi}_{\mathrm{c1}}=k_{0}\left(\frac{\partial\omega\left(\xi_{\mathrm{c1}}\right)}{\partial\xi_{\mathrm{c1}}}\right)^{\mathrm{T}}$
(16)

where $k_{0}$ is a positive gain.

Constraint task 2: Joint limits are physical constraints on robots that must
be carefully considered in order to avoid damaging the robotic system.
Certainly, only the upper manipulator joints have physical limitations. All
joint angles are restricted to a range of $-180^{\circ}$ to $180^{\circ}$. To
avoid joint limits, the artificial potential field technique [30] is used to
computes the distance between the $i$th joint and its limits.

$d_{i}=\min\left(\left\|\gamma_{i}-\gamma_{li}\right\|,\left\|\gamma_{i}-\gamma_{\mathrm{ui}}\right\|\right),$
(17)

where $\gamma_{i}$ denotes joint angle of the $i$-th joint, $\gamma_{11}$ and
$\gamma_{\text{ul }}$ denote the lower and upper joint limits of the $i$-th
joint, respectively.

The $i$-th joint’s ”repulsive velocity” is defined as follows:

$\dot{\xi}_{i,c2}=\left\\{\begin{array}[]{ll}k_{i}d_{i}^{2},&d_{i}\leqslant\gamma_{\text{start
}}\\\ 0,&d_{i}>\gamma_{\text{start }}\end{array}\right.$ (18)

where $k_{i}$ denotes a positive gain, and $\gamma_{\text{start }}$ denotes
the minimum distance to be free of repulsive force.

So far, we have modeled the main task (Rolling panda’s end effector approach
grapevine pruning point ) and low-level tasks (constraint task 1 and
constraint task 2) at the velocity level. The application of mull-space
projection technology is able to project constrained tasks to the null space
of main task, so that the robot can perform these tasks simultaneously and
ensure the priority of the tasks.

###  V-B Results and Discussion

The proposed controller for the non-holonomoic mobile manipulator can deal
with grapevine pruning tasks while satisfying singularity avoidance and joint
limitation avoidance, as shown in Fig. 6. The main task is to move the end-
effector along the direction of $x$-axis to approach the pruning point and
then rotate around $pitch$-axis to match the orientation of the pruning point.
When the target pruning point is generated by the visual perception system,
the whole-body controller can control the overall coordinated movement of the
robot to reach the pruning point smoothly.

##  VI Conclusions

In this paper, which addresses the problem of grapevine winter pruning, we
present a task-priority coordinated whole-body motion controller for a non-
holonomic mobile manipulator. The controller can plan and schedule whole-body
coordinated motion to complete the grapevine pruning task. The top priority
task can be executed by employing all capabilities (manipulation and
locomotion) of the robotic system. The second (lower) priority task is then
projected into the null space of the top priority task, hence they have no
impact on its execution. We conducted grapevine pruning experiment to
demonstrate the performance of the proposed framework using a two-wheeled
mobile base with a 7-DoF robot manipulator. Our future work will involve the
extension of the proposed framework for whole-body motion and perception
coupling.

## ACKNOWLEDGMENT

We thank Sunny Katyara (Istituto Italiano di Tecnologia), Carlo Rizzardo
(Istituto Italiano di Tecnologia) and Antonello Scaldaferri (Istituto Italiano
di Tecnologia) for their assistance in experiment setup.

## References

  * [1] A. McBratney, B. Whelan, T. Ancev, and J. Bouma, “Future directions of precision agriculture,” _Precision agriculture_ , vol. 6, no. 1, pp. 7–23, 2005.
  * [2] J. V. Stafford, “Implementing precision agriculture in the 21st century,” _Journal of agricultural engineering research_ , vol. 76, no. 3, pp. 267–275, 2000.
  * [3] N. Zhang, M. Wang, and N. Wang, “Precision agriculture¡ªa worldwide overview,” _Computers and electronics in agriculture_ , vol. 36, no. 2-3, pp. 113–132, 2002.
  * [4] S. G. Vougioukas, “Agricultural robotics,” _Annual Review of Control, Robotics, and Autonomous Systems_ , vol. 2, pp. 365–392, 2019.
  * [5] A. Kumar, R. S. Deepak, and D. D. Sagar Kusuma, “Review on multipurpose agriculture robot,” _International Journal for Research in Applied Science and Engineering Technology_ , vol. 8, pp. 1314–1318, 2020.
  * [6] J. Fourie, C. Bateman, J. Hsiao, K. Pahalawatta, O. Batchelor, P. E. Misse, and A. Werner, “Towards automated grape vine pruning: Learning by example using recurrent graph neural networks,” _International Journal of Intelligent Systems_ , vol. 36, no. 2, pp. 715–735, 2021.
  * [7] T. Botterill, S. Paulin, R. Green, S. Williams, J. Lin, V. Saxton, S. Mills, X. Chen, and S. Corbett-Davies, “A robot system for pruning grape vines,” _Journal of Field Robotics_ , vol. 34, no. 6, pp. 1100–1122, 2017.
  * [8] R. Raja, A. Dutta, and B. Dasgupta, “Learning framework for inverse kinematics of a highly redundant mobile manipulator,” _Robotics and Autonomous Systems_ , vol. 120, p. 103245, 2019.
  * [9] T. J. Tsay, Y.-F. Lai, and Y.-L. Hsiao, “Material handling of a mobile manipulator using an eye-in-hand vision system,” in _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2010, pp. 4743–4748.
  * [10] F. Chen, M. Selvaggio, and D. G. Caldwell, “Dexterous grasping by manipulability selection for mobile manipulator with visual guidance,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 2, pp. 1202–1210, 2018.
  * [11] S. Katyara, F. Ficuciello, D. G. Caldwell, F. Chen, and B. Siciliano, “Reproducible pruning system on dynamic natural plants for field agricultural robots,” in _Human-Friendly Robotics 2020_. Cham: Springer International Publishing, 2021, pp. 1–15.
  * [12] M. Wrock and S. B. Nokleby, “Decoupled teleoperation of a holonomic mobile-manipulator system using automatic switching,” in _2011 24th Canadian Conference on Electrical and Computer Engineering (CCECE)_. IEEE, 2011, pp. 001 164–001 168.
  * [13] S. Kim, K. Jang, S. Park, Y. Lee, S. Y. Lee, and J. Park, “Whole-body control of non-holonomic mobile manipulator based on hierarchical quadratic programming and continuous task transition,” in _2019 IEEE 4th International Conference on Advanced Robotics and Mechatronics (ICARM)_. IEEE, 2019, pp. 414–419.
  * [14] B. Hamner, S. Koterba, J. Shi, R. Simmons, and S. Singh, “An autonomous mobile manipulator for assembly tasks,” _Autonomous Robots_ , vol. 28, no. 1, pp. 131–149, 2010.
  * [15] A. De Luca, G. Oriolo, and P. R. Giordano, “Kinematic modeling and redundancy resolution for nonholonomic mobile manipulators,” in _Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006._ IEEE, 2006, pp. 1867–1873.
  * [16] M. Li, Z. Yang, F. Zha, X. Wang, P. Wang, P. Li, Q. Ren, and F. Chen, “Design and analysis of a whole-body controller for a velocity controlled robot mobile manipulator,” _SCIENCE CHINA Information Sciences_ , vol. 63, no. 7, p. 170204, 2020.
  * [17] R. Ancona, “Redundancy modelling and resolution for robotic mobile manipulators: a general approach,” _Advanced Robotics_ , vol. 31, no. 13, pp. 706–715, 2017.
  * [18] S. Katyara, F. Ficuciello, T. Teng, F. Chen, D. G. Caldwell, and B. Siciliano, “Formulating intuitive stack-of-tasks with visuo-tactile perception for collaborative human-robot fine manipulation,” _arXiv preprint arXiv:2103.05676_ , 2021.
  * [19] A. Rocchi, E. M. Hoffman, D. G. Caldwell, and N. G. Tsagarakis, “Opensot: a whole-body control library for the compliant humanoid robot coman,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 6248–6253.
  * [20] S. Katyara, F. Ficuciello, T. Teng, F. Chen, B. Siciliano, and D. G. Caldwell, “Intuitive tasks planning using visuo-tactile perception for human robot cooperation,” _arXiv preprint arXiv:2104.00342_ , 2021.
  * [21] Y. Liu, Z. Li, H. Su, and C.-y. Su, “Whole body control of an autonomous mobile manipulator using series elastic actuators,” _IEEE/ASME Transactions on Mechatronics_ , 2021.
  * [22] J. Zhao, K. Liu, F. Zhao, and Z. Sun, “Design and kinematic analysis on a novel serial-parallel hybrid leg for quadruped robot,” in _International Conference on Intelligent Robotics and Applications_. Springer, 2019, pp. 436–447.
  * [23] S. B. Slotine and B. Siciliano, “A general framework for managing multiple tasks in highly redundant robotic systems,” in _proceeding of 5th International Conference on Advanced Robotics_ , vol. 2, 1991, pp. 1211–1216.
  * [24] B. G. Coombe and P. Dry, “Viticulture. volume 2. practices,” _Winetitles, Adelaide_ , 1992.
  * [25] S. Poni, S. Tombesi, A. Palliotti, V. Ughini, and M. Gatti, “Mechanical winter pruning of grapevine: physiological bases and applications,” _Scientia Horticulturae_ , vol. 204, pp. 88–98, 2016.
  * [26] P. Guadagna, T. Frioni, F. Chen, T. Incerti Delmonte, Andreaand Teng, M. I. Ferreira Fernandes, A. Scaldaferri, C. Semini, S. Poni, and M. Gatti, “Designing an algorithm for pruning grapevine based on 3d image processing,” _15th European Conference on Precision Agriculture, ECPA 2021_ , in press, 2021.
  * [27] D. S. Pérez, F. Bromberg, and C. A. Diaz, “Image classification for detection of winter grapevine buds in natural conditions using scale-invariant features transform, bag of features and support vector machines,” _Computers and electronics in agriculture_ , vol. 135, pp. 81–95, 2017.
  * [28] S. M. Hosseini and A. Jafari, “Designing an algorithm for pruning grapevine based on 3d image processing,” _Iranian Journal of Biosystems Engineering_ , vol. 48, no. 3, pp. 289–297, 2017.
  * [29] Y. Guan, K. Yokoi, O. Stasse, and A. Kheddar, “On robotic trajectory planning using polynomial interpolations,” in _2005 IEEE International Conference on Robotics and Biomimetics-ROBIO_. IEEE, 2005, pp. 111–116.
  * [30] O. Khatib, “Real-time obstacle avoidance for manipulators and mobile robots,” in _Proceedings. 1985 IEEE International Conference on Robotics and Automation_ , vol. 2. IEEE, 1985, pp. 500–505.