# Strain effect on Goos–Hänchen shifts and group delay time in gapped graphene
barrier

Miloud Mekkaoui  Laboratory of Theoretical Physics, Faculty of Sciences,
Chouaïb Doukkali University, PO Box 20, 24000 El Jadida, Morocco  Youssef
Fattasse  Laboratory of Theoretical Physics, Faculty of Sciences, Chouaïb
Doukkali University, PO Box 20, 24000 El Jadida, Morocco  Ahmed Jellal
<EMAIL_ADDRESS>Laboratory of Theoretical Physics, Faculty of Sciences,
Chouaïb Doukkali University, PO Box 20, 24000 El Jadida, Morocco  Canadian
Quantum Research Center, 204-3002 32 Ave Vernon, BC V1T 2L7, Canada

###### Abstract

We investigate the strain effect on the Goos–Hänchen (GH) shifts and group
delay time for transmitted Dirac fermions in gapped graphene through a single
barrier potential. The solutions of energy spectrum are used to compute the
transmission probabilities together with the GH shifts and group delay time.
Our results show that the two last quantities are strongly depending to
weather the strain is applied along armchair or zigzag directions. In
particular it found that both of quantities can be enhanced with the applied
strain.

###### pacs:

##  I Introduction

Since its successful isolation in 2004 [1, 2], graphene has attracted a
considerable attention from both experimental and theoretical investigations.
This is because of its unique and outstanding mechanical, electronic, optical,
and thermal properties [3]. On the other hand, there is a big progress in
studying quantum phenomena in graphene systems among them we cite the quantum
version of the Goos-Hänchen (GH) effect originating from the reflection of
particles from interfaces [4]. Many works in various graphene-based
nanostructures, including single [5], double barrier [6], and superlattices
[7], showed that the GH shifts can be enhanced by the transmission resonances
and controlled by varying the electrostatic potential and induced gap [5].
Another crucial physical quantity is the group delay time, which remains among
the important quantities related to the dynamic aspect of the tunneling
process [8, 9]. This in fact is often referred to as the Hartman effect, which
implies that for sufficiently large barriers the effective group velocity of
the particle can become superluminal [10, 9].

Moreover, the electronic properties of graphene based nanostructures can be
adjusted by distorting a deformation on the graphene sample [11, 12, 13, 14].
Indeed, since its discovery researchers have conducted extensive research on
the in influence of elastic strain on mechanical and physical properties of
graphene [15, 16]. It is showed that graphene has an effective young’s modulus
and simultaneously can reversibly support elastic strain up to $25\%$ [17].
Also it is found that the strain applied to graphene allows for producing an
energy gap and changes the Dirac points, which resulted in having asymmetrical
effective Fermi velocities $(v_{x}^{\eta},v_{y}^{\eta})$ for fermions [18, 19,
20]. Here $\eta=A,Z$ refers to applied strain along armchair direction or
zigzag one, respectively.

We address the question of how can strain affect the GH shifts and group delay
time in graphene under constraints. Then let us consider a gapped graphene
barrier and apply in the intermediate region a strain along armchair and
zigzag directions. Solving Dirac equation, we establish the solutions of
energy spectrum for three regions. From the continuity conditions, we
determine two transmission probabilities referred to armchair and zigzag
directions. These are used to compute the corresponding GH shifts and group
delay time. As results, we show that the strain causes some changes on the GH
shifts and group delay time in transmission along the armchair direction, but
it produces remarkable influence along the zigzag direction. Consequently, we
conclude that both of these quantities can be controlled by adjusting the
strength of strain along each direction.

The present paper is organized as follows. In section II, we formulate our
theoretical problem by writing the corresponding Hamiltonian and determine the
eigenspinors and eigenvalues. In section III, we compute the transmission
probabilities from which we derive the phase shifts. These are used to obtain
the GH shifts and group delay time. We numerically discuss our results by
showing different illustrations under suitable choices of the physical
parameters, in section IV. Finally, we conclude our results.

##  II Theoretical model

We consider a system made of graphene having three regions labeled by
$j=1,2,3$ where the intermediate one is subject to a mass term, scalar
potential and applied strain, as geometrically presented in FIG. 1. The mass
term $\Delta$ can be induced by breaking the sublattice symmetry through
potentials or spin rotational symmetry via intrinsic spin orbit coupling [21,
22, 23].

Figure 1: (color online) Schematically profile of three regions of graphene.
The central region is a gapped graphene subjected to a scalar potential $V(x)$
together with a strain strength $S$ applied armchair ($y$-axis) and zigzag
($x$-axis) directions.

In the framework of the tight-binding approximation, the Hamiltonian governing
the motion of the electron in our system can be written as

$\displaystyle
H=v_{x}^{\eta}\sigma_{x}p_{x}+v_{y}^{\eta}\sigma_{y}p_{y}+\left({V}\mathbb{I}_{2}+{\Delta}\sigma_{z}\right)\Theta\left(xd-x^{2}\right)$
(1)

where $(\sigma_{x},\sigma_{y})$ are the usual Pauli matrices,
${\mathbb{I}}_{2}$ the $2\times 2$ unit matrix and $\Theta$ is the Heaviside
step function. The tensional strain affects the Fermi velocity components to
be differently as $v_{x}^{\eta}$ and $v_{y}^{\eta}$ [24, 25]. According to the
geometry of our system, we distinguish between applied strain along armchair
(A) and zigzag (Z) directions. Consequently, we have the Fermi velocities

$\displaystyle v_{x}^{A}=\frac{\sqrt{3}}{2\hbar}a(1-\sigma S)t,\quad
v_{y}^{A}=\frac{3}{2\hbar}a(1+S)t^{\prime}_{3}$ (2) $\displaystyle
v_{x}^{Z}=\frac{\sqrt{3}}{2\hbar}a(1+S)t,\quad
v_{y}^{Z}=\dfrac{3}{2\hbar}a(1-\sigma S)t^{\prime}_{3}$ (3)

with $t=\sqrt{4{t^{\prime}_{1}}^{2}-{t^{\prime}_{3}}^{2}}$, $a=0.142$ nm is
the distance of the nearest neighbors without any deformation, $\sigma=0.165$
is the Poisson ratio, and $S$ is the strain strength. In the tight binding
approximation, the only effect of strain is to modify the altered hopping
integral parameter $t^{\prime}_{i}$ given by a empirical relation

$\displaystyle
t^{\prime}_{i}=t_{0}e^{-3.37\frac{\delta^{\prime}_{i}}{a-1}},\qquad i=1,2,3$
(4)

resulted from stretching or shrinking of the distance vectors between the
nearest neighbor carbon atoms [26] and $t_{0}\approx 2.7$ eV [27] is the
transfer energy without deformation. As a consequence, the strain changes the
distance of nearest neighbors as depicted in FIG. 1 with solid and dashed
circles denote sublattices A and B in undeformed and deformed configurations.
As a result the three nearest neighbor vectors $\delta_{i}$ change to the new
ones $\delta^{\prime}_{i}$, such as

$\displaystyle\left|\delta^{\prime A}_{1}\right|=a\left(1-\dfrac{3}{4}\sigma
S+\dfrac{1}{4}\ S\right),\quad\left|\delta^{\prime
A}_{3}\right|=a\left(1+S\right)$ (5) $\displaystyle\left|\delta^{\prime
Z}_{1}\right|=a\left(1+\dfrac{3}{4}\sigma S-\dfrac{1}{4}\
S\right),\quad\left|\delta^{\prime Z}_{3}\right|=a\left(1-\sigma S\right)$ (6)

with $\left|\delta^{\prime A}_{1}\right|=\left|\delta^{\prime A}_{2}\right|$
and $\left|\delta^{\prime Z}_{1}\right|=\left|\delta^{\prime Z}_{2}\right|$.

To solve the eigenvalue problem, we proceed by separating variables and then
write the eigenspinors as
$\psi_{2}(x,y)=e^{ik_{y}y}\left(\varphi_{2}^{+},\varphi_{2}^{-}\right)^{T}$,
with $k_{y}$ being a real parameter that stands for the wave number of the
excitations along the $y$-axis. Consequently, the resulting reduced time
independent Dirac equation is given by

$\displaystyle\begin{pmatrix}V+\Delta-E&-i\hbar(v_{x}^{\eta}\frac{\partial}{\partial
x}+v_{y}^{\eta}k_{y})\\\ -i\hbar(v_{x}^{\eta}\frac{\partial}{\partial
x}-v_{y}^{\eta}k_{y})&V-\Delta-E\\\
\end{pmatrix}\begin{pmatrix}\varphi_{2}^{+}\\\ \varphi_{2}^{-}\\\
\end{pmatrix}=0$ (7)

and here the conservation of the momentum $p_{y}$ has been taken into account
due to the vanishing commutator $[p_{y},H]$. As a result, in region 2
($0<x<d$) we get the eigenvalues

$E=V+s{{}^{\prime}}\sqrt{(v_{x}^{\eta}\hbar
k_{x}^{\eta})^{2}+(v_{y}^{\eta}\hbar k_{y})^{2}+\Delta^{2}}$ (8)

with the sign $s{{}^{\prime}}=\mbox{sgn}(E-V)$ refers to conduction and
valence bands of region. The associated eigenspinors are found to be

$\displaystyle\psi_{2}=\left[a_{0}\begin{pmatrix}\alpha^{\eta}_{+}\\\
\alpha^{\eta}_{-}z^{\eta}\end{pmatrix}e^{ik_{x}^{\eta}x}+b_{0}\begin{pmatrix}\alpha^{\eta}_{+}\\\
-\frac{\alpha^{\eta}_{-}}{z^{\eta}}\end{pmatrix}e^{-ik_{x}^{\eta}x}\right]e^{ik_{y}y}$
(9)

where we have set the parameters $\alpha^{\eta}_{\pm}$, $k^{\eta}_{F}$ and the
complex number $z^{\eta}$

$\displaystyle\alpha^{\eta}_{\pm}=\left[1\pm\frac{s{{}^{\prime}}\Delta}{\sqrt{\Delta^{2}+\hbar^{2}\left(v_{x}^{\eta}k^{\eta}_{F}\right)^{2}}}\right]^{\frac{1}{2}}$
(10) $\displaystyle
k^{\eta}_{F}=\sqrt{\frac{\left(E-V\right)^{2}-\Delta^{2}}{(\hbar
v^{\eta}_{x})^{2}}}$ (11) $\displaystyle
z^{\eta}=s{{}^{\prime}}e^{\textbf{-\emph{i}}\phi_{\eta}},\qquad\phi_{\eta}=\tan^{-1}\frac{v_{y}^{\eta}k_{y}}{v_{x}^{\eta}k_{x}^{\eta}}$
(12)

with the wave vector

$k_{x}^{\eta}=s{{}^{\prime}}\sqrt{\left(k^{\eta}_{F}\right)^{2}-\left(\frac{v_{y}^{\eta}}{v_{x}^{\eta}}k_{y}\right)^{2}}$
(13)

$a_{0}$ and $b_{0}$ are two constants. The ration
$\frac{v_{y}^{\eta}}{v_{x}^{\eta}}$ shows a manifestation of the anisotropy in
our system that will play a crucial role in the forthcoming analysis.

Regions $1$ and $3$ are assumed to be the infinite pristine graphene stripes
with $S=0$ and an isotropic Fermi velocity, i.e. $v_{x}=v_{y}=v_{F}$. The
eigenspinors in region $1$ ($x<0$) consists of the incident and reflected
plane waves $\psi_{1}=\psi_{\text{in}}+\psi_{\text{re}}$

$\psi_{1}(x,y)=\left[\begin{pmatrix}1\\\
z_{0}\end{pmatrix}e^{ik_{x}x}+r\begin{pmatrix}1\\\
-\frac{1}{z_{0}}\end{pmatrix}e^{-ik_{x}x}\right]e^{ik_{y}y}$ (14)

and for region ${\sf 3}$ $(x>d)$, we have $\psi_{3}=\psi_{\text{tr}}$

$\psi_{3}(x,y)=t\begin{pmatrix}1\\\ z_{0}\end{pmatrix}e^{ik_{x}x}e^{ik_{y}y}$
(15)

where the incident wave vector and $z_{0}$ are given by

$\displaystyle k_{x}=\sqrt{k_{F}^{2}-k_{y}^{2}}$ (16) $\displaystyle
z_{0}=se^{\textbf{\emph{i}}\phi},\qquad\phi=\tan^{-1}\frac{k_{y}}{k_{x}}$ (17)

with $r$ and $t$ denote the reflection and transmission coefficients,
respectively, $s=\mbox{sgn}(E)$ and the Fermi wave vector
$k_{F}=\frac{E}{\hbar v_{F}}$.

##  III Transport properties

As usual to determine the transmission coefficients one uses the boundary
conditions at $x=0$ and $x=d$. This process yields to the result

$\displaystyle
t_{\eta}=\frac{e^{ik_{x}d}\cos\phi_{\eta}\cos\phi}{\cos\phi_{\eta}\cos\phi\cos(k_{x}^{\eta}d)+i\sin(k_{x}^{\eta}d)(1+\sin\phi_{\eta}\sin\phi)}$
(18)

which can be cast to a complex notation

$\displaystyle t_{\eta}=\rho e^{i\varphi_{t}^{\eta}}$ (19)

of amplitude $\rho$ and phase shifts

$\displaystyle\varphi_{t}^{\eta}=\tan^{-1}\left(i\frac{t_{\eta}^{\ast}-t_{\eta}}{t_{\eta}+t_{\eta}^{\ast}}\right).$
(20)

At this stage we are ready for computing the corresponding transmission
probabilities $T_{\eta}$. Indeed, let us introduce the current density $J$,
which defines $T_{\eta}=\frac{J_{\text{tr}}}{J_{\text{in}}}$, with the
incident $J_{\text{in}}$ and transmitted $J_{\text{tr}}$ components of $J$. As
for our system, we find

$\displaystyle J=e\upsilon_{F}\psi^{+}\sigma_{x}\psi$ (21)

giving rise to the two transmissions

$\displaystyle T_{\eta}=\left|t_{\eta}\right|^{2}$ (22)

and $R_{\eta}=1-T_{\eta}$, which resulted from the conservation law.

Next, we study the GH shift and group delay by considering some transverse
wave vector $k_{y}=k_{y_{0}}$ together with an incident angle
$\phi\left(k_{y_{0}}\right)\in\left[0,\frac{\pi}{2}\right]$, denoted by the
subscript 0. An actual finite pulsed electron beam can be represented as a
temporo-spatial wave packet, which is the weighed superposition of plane wave
spinors. Therefore, the wave function of the incident, refelected at $x=0$ and
transmitted wave packets at $x=d$ can be expressed as double Fourier integral
over $\omega$ and $k_{y}$ [28]

$\displaystyle\Phi_{\text{in}}(x,y,t)=\iint f(k_{y},\omega)\
\psi_{\text{in}}(x,y)\ e^{-i\omega t}\ dk_{y}d\omega$ (23)
$\displaystyle\Phi_{\text{re}}(x,y,t)=\iint f(k_{y},\omega)\
\psi_{\text{re}}(x,y)\ e^{-i\omega t}\ dk_{y}d\omega$ (24)
$\displaystyle\Phi_{\text{tr}}(x,y,t)=\iint f(k_{y},\omega)\
\psi_{\text{tr}}(x,y)\ e^{-i\omega t}\ dk_{y}d\omega$ (25)

where the three spinors $\psi_{\text{in}},\psi_{\text{re}}$ and
$\psi_{\text{tr}}$ are given in (14) and (15), respectively. The frequency of
wave is $\omega=E/\hbar$ and the angular spectral distribution takes the form
$f(k_{y},\omega)=w_{y}e^{-w_{y}^{2}(k_{y}-\omega)^{2}}$ with the half beam
width at waist $w_{y}$ [29]. As a result, the total phases for the reflected
and transmitted waves at $x=0,d$ are, respectively,

$\displaystyle\Phi_{r}^{\eta}=\varphi_{r}^{\eta}+k_{y}-\omega t$ (26)
$\displaystyle\Phi_{t}^{\eta}=\varphi_{t}^{\eta}+k_{y}-\omega t.$ (27)

Next, we use the stationary phase approximation [30, 31] to establish the
expressions of GH shifts and group delay time. Then the GH shifts in
transmissions are written as

$S_{t}^{\eta}=-\frac{\partial\varphi_{t}^{\eta}}{\partial k_{y}}.$ (28)

The equation of motion is obtained using the condition
$\partial\Phi_{t}^{\eta}/\partial\omega=0$ for keeping the good shape during
the propagation, provide the group delay

$\displaystyle\tau_{t}$
$\displaystyle=\frac{\partial\varphi_{t}^{\eta}}{\partial\omega}+\frac{\partial
k_{y}}{\partial\omega}S_{t}^{\eta}$ (29)
$\displaystyle=\tau_{t}^{s}+\tau_{t}^{\varphi}$ (30)

where $\tau_{t}^{\varphi}$ resulted from time derivative of phase shifts and
$\tau_{t}^{s}$ is originated from the contribution of $S_{t}$. As a
consequence, we end up with

$\tau_{t}^{\varphi}=\hbar\frac{\partial\varphi_{t}^{\eta}}{\partial
E}+\frac{\hbar}{2}\frac{\partial\phi}{\partial
E},\quad\tau_{t}^{s}=\frac{\sin\phi}{\upsilon_{F}}S_{t}^{\eta}.$ (31)

These quantities will be numerically computed under suitable conditions of the
physical parameters.

##  IV Results and discussions

The properties of GH shifts $S_{t}^{\eta}$, transmission probabilities
$T^{\eta}$ and group delay time $\tau_{t}^{\eta}/\tau_{0}$ will be discussed
for electrons transmitting across a barrier in gapped-strained graphene. Here
we introduced the scaled Fermi wavelength $\lambda=\frac{2\pi}{k_{F}}$ and
time scale $\tau_{0}=\frac{d\cos\phi}{v_{F}}$. Note that according to (13) one
should have the condition

$\displaystyle\left(k_{F}^{\eta}\right)^{2}-\left(\frac{v_{y}^{\eta}}{v_{x}^{\eta}}k_{y}\right)^{2}\geq
0$ (32)

in order to have a real wave vector $k_{x}^{\eta}$ . Then beyond this,
$k_{x}^{\eta}$ will be imaginary, which physically entails the evanescence of
the wave function inside the barrier. In contrast, when the strain along
armchair and zigzag directions satisfies (32), the evanescent wave function
exists, but still propagating inside the transmission region.

###  IV.1 The GH shifts in transmissions

Figure 2: (color online) The GH shifts in transmissions a) and transmission
probabilities b) as a function of the strain strength $S$ for $V=125$ meV,
$E=75$ meV, $\Delta=0$ meV, $d=80$ nm and $\phi=20^{\circ}$. Strain along
armchair direction (green line) and zigzag (red line).

FIG. 2 shows the influence of the strain along armchair and zigzag directions
on the GH shifts in transmissions and transmission probabilities. This has
been performed by fixing the barrier height $V=125$ meV, incident energy
$E=75$ meV, band gap $\Delta=0$ meV, incident angle $\phi=20^{\circ}$ and
barrier width $d=80$ nm. From FIG. 2a, we observe that for small values of
$S$, the GH shifts in the propagating mode can be enhanced by transmission
resonances. We notice that the GH shifts decrease by increasing $S$ for the
zigzag case but increase in the armchair one. It is clearly seen that the GH
shifts in transmissions for zigzag survive beyond the ratio S=23.6% and vanish
at larger ratio $S>23.6\%$. In FIG. 2b under the condition $S>23.6\%$ every
incoming state is fully reflected for the zigzag case. The strain along
armchair direction (green line) shows much less impact on the transmission
than strain along zigzag direction. This latter makes the transmissions
oscillate with small amplitudes but high frequencies.

In FIG. 3 we plot the GH shifts and the transmissions as a function of the
incident energy $E$ in the strainless $S=0\%$ and strain graphene $S=22\%$ for
$V_{0}=120$ meV, $\Delta=0$ meV, $d$=80 nm and $\phi$= $20^{\circ}$. We
observe that the GH shifts are closely related to the transmissions. FIG. 3a
indicates that the GH shifts change sign near the Dirac point $E=V$, and
become large at certain resonance points. In fact, the change in sign of the
GH results from the fact that the Dirac point $E=V$ signifies the transition
between the Klein effect $(E<V)$ and the classical motion $(E>V)$. The GH
shifts present a maximum peak for the zigzag case compared to armchair case
and become constant after certain threshold energy, which is compatible with a
maximum of transmission in FIG. 3b. We notice that the oscillating
transmissions decrease for armchair (green line) and increase for the zigzag
(red line) compared to the strainless graphene (blue line).

Figure 3: (color online) The GH shifts in transmissions a) and transmission
probabilities b) as a function of the incident energy $E$ for $V_{0}=120$ meV,
$\Delta=0$ meV, $d$=80 nm, $\phi$= $20^{\circ}$, $S=0\%,22\%$. Strainless
(blue line), strain along armchair direction (green line) and zigzag (red
line).

Figure 4: (color online) The GH shifts in transmissions for zigzag direction
as a function of the incident energy $E$ for $V=120$ meV, $d=80$ nm,
$\phi=20^{\circ}$, $\Delta=0$ meV, $S=0\%$ (magenta line), $S=5\%$ (blue
line), $S=10\%$ (green line) and $S=20\%$ (red line).

FIG. 4 presents the GH shifts in transmissions as a function of the incident
energy $E$ for strain along zigzag direction with $S=(5\%,10\%,20\%)$, $V=120$
meV, $d=80$ nm, $\phi=20^{\circ}$ and $\Delta=0$ meV. Overall, the GH shifts
evolve in a similar tendency as that in the strainless case (magenta line)
regardless of the strain along zigzag direction being $S=5\%$ (blue line),
$S=10\%$ (green line) and $S=20\%$ (red line). However, we observe that the GH
shifts sensitively depend on the strain strength and show a remarkable
difference between the two values $S=10\%$ and $S=20\%$. It turns out that the
strain effect results in the deformation of the Dirac cones and for that the
modulation of GH shifts can be realized by changing $S$.

###  IV.2 Group delay time

Figure 5: (color online) The group delay time in transmissions
$\tau_{t}/\tau_{0}$ as a function of the strain strength $S$ for a):
$\phi=20^{\circ}$ and b): $\phi=0^{\circ}$, with $E=75$ meV, $V=123$ meV,
$d=100$ nm, $\Delta=0$. Strain along armchair direction (green line) and
zigzag (red line).

Now, we investigate the group delay time in transmissions for graphene in the
presence of strain along armchair and zigzag directions. As a result we will
discuss the modulation of group delay by changing the height of barrier and
strain strength $S$ in FIG. 5. For an incident angle $\phi=20^{\circ}$ in FIG.
5a, we observe the group delay increases by oscillating for strain along
zigzag direction (red line). As for armchair case (green line), the group
delay is approximately to unity, meaning that the particles propagate through
the barrier with the Fermi velocity $v_{F}$ ($\tau_{t}/\tau_{0}\simeq 1$).
FIG. 5b shows the group delay as a function of strain strength $S$ at normal
incidence $\phi=0^{\circ}$, i.e. $k_{y}=0$, for both armchair and zigzag
directions in the same choice of parameters as in FIG. 5a. It is clearly seen
that the oscillations of group delay disappeared for zigzag direction but for
armchair we still have the same behavior as in FIG. 5a. In addition to these
properties, we notice that the absolute values of the group delay are strongly
dependent on the incident angle.

Figure 6: (color online) The group delay time in transmissions
$\tau_{t}/\tau_{0}$ as a function of a) the incident angle $\phi$ for
$\Delta=0$ and b) the gap $\Delta$ for $\phi=20^{\circ}$. Here we choose
$V=120$ meV, $E=75$ meV $d=100$ nm, $S=6.5\%$. Strain along armchair direction
(green line) and zigzag (red line).

FIG. 6a we show the influence of incident angle $\phi$ on the group delay time
$\tau_{t}/\tau_{0}$ in transmission for strain along armchair and zigzag
directions. The group delay in transmission become mostly constant up to some
value then show sharp picks. It is found that the group delay in transmission
can be enhanced by a certain incident angle. Indeed, by increasing $\phi$, we
notice there is modulation of $\tau_{t}/\tau_{0}$ for strain along armchair
and zigzag directions. One sees that $\tau_{t}/\tau_{0}$ vanishes after
$\phi>45^{\circ}$ for strain along zigzag and $\phi>50^{\circ}$ for armchair.
In FIG. 6b we show the influence of the band gap $\Delta$ on the group delay
$\tau_{t}/\tau_{0}$ for strain along armchair (green line) and zigzag (red
line) with $\phi=20^{\circ}$ degree, $V=120$ meV, $E=75$ meV $d=100$ nm,
$S=6.5\%$. We observe that for $\Delta=0$ the particles propagate through the
barrier with the Fermi velocity $v_{F}$, (i.e. $\tau_{t}/\tau_{0}=1$).
Increasing now $\Delta$, $\tau_{t}/\tau_{0}$ oscillates for the both strain
directions. Additionally, $\tau_{t}/\tau_{0}$ in the case of zigzag strain
exceeds that one of armchair for $\Delta<36$ meV. However, for
$36\leq\Delta\leq 50$ meV and by increasing $\Delta$ the former hierarchy is
inverted and therefore $\tau_{t}/\tau_{0}$ for armchair strain exceeds the
zigzag one. Subsequently, as soon as $\Delta$ increases for $\Delta>50$ meV,
$\tau_{t}/\tau_{0}$ will be frozen, which means that it becomes independent on
$\Delta$.

Figure 7: (color online) The group delay time in transmissions
$\tau_{t}/\tau_{0}$ as a function of a) the incident energy $E$ for $V=120$
meV and b) the barrier height $V$ for $E=120$ meV. Here we choose $d=100$ nm,
$\phi=20^{\circ}$, $\Delta=0$, $S=0\%$ (strainless) and $S=5\%$ (strain).
Strainless (color blue), strain along armchair direction (green line) and
zigzag (red line).

In FIG. 7 we discuss the modulation of group delay time in transmissions by
varying the incident energy and the barrier height. Indeed, FIG. 7a presents
$\tau_{t}/\tau_{0}$ as a function the incident energy $E$ for different values
of strain strength, strainless $S=0\%$ and $S=5\%$ for armchair and zigzag
directions. It turns out that the modulation of $\tau_{t}/\tau_{0}$ can be
realized by changing strain $S$. We observe that the amplitude of oscillations
or peaks increases for zigzag direction and decreases for armchair compared to
the strainless case. FIG. 7b presents $\tau_{t}/\tau_{0}$ as a function of the
barrier height $V$. By increasing the strain strength to $S=5\%$,
$\tau_{t}/\tau_{0}$ decreases for armchair direction and becomes less than
that for strainless $S=0$. In contrast, for the zigzag direction with $S=5\%$,
$\tau_{t}/\tau_{0}$ increases with respect to $S=0$. The group delay time in
transmission in the propagating mode can be enhanced by transmission
resonances and a null $\tau_{t}/\tau_{0}$ corresponds to a total reflection.

##  V Conclusion

We have studied the strain effect applied along armchair and zigzag directions
on the GH shifts and group delay time for transmitted Dirac fermions in gapped
graphene through a single barrier structure. In the first stage, we have
determined the eigenvalues and eigenspinors, which were used to compute the
corresponding transmission probabilities. Subsequently, we have analytically
derived the GH shifts and group delay time.

We have numerically analyzed the GH shifts and group delay time by considering
various choice of the physical parameters. Moreover, for strain along zigzag
direction, there are increasing of oscillations in transmission probabilities,
GH shifts and group delay time compared to the strainless graphene. In
contrast, it is found that such oscillations decrease for strain along
armchair direction. We have showed that the group delay time in transmission
approaches unity for a certain critical value of the barrier height, incident
energy, band gap, incident angle and barrier width. We have concluded that the
group delay time in transmission in the propagating mode can be enhanced by
transmission resonances.

May our findings could help to use graphene as a feasible setup to measure the
superluminal group delay in solid state physics. In addition, tuning the group
delay time by scalar potential, strain strength and gap could provide some
applications in high-speed graphene-based nanoelectronics [32].

## References

  * [1] K. S. Novoselov, A. K. Geim, S. V. Morozov, D. Jiang, Y. Zhang, S. V. Dubonos, I. V. Grigorieva and A. A. Firsov, Science 306, 666 (2004).
  * [2] A. K. Geim and K. S. Novoselov, Nat. Mater. 6, 183 (2007).
  * [3] A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov and A. K. Geim, Rev. Mod. Phys. 81, 109 (2009).
  * [4] F. Goos and H. Hänchen, Ann. Phys. 436, 333 (1947).
  * [5] X. Chen, J.-W. Tao, and Y. Ban, Eur. Phys. J. B 79, 203 (2011).
  * [6] Y. Song, H-C. Wu and Y. Guo, Appl. Phys. Lett. 100, 253116 (2012).
  * [7] X. Chen, P-L. Zhao, X-J. Lu and L-G. Wang, Eur. Phys. J. B 86, 223 (2013).
  * [8] T. E. Hartman, J. Appl. Phys. 33, 3427 (1962).
  * [9] Zhenhua Wu, Kai Chang, J. T. Liu, X. J. Li, and K. S. Chan, J. Appl. Phys. 105, 043702 (2009).
  * [10] V. S. Olkhovsky and E. Recami, Phys. Rep. 214, 339 (1992).
  * [11] H. Haugen, D. H. Hernando, and A. Brataas, Phys. Rev. B 77, 115406 (2008).
  * [12] Z. H. Ni, T. Yu, Y. H. Lu, Y. Y. Wang, Y. P. Feng, and Z. X. Shen, ACS Nano 2, 2301 (2008).
  * [13] T. M. G. Mohiuddin, A. Lombardo, R. R. Nair, A. Bonetti, G. Savini, R. Jalil, N. Bonini, D. M. Basko, C. Galiotis, N. Marzari, K. S. Novoselov, A. K. Geim, and A. C. Ferrari, Phys. Rev. B 79, 205433 (2009).
  * [14] M. Y. Huang, H. G. Yan, C. Y. Chen, D. H. Song, T. F. Heinz, and J. Hone, Proceedings of the National Academy of Sciences 106, 7304 (2009).
  * [15] K. Sasaki, Y. Kawazoe and R. Saito, Prog. Theor. Phys. 113, 463 (2005).
  * [16] J. L. Maenes, Phys. Rev. B 76, 045430 (2007).
  * [17] C. Lee, X. Wei, J. W. Kysar, and J. Hone, Science 321, 385 (2008).
  * [18] S. M. Choi, S. H. Jhi, and Y. W. Son, Phys. Rev. B 81, 081407 (2010).
  * [19] B. Soodchomshom, P. Chantngarm, J. Supercond. Nov. Magn. 24, 1885 (2011).
  * [20] W.-X. Yan and L.-N. Ma, Physica B 445, 28 (2014).
  * [21] C. L. Kane and E. J. Mele, Phys. Rev. Lett. 95, 226801 (2005).
  * [22] C. L. Kane and E. J. Mele, Phys. Rev. Lett. 95, 146802 (2005).
  * [23] J. Sichau, M. Prada, T. Anlauf, T. J. Lyon, B. Bosnjak, L. Tiemann, and R. H. Blick, Phys. Rev. Lett. 122, 046402 (2019).
  * [24] W. Yan, Physica B 504, 23 (2017).
  * [25] J. H. Wong, B. R. Wu, and M. F. Lin, J. Phys. Chem. C 116, 8271 (2012).
  * [26] V. M. Pereira, A. C. Neto, and N. M. R. Peres, Phys. Rev. B 80, 045401 (2009).
  * [27] K. S. Novoselov and A. K. Geim, Rev. Mod. Phys. 81, 109 (2009).
  * [28] Y. Fattasse, M. Mekkaoui, A. Jellal, and A. Bahaoui, Physica E 134, 114924 (2021).
  * [29] C. W. J. Beenakker, R. A. Sepkhanov, A. R. Akhmerov, and J. Tworzydlo, Phys. Rev. Lett. 102, 146804 (2009).
  * [30] A. M. Steinberg and R. Y. Chiao, Phys. Rev. A 49, 3283 (1994).
  * [31] C.-F. Li, Phys. Rev. A 65, 066101 (2002).
  * [32] Xi Chen, Zhi-Yong Deng, and Yue Ban, Appl. Phys. 115, 173703 (2014).


# Autonomous, Mobile Manipulation in a Wall-building Scenario: Team LARICS at
MBZIRC 2020

Ivo Vatavuk, Marsela Polic, Ivan Hrabar, Frano Petric, Matko Orsag, Stjepan
Bogdan
Laboratory for Robotics and Intelligent Control systems
University of Zagreb Faculty of Electrical Engineering and Computing
Unska 3, Zagreb, Croatia
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper we present our hardware design and control approaches for a
mobile manipulation platform used in Challenge 2 of the MBZIRC 2020
competition. In this challenge, a team of UAVs and a single UGV collaborate in
an autonomous, wall-building scenario, motivated by construction automation
and large-scale robotic 3D printing. The robots must be able, autonomously, to
detect, manipulate, and transport bricks in an unstructured, outdoor
environment. Our control approach is based on a state machine that dictates
which controllers are active at each stage of the Challenge. In the first
stage our UGV uses visual servoing and local controllers to approach the
target object without considering its orientation. The second stage consists
of detecting the object’s global pose using OpenCV-based processing of RGB-D
image and point-cloud data, and calculating an alignment goal within a global
map. The map is built with Google Cartographer and is based on onboard LIDAR,
IMU, and GPS data. Motion control in the second stage is realized using the
ROS Move Base package with Time-Elastic Band trajectory optimization. Visual
servo algorithms guide the vehicle in local object-approach movement and the
arm in manipulating bricks. To ensure a stable grasp of the brick’s magnetic
patch, we developed a passively-compliant, electromagnetic gripper with
tactile feedback. Our fully-autonomous UGV performed well in Challenge 2 and
in post-competition evaluations of its brick pick-and-place algorithms.

##  1 Introduction

The focus of many robotics competitions in recent years has been on tasks
closely related to warehouse automation or service robotics. Such competitions
include the Amazon Picking Challenge [Correll et al., 2018] and the
RoboCup@Home challenge [Matamoros et al., 2019], which both emphasize object-
detection and grasp-planning for pick-and-place tasks. The Mohamed Bin Zayed
International Robotics Challenge [MBZIRC, 2020] builds upon these ideas and
extends the complexity by placing the robots outdoors in a relatively
unstructured environment. For the 2020 edition of MBZIRC, one of the
challenges emulates a construction site where a heterogeneous multi-robot
team, consisting of multiple, unmanned, aerial vehicles (UAVs) and a single,
unmanned, ground vehicle (UGV) must build a wall with a predefined blueprint.

The wall-building scenario can be easily generalized to other real-world
problems involving navigation, object perception, motion, grasp-planning and
task-planning. Warehouse automation and search-and-rescue missions are a few
among a plethora of such everyday work environments that have not yet been
automated. As with all robotics competitions, the challenges presented should
be viewed as a validation of robotic progress and its adaptability to
unstructured environments with unmodeled disturbances, and outside of
laboratory conditions. The objective of these competitions is to provide a
challenge to the robotics research community, to test and validate the
existing state of the art, to identify problem areas, and to accelerate
scientific progress. No less importantly, the competition can be seen as an
opportunity for the less-prominent research groups to gain visibility.

Heterogeneous multi-robot teams constitute a well-researched concept with a
focus on coordination and planning, as discussed in our earlier paper
[Krizmancic et al., 2020]. Herein, the focus is on the problems faced by the
UGV in the Challenge 2 of MBZIRC 2020. As stated in the challenge rules, the
UGV must find the stacks of bricks, which consist of bricks of different
colors, transport them to the building site marked by a distinctive pattern on
the ground (see Figure 1), and build a wall with a specified brick
arrangement. Points are awarded for the correct arrangement and precision of
bricklaying and, most importantly, for autonomous task execution.

To complete Challenge 2 the UGV must solve several, nested problems. First,
the UGV must navigate precisely in an outdoor environment, finding key
locations, transporting bricks, and avoiding brick stacks and any already-
constructed wall. In addition, the UGV must autonomously detect the brick
stack, a single, target brick atop the stack, and, finally the construction
location. Once these tasks are handled, the UGV must accurately determine the
grasping point to pick up a brick and then move to position itself in a pose
that makes the brick reachable. Similar tasks are required for the brick-
laying part of the Challenge. The robot must accurately determine the location
of building site and navigate and position itself to enable successful and
precise brick placement.

Figure 1: MBZIRC Challenge 2 scenario: A UGV must autonomously find a stack of
bricks (left), select and pick up a brick, transport it to a building site
marked with a checkered pattern, and place it precisely on the building
location (right).

This paper discusses our contributions in each of the aforementioned tasks.
For precise navigation in an outdoor environment, building on our previous
research [Vasiljević et al., 2016], we used a highly-precise graph-SLAM
navigation and mapping algorithm based on Google Cartographer [Hess et al.,
2016]. The algorithm combines 3D data from LIDAR, IMU, and GPS data to build a
3D-map of the environment, which is then projected into a 2D costmap for
faster path-planning. Visual and depth data from an RGB-D camera is used to
autonomously identify the objects of interest and their positions within the
UGV map. The geometry of the objects and properties of the Challenge scenario
are used to estimate precisely navigation waypoints that ensure reachability
of bricks and building site. Finally, to ensure the robustness of picking up
and placing of the bricks, an eye-in-hand visual servo approach is deployed,
for which a gripper was designed with an integrated depth camera, tactile
feedback, and passive-compliance components to compensate for uneven surfaces
and small differences in the orientation of the UGV with respect to the
bricks. The approach to solving the technical challenges posed by the
equipment used is also discussed.

The remainder of this paper is organized as follows: Section 2 positions our
approach with respect to the state-of-the-art in the field. Sections 4, 5 and
6 present a description of the presented developed algorithms for high-level
control based on a state machine, object detection and low-level control for
navigation and manipulation. Section 7 presents the experimental results,
testing the performance of brick pick-up and pattern drop-off individually,
along with the performance of the autonomous system as a whole, performing the
entire Challenge 2 task autonomously. Section 8 concludes the paper with some
comments on future work.

##  2 Related Work

The manipulation of objects using a robotic ground platform is a well-known
concept. Pioneering work in the development of an automated brick laying
robotic system in building construction was done as early as 1996 [Pritschow
et al., 1996]. Although robotic automation is widely used in the automotive
industry, it is not yet very present in the construction industry. There are
many existing solutions in terms of stationary or fixed automation solutions.
However, there are also multi-purpose mobile ground robot systems, consisting
of mobile robot bases and manipulators, enriched with task-specific end
effectors and sensors such as [Gawel et al., 2019]. Fully autonomous robots
for building construction are mostly demonstrated under laboratory conditions,
but there is a strong industrial need for automation in building construction
[Davila Delgado et al., 2019] [Melenbrink et al., 2020].

Nowadays, object detection is usually performed with neural networks that
provide fast and reliable detection, such as Yolo [Bochkovskiy et al., 2020]
or Fast R-CNN [Girshick, 2015]. However, taking advantage of the fact that the
objects in the Challenge 2 of MBZIRC 2020 are color-coded, the approach in
this work is based on classical detection in HSV space, with several
extensions for increased reliability, similar to those presented in
[Nieuwenhuisen et al., 2017].

Since the goal of the Challenge is to detect and manipulate objects, the robot
must estimate the pose of the brick and evaluate whether the brick is
reachable before planning the grasping maneuver. Since the gripper is equipped
with an RGB-D camera, the possibility of using depth data to estimate the pose
of the brick was also explored. The most obvious approach is to generate a
point cloud from depth data and then perform template matching to extract the
pose of a brick [Vock et al., 2019][Opromolla et al., 2015]. However, this is
computationally expensive for dense point clouds generated by depth cameras.
Given the noise observed in depth image data in outdoor experiments, and the
fact that the variance in depth estimation for one model of Intel Realsense
camera grows quadratically with distance [Ahn et al., 2019], brick-pose
estimation could not rely on depth information, but rather on image based
estimation.

Similar to object detection, the estimation of objects pose from RGB images is
most commonly approached with neural networks, as presented in [Grabner et
al., 2018] and [Xiang et al., 2018]. While offering ever-increasing
performance, neural network-based solutions still suffer from the lack of
annotated data (see [Kundu et al., 2018] and [Sock et al., 2020]). A priori
knowledge that the objects of interest are positioned in a certain way in the
environment, and knowledge of the pose of the camera with respect to the same
environment, enable solutions that do not require neural networks. The
approach of this work to determining the object’s pose is to invert a well-
known Perspective-N-Point problem [Fischler and Bolles, 1981] and extract the
position of 3D points knowing the position of a calibrated camera in the
environment, enabled by precise robot localization and manipulator kinematics.

Once the object of interest is detected and its pose is estimated, the goal of
the system is to navigate the mobile robot to the vicinity of the object,
followed by precise control of the manipulator to execute a given task. This
two-stage approach, where the mobile robot and the arm are controlled
separately (while the robot moves the arm is stationary and vice-versa), is
inspired by the split-range control concept [Reyes-Lúa et al., 2019]. For the
navigation part, the TEB planner [Rösmann et al., 2017] is used in a 2D
costmap obtained from a 3D-map of the environment generated by Google
Cartographer. This approach was inspired by the work in [Wulf et al., 2004]
where the authors use 3D range data to generate 2D maps directly. Since the
generation of waypoints in the map was based on image processing, i.e.
estimation of objects pose in robot’s coordinate frame, the navigation of the
robot could be compared to position-based visual servoing [Chaumette and
Hutchinson, 2006]. However, in our approach only the reference to the
controller is provided by the image processing stack, while the feedback is
provided by the map localization using Lidar. Control architecture used for
both approaching the bricks and the wall, as well as for control of the arm
during loading and unloading of the bricks, can be classified as eye-in-hand
image-based visual servoing, which is often used in the field [Muslikhin et
al., 2020], [Tsay and Chang, 2004].

##  3 Equipment

Our hardware architecture was driven by the MBZIRC experimental setup
specifications, which required a robust mobile manipulation platform capable
of driving over areas partially covered with pebbles.

Based on these requirements and our previous experience, we selected the Husky
A200, a four-wheeled skid-steered field research platform, manufactured by
Clearpath Robotics Inc. The existing Husky hardware was equipped with an
Intel® NUC i7DNHE running Ubuntu 18.04 with ROS Melodic, which served as the
primary processing and control computer. Robosense RS-LiDAR-16, a lightweight
omnidirectional laser radar with 16 beams, was mounted on the front of the
robot (see Figure 2a). To increase mapping accuracy, a Pixhawk autopilot was
mounted on the robot, which provided IMU measurements and served as an
interface between the Husky on-board control and the remote control unit (RC),
which was programmed to immediately switch to manual mode in case of
emergency.

(a) Husky A200 with Schunk LWA4P arm and sensors

(b) The breakdown of the developed electromagnetic gripper with integrated
eye-in-hand depth camera and compliant components which ensure sufficient
contact area

Figure 2:  Equipment used by LARICS team in MBZIRC2020.

A 6-DoF extended Schunk Powerball LWA4P lightweight arm with integrated joint
drives, controlled via a CAN interface, was mounted on the Husky. Having
integrated joint drives eliminates the need for additional external control
units or power converters. The Schunk manipulator is characterized by its
reach of 1460 mm, which is significantly greater than that of other
commercially available lightweight arms. To increase the autonomy of the Husky
equipped with the Schunk arm, two separate battery packs were installed, with
the first primarily powering the robot’s drive train, and the second powering
the manipulator. In addition to powering a high-current device (the robot
drive train or the manipulator), each battery pack powers some low-current
electronic components. The output grounds of the battery packs are connected
to each other with star-connected grounding cables. To achieve faster
turnaround time for placing bricks, additional cargo baskets are installed on
both sides of the platform, allowing the Husky to transport up to four bricks,
as shown in Figure 2a.

###  3.1 Passively compliant magnetic gripper with integrated depth camera and
tactile feedback

The bricks in the Challenge 2 of the MBZIRC2020 were designed to have a
ferromagnetic area on the top side, which required the design and fabrication
of a controllable electromagnetic gripper. Several constraints had to be met
in the design of the gripper, which is shown in Figure 2b. Aside from using
controllable electromagnets, the gripper had to be lightweight, low-powered
and strong enough to lift bricks that can weigh up to 2 kg.

With weight in mind, ten small electromagnets were selected, each nominally
capable of lifting 1 kg. The use of small and low-power magnets in conjunction
with a slender ferromagnetic patch on the bricks required that at least six
magnets be in contact with the brick to generate magnetic force of sufficient
strength to lift the brick. This poses a hard constraint on the orientation of
the gripper with respect to the brick when in contact, which could not be
satisfied via control algorithms. Due to the stiffness of the Schunk arm, and
consequently of the mobile manipulator as a whole, any irregularities on the
surface or even slightly different pressures in the robot’s tires could cause
misalignment of the gripper and the brick, reducing the number of magnets in
contact with the ferromagnetic patch. This in turn resulted in the robot being
unable to pick up the brick.

To overcome this problem and ensure full contact of the electromagnetic
gripper with the brick at all times, a passive compliance was incorporated
into the gripper with four shock-absorbing rubber balls. The rubber balls
having clearance between the rigid parts of 15 mm provide the gripper with the
ability to compensate for orientation differences of more than $10^{\circ}$,
which ensures optimal contact between the magnets and the surface of the
ferromagnetic patch on the brick. The size of the gripper is 10x15cm and it
weighs about 700g.

From the control perspective, laser-based navigation in free space and control
based on visual servo principles around the objects of interest are used. An
Eye-in-hand approach was implemented, with the Intel Realsense D435 integrated
into the gripper, which was used both for detecting objects during navigation
and for controlling the mobile manipulator during pickup. To provide the
control algorithm with stable references while Husky is stationary, Realsense
was mounted on the rigid part of the gripper, right next to the rubber shock
absorbers. To mitigate the major drawback of the eye-in-hand visual servo
approach, which is loosing feedback when close to the object of interest, a
contact sensor in form of a microswitch was added to the gripper, mounted in
the center of the end-effector. The role of the tactile feedback was to signal
to the visual servo controller that contact had been established, and to
provide high-level control with information in case the brick was dropped
during transport.

##  4 High-level control

A challenge-specific state machine has been designed that dictates which
controllers are used at each stage of the Challenge. In certain states, both
the UGV and the manipulator arm must collaborate, while in other states either
UGV or manipulator is controlled while the other remains idle. This approach
is described using a high-level state machine (Figure 3) .

Figure 3: High-level state machine designed for the Challenge. After reaching
the brick stack initial position, the desired bricks are loaded into the brick
containers. The robot navigates to the wall pattern position, and unloads the
bricks, building a portion of the wall. This process is repeated until the
mission is completed.

####  4.0.1 UGV-only actions

The mission planner 4.1 specifies which bricks to load and unload, providing
an input for the high-level state machine (see Figure 7). The states Go To
Initial Brick Stack Position and Go To Wall Pattern are executed by the UGV,
while the robot arm remains idle. During execution of these two states, the
platform navigates through the map to the approximate position of the brick
stacks, or the wall pattern, respectively.

####  4.0.2 Collaborative control

Operations in which the platform and the robot arm collaborate are Load Bricks
and Unload Bricks. These two behaviors have similar structures, as shown in
Figures 4 and 5. Here, the platform is controlled by Two-Stage Approach, whose
objective is to guide the UGV to a pose where the desired object is within
reach of the robot arm. In other words, the emphasis is on the correct
orientation of the UGV. Although it is not critical when picking up smaller
objects such as the red brick, the correct UGV orientation becomes very
important when manipulating larger objects or when approaching the wall
pattern.

Figure 4: Inside the Load Bricks block of the state machine Figure 5: Inside
the Unload Bricks block of the state machine

Both Load Bricks and Unload Bricks include Initial Approach and Final Approach
states, in which the robot locally approaches a desired object, without taking
its orientation into account. This kind of approach is executed by directly
controlling the robot drive and is referred to as Local Object Approach in the
remainder of this paper. Initial Approach is used to get close enough to the
desired object to detect its pose (Pose Detection state), after which a goal
pose for the UGV, aligned with the object pose, is calculated (Figure 6).
Here, the local approach controller is switched off, and the vehicle is
stopped. The Alignment is conducted with a navigation planner that ensures
global orientation of the platform within the map, in close vicinity of the
manipulated object. Upon reaching the proper pose, the local approach
controller is again switched on. The local control in the Final Approach
visually guides the platform within the object reach retaining the acquired
global orientation.

The main difference between the Load Bricks and Unload Bricks behaviors is
that the whole two-stage approach is executed only once when unloading bricks
onto the wall pattern. All of the subsequent Unload Bricks behaviors rely on
the pattern’s pose saved in the global map frame $L_{M}$ during the first pose
detection state, directly proceeding to the second, Alignment stage.

Figure 6: In the Alignment state we use target pose detection to calculate an
aligned goal for the UGV. The pose of the magnetic patch is obtained in the
robot’s body frame $L_{B}$ and transformed into a map frame $L_{M}$, from
which a goal pose is constructed at a fixed distance from the patch with the
desired orientation of the robot aligned with the orientation of the brick.
This magnetic patch alignment is used in the Load Bricks block. A similar
principle is employed for the wall footprint alignment in the Unload Bricks
block.

####  4.0.3 Manipulator-only actions

Finally, procedures where only the arm is controlled while the UGV is idle are
Brick Pickup and Brick Drop. Brick Pickup includes running the visual servo
algorithm used to grasp the brick with the electromagnetic gripper and placing
the brick into an unoccupied brick container. Individual brick placement
inside the containers is memorized and used to determine the starting position
for the Brick Drop visual servo in navigating the arm to the wall pattern.

###  4.1 Planning and laying bricks in the correct order

The blueprint for the wall provided by the referees is parsed by a planner
discussed in an earlier paper [Krizmancic et al., 2020], and the optimal
sequence of bricks is calculated, taking into account the payload capacity of
the robot. Algorithm that generates the optimal brick laying order relies on
the hierarchical task representation through TÆMS (Task Analysis, Environment
Modeling, and Simulation) language [Horling et al., 1999][Lesser et al., 2004]
and uses the Generalized Partial Global Planning (GPGP) coordination framework
[Decker and Lesser, 1995] to find the optimal sequence of tasks with respect
to the robot’s capabilities and user-specific cost and/or rewards, which can
be specified to minimize the distance traveled, energy expended or in this
case, to maximize the number of points in the competition. The mission for the
robot is specified from the optimal brick laying order as shown in Figure 7.

Figure 7: Execution of a wall building scenario. Upon receiving the blueprint,
the planner generates a sequence of tasks coded Lc ( where L stands for Load
and c is the color of the brick) or Bc (where B stands for build and c is the
color of the brick) as the robot searches for bricks (or drives towards bricks
if the location is provided by the UAV). Task Bc can first appear after
several Lc tasks which can be executed several times until capacity of the
robot is filled. In the first building task, the robot searches for the wall
footprint (or the location can be provided externally by UAV), while in
subsequent runs it uses known brick positions within the blueprint and
calculates waypoints based on the known location of the wall footprint and the
dimensions of the bricks. Solid arrows show the sequence of actions, while
dashed arrows represent non-sequential information flow.

Figure 7 also shows how the state machines from Figures 3, 4 and 5 integrate
to form a complex behavior of the robot. As can be seen, upon receiving the
blueprint, the planner generates a sequence of actions (load brick, build
brick), that are executed according to the aforementioned state machines. It
can be observed that both the brick stacks and the wall footprint are searched
for only once, and their location in the global map is used in subsequent
trips to the brick stacks and to calculate alignment poses for brick
unloading. The alignment poses for unloading are calculated from the now-known
pose of the rightmost corner of the wall footprint and the brick positions
within the wall blueprint, taking into account a priori known brick
dimensions.

This approach is facilitated by the underlying mapping and localization stack
based on the Cartographer, for which we have shown that localization error and
drift are minimal (less than 25 cm) due to loop-closing capabilities of
Cartographer, even for a more dynamic robot (UAV), large maps (250 m $\times$
100 m) and large distances covered (more than 600 m), both of which are
significantly larger than the arena designated for the ground vehicle in
Challenge 2 of MBZIRC2020. For more details on this evaluation, interested
reader is invited to read [Milijas et al., 2020].

##  5 Detection of objects of interest

For Local Object Approach, a visual servo algorithm is used to keep observing
an object while approaching it. This requires detecting the position of the
wall footprint pattern, brick stacks, and magnetic patches in an image
captured by a camera. Magnetic patch-detection is also used for the visual
servo manipulation algorithm. Here, along with its position, the orientation
is also controlled. For the Two-Stage Approach, the global pose of the
approached object in map frame $L_{M}$ must be estimated in order to generate
an aligned goal pose for the mobile base.

Figure 8: Convex hull of the selected brick stack contour is shown with a
green line. The image position of the brick stack is denoted as
$(x_{b},y_{b})$. IDs are assigned to the detected magnetic patches and their
positions are tracked from frame to frame. In this particular case, patch
number $1$ is selected for approach by using a scoring scheme that takes into
account the area of the patch and the patch position in the image (denoted as
$(x_{p},y_{p})$).

###  5.1 Brick Stack and Wall Footprint Detection

To detect the brick stacks, the camera image is filtered using different Hue-
Saturation-Value thresholds for red, green, blue and orange bricks. Contour
detection is performed on the filtered image, and contours with a sufficiently
large surface area are selected as brick stack candidates. As a measure of the
quality of a brick stack candidate, it’s surface area is used, the ones with a
bigger area being of higher quality. The center of the selected brick stack
contour is used as its image position ($x_{b}$, $y_{b}$), which is used for
the local brick stack approach (Figure 8).

Wall footprint detection is performed in a similar way, combining Hue-
Saturation-Value bounds for yellow and magenta. To start unloading bricks onto
the wall footprint, one of its edges must be reached first, and the image
position of the rightmost contour point is chosen as the goal for the Local
Object Approach corresponding to the upper right corner of the wall footprint.
If the rightmost edge is not visible in the image, the camera rotates until
the edge comes into view.

###  5.2 Detecting the magnetic patch and calculating its pose

Contour detection is performed inside the convex hull of a selected brick
stack, and detected contours that resemble a rectangle are selected as
magnetic patch candidates. The center of the detected rectangle is used as its
position in the image ($x_{p}$, $y_{p}$) (Figure 8).

Since the magnetic patch candidate with the largest area is not necessarily
the one the robot must pursue, a scoring scheme is used to calculate the
candidate’s desirability $S_{i}$ based on its area and image position. To
minimize the rotation required to approach the object, this scoring scheme
prefers patch candidates positioned in the center of the image on the $x$
axis. Since objects detected in the lower part of the image are usually closer
to the robot, the scoring scheme also favors patches positioned as low as
possible on the $y$ axis. The aforementioned logic yields the following
equation to calculate the score of the $i$-th patch candidate:

$S_{i}=-w_{x}|x_{p,i}|+w_{y}y_{p,i}+w_{A}A_{i}$ (1)

where $A_{i}$ is the area, $w_{x}$, $w_{y}$ and $w_{A}$ are non-negative
weights used to tune the scoring scheme and correspond to the $x$ and $y$
position in the image and area, respectively.

To prevent rapid switching from one patch candidate selection to another,
hysteresis is introduced. This forces the new candidate score to be larger
than the current one by some margin. The hysteresis introduces a memory
element into the candidate selection process as candidate scores are tracked
over a period of time. It is therefore necessary to keep track of the detected
magnetic patches IDs, and update them from frame to frame.

The scoring scheme finally yields a single best patch candidate described with
its corresponding rectangle. The rectangle’s pose is determined by the pose of
its major axis. In fact, only the endpoints of this axis in the RGB camera
image are considered ($p_{1}$ and $p_{2}$ in Figure 9), which eliminates the
effect of perspective distortion. As a first step in obtaining the rectangle’s
pose in the robot body frame $L_{B}$, the pixel positions $(x_{px},y_{px})$ of
both endpoints’ image projections are transformed into distance measure using
camera intrinsic parameters, namely the focal length in pixels $z_{px}$ and
millimeters $z_{mm}$, along with the camera image resolution in pixels
$x_{px}\times y_{px}$ using equation 2 for both $x$ and $y$ coordinate. The
complete image plane projections of the endpoints $p_{1}$ and $p_{2}$ in the
camera local frame $L_{C}$ are then obtained as $p_{1c}$ and $p_{2c}$, with
the $z$ coordinate in the image plane $\pi_{c}:z=z_{mm}$.

$x_{mm}=x_{px}\frac{z_{mm}}{z_{px}}$ (2)

Figure 9: Endpoints of a patch in the image plane are transformed into the
robot base frame $L_{B}$ using camera intrinsic parameters. The pose of the
patch is calculated by intersecting the lines from the camera center through
the rectangle endpoints in the image plane with the horizontal plane whose
height is extracted from the 3D point cloud obtained from Realsense.

With a known position of the axis endpoints’ projections in the local camera
frame $L_{C}$, the same are obtained in the robot base frame, denoted $L_{B}$
in Fig. 9, using the direct kinematics transform matrix $T_{B}^{C}$. Camera
position $p_{cam}|_{L_{B}}=(x_{cam},y_{cam},z_{cam})$ is also obtained in the
robot base frame $L_{B}$ under the camera pin-hole model assumption, with the
local camera position in the $L_{C}$ frame origin, using the same transform
$T_{B}^{C}$. Now, lines passing from the camera origin through the projected
endpoints $p_{1c}$ and $p_{2c}$ can be described in the x-z plane in the robot
base frame with eq. 3. Here, the index $i\in\\{1,2\\}$ denotes both axis
endpoints $p_{1}$ and $p_{2}$. The equivalent linear equation holds for the
y-z plane in the robot base frame.

$x=x_{cam}-\frac{x_{cam}-x_{p_{i}c}}{z_{cam}-z_{p_{i}c}}(z_{cam}-z)$ (3)

What remains in determining the pose of the brick patch in $L_{B}$ is to find
the $x$ and $y$ coordinates of $p_{1}$ and $p_{2}$ in $L_{B}$, when the patch
height $z=patch\\_height$ is substituted into the projection equations 3 for
both $x$ and $y$ coordinates of $p_{1}$ and $p_{2}$. The patch height can be
obtained from the organized point cloud generated from the Realsense depth
image upon transformation into the camera and the base frame. Using the known
location of the patch in the image, the height of the brick stack can be read
in the organised point cloud.

The pose of the wall pattern edge is obtained in a similar manner. This pose
is detected once, transformed into the global map frame using the
transformation matrix $T_{M}^{L}$ that contains the estimate of robots
position and orientation in the map frame $L_{M}$, and reused for each
subsequent brick drop. Similarly, the pose of the magnetic patch is also
transformed into the global map frame using $T_{M}^{L}$ for calculation of the
navigation goal pose during the $Alignment$ phase.

##  6 Low-level control for navigation and manipulation

The UGV is controlled in two ways: a) Map Navigation uses Move Base motion
planning to navigate to a desired goal in the 2D map while avoiding obstacles,
b) Local Object Approach controls the robot’s forward and angular velocity
directly without taking the obstacles into account.

###  6.1 Map Navigation

Google Cartographer SLAM [Hess et al., 2016] is used to build a map of the
unknown environment and localize the robot in it. Since the Move Base motion
planning package needs a 2D costmap, it is generated using the 3D SLAM
algorithm (Figure 10) by filtering the Cartographer submap points based on
their height. Points below the lower threshold are filtered out to avoid
labeling ground as an obstacle. Similarly, points above the robot maximum
height are removed as they do not present a real obstacle in platform
navigation. Ground points that are far from the robot may be perceived as
false positive obstacles if the ground is not completely flat, which can be
solved by building ”smaller” submaps with $30$ laser scans per submap.

Figure 10: Point cloud of the Challenge 2 arena in Abu Dhabi, built by the
Cartographer 3D SLAM algorithm is shown on the left. The 2D costmap generated
from the 3D point cloud, used for the Move Base package, is shown on the
right.

Although the Husky A200 is a differential drive mobile robot and has the
capability to rotate in place, in practice this motion causes significant
vibration, particularly in cases when there is high friction between the
rubber wheels and the terrain. These vibrations severely affect certain
aspects of the system, most notably mapping and localization, as well as image
processing algorithms. To mitigate this issue, the Timed-Elastic Band (TEB)
planner for mobile robots is used as a local planner in the ROS navigation
stack [Rösmann et al., 2017]. The TEB planner uses online trajectory
optimization to generate a local plan for the robot subject to a minimum-
turning-radius constraint. The trajectory optimization used by the planner
incorporates soft constraints on velocity limits, but it was found that the
produced plan often does not satisfy these constraints. The algorithm was
modified to constrain the UGV’s forward and angular velocity, $v_{x}$ and
$\omega_{z}$ respectively

$\underline{v}_{x}\leq v_{x}\leq\overline{v}_{x}$ (4)
$|\omega_{z}|\leq\overline{\omega}_{z},$ (5)

while keeping the planned turning-radius

$\frac{v_{x}}{\omega_{z}}=\frac{v_{x,planned}}{\omega_{z,planned}}.$ (6)

The initial trajectory generated by TEB satisfies the imposed turning-radius
constraints by producing a motion plan with a high number of back and forth
motion switches. In the original implementation, the initial trajectory is
pursued and refined through iterative online optimization, reducing the number
of forward-backward switches. In contrast to the original implementation, this
work introduces initial conditions into the algorithm, suppressing the
velocity commands until the iterative trajectory optimization yields a plan
with a satisfactory low number of forward-backward switches. The initial
conditions can also be generated by a timeout if the optimization gets stuck
in a local optimum.

###  6.2 Local Object Approach

Local Approach is used to track a desired object under the assumption that the
object is in the camera field of view. During servo control, the control
inputs that achieve this assumption are the UGV’s forward velocity $v_{x}$,
the UGV’s angular velocity $\omega_{z}$, and the displacement of the camera
pitch angle $\Delta{\theta}$ controlled by the robot manipulator (Figure 11).
In the competition, a rough estimate of the initial brick stack position was
known a priori, and the mobile manipulator was navigated to a point in that
area of the arena. Once there, the manipulator arm performed a rotational
scanning motion searching for the stack of a particular color, if it was not
already in the field of view.

In a more general scenario, as originally announced in the MBZIRC challenge
rules, the initial information about the arena, including the map and the
positions of the brick stacks and the wall pattern, would be shared within the
robot team. In this way, a quick search flight would be conducted by the UAVs
in the robot team, providing the UGV with the necessary navigation
information. Since the operation of the different types of robots was
ultimately independent, a lawnmower search trajectory was planned for the UGV
with the brick stack detection and localisation as objective. However, due to
the time consumption of such a search pattern within a short experimental time
slot, and thanks to the easily located brick stacks, the lawnmower search was
not deployed.

Figure 11: Control inputs used for the local approach: the top view of the
mobile manipulator with its forward and angular velocity is shown on the left,
and its side view with the camera pitch angle is shown on the right.

The position of the object in the image ($x_{img}$, $y_{img}$), and object
distance $d_{x}$, measured in the robot frame $L_{B}$, are used as the
feedback. Depending on the current state, $x_{img}$ and $y_{img}$ are acquired
from either the brick stack, the magnetic patch, or the wall footprint
detection. The object distance is obtained by filtering the Realsense camera
point cloud with the detected object mask as the closest point in the UGVs
forward direction. To deal with occasional poor distance measurements, a
constant velocity model Kalman filter is used. Since the distance is measured
with the RGBD camera, it is important to keep the desired object in the camera
field of view during the approach.

Proportional controllers are used to approach the desired object, while
keeping it in the center of the image:

$v_{x}=-K_{d_{x}}^{A}(d_{r}-d_{x})$ (7) $\omega_{z}=-K_{x_{img}}^{A}x_{img}$
(8) $\Delta{\theta}=K_{y_{img}}^{A}y_{img},$ (9)

where $d_{r}$ is the desired object distance, $K_{d_{x}}^{A}$,
$K_{x_{img}}^{A}$ and $K_{y_{img}}^{A}$ are proportional gains of the
distance, $x$ and $y$ image position controllers respectively.

###  6.3 Visual Servo Brick Pickup

Once the Two-Stage Approach to the brick is finished, the reachability of the
magnetic patch is checked, and if the patch is reachable, the Visual Servo
Brick Pickup algorithm is executed. This state is performed solely by the
robot manipulator while the UGV remains idle. The brick pickup motion is
divided into four different stages: $x$ and $Pitch$ Visual Servo, $y$ Visual
Servo, $yaw$ Visual Servo and $z$ Approach, all of which are executed in the
robot frame $L_{B}$, to decouple the controllers used for the different
stages, simplifying the tuning process.

####  6.3.1 $x$ and $Pitch$ Visual Servo

During the $x$ and $Pitch$ Visual Servo stage of the visual servo algorithm,
the $x$ position of the electromagnetic gripper and the camera pitch angle are
controlled (Figure 12). The goal of this stage is to orient the camera
downward, while keeping the magnetic patch inside the camera image. Increasing
the camera pitch angle causes the magnetic patch to move upward in the image.
This is compensated for by moving the camera forward in the $x$ direction.

Figure 12: x and Pitch Visual Servo stage of the Brick Pickup behavior. The
end effector movement direction is described with respect to the robot base
frame $L_{B}$. Filtered camera images at the beginning and at the end of this
stage are shown on the left and right, respectively. In both images, the
position of the magnetic patch on the image $y$ axis is close to zero, the
main difference being the camera pitch angle, denoted $\theta_{start}$ and
$\theta_{end}$ in the middle image.

Two proportional controllers are active at the same time:

$\Delta\theta=K_{\theta}^{M}(\theta_{d}-\theta)$ (10) $\Delta
x=-K_{y_{p}}^{M}y_{p},$ (11)

where $\theta_{d}$ is the desired camera pitch angle, which is set to $\pi/2$
(camera facing downwards). $\Delta x$ represents the end effector displacement
in the $x$ direction.

This stage of the visual servo pickup is completed when the camera pitch angle
reaches $\pi/2$ and the $y$ image coordinate of the magnetic patch $y_{p}$ is
sufficiently close to zero (Figure 12).

####  6.3.2 $y$ and $yaw$ Visual Servo

Once $x$ and $Pitch$ Visual Servo is finished, the remaining $x_{p}$ and
orientation errors are eliminated with the $y$ Visual Servo and $yaw$ Visual
Servo proportional controllers. The following proportional controllers are
designed:

$\Delta y=-K_{x_{p}}^{M}x_{p}$ (12) $\Delta\psi=-K_{\psi_{p}}^{M}\psi_{p},$
(13)

where $\psi_{p}$ is the magnetic patch camera orientation, and $\Delta\psi$ is
the end effector yaw displacement (Figure 13).

Figure 13: Filtered camera image after the x and Pitch Visual Servo (left), y
Visual Servo (middle), and Yaw Visual Servo (right). The sequence shows the
progress of the pickup procedure. First, the end effector is positioned
directly over the center of the magnetic patch, then the effector is aligned
with the axes of the patch to ensure proper orientation of the bricks. The
final offset of the camera from the center of the patch accounts for the
displacement of the camera with respect to the center of the end effector.

####  6.3.3 $z$ Approach

$z$ Approach starts once the magnetic patch is sufficiently close to the image
center, with a sufficiently small $\psi_{p}$. At this stage, the center of the
magnetic patch is located directly bellow the RGB camera, so the
electromagnetic gripper is displaced to account for the offset between the
camera and gripper centers.

Distance measurements from the Realsense camera’s depth module are used to
estimate the distance between the magnetic patch and the electromagnetic
gripper $d_{z}$. Knowing that the height of the magnetic patch in the robot
base frame $L_{B}$ is $N$ times the height of a single brick $h_{b}$, where
$N$ is a positive integer, the final value of measurement $d_{z}$ is set to
the closest $Nh_{b}$ value. The following proportional controller

$\Delta z=K_{d_{z}}^{M}d_{z}$ (14)

is designed for the $z$ Approach and finishes at the first triggering of the
contact sensor.

##  7 Results

Discussion of the performance in the competition is given in section 7.1 and
presents results in an outdoor scenario under realistic, real-world
conditions. We conducted additional, post-competition experiments to evaluate
some of the previously described algorithms in an indoor environment, using
the Optitrack motion capture system to provide ground truth measurements.

###  7.1 Competition performance

The first real performance test for our system was the competition. The
Challenge 2 environment was designed as a single arena with two separate wall-
building sites, one for the UAVs and one for the UGV. Each of the two walls
was to be built using dedicated bricks. The bricks designated for the UGV were
set up as structured brick stacks, separated by color.

Each team had three $25$ minute runs to attempt the Challenge, either in
autonomous or manual mode. The location of the brick stacks and the wall
footprint changed in each run, but the layout of the Challenge was similar:
the brick stacks were positioned on the side of the arena closest to the UGV
and were visible from the starting position (especially since the starting
orientation of the UGV could be set arbitrarily by the team) while the
building site was somewhere on the other side of the arena, narrowing the area
for search.

In the three runs of the Challenge, the UGV managed to autonomously pick up
multiple bricks and place them in the baskets. In the second run a green brick
was placed on the wall footprint fully autonomously by the UGV, giving the
team a score of $0.44444$ points, which was enough for 5th place in the
Challenge 2 rankings.

It was difficult to collect data in the competition, and, consequently,
evaluate performance. However, certain observational conclusions were reached.
No significant problems were noticed regarding the navigation and mapping
algorithms. The two-dimensional maps built by Cartographer seemed to
correspond to the environment, without any evident flaws. An example of a
Challenge 2 arena map built during the competition is shown in Figure 10. The
TEB planner managed to produce and execute a feasible plan in every instance.
The planned trajectories satisfied the minimum-turning-radius constraint and
thus successfully resolved a previously described base-vibration problem. The
only evident drawback of move-base navigation is time-consumption, discussed
in the ”Limitations” section 8.1.

Variable lighting conditions caused detection problems that negatively
affected visual-feedback dependent algorithms. This issue was less pronounced
during local approach and more so in manipulation tasks. Apart from that, when
detection performed well, there were no problems with the manipulation.

###  7.2 Post-competition evaluation

To further test overall system performance, we conducted additional post-
competition experiments, using the Optitrack system as a benchmark to provide
ground truth data. In the first set of experiments, the goal was to evaluate
the performance of the Brick Loading process with two measures: error in
brick-pose estimation and brick-loading success rate.

The second set of experiments focused on evaluating the Brick Unloading
performance with similar measures: differences in estimated wall pose compared
to ground truth data and success rate of the algorithm. For both of the
aforementioned experiment sets, ground truth data was obtained by placing
Optitrack fiducials on the Husky, the bricks, and the wall pattern.

In the final experiment, the goal was to evaluate the autonomy of the system
performing the entire Challenge as well as the time required to complete the
mission.

###  7.3 Brick loading experiment

In this experimental scenario the robot is assumed to have located the brick
stack. To complete the brick loading task, the robot must identify an
individual brick of a given color, estimate its pose, navigate to it and pick
it up. All of these actions are performed fully autonomously. To evaluate the
brick pose-estimation, the brick and the UGV poses are recorded via Optitrack
at the beginning of the Alignment phase of the two-stage approach, and
compared to the pose obtained using the algorithm described in section 5. The
results of each experiment are shown in Table 1.

The experiment was repeated seven times, with a $100\%$ success rate. Table 1
shows that the developed algorithm can cope with various initial relative
orientations between the UGV and the brick. Since the estimated brick pose is
used only to navigate to a starting point for the Final Approach visual servo,
the system is robust enough to successfully pick up the brick even if the
distance to the brick is underestimated.

We determined experimentally that the configuration of the platform and the
robot arm allows for a perpendicular approach error of approximately
$26^{\circ}$. All observed errors fall comfortably within this range.

Table 1: Brick approach and loading experiment showing differences between the true (relative to the robot, measured in the Optitrack frame) and the estimated object pose (relative to the robot, measured in the robot frame $L_{B}$) for each of the experiments and the mean absolute error (MAE) across all experiments. Since a success rate of $100\%$ indicates that the position of the brick is correctly identified, we show only the total distance of the brick from the Husky and error in estimating this distance. Run no. |  Patch distance [$m$] |  Patch distance error [$m$] |  Patch orientation [∘] |  Patch orientation error [∘] |  Success
---|---|---|---|---|---
1 | 1.632 | -0.153 | -87.72 | -3.00 | Yes
2 | 1.673 | -0.173 | -105.37 | -5.43 | Yes
3 | 1.670 | -0.144 | -138.48 | -6.23 | Yes
4 | 1.702 | -0.159 | -161.46 | -7.96 | Yes
5 | 1.761 | -0.249 | -66.69 | -2.89 | Yes
6 | 1.625 | -0.143 | -89.38 | -6.47 | Yes
7 | 1.677 | -0.144 | -52.12 | 6.90 | Yes
| MAE[m] = 0.166 |  | MAE[∘]=5.5 |

###  7.4 Brick unloading experiment

The effectiveness of the brick unloading algorithm was evaluated in a similar
manner. The UGV is initially placed at an arbitrary distance from the wall
footprint, with an arbitrary relative orientation between the two. The robot
must position itself perpendicular to the footprint’s long axis and at a
predefined point relative to the rightmost point of the pattern. The
scenario’s final step places a brick on the wall, an action that comprises of
picking the brick up from the basket and unloading it in an appropriate
location on the template or wall. The results of each run and some summary
statistics are shown in Table 2.

The experiment was performed ten times with a success rate of $100\%$. The
outcome of the experiment is considered successful only if at least half of
the brick is placed inside the wall pattern. With respect to the estimation of
distance to the pattern, the system behaves similarly to brick loading, since
the final step is a visual servo to the patch that keeps the brick centered on
the pattern across its shorter axis. However, since the starting point for
this servo procedure is constructed based on the first pattern servo goal
(which is the rightmost point of the pattern), the error in brick placement
along the longer axis of the pattern is not accounted for, generating gaps
between bricks. As can be seen in the following section, the gaps between
bricks do not increase over time, suggesting that the longitudinal errors in
brick placement are, for the most part, generated by the TEB planner’s goal
tolerance and, to a lesser degree, by errors or drift in localization.

Table 2: Brick unloading experiment showing the effectiveness of the brick unloading procedure. The errors in orientation of the pattern are measured by comparing the orientation of the placed brick to the orientation of the pattern, measured in the Optitrack frame. The distance and the orientation of the pattern with respect to the robot when it is first detected is also shown. Run no. |  Pattern distance [$m$] |  Pattern orientation [∘] |  Brick placement orientation error [∘] |  Success
---|---|---|---|---
1 | 1.232 | 2.79 | 4.96 | Yes
2 | 0.934 | 52.22 | 1.41 | Yes
3 | 1.208 | 1.03 | 1.03 | Yes
4 | 1.263 | 28.05 | 4.91 | Yes
5 | 1.125 | 3.74 | 4.91 | Yes
6 | 1.269 | 0.57 | 6.85 | Yes
7 | 1.211 | 6.65 | 9.55 | Yes
8 | 1.321 | 1.68 | 9.88 | Yes
9 | 1.457 | 2.55 | 2.38 | Yes
10 | 1.115 | 2.73 | 4.31 | Yes
| MAE[∘]=5.02 |

###  7.5 Wall Building Scenario

An experiment to evaluate the entire autonomous wall building scenario was
performed in the Challenge mock-up environment built on the University campus,
shown in Figure 1, with the experiment layout as shown in Figure 14. The
experiment layout is inspired by the one in the Challenge. The bricks are set
up as structured brick stacks separated by color, and the wall pattern is
placed on the other side of the arena. Similarly to the competition, there are
no obstacles between the brick stacks and the building site.

Figure 14: Path traversed during the fully autonomous wall building
experiment. The wall footprint pattern, brick stack position, and the UGV
starting position are shown in a 2D map built during the experiment.
Dimensions of the arena are $10\times 7.5m$. Distance between the UGV start
position and the brick stacks is $7m$. The distance between the starting
position and the wall footprint is $3.5m$.

The system is provided with a desired wall configuration, along with the
approximate initial position of the brick stacks, and that of the wall
footprint. The wall building speed highly depends on the desired wall
structure, as different colored bricks take up different amounts of space in
the brick baskets. This situation makes some wall structures less desirable
speed-wise, as they require multiple trips between the brick stacks and the
wall footprint to complete the wall. Another factor that affects the overall
speed of the system is the Challenge layout: the initial brick stacks
position, the initial UGV position, and the wall-footprint position.
Performing full runs of the experiment is time consuming, but from the several
runs that were completed, it was observed that the UGV is capable of
transporting and laying 6 to 10 bricks within $25$ minutes 111Time limit for
the Challenge, depending on the wall configuration. The final result of such a
run is shown in Figure 15.

Figure 15: Six brick wall built in an autonomous full-scale MZBIRC 2020
Challenge 2 experiment. Total of three trips between brick stacks and wall
were required and all bricks were placed within the boundaries as proposed by
the competition.

The wall, consisting of two blue, two green, and two red bricks was
successfully built without requiring any type of manual intervention. It took
the robot $23$ minutes and $37$ seconds to perform the entire experiment, and
all bricks were placed with sufficient precision to satisfy Challenge rules. A
video of the experiment is available at https://youtu.be/0C_E7tGCZ5k.

Overall, the experiments shown in this paper validate our approach and confirm
that precision of the constrols, 3D-map localization, navigation and the pose-
estimation from RGB images satisfy the Challenge requirements. Finally, our
custom-developed, passively-compliant, electromagnetic gripper successfully
addressed the end-effector misalignment problem in every brick loading and
unloading procedure.

##  8 Conclusion

This paper presented the LARICS team’s approach to the Challenge 2 of MBZIRC
2020 competition, as well as the post-competition experimental setup and
results. Both the brick loading and unloading experiments were performed with
a $100\%$ success rate, the latter of which resulted in brick placement
sufficiently precise to satisfy Challenge rules. During the full Challenge
experiment, a wall consisting of two red, two green and two blue bricks, shown
in Figure 15, was built autonomously. The entire trial lasted $23$ minutes and
$37$ seconds.

###  8.1 Limitations

Our approach described in this work is specifically designed for a semi-
structured scenario, as described by the Challenge 2 rules, and as such does
not generalize to other non-structured scenarios.

The only obstacles in the competition arena were the brick stacks and an UAV
wall-building site (as seen in Figure 10). Even though the TEB planner takes
obstacles into account, a more crowded environment could introduce obstacle-
avoidance problems. Since the planner is configured to use a large minimal-
turning-radius, in environments with many obstacles it may be unable to
produce a feasible plan due to numerous constraints. This conflict could be
tackled by reducing the minimum-turning-radius or using a different planner.

In the Challenge, the bricks were initially structured in stacks separated by
color, which organization was exploited in the local approach and manipulation
algorithms. The local approach does not take obstacles into account, as long
as they do not interfere with the visual feedback. This constraint could
present a problem in a more general scenario, if there are obstacles in close
proximity to the target object. Likewise, the manipulation algorithms were
designed with a structured brick stack in mind and would require modifications
in the case of a chaotic brick pile.

Object detection based on HSV filtering presents an obvious general-case
limitation in terms of object and environment texture. The bricks in the
competition were color-coded to stand out from the environment. Nevertheless,
detection proved to be difficult in the Challenge 2 scenario. Each trial was
conducted under different sunlight conditions, forcing the adjustment of HSV
thresholds used for both brick and wall-footprint detection. Brick-stack and
wall-footprint detection, which are solely color-based, performed well even
with a less than optimal HSV thresholds, while grasp-point patch-detection,
which is based on both color and shape, was more error-prone, especially in
cases where the manipulator creates a moving shadow on the brick.

In our post-Challenge tests, bricks were placed on the wall-footprint with an
orientation MAE of $5.02^{\circ}$, and a more significant position error as
evidenced by gaps between bricks. These errors were more pronounced in a full
challenge scenario, due to the fact that the UGV does not detect the wall
pattern’s pose at each brick drop, but memorizes its map pose and reuses it.
This issue could be avoided using fresh detection of the previously placed
bricks or the wall pattern edge during the alignment, but was not found to be
necessary.

There is room for improvement in speed. Most of the time is spent in the
navigation stages of the Challenge, mainly in the stages that involve Move
Base motion planning. This situation is due to the TEB planner re-planning
until the goal pose-tolerance is satisfied. Since the trajectory produced by
the TEB planner is constrained by a minimum-turning-radius, a small error in
the achieved pose requires a non-trivial maneuver to eliminate it.
Furthermore, we suspect that the manipulation speed could be improved using
not only the image position of the detected magnetic patch, but also its
estimated global pose to plan end-effector motion.

#### Acknowledgments

Research work presented in this article has been supported in part by Khalifa
University of Science and Technology, Abu Dhabi, UAE, by European Commission
Horizon 2020 Programme through project under G. A. number 810321, named
Twinning coordination action for spreading excellence in Aerial Robotics -
AeRoTwin, and project Heterogeneous autonomous robotic system in viticulture
and mariculture (HEKTOR) financed by the European Union through the European
Regional Development Fund-The Competitiveness and Cohesion Operational
Programme (KK.01.1.1.04.0041). Work of Frano Petric was supported by the
European Regional Development Fund under the grant KK.01.1.1.01.0009
(DATACROSS).

## References

  * Ahn et al., 2019 Ahn, M. S., Chae, H., Noh, D., Nam, H., and Hong, D. (2019).  Analysis and noise modeling of the intel RealSense d435 for mobile robots.  In 2019 16th International Conference on Ubiquitous Robots (UR). IEEE.
  * Bochkovskiy et al., 2020 Bochkovskiy, A., Wang, C.-Y., and Liao, H.-Y. M. (2020).  Yolov4: Optimal speed and accuracy of object detection.
  * Chaumette and Hutchinson, 2006 Chaumette, F. and Hutchinson, S. (2006).  Visual servo control. i. basic approaches.  IEEE Robotics Automation Magazine, 13(4):82–90.
  * Correll et al., 2018 Correll, N., Bekris, K. E., Berenson, D., Brock, O., Causo, A., Hauser, K., Okada, K., Rodriguez, A., Romano, J. M., and Wurman, P. R. (2018).  Analysis and observations from the first amazon picking challenge.  IEEE Transactions on Automation Science and Engineering, 15(1):172–188.
  * Davila Delgado et al., 2019 Davila Delgado, J. M., Oyedele, L., Ajayi, A., Akanbi, L., Akinade, O., Bilal, M., and Owolabi, H. (2019).  Robotics and automated systems in construction: Understanding industry-specific challenges for adoption.  Journal of Building Engineering, 26:100868.
  * Decker and Lesser, 1995 Decker, K. S. and Lesser, V. R. (1995).  Designing a family of coordination algorithms.  In Proceedings of the First International Conference on Multi-Agent System, pages 73–80. AAAI Press.
  * Fischler and Bolles, 1981 Fischler, M. A. and Bolles, R. C. (1981).  Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.  Commun. ACM, 24(6):381–395.
  * Gawel et al., 2019 Gawel, A., Blum, H., Pankert, J., Krämer, K., Bartolomei, L., Ercan, S., Farshidian, F., Chli, M., Gramazio, F., Siegwart, R., Hutter, M., and Sandy, T. (2019).  A fully-integrated sensing and control system for high-accuracy mobile robotic building construction.
  * Girshick, 2015 Girshick, R. (2015).  Fast r-CNN.  In 2015 IEEE International Conference on Computer Vision (ICCV). IEEE.
  * Grabner et al., 2018 Grabner, A., Roth, P. M., and Lepetit, V. (2018).  3d pose estimation and 3d model retrieval for objects in the wild.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3022–3031.
  * Hess et al., 2016 Hess, W., Kohler, D., Rapp, H., and Andor, D. (2016).  Real-time loop closure in 2d lidar slam.  In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 1271–1278.
  * Horling et al., 1999 Horling, B., Lesser, V., Vincent, R., Wagner, T., Raja, A., Zhang, S., Decker, K., and Garvey, A. (1999).  The taems white paper.
  * Krizmancic et al., 2020 Krizmancic, M., Arbanas, B., Petrovic, T., Petric, F., and Bogdan, S. (2020).  Cooperative aerial-ground multi-robot system for automated construction tasks.  IEEE Robotics and Automation Letters, 5(2):798–805.
  * Kundu et al., 2018 Kundu, J. N., RahulM., V., Ganeshan, A., and Babu, R. V. (2018).  Object pose estimation from monocular image using multi-view keypoint correspondence.  ArXiv, abs/1809.00553.
  * Lesser et al., 2004 Lesser, V., Decker, K., Wagner, T., Carver, N., Garvey, A., Horling, B., Neiman, D., Podorozhny, R., Prasad, M. N., Raja, A., Vincent, R., Xuan, P., and Zhang, X. Q. (2004).  Evolution of the GPGP/TAEMS domain-independent coordination framework.  Autonomous Agents and Multi-Agent Systems, 9(1/2):87–143.
  * Matamoros et al., 2019 Matamoros, M., Seib, V., and Paulus, D. (2019).  Trends, challenges and adopted strategies in robocup@home.  In 2019 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pages 1–6.
  * MBZIRC, 2020 MBZIRC (2020).  The challenge 2020.  Available: https://www.mbzirc.com/challenge/2020, Accessed: 30 September 2020.
  * Melenbrink et al., 2020 Melenbrink, N., Werfel, J., and Menges, A. (2020).  On-site autonomous construction robots: Towards unsupervised building.  Automation in Construction, 119:103312.
  * Milijas et al., 2020 Milijas, R., Markovic, L., Ivanovic, A., Petric, F., and Bogdan, S. (2020).  A comprehensive lidar-based slam comparison for control of unmanned aerial vehicles.
  * Muslikhin et al., 2020 Muslikhin, Horng, J.-R., Yang, S.-Y., and Wang, M.-S. (2020).  Object localization and depth estimation for eye-in-hand manipulator using mono camera.  IEEE Access, 8:121765–121779.
  * Nieuwenhuisen et al., 2017 Nieuwenhuisen, M., Beul, M., Rosu, R. A., Quenzel, J., Pavlichenko, D., Houben, S., and Behnke, S. (2017).  Collaborative object picking and delivery with a team of micro aerial vehicles at MBZIRC.  In 2017 European Conference on Mobile Robots (ECMR). IEEE.
  * Opromolla et al., 2015 Opromolla, R., Fasano, G., Rufino, G., and Grassi, M. (2015).  A model-based 3d template matching technique for pose acquisition of an uncooperative space object.  Sensors, 15(3):6360–6382.
  * Pritschow et al., 1996 Pritschow, G., Dalacker, M., Kurz, J., and Gaenssle, M. (1996).  Technological aspects in the development of a mobile bricklaying robot.  Automation in Construction, 5(1):3–13.  12th ISARC.
  * Reyes-Lúa et al., 2019 Reyes-Lúa, A., Zotică, C., Forsman, K., and Skogestad, S. (2019).  Systematic design of split range controllers.  IFAC-PapersOnLine, 52(1):898–903.
  * Rösmann et al., 2017 Rösmann, C., Hoffmann, F., and Bertram, T. (2017).  Integrated online trajectory planning and optimization in distinctive topologies.  Robotics and Autonomous Systems, 88:142–153.
  * Sock et al., 2020 Sock, J., Castro, P., Armagan, A., Garcia-Hernando, G., and Kim, T.-K. (2020).  Tackling two challenges of 6d object pose estimation: Lack of real annotated rgb images and scalability to number of objects.
  * Tsay and Chang, 2004 Tsay, T. and Chang, C. (2004).  Pose control ofmobile manipulators with an uncalibrated eye-in-hand vision system.  In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566). IEEE.
  * Vasiljević et al., 2016 Vasiljević, G., Miklić, D., Draganjac, I., Kovačić, Z., and Lista, P. (2016).  High-accuracy vehicle localization for autonomous warehousing.  Robotics and Computer-Integrated Manufacturing, 42:1–16.
  * Vock et al., 2019 Vock, R., Dieckmann, A., Ochmann, S., and Klein, R. (2019).  Fast template matching and pose estimation in 3d point clouds.  Computers & Graphics, 79:36–45.
  * Wulf et al., 2004 Wulf, O., Brenneke, C., and Wagner, B. (2004).  Colored 2d maps for robot navigation with 3d sensor data.  In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), volume 3, pages 2991–2996 vol.3.
  * Xiang et al., 2018 Xiang, Y., Schmidt, T., Narayanan, V., and Fox, D. (2018).  PoseCNN: A convolutional neural network for 6d object pose estimation in cluttered scenes.  In Robotics: Science and Systems XIV. Robotics: Science and Systems Foundation.