# Tunneling in soft waveguides: closing a book

Pavel Exner  Nuclear Physics Institute, Czech Academy of Sciences, Hlavní 130,
25068 Řež near Prague, Czech Republic  Doppler Institute, Czech Technical
University, Břehová 7, 11519 Prague,
Czech Republic<EMAIL_ADDRESS>David Spitzkopf  Nuclear Physics Institute,
Czech Academy of Sciences, Hlavní 130,
25068 Řež near Prague, Czech Republic  Faculty of Mathematics and Physics,
Charles University, V Holešovičkách 2, 18000 Prague
Czech Republic<EMAIL_ADDRESS>
###### Abstract

We investigate the spectrum of a soft quantum waveguide in two dimensions of
the generalized ‘bookcover’ shape, that is, Schrödinger operator with the
potential in the form of a ditch consisting of a finite curved part and
straight asymptotes which are parallel or almost parallel pointing in the same
direction. We show how the eigenvalues accumulate when the angle between the
asymptotes tends to zero. In case of parallel asymptotes the existence of a
discrete spectrum depends on the ditch profile. We prove that it is absent in
the weak-coupling case, on the other hand, it exists provided the transverse
potential is strong enough. We also present a numerical example in which the
critical strength can be assessed.

###### pacs:

03.65.Ge, 03.65Db

††: J. Phys. A: Math. Gen.

Keywords: Quantum waveguides, Schrödinger operators, ditch-shaped potentials,
discrete spectrum, tunneling effect

##  1 Introduction

Quantum waveguides have been a subject of intense investigation in the
previous three or four decades; for a survey (except the last few years) and
an extensive bibliography we refer to [EK15]. They serve as models of numerous
physical systems, and at the same time, they represent a source of interesting
mathematical problems. The nature of the interaction responsible for the
guiding may differ. Often the walls of the waveguide are hard, meaning
mathematically that the Hamiltonian is Dirichlet Laplacian in an appropriate
spatial region. Alternatively singular Schrödinger operators are used with an
attractive interaction supported by curves, surfaces, or more complicated sets
of lower dimensionality in the configuration space.

Recently an alternative, more realistic model of ‘soft’ waveguides attracted
attention, cf. [Ex20] and further developments in [KKK21, EL21, EKL22, EV23],
where the interaction term is a regular potential ‘ditch’ the axis of which is
a fixed curve. In such a description the tunneling between different parts of
the guide is not suppressed while its transverse width need not be zero. The
mentioned work dealt with the two-dimensional situation; there are extensions
to three dimensions describing soft quantum tubes [Ex22] and layers [EKP20,
KK23], or even to situations where the ditch is replaced by an array of
potential wells [Ex23].

The most remarkable feature of these systems are relations between their
geometry and the energy spectrum, in particular, the fact that bending gives
rise to an effective attractive interaction able to produce localized states
in an infinitely extended ditch. This property is shared with the other
waveguide types mentioned above, however, in contrast to them, the existence
results obtained so far lack the universal character of their ‘hard’ or
‘singular’ counterparts. For two-dimensional Schrödinger operator with a
$\delta$ interaction on a non-straight curve, for instance, Birman-Schwinger
method yields the discrete spectrum existence under rather weak regularity and
asymptotic straightness requirements [EI01], while the sufficient condition
obtained by the same method in the soft case [Ex20] is much weaker.

As an alternative, one may apply the variational method to the operator
directly; the difficulty with this approach is that in general it is not easy
to construct a suitable trial function. In [KKK21] a simple case, usually
dubbed _bookcover_ in the literature [SM90], was treated; the existence of
bound states was proved for any bending angle $\theta\in(0,\pi)$ between the
straight parts of the guide and any channel profile such that the transverse
part of the operator has a discrete eigenvalue. Two questions have been left
open, both related to the tunneling between the guide arms, namely the
accumulation rate of the eigenvalues as the ‘book’ closes, $\theta\to\pi-$,
and the existence of the discrete spectrum in the limiting case, $\theta=\pi$.

A considerably more general existence result was obtained by variational
method in [EV23] allowing the finite non-straight part to be built over a
$C^{3}$ smooth curve, while the semi-infinite asymptotic parts were supposed
to be straight as before. The above two questions remained open again, and
goal of this paper is to address them in the less restrictive geometrical
setting of [EV23] where the ‘spine’ of the book may have an arbitrary shape
provided it is finite and sufficiently regular. As in [EL21] we characterize
the ditch profile by a measure type potential which allows us to treat on the
same footing regular and singular interactions; recall that for the latter the
said two questions are also open.

In the next section we collect the necessary notions and hypotheses, we also
find the essential spectrum which differs in the parallel and non-parallel
case due to the different character of the tunneling in these two situations.
Section 3 is devoted to ‘closing’ the book. Using the angle $\beta=\pi-\theta$
as a more natural parameter in this case, we show the curvature-induced
eigenvalues fill the difference between the two essential spectra in the limit
$\beta\to 0$, their number in any subinterval behaving as $\sim\beta^{-1}$.

After that we turn to the situation when the asymptotes are parallel,
$\beta=0$. In Section 4 we consider the weak-coupling case in which the
channel profile is shallow; we show that the discrete spectrum is then empty.
The opposite extreme is the topic of Section 5. The strong-coupling
asymptotics offer more than one regime and we discuss three of them, a
singular $\delta$ potential, a rectangular well of a non-flat bottom, and a
sufficiently deep and narrow profile; in all of them we prove that strong
coupling gives rise to discrete eigenvalues. As a byproduct, we generalize in
Sec. 5.2 the well-known result about the discrete spectrum existence [EK15,
Thm. 1.1] to a class of Schrödinger operators in curved Dirichlet strips. As
finding the critical interaction strength for U-shaped soft waveguides and
their generalizations is out of reach analytically, we present in Section 6 a
numerical example of an U-shaped channel with a specific transversal profile
to illustrate bound states. Furthermore, we present computations regarding the
dependence of the critical interaction strength on the geometry of the
waveguide.

##  2 Problem setting

The object of our interest are two-dimensional Schrödinger operators with a
potential in the shape of a ‘ditch’ of a finite width and fixed profile. Let
us describe the setting in more precise terms starting from the channel axis
$\Gamma$. We suppose it is an infinite and smooth planar curve without self-
intersections, naturally parametrized by its arc length $s$, that is, the
graph of a $C^{1}$ function $\Gamma:\>\mathbb{R}\to\mathbb{R}^{2}$ such that
$\dot{\Gamma}(s)=1$, where the dot conventionally denotes the derivative with
respect to $s$. With an abuse of notation we employ the same symbol for the
map $\Gamma$ and for its range. If $\Gamma$ is $C^{2}$, its signed curvature
is
$\gamma:\>\gamma(s)=(\dot{\Gamma}_{2}\ddot{\Gamma}_{1}-\dot{\Gamma}_{1}\ddot{\Gamma}_{2})(s)$
being naturally independent of the Cartesian coordinates used. The curve
$\Gamma$ is supposed to satisfy the following assumption:

  1. (a)

$\Gamma$ is piecewise $C^{2}$-smooth and its curvature is of compact support.

Without loss of generality we may suppose that the straight parts of $\Gamma$
are the halflines
$\Gamma_{\pm}:=\big{\\{}\big{(}x,\pm(\rho+x\tan(\frac{1}{2}\beta)\big{)}:\,x\geq
0\big{\\}}$ for a positive $\rho$ and $\beta\in[0,\frac{1}{2}\pi)$, symmetric
with respect to the $x$ axis, parametrized as

$\Gamma_{\pm}:=\Big{\\{}\big{(}(\pm
s-s_{0})\cos(\textstyle{\frac{1}{2}}\beta),\rho+(s\mp
s_{0})\sin(\textstyle{\frac{1}{2}}\beta)\big{)}:\,\pm s\geq s_{0}\Big{\\}}$
(2.1)

for some $s_{0}>2\rho$. The middle part of the curve referring to
$s\in(-s_{0},s_{0})$ will be denoted as $\Gamma_{\mathrm{int}}$; without loss
of generality again, we may suppose that it is contained in the open left
halfplane, $x<0$. Recall that
$\theta(s_{2},s_{1}):=\int_{s_{1}}^{s_{2}}\gamma(s)\,\mathrm{d}s$ is the angle
between the tangents at the points $\Gamma(s_{j}),\,j=1,2$, hence

$\beta=\pi-\int_{-s_{0}}^{s_{0}}\gamma(s)\,\mathrm{d}s\,;$ (2.2)

we will be interested in the situations when $\beta=0$ and in the asymptotic
regime $\beta\to 0+$.

Let us turn to the potential. We suppose that it is restricted transversally
being a subset of the strip

$\Omega^{a}:=\\{\textbf{x}\in\mathbb{R}^{2}:\>\mathrm{dist}(\textbf{x},\Gamma)<a\\},$
(2.3)

of a halfwidth $a>0$ built over $\Gamma$. We suppose that $a$ is small enough
so that we can parametrize $\Omega^{a}$ using the parallel (often called
Fermi) coordinates, given by the arc length $s$ and the distance $u$ along the
normal $N(s)=(-\dot{\Gamma}_{2}(s),\dot{\Gamma}_{1}(s))$ to $\Gamma$ at the
point $s$,

$\textbf{x}(s,u)=\big{(}\Gamma_{1}(s)-u\dot{\Gamma}_{2}(s),\Gamma_{2}(s)+u\dot{\Gamma}_{1}(s)\big{)},$
(2.4)

in such a way that

  1. (b)

the map $\mathbb{R}\times I_{a}\to\Omega^{a}$ defined by (2.4) is a
diffeomorphism,

where $I_{a}:=(-a,a)$; note that it can be satisfied, in particular, only if
$a<\rho$.

The channel profile will be determined by the one-dimensional Schrödinger
operator

$h_{v}=-\textstyle{\frac{\mathrm{d}^{2}}{\mathrm{d}x^{2}}}-v(x).$ (2.5)

The potential $v$ is always supposed to be _nontrivial_ ; when the interaction
strength will be of importance we will replace $h_{v}$ by $h_{\lambda v}$ with
$\lambda>0$. Mostly we will not strive for the maximum generality focusing
primarily on two situations; in the first

  1. (c)

$v\in L^{\infty}(\mathbb{R})$ with $\mathrm{supp}\,v\subset(-a,a)$ and
$\inf\sigma(h_{v})<0$ ;

needless to say one then has $D(h_{v})=H^{2}(\mathbb{R})$. Moreover, we have
$\sigma_{\mathrm{ess}}(h_{v})=[0,\infty)$ and
$\epsilon_{v}:=\inf\sigma(h_{v})<0$ is a simple eigenvalue. Using then the map
(2.4) we define

$\displaystyle V:\>\Omega^{a}$ $\displaystyle\to\mathbb{R}_{+},\quad
V(\textbf{x}(s,u))=v(u),$ (2.6a) $\displaystyle H_{\Gamma,V}$
$\displaystyle=-\Delta-V(\textbf{x});$ (2.6b)

in view of assumption (c) the operator domain is
$D(-\Delta)=H^{2}(\mathbb{R}^{2})$. The second case of interest is the _leaky
curve_ when instead of (2.6b) we consider the operator

  1. (d)

$H_{\Gamma,\alpha}$ associated with the quadratic form
$\psi\mapsto\int_{\mathbb{R}^{2}}|\nabla\psi(\textbf{x})|^{2}\mathrm{d}\textbf{x}-\alpha\int_{\mathbb{R}}|\psi(\Gamma(s))|^{2}\mathrm{d}s$,
where $\alpha>0$, defined on $H^{1}(\mathbb{R}^{2})$.

Alternatively one can say that the self-adjoint operator $H_{\Gamma,\alpha}$
acts as the negative Laplacian on functions $\psi\in H^{1}(\mathbb{R}^{2})\cap
H^{2}_{\mathrm{loc}}(\mathbb{R}^{2}\setminus\Gamma)$ which have at points of
$\Gamma$ the jump of the normal derivative equal to $-\alpha\psi(\textbf{x})$;
formally one can write
$H_{\Gamma,\alpha}=-\Delta-\alpha\delta(\textbf{x}-\Gamma)$. It may be
regarded as a singular version of the previous case, namely the family
$H_{\Gamma,V_{\varepsilon}}$ corresponding to the scaled transverse potentials
$v_{\varepsilon}(u)=\frac{1}{\varepsilon}v(\frac{u}{\varepsilon})$ converges
in the norm resolvent sense to $H_{\Gamma,\alpha}$ with
$\alpha:=\int_{\mathbb{R}}v(u)\,\mathrm{d}u\>$ as $\varepsilon\to 0\>$ [EI01,
BEHL17]. The analogue of the operator (2.5), formally
$-\textstyle{\frac{\mathrm{d}^{2}}{\mathrm{d}x^{2}}}-\alpha\delta(x)$, will be
denoted as $h_{\alpha}$; its ground state is in this case known explicitly,
$\epsilon_{\alpha}:=\inf\sigma(h_{\alpha})=-\frac{1}{4}\alpha^{2}$.

While the two situations are of primary interest to us, they are particular
cases of a more general channel profile. In the spirit of [EL21] one can
consider the operator

  1. (e)

$H_{\Gamma,\mu}$ associated with the quadratic form
$\psi\mapsto\int_{\mathbb{R}^{2}}|\nabla\psi(\textbf{x})|^{2}\mathrm{d}\textbf{x}-\int_{\mathbb{R}^{2}}|\psi(\textbf{x})|^{2}\mathrm{d}\mu(\textbf{x})$
defined on $H^{1}(\mathbb{R}^{2})$, where
$\mathrm{d}\mu(\textbf{x})=(1-u\gamma(s))\,\mathrm{d}s\mathrm{d}\mu_{\perp}(u)$
corresponds to a finite positive Borel measure $\mu_{\perp}$ supported on the
interval $(-a,a)$;

the assumptions (c) and (d) correspond respectively to
$\mathrm{d}\mu_{\perp}(u)=v(u)\mathrm{d}u$ and
$\mathrm{d}\mu_{\perp}(u)=\alpha\delta$, where $\delta$ is the Dirac measure
supported at the origin. The analogue of (2.5), the self-adjoint operator
associated with the form
$f\mapsto\int_{\mathbb{R}}|f^{\prime}(u)|^{2}\mathrm{d}u-\int_{\mathbb{R}}|f(u)|^{2}\mathrm{d}\mu_{\perp}(u)$
defined on $H^{1}(\mathbb{R})$, will be denoted as $h_{\mu_{\perp}}$; by
assumption we have $\epsilon_{\mu_{\perp}}:=\inf\sigma(h_{\mu_{\perp}})<0$.

In addition to (2.5) we need also its double-well counterpart

$h_{v,\rho}=-\textstyle{\frac{\mathrm{d}^{2}}{\mathrm{d}x^{2}}}-v(\rho+x)-v(-\rho-x)$
(2.7)

and its analogues $h_{\alpha,\rho}$ and $h_{\mu_{\perp},\rho}$ corresponding
to assumptions (d) and (e), respectively. The essential spectrum of these
operators is, of course, $\mathbb{R}_{+}$ and their ground-state eigenvalues
will be denoted as $\epsilon_{v,\rho}$ and analogously in the other two
situations.

######  Proposition 2.1.

Assume (a), (b) and (e), then
$\sigma_{\mathrm{ess}}(H_{\Gamma,\mu})=[\epsilon_{\mu_{\perp}},\infty)$ holds
provided $\beta>0$. If $\beta=0$, we have
$\sigma_{\mathrm{ess}}(H_{\Gamma,\mu})=[\epsilon_{\mu_{\perp},\rho},\infty)$
where the function $\rho\mapsto\epsilon_{\mu_{\perp},\rho}$ is monotonously
increasing with
$\epsilon_{\mu_{\perp},\infty}:=\lim_{\rho\to\infty}\epsilon_{\mu_{\perp},\rho}=\epsilon_{\mu_{\perp}}$.

###### Proof.

Under the assumptions (c) or (d) the essential spectrum was determined in
[Ex20, Prop. 3.1] and [EI01, Prop. 5.1] for $\beta>0$, the case $\beta=0$ in
the bookcover situation was dealt with in [KKK21, Prop. 2]. The more general
waveguide satisfying the present assumptions can be treated similarly; the
core of the argument is the existence of an increasing family of rectangles on
which the potential channel is straight (or the parallel potential channels
are straight for $\beta=0$) so that one can separate the variables and
construct a suitable Weyl sequence combining transversally the ground-state
eigenfunction of $h_{\mu_{\perp}}$ or $h_{\mu_{\perp},\rho}$, restricted to
intervals of a growing length, with a mollifier in the longitudinal direction.
The interaction term in $h_{\mu_{\perp},\rho}$ is mirror-symmetric, hence its
ground state coincides with that of the single well and the Neumann boundary
condition at the distance $\rho$ from its ‘center’. By the bracketing argument
[RS78, Sec. XIII.15] the function $\rho\mapsto\epsilon_{\mu_{\perp},\rho}$ is
non-decreasing. In fact, it is increasing with the indicated limit; one way to
see that is to use the formula
$\frac{\mathrm{d}}{\mathrm{d}\rho}\epsilon_{\mu_{\perp},\rho}=-\varphi_{\mu_{\perp},\rho}(0)^{2}\,\epsilon_{\mu_{\perp},\rho}$,
where $\varphi_{\mu_{\perp},\rho}$ is the real-valued eigenfunction associated
with $\epsilon_{\mu_{\perp},\rho}$, obtained by an easy modification of the
argument used in [DH93] (an alternative is to employ an Agmon-type estimate
which shows that the convergence is even exponentially fast). ∎

##  3 Closing the book

Let us now investigate what happens if we are ‘closing the book’, in other
words, how the spectrum of $H_{\Gamma,\mu}$ behaves asymptotically in the
limit $\beta\to 0$; our aim is to show that the eigenvalues will then fill the
gap between the corresponding essential spectrum thresholds,
$\epsilon_{\mu_{\perp},\rho}$ and $\epsilon_{\mu_{\perp}}$. Specifically, we
are going to show that for any
$\nu\in(\epsilon_{\mu_{\perp},\rho},\epsilon_{\mu_{\perp}})$ there are
eigenvalues of $H_{\Gamma,\mu}$ in the interval
$(\epsilon_{\mu_{\perp},\rho},\nu)$ for $\beta$ small enough and to estimate
their accumulation rate.

######  Theorem 3.1.

Assume (a), (b) and (e), then for any
$\nu\in(\epsilon_{\mu_{\perp},\rho},\epsilon_{\mu_{\perp}})$ there is a
constant $C_{\nu}>0$ such that $\dim
E_{H_{\Gamma,\mu}}(\epsilon_{\mu_{\perp},\rho},\nu)\geq C_{\nu}\beta^{-1}$
holds for the corresponding spectral projection of $H_{\Gamma,\mu}$ provided
that $\beta$ is small enough.

###### Proof.

It is sufficient to find the appropriate number of linearly independent test
functions for which the quadratic form associated with $H_{\Gamma,\mu}$
satisfies $\mathbf{q}[\phi]<\nu\|\phi\|^{2}$. We will construct such functions
with the support in the part of the plane where $\Omega^{a}$ is straight,
choosing them even with the respect to the axis of the angle between the two
asymptotes, $\phi(x,y)=\phi(x,-y)$. In the upper halfplane we choose a pair of
points of $\Gamma$ with the coordinates $s_{1}$, which can without loss of
generality refer to the place where the straight parts begins, and
$s_{2}>s_{1}$. The support of $\phi$ for $y\geq 0$ will be a semiinfinite
strip $\Sigma$ the boundaries of which will be lines normal to $\Gamma$ at the
points $s_{1},s_{2}$; we divide it into the rectagular part $\Sigma_{r}$ and
the wedge-shaped $\Sigma_{w}$ as sketched in Fig. 1.

Figure 1: The regions used in the proof of Theorem 3.1

It is obvious that in $\Sigma$ one can use the parallel coordinates; those
referring to Cartesian $(x,y)$ will be denoted as $s(x,y)$ and $u(x,y)$. With
this licence we use the following test function Ansatz,

$\phi(x,y)\vcentcolon=\big{[}\chi_{\Sigma_{r}}(x,y)\varphi_{\rho_{\beta}}(u(x,y)+\rho_{\beta})+\chi_{\Sigma_{w}}(x,y)\varphi_{\rho_{\beta}}(0)\big{]}f(s(x,y)),\;\;y\geq
0,$ (3.1)

where $\varphi_{\rho_{\beta}}$ is the eigenfunction associated with the
eigenvalue $\epsilon_{\mu_{\perp},\rho_{\beta}}$ of operator (2.7) (for
simplicity we drop the subscript $\mu_{\perp}$),
$\rho_{\beta}=\rho\sec\frac{\beta}{2}$, and

$f(s)\in
C^{2}\left(\mathbb{R}\right)\quad\text{with}\quad\mathrm{supp\,}f\subset\left[s_{1},s_{2}\right],\;f\left(s_{1}\right)=f\left(s_{2}\right)=0\,;$
(3.2)

the functions (3.1) are obviously admissible test functions belonging to
$H^{1}(\mathbb{R}^{2})$.

Let us now express the quadratic form starting from its kinetic term,
$\|\nabla\phi\|^{2}$. Using the symmetry of $\phi$ and the fact that the
parallel coordinates in $\Sigma$ are obtained by rotation of the Cartesian
ones, we get

$\displaystyle\|\nabla\phi\|^{2}$
$\displaystyle=2\int_{\Sigma}|\nabla_{s,u}\phi(x(s,u),y(s,u))|^{2}\,\mathrm{d}s\mathrm{d}u$
(3.3)
$\displaystyle=2\int_{\Sigma_{r}}\big{[}|f^{\prime}(s)|^{2}|\varphi_{\rho_{\beta}}(u+\rho_{\beta}))|^{2}+|f(s)|^{2}|\varphi_{\rho_{\beta}}^{\prime}(u+\rho_{\beta}))|^{2}\big{]}\,\mathrm{d}s\mathrm{d}u$
$\displaystyle\quad+2\int_{\Sigma_{w}}|f^{\prime}(s)|^{2}|\varphi_{\rho_{\beta}}(0)|^{2}\,\mathrm{d}s\mathrm{d}u$
$\displaystyle=\|f^{\prime}\|^{2}\|\varphi_{\rho_{\beta}}\|^{2}+\|f\|^{2}\|\varphi^{\prime}_{\rho_{\beta}}\|^{2}+\|f^{\prime}\|^{2}|\varphi_{\rho_{\beta}}(0)|^{2}L\tan\textstyle{\frac{\beta}{2}},$

where $L:=s_{2}-s_{1}$, the norms refer to $L^{2}(s_{1},s_{2})$ and
$L^{2}(\mathbb{R})$, respectively, and when integrating over $u$ we used the
fact that $\varphi_{\rho_{\beta}}$ is an even function. For the interaction
part of the quadratic form we get similarly

$\|\phi\|^{2}_{\mu}=2\int_{s_{1}}^{s_{2}}\int_{-a}^{a}|f(s)|^{2}|\varphi_{\rho_{\beta}}(u)|^{2}\,\mathrm{d}\mu_{\perp}(u)\mathrm{d}s=\|f\|^{2}\int_{\mathbb{R}}|\varphi_{\rho_{\beta}}(u)|^{2}\,\mathrm{d}\mu_{\perp}(u)$
(3.4)

and the norm of the test function is

$\|\phi\|^{2}=\|f\|^{2}\big{(}\|\varphi_{\rho_{\beta}}\|^{2}+|\varphi_{\rho_{\beta}}(0)|^{2}L\tan\textstyle{\frac{\beta}{2}}\big{)}.$
(3.5)

Putting now (3.3)–(3.5) together, we can express the shifted quadratic form as

$\displaystyle\mathbf{q}[\phi]-\nu\|\phi\|^{2}=\|f^{\prime}\|^{2}\Big{[}\|\varphi_{\rho_{\beta}}\|^{2}+|\varphi_{\rho_{\beta}}(0)|^{2}L\tan\textstyle{\frac{\beta}{2}}\Big{]}$
$\displaystyle\qquad+\|f\|^{2}\Big{[}\|\varphi^{\prime}_{\rho_{\beta}}\|^{2}-\int_{\mathbb{R}}|\varphi_{\rho_{\beta}}(u)|^{2}\,\mathrm{d}\mu_{\perp}(u)-\nu\big{(}\|\varphi_{\rho_{\beta}}\|^{2}+|\varphi_{\rho_{\beta}}(0)|^{2}L\tan\textstyle{\frac{\beta}{2}}\big{)}\Big{]}$
$\displaystyle=\|f^{\prime}\|^{2}\Big{[}\|\varphi_{\rho_{\beta}}\|^{2}+|\varphi_{\rho_{\beta}}(0)|^{2}L\tan\textstyle{\frac{\beta}{2}}\Big{]}+\|f\|^{2}\Big{[}\big{(}\epsilon_{\mu_{\perp},\rho_{\beta}}-\nu\big{)}\|\varphi_{\rho_{\beta}}\|^{2}-\nu|\varphi_{\rho_{\beta}}(0)|^{2}L\tan\textstyle{\frac{\beta}{2}}\big{)}\Big{]},$

where we have used the fact that $\varphi_{\rho_{\beta}}$ is by assumption the
ground-state eigenfunction of $h_{\mu_{\perp},\rho_{\beta}}$. The last
expression is negative provided

$\frac{\|f^{\prime}\|^{2}}{\|f\|^{2}}<\frac{\nu-\epsilon_{\mu_{\perp},\rho_{\beta}}+\nu\eta^{2}_{\rho_{\beta}}L\tan\textstyle{\frac{\beta}{2}}}{1+\eta^{2}_{\rho_{\beta}}L\tan\textstyle{\frac{\beta}{2}}},$
(3.6)

where we set
$\eta_{\rho_{\beta}}:=\frac{|\varphi_{\rho_{\beta}}(0)|}{\|\varphi_{\rho_{\beta}}\|}$
for the sake of brevity; this quantity is bounded as a function of the angle
$\beta$ and $\lim_{\beta\to 0}\eta_{\rho_{\beta}}=\eta_{\rho}>0$. The right-
hand side of inequality (3.6), which we denote as $R_{\nu}(L,\beta)$, is
continuous with respect to the parameters and for small enough $\beta$ it is
positive tending to $\nu-\epsilon_{\mu_{\perp},\rho}$ as $\beta\to 0$.

The left-hand side is in view of (3.2) nothing else than the normalized
quadratic form of the Dirichlet Laplacian on an interval of length $L$, hence
the maximum number $n_{\nu}$ of mutually orthogonal trial functions making the
form $\mathbf{q}[\cdot]-\nu\|\cdot\|^{2}$ negative has to satisfy the
inequality $\big{(}\frac{\pi n_{\nu}}{L}\big{)}^{2}<R_{\nu}(L,\beta)$. The
indicated properties of the right-hand side imply, in particular, that there
are positive $a_{\nu},\,b_{\nu}$ such that $R_{\nu}(L,\beta)>a_{\nu}$ holds
whenever $L\tan\frac{\beta}{2}<b_{\nu}$, and consequently, we get for
$n_{\nu}$ the bound

$n_{\nu}<\frac{L}{\pi}\sqrt{a_{\nu}}<\frac{\sqrt{a_{\nu}}}{\pi}\,\frac{b_{\nu}}{\tan\frac{\beta}{2}}<\frac{2b_{\nu}\sqrt{a_{\nu}}}{\pi}\,\beta^{-1},$

which is the result we have set out to prove. ∎

##  4 Parallel asymptotes, weak coupling

Let us now turn to the situation when the ‘book is closed’, $\beta=0$. We know
from Proposition 2.1 how the essential spectrum looks in this case; what we
want to know is whether the discrete spectrum is nonempty. It appear that the
answer depends on the strength of the transversal coupling. To this aim we
introduce a coupling constant $\lambda>0$ into the picture; modifying
assumption (e) we consider operator $H_{\Gamma,\lambda\mu}$ associated with
the quadratic form
$\psi\mapsto\int_{\mathbb{R}^{2}}|\nabla\psi(\textbf{x})|^{2}\mathrm{d}\textbf{x}-\lambda\int_{\mathbb{R}^{2}}|\psi(\textbf{x})|^{2}\mathrm{d}\mu(\textbf{x})$
defined on $H^{1}(\mathbb{R}^{2})$. First we focus on the situation when the
coupling is weak.

######  Theorem 4.1.

Assume (a), (b) and (e). Then
$\sigma_{disc}\left(H_{\Gamma,\lambda\mu}\right)$ is empty for all $\lambda$
small enough.

We will need the following claim extending the well-known result of [Si76]:

######  Lemma 4.1.

Let $h_{\lambda\eta}$ be the Schrödinger operator in $L^{2}(\mathbb{R})$
associated with the quadratic form
$f\mapsto\int_{\mathbb{R}}|f^{\prime}(u)|^{2}\mathrm{d}u-\lambda\int_{\mathbb{R}}|f(u)|^{2}\mathrm{d}\eta(u)$
defined on $H^{1}(\mathbb{R})$, where $\eta$ is a compactly supported finite
positive Borel measure. For all sufficiently small positive $\lambda$, the
operator has exactly one eigenvalue $\epsilon(\lambda)$ which behaves
asymptotically as

$\sqrt{-\epsilon(\lambda)}=\frac{1}{2}\lambda\eta(\mathbb{R})+\mathcal{O}(\lambda^{2}).$
(4.1)

###### Proof.

As in the regular case, on can use the Birman-Schwinger principle: by [BEKŠ94,
Lemma 2.3] we have to check that the operator $\lambda R_{\eta\eta}(\kappa)$,
the trace of the resolvent
$\big{(}-\frac{\mathrm{d}^{2}}{\mathrm{d}x^{2}}+\kappa^{2}\big{)}^{-1}$ at the
space $L^{2}(\mathbb{R},\mathrm{d}\eta)$, has for small $\lambda$ a simple
eigenvalue one. The trace acts on $f\in L^{2}(\mathbb{R},\mathrm{d}\eta)$ as
$R_{\eta\eta}(\kappa)f=G_{\kappa}\star f$ $\,\eta$-almost everywhere with
$G_{\kappa}=\frac{1}{2\kappa}\rme^{-\kappa|\cdot|}$ defines the kernel of the
one-dimensional Laplacian resolvent. As in [Si76] we split the operator into
two parts, $R(\kappa)=L_{\kappa}+M_{\kappa}$, where $L_{\kappa}$ is the
integral operator with the kernel $\frac{1}{2\kappa}$ whose range is one-
dimensional consisting of constant functions, and $M_{\kappa}$ with the kernel
$\frac{1}{2\kappa}\big{(}\rme^{-\kappa|x-y|}-1\big{)}$. The last opertor is
obviously bounded which means that $I-\lambda M_{\kappa}$ is invertible for
$\lambda$ small enough; using then the identity

$I-\lambda R(\kappa)=(I-\lambda M_{\kappa})\big{(}I-(I-\lambda
M_{\kappa})^{-1}\lambda L_{\kappa}\big{)},$

we see that the task is reduced to finding $\mathrm{Ker}\big{(}I-(I-\lambda
M_{\kappa})^{-1}\lambda L_{\kappa}\big{)}_{\eta\eta}$, and since we have noted
that the operator is one-dimensional, the explicit form of $L_{\kappa}$ yields
the equation

$\kappa\eta(\mathbb{R})=\frac{\lambda}{2}\int_{\mathbb{R}^{2}}(I-\lambda
M_{\kappa})^{-1}(x,y)\,\mathrm{d}\eta(x)\,\mathrm{d}\eta(y).$

The rest of the proof proceeds as in the regular case; to complete the proof
one has to check that the right-hand side is a real analytic function of
$\lambda$ up to the point $\lambda=0$ and to expand it there in the leading
order. ∎

###### Proof of Theorem 4.1.

We use again a bracketing argument and impose additional Neumann boundary
conditions on the $y$-axis obtaining thus the inequality

$H_{\Gamma,\lambda\mu}\geq H_{\Gamma,\lambda\mu}^{\mathrm{c}}\oplus
H_{\Gamma,\lambda\mu}^{\mathrm{a}},$ (4.2)

where the first operator in the direct sum refers to the curved part in the
left halfplane and the second to the parallel channels in the right one. It is
thus sufficient to check that none of the two operators has spectral points
below $\epsilon_{\lambda\mu_{\perp},\rho}$. The spectrum of
$H_{\Gamma,\lambda\mu}^{\mathrm{a}}$ is purely absolutely continuous covering
the interval $[\epsilon_{\lambda\mu_{\perp},\rho},\infty)$, and choosing for
$\eta$ in Lemma 4.1 the ‘double-well measure’,
$\mathrm{d}\eta(u)=\mathrm{d}\mu_{\perp}(\rho+u)+\mathrm{d}\mu_{\perp}(-\rho-u)$,
we see that

$\inf\sigma(H_{\Gamma,\lambda\mu}^{\mathrm{a}})=\epsilon_{\lambda\mu_{\perp},\rho}=\lambda^{2}\mu_{\perp}(\mathbb{R})^{2}+\mathcal{O}(\lambda^{3})\quad\text{as}\;\;\lambda\to
0.$ (4.3)

Consider next the operator $H_{\Gamma,\lambda\mu}^{\mathrm{c}}$ and compare it
with $H_{\tilde{\Gamma},\lambda\mu}$ where $\tilde{\Gamma}$ a closed curve
being a union of $\Gamma_{int}$ and its mirror image with respect to the
$y$-axis. By construction $\tilde{\Gamma}$ is $C^{1}$-smooth, and since it is
a finite loop, the negative spectrum of $H_{\tilde{\Gamma},\lambda\mu}$ is
discrete. In view of the symmetry, the operator allows for a parity
decomposition in the $x$ direction,
$H_{\tilde{\Gamma},\lambda\mu}=H_{\tilde{\Gamma},\lambda\mu}^{\mathrm{sym}}\oplus
H_{\tilde{\Gamma},\lambda\mu}^{\mathrm{asym}}$ and it holds
$H_{\tilde{\Gamma},\lambda\mu}^{\mathrm{sym}}\leq
H_{\tilde{\Gamma},\lambda\mu}^{\mathrm{asym}}$, in particular,
$\inf\sigma(H_{\tilde{\Gamma},\lambda\mu})=\inf\sigma(H_{\tilde{\Gamma},\lambda\mu}^{\mathrm{sym}})=\inf\sigma(H_{\Gamma,\lambda\mu}^{\mathrm{c}})$.
By assumption, the measure $\mu_{\perp}$ is positive, and the same is, of
course, true for $\mu$. This allows us to use the result of [KL14] by which
$H_{\tilde{\Gamma},\lambda\mu}$ has for all $\lambda$ small enough a single
eigenvalues which behaves asymptotically as follows,

$\epsilon_{0}(\lambda)=-\big{(}C_{\mu}+o(1)\big{)}\exp\Big{(}-\frac{4\pi}{\lambda\mu(\mathbb{R}^{2})}\Big{)}\quad\text{as}\;\;\lambda\to
0,$ (4.4)

where $C_{\mu}$ is a positive constant depending on the measure $\mu$. A
comparison of (4.3) and (4.4), taking into account that
$\mu_{\perp}(\mathbb{R})>0$, implies in view of (4.2) that for sufficiently
small $\lambda$ the spectrum of $H_{\Gamma,\lambda\mu}$ below
$\epsilon_{\lambda\mu_{\perp},\rho}$ is empty; this concludes the proof. ∎

######  Remark 4.1.

The assumption (e) includes positivity of the measure. This may not be
necessary, for instance, a regular potential of assumption (c) might be sign-
changing. The claim of Theorem 4.1 remains nevertheless valid in such a
situation even if $\int_{\mathbb{R}}v(x)\,\mathrm{d}x=0$. Indeed, we know from
[Si76] that the weakly coupled state then still exists and it satisfies
$\sqrt{-\epsilon(\lambda)}=\frac{1}{4}\lambda^{2}\int_{\mathbb{R}^{2}}v(x)|x-y|v(y)\,\mathrm{d}x\,\mathrm{d}y+\mathcal{O}(\lambda^{3})$,
hence the essential spectrum threshold behaves as $\mathcal{O}(\lambda^{4})$.
We cannot apply the result of [KL14] directly to get (4.4) because the
asymptotics was obtained under the positivity assumption – see, however,
Remark 3.5 of the paper – it is easy to bypass this limitation. If
$\mu_{\perp}$ is sign-changing, we can write it as a difference of two
positive measures with disjoint supports,
$\mu_{\perp}=\mu_{\perp}^{(+)}-\mu_{\perp}^{(-)}$, and the same decomposition
applies to $\mu$. Since $H_{\tilde{\Gamma},\lambda\mu}\geq
H_{\tilde{\Gamma},\lambda\mu^{(+)}}$, the eigenvalue $\epsilon_{0}(\lambda)$
is bounded from below by the right-hand side of (4.4) with
$\mu(\mathbb{R}^{2})$ replaced by $\mu^{(+)}(\mathbb{R}^{2})$ tending
exponentially fast to zero, so for small enough $\lambda$ it is again the
essential spectrum threshold which dominates.

##  5 Parallel asymptotes, strong coupling

In contrast to the weak coupling, the strong one offers a wider variety of
asymptotic regimes referring to particular subclasses of potentials. In this
section we focus on three of them.

###  5.1 Leaky curves

Consider first a ‘leaky curve’, that is, operator $H_{\Gamma,\alpha}$ with the
attractive $\delta$ interaction supported by the curve $\Gamma$ with parallel
asymptotes, $\beta=0$.

######  Proposition 5.1.

Assume (a) and (d). If, in addition, $\Gamma\in C^{4}(\mathbb{R})$, then
$\sigma_{\mathrm{disc}}(H_{\Gamma,\alpha})\neq\emptyset$ holds for all
$\alpha$ large enough. Moreover, the number of eigenvalues (with the
multiplicity taken into account) does not exceed that of the Schrödinger
operator
$S_{\Gamma}=-\frac{\mathrm{d}^{2}}{\mathrm{d}s^{2}}-\frac{1}{4}\gamma(s)^{2}$
on $L^{2}(\mathbb{R})$, the bound being saturated as $\alpha\to\infty$.

###### Proof.

The claim follows from the strong-coupling asymptotics of
$\sigma_{\mathrm{disc}}(H_{\Gamma,\alpha})$ derived in [EY01], see also [EK15,
Cor. 10.3.1]. It was stated there for curves the asymptotes of which were not
parallel, however, this assumption was not used in its proof; the vital
assumptions are satisfied since $\Gamma\in C^{4}(\mathbb{R})$ and its
curvature is compactly supported. The $j$th eigenvalues of $H_{\Gamma,\alpha}$
then behaves asymptotically as

$\epsilon_{j}(\alpha)=-\frac{1}{4}\alpha^{2}+\epsilon_{j}+\mathcal{O}(\alpha^{-1}\ln\alpha)\quad\text{for}\;\;\alpha\to\infty,$
(5.1)

where $\epsilon_{j}<0$ is the $j$th eigenvalue of the comparison operator
$S_{\Gamma}$. The curvature-induced potential of $S_{\Gamma}$ is nonzero,
attractive, bounded and compactly supported, hence
$\sigma_{\mathrm{disc}}(S_{\Gamma})$ is nonempty and finite. ∎

###  5.2 An interlude: Dirichlet guides

To consider another strong-coupling asymptotic, we shall extend a known result
about the discrete spectrum of curved Dirichlet strips [EK15, Thm. 1.1].
Comparing the assumption concerning the Dirichlet strips in that book with
those about the potential support used here, we see that _(i)_ of [EK15, Sec.
1.1] guaranteeing the existence of parallel coordinates coincides with (b),
while (a) is weaker than _(ii)_ because it requires the curvature only to be
piecewise continuous. For the needs of the present paper it would be
sufficient to have the curvature compactly supported which would imply
assumptions _(iii)_ – _(v)_ but we will not limit ourselves here to this case
and prove the result for strips $\Omega^{a}$ which are only asymptotically
straight.

The object to consider is a Dirichlet waveguide with a generally non-flat
bottom, that is, the operator
$H_{\mathrm{D},v}:=-\Delta_{\mathrm{D}}^{\Omega^{a}}-V(x)$, where $V(x)$ is
the potential (2.6a) corresponding to $v$ satisfying assumption (c). Let
$h_{\mathrm{D},v}$ be the operator on $L^{2}(I_{a})$ with the domain
$H^{2}(I_{a})\cap H_{0}^{1}(I_{a})$ acting as (2.5), in other words, the
transverse Schrödinger operator on $I_{a}$ with Dirichlet boundary conditions
at $\pm a$. The spectrum of this operator is simple and purely discrete; we
denote $\epsilon_{\mathrm{D},v}:=\inf\sigma(h_{\mathrm{D},v})$ and $\chi_{0}$
will be the eigenfunction corresponding to this eigenvalue.

######  Theorem 5.1.

Let $\Omega^{a}$ satisfy assumptions (a) and (b), and $v$ satisfy assumption
(c). Let further $\lim_{|s|\to\infty}\gamma(s)=0$, then the operator
$H_{\mathrm{D},v}$ has at least one eigenvalue below
$\inf\sigma_{\mathrm{ess}}(H_{\mathrm{D},v})=\epsilon_{\mathrm{D},v}$ unless
$\gamma=0$ identically.

###### Proof.

The argument follows the proof of the mentioned Theorem 1.1 in [EK15] except
that we weaken the regularity requirements on $\Gamma$ in the spirit of
[KŠ12]. The main tool is the quadratic form of the operator
$H_{\mathrm{D},v}-\epsilon_{\mathrm{D},v}I$ which can be, using the parallel
coordinates, written as

$\mathbf{q}\left[\psi\right]\vcentcolon=\lVert
g^{-1/4}\partial_{s}\psi\rVert^{2}+\lVert
g^{1/4}\partial_{u}\psi\rVert^{2}-\lVert
g^{1/4}V\psi\rVert^{2}-\epsilon_{\mathrm{D},v}\lVert g^{1/4}\psi\rVert^{2}$
(5.2)

for any $\psi\in H^{1}(\mathbb{R}\times I_{a})$; we recall that
$\sqrt{g(s,u)}=(1-u\gamma(s))$. To find the essential spectrum threshold, we
use Neumann bracketing, adding Neumann condition at the segments perpendicular
to $\Gamma$ at $s=\pm s_{0}$. This allows us to estimates $H_{\mathrm{D},v}$
from below by the direct sum $H_{\mathrm{D},v}^{(-)}\oplus
H_{\mathrm{D},v}^{(0)}\oplus H_{\mathrm{D},v}^{(+)}$. The middle part refers
to a compact region being thus irrelevant from the point of the essential
spectrum. The shifted quadratic forms of the other two parts can be estimated
from below as follows,

$\mathbf{q}^{(\pm)}\left[\psi\right]\geq(1-a\|\gamma_{\pm}\|_{\infty})^{1/2}\big{[}\lVert\partial_{s}\psi\rVert^{2}_{\pm}+\lVert\partial_{u}\psi\rVert^{2}_{\pm}-\lVert
V\psi\rVert^{2}_{\pm}\big{]}-\epsilon_{\mathrm{D},v}(1+a\|\gamma_{\pm}\|_{\infty})^{1/2}\lVert\psi\rVert^{2}_{\pm},$

where the norms refer to $L^{2}((s_{0},\infty)\times I_{a})$ and
$L^{2}((-\infty,-s_{0})\times I_{a})$, respectively, and $\gamma_{\pm}$ are
the restrictions of $\gamma$ to the appropriate intervals, and we have used
the inequality
$(1-a\|\gamma_{\pm}\|_{\infty})^{1/2}\leq(1+a\|\gamma_{\pm}\|_{\infty})^{-1/2}$.
The parts in the square bracket are nothing but quadratic forms of the
operators on the respective parts of the straight channel and as such they
they bounded from below by
$\epsilon_{\mathrm{D},v}\lVert\psi\rVert^{2}_{\pm}$, so we have

$\mathbf{q}^{(\pm)}\left[\psi\right]\geq-\epsilon_{\mathrm{D},v}\big{[}(1+a\|\gamma_{\pm}\|_{\infty})^{1/2}-(1-a\|\gamma_{\pm}\|_{\infty})^{1/2}\big{]}\lVert\psi\rVert^{2}_{\pm},$

and since $\gamma(s)$ tends to zero as $|s|\to\infty$ by assumption, the left-
hand side can be made arbitrarily close to zero by choosing $s_{0}$ large
enough.

The prove the second claim, one has to find a trial function from the form
domain of $H_{\mathrm{D},v}$ which makes the left-hand side of (5.2) negative.
We use the idea proposed in this context in [GJ92] and choose the function in
the form $\psi\vcentcolon=\phi_{\varsigma}\chi_{0}+\varepsilon f$, where $f$
will be chose later and $\phi_{\varsigma}$ is a suitable mollifier, say

$\phi_{\varsigma}\vcentcolon=\begin{cases*}\phi(s)&for $|s|\leq s_{1}$\\\
\phi\left(s_{0}\,\mathrm{sgn}s+\varsigma(s-s_{0}\,\mathrm{sgn}s)\right)&for
$|s|>s_{1}$\end{cases*},$ (5.3)

with $\phi\in C_{0}^{\infty}(\mathbb{R})$ such that $\phi(s)=1$ holds for
$s\in[-s_{1},s_{1}]$ for some $s_{1}>0$. We have

$\displaystyle\mathbf{q}\left[\phi_{\varsigma}\chi_{0}\right]=\int_{\mathbb{R}}\langle
g^{-1/2}\rangle(s)|\phi^{\prime}_{\varsigma}(s)|^{2}\mathrm{d}s+\iint
g^{1/2}(s,u)|\chi^{\prime}_{0}(u)|^{2}|\phi_{\varsigma}(s)|^{2}\mathrm{d}s\mathrm{d}u$
(5.4) $\displaystyle\quad-\iint
g^{1/2}(s,u)v(u)|\chi_{0}(u)|^{2}|\phi_{\varsigma}(s)|^{2}\mathrm{d}s\mathrm{d}u-\epsilon_{\mathrm{D},v}\iint
g^{1/2}(s,u)|\chi_{0}(u)|^{2}|\phi_{\varsigma}(s)|^{2}\mathrm{d}s\mathrm{d}u,$

where $\langle g^{-1/2}\rangle(s)=\int_{-a}^{a}g^{-1/2}(s,u)\,\mathrm{d}u$ is
the transverse average of the inverse Jacobian. By Fubini theorem the order of
integration in the last three terms makes no difference. Eigenfunctions of
$h_{\mathrm{D},v}$ may be chosen real and by assumption we have

$-\chi_{0}^{\prime\prime}(u)-v(u)\chi_{0}(u)=\epsilon_{\mathrm{D},v}\chi_{0}(u)\,;$

an easy integration by parts with $(1-u\gamma(s))\chi_{0}(u)\mathrm{d}u$ using
the fact, that $\chi_{0}(\pm a)=0$, shows that the sum of the three terms is
zero. The remaining term is easily estimated; we get the bound

$\mathbf{q}\left[\phi_{\varsigma}\chi_{0}\right]\leq\frac{\varsigma}{(1-a\lVert\gamma\rVert_{\infty})^{1/2}}\lVert\phi^{\prime}\rVert^{2}$
(5.5)

showing that by the choice of $\varsigma$ the positive contribution from the
trial function tails can be made arbitrarily small.

To complete the construction, we have to choose the function $f$. We pick it
from $C_{0}^{\infty}((-s_{1},s_{1})\times I_{a})$ in which case $\psi\in
H^{1}(\mathbb{R}\times I_{a})$ and

$\mathbf{q}[\psi]=\mathbf{q}\left[\phi_{\varsigma}\chi_{0}\right]+2\varepsilon\,\mathrm{Re}\,\mathbf{q}(\phi_{\varsigma}\chi_{0},f)+\varepsilon^{2}\mathbf{q}[f],$
(5.6)

where $\mathbf{q}(\cdot,\cdot)$ is the corresponding sesquilinear form. Using
the fact that the supports of $\phi^{\prime}_{\varsigma}\chi_{0}$ and $f$ are
disjoint, we evaluate easily the linear term coefficient,

$2\,\mathrm{Re}\,\mathbf{q}(\phi_{\varsigma}\chi_{0},f)=\big{(}g^{1/2}(\chi^{\prime}_{0}-V\chi_{0}-\epsilon_{\mathrm{D},v}\chi_{0}),f\big{)}.$

Since $f$ is picked from an infinitely dimensional space, we can obviously
choose it in such a way that the coefficient is nonzero and, say, negative.
For small $\varepsilon>0$ the linear term dominates over the quadratic one;
then we can choose $\varepsilon$ so that the sum of the last two terms on the
right-hand side of (5.6) is negative, and subsequently to choose $\varsigma$
in order not to spoil the negativity of the whole estimating expression. ∎

Note that we have identified only the threshold of
$\sigma_{\mathrm{ess}}(H_{\mathrm{D},v})$. Under stronger regularity
assumptions one can prove that the essential spectrum covers the whole
interval $[\epsilon_{\mathrm{D},v},\infty)$ but we will not need it here.

###  5.3 Making the ditch deeper

Let us return to our problem in the situation with parallel asymptotes,
$\beta=0$. A natural way to achieve strong coupling for a regular potential
channel consists of modifying its depth without changing the profile of its
bottom. For the sake of brevity we denote by $\chi_{a}$ the characteristic
function of the set $\Omega^{a}\subset\mathbb{R}^{2}$, with an abuse of
notation we will use the same symbol the characteristic function of the
interval $I_{a}\subset\mathbb{R}$ and for the measure $\chi_{a}(x)\mathrm{d}x$
on $\mathbb{R}$; then we consider the behavior of the operator
$H_{\Gamma,V+\lambda\chi_{a}}$.

######  Theorem 5.2.

Let assumptions (a), (b) and (c) be satisfied, then we have

$\inf\sigma_{\mathrm{ess}}(H_{\Gamma,V+\lambda\chi_{a}})=\epsilon_{\mu_{\perp}+\lambda\chi_{a},\rho}=-\lambda+\epsilon_{\mathrm{D},v}+\mathcal{O}\big{(}\rme^{-c\sqrt{\lambda}}\,\big{)}$
(5.7)

as $\lambda\to\infty$ for some $c>0$, and
$\sigma_{\mathrm{disc}}(H_{\Gamma,V+\lambda\chi_{a}})\neq\emptyset$ for all
$\lambda$ large enough.

###### Proof.

Consider the operator family $\\{H_{\Gamma,V+\lambda\chi_{a}}+\lambda
I:\>\lambda\geq 0\\}$. The corresponding family of quadratic forms is
monotonously increasing, then it follows from [RS80, Thm. S.14] that these
operators converge to $H_{\mathrm{D},V}$ as $\lambda\to\infty$ in the
generalized strong resolvent sense [W00, Sec. 9.3]. By the same argument,
$h_{\Gamma,v+\lambda\chi_{a}}+\lambda I\to h_{\mathrm{D},v}$ as
$\lambda\to\infty$ in the generalized strong resolvent sense. We will use the
last result to prove (5.7). Its first relation follows form Proposition 2.1.
To get the second one, we note that by the standard double-well estimate
[Si84, Thm. 1.5] we have
$\epsilon_{\mu_{\perp}+\lambda\chi_{a},\rho}=\epsilon_{\mu_{\perp}+\lambda\chi_{a}}+\mathcal{O}\big{(}\rme^{-c\sqrt{\lambda}}\,\big{)}$
where the constant $c$ is given by the associated Agmon metric.

The second claim follows from Theorem 5.1. As the curvature $\gamma$ is by
assumption nonzero and compactly supported, the assumptions are satisfied and
the limiting operator $H_{\mathrm{D},V}$ has at least one eigenvalue below
$\epsilon_{\mathrm{D},v}$, hence $H_{\Gamma,V+\lambda\chi_{a}}+\lambda I$ has
an eigenvalue below $\epsilon_{\mu_{\perp}+\lambda\chi_{a},\rho}$, and
consequently, below $-\lambda+\epsilon_{\mathrm{D},v}$ for $\lambda$ large
enough. ∎

###  5.4 Scaling with increasing volume

Another strong coupling situation, extending the result of Proposition 5.1 to
regular potentials, arises if we combined linear scaling with a sufficiently
fast increasing mean strength, more specifically, if we consider the following
family of operators,

$H_{\Gamma,V_{g(\lambda)}}\quad\text{with}\quad
v_{g(\lambda)}(x):=g(\lambda)v(\lambda x),$ (5.8)

with $g$ being a suitable function such that
$\lim_{\lambda\to\infty}\frac{g(\lambda)}{\lambda}=\infty$ as
$\lambda\to\infty$.

######  Proposition 5.2.

Adopt assumptions (a)–(c) and suppose, in addition, that $\Gamma\in
C^{4}(\mathbb{R})$. Then there is a function $g_{0}$ with the indicated
properties such that $\sigma_{\mathrm{disc}}(H_{\Gamma,V_{g(\lambda)}})$ is
nonempty for any $g\geq g_{0}$ and all $\lambda$ large enough.

###### Proof.

Instead of (5.8), consider the two-parameter family of operators
$H_{\Gamma,V_{\xi,\lambda}}$ with the profile potential
$v_{\xi,\lambda}(x):=\xi\lambda v(\lambda x)$. As we have already mentioned,
for a fixed $\xi>0$ these operators converge in the norm-resolvent sense to
$H_{\Gamma,\alpha_{\xi}}$ with $\alpha_{\xi}:=\xi\|v\|_{1}$ as
$\lambda\to\infty$, cf. [EI01, BEHL17]. The difficulty to deal with is that
the essential spectrum threshold depend on $\xi$. Given a below bounded self-
adjoint operator $A$, we denote by $\mu_{k}(A)$ the corresponding numbers
obtained from the minimax principle111This is a traditional notation and we
are sure there is no danger of confusion with the symbol used to describe
measure-type potentials. [RS78, Sec.XIII.1] and by $\mu_{\infty}(A)$ the
limiting value of this sequence (for operators with a finite discrete spectrum
we consider here, the sequence is constant from some index on). Then we
introduce the operators

$\tilde{H}_{\Gamma,V_{\xi,\lambda}}:=H_{\Gamma,V_{\xi,\lambda}}-\mu_{\infty}(H_{\Gamma,V_{\xi,\lambda}})\quad\text{and}\quad\tilde{H}_{\Gamma,\alpha_{\xi}}:=H_{\Gamma,\alpha_{\xi}}-\mu_{\infty}(H_{\Gamma,\alpha_{\xi}})$

whose essential spectrum threshold is zero by construction; the mentioned
result implies that $\tilde{H}_{\Gamma,V_{\xi,\lambda}}$ converges for a fixed
$g$ to $\tilde{H}_{\Gamma,\alpha_{\xi}}$ as $\lambda\to\infty$. At the same
time,
$\mu_{k}(\tilde{H}_{\Gamma,\alpha_{\xi}})=\epsilon_{k}(\alpha_{\xi})+\frac{1}{4}\alpha_{\xi}^{2}$
converges by (5.1) to $\epsilon_{k}$, the $k$th eigenvalue of $S_{\Gamma}$.
Next we use a simple telescopic estimate,

$\big{|}\mu_{k}(\tilde{H}_{\Gamma,V_{\xi,\lambda}})-\mu_{k}(S_{\Gamma})\big{|}\leq\big{|}\mu_{k}(\tilde{H}_{\Gamma,V_{\xi,\lambda}})-\mu_{k}(\tilde{H}_{\Gamma,\alpha_{\xi}})\big{|}+\big{|}\mu_{k}(\tilde{H}_{\Gamma,\alpha_{\xi}})-\mu_{k}(S_{\Gamma})\big{|}.$
(5.9)

Given an arbitrary $\varepsilon>0$, one can find a $\xi_{0}>0$ such that the
second term on right-hand side of (5.9) is smaller that
$\frac{1}{2}\varepsilon$ for all $\xi>\xi_{0}$, and to such a $\xi$ there is a
$\lambda_{\xi}>0$ with the property that the first term on right-hand side of
(5.9) is smaller than $\frac{1}{2}\varepsilon$ for $\lambda>\lambda_{\xi}$.
The map $\xi\mapsto\lambda_{\xi}$ is obviously monotonous which allows us to
define $f_{0}$ as its pull-back and $g_{0}(\lambda):=\lambda f(\lambda)$. For
any function $g\geq g_{0}$ we then have
$\mu_{k}(\tilde{H}_{\Gamma,V_{g(\lambda)})})\to\mu_{k}(S_{\Gamma})$ as
$\lambda\to\infty$, and since
$\mu_{1}(S_{\Gamma})=\mu_{1}(S_{\Gamma})-\mu_{\infty}(S_{\Gamma})<0$ holds for
our non-straight $\Gamma$, the same must be true for
$\mu_{k}(\tilde{H}_{\Gamma,V_{g(\lambda)})})$ if $\lambda$ is large enough, in
other words, we conclude $\sigma_{\mathrm{disc}}(H_{\Gamma,V_{g(\lambda)}})$
must be nonempty for large $\lambda$. ∎

Note that one could get a better idea of the function $g_{0}$ looking into the
error terms of the tow limits involved, but we will not follow this route
here.

##  6 A numerical example

While we have been able to establish the weak/strong coupling dichotomy, finer
properties are out of reach for an analytical treatment, for instance, it is
not easy to find the critical strength needed to produce the discrete
spectrum. They can be dealt with numerically as we are going to illustrate
here. As an example, we consider the U-shaped channel with a polynomial
profile, a multiple of

$v_{\alpha}(x)\vcentcolon=\min\Big{\\{}\Big{(}\frac{|x|-\rho}{a}\Big{)}^{\alpha}-1,\>0\Big{\\}}.$
(6.1)

where
$\Omega\vcentcolon=\left(-\rho-a,-\rho+a\right)\bigcup\left(\rho-a,\rho+a\right)$
and $\alpha\geq 2$ is an even integer. Such a profile obviously tends to a
rectangular well as $\alpha\to\infty$ but we work with finite values as it
makes the numerics easier.

Naturally, we have to solve the equation in a finite region. We do it for both
the Dirichlet and Neumann boundaries which allows us to control the precision
using the gap between the two. In computations, we made use of the Spectral
Method [T00, Chap. 9]. Looking for the critical coupling constant of the
potential $\lambda v_{\alpha}$, we plot in Figure 2

Figure 2: The critical interaction strength _vs._ the relative channel width

the dependence of two dimensionless quantities, $-\sqrt{|\lambda|}A$, where
$A:=\frac{1}{\pi}\int_{-a}^{a}\sqrt{v_{\alpha}(x)}\,\mathrm{d}x$, and
$a/\rho$, the relative width of the channel, for two values of $\alpha$. The
former is, of course, an upper estimate, asymptotically exact, of the number
of bound states minus one in the one-dimensional potential well of that
profile. Predictably, the curves are monotonously decreasing since the
tunnelling between the parallel channels plays a more prominent role as $a$
increases and a stronger coupling is needed to compete with it. Also the
dependence on $\alpha$ makes sense, as the dimensionless ‘volume’ increases as
$\alpha$ grows. We also see that for larger $\alpha$ the curve becomes steeper
as we approach the point $a=\rho$ as the ‘residual’ barrier is thinner then.

For illustration we also plot in Figure 3

Figure 3: Bound state eigenfunction $\rho=0.25$, $a=0.1$, $\lambda=-225$ and
$\alpha=2$.

the eigenfunction in the situation when the coupling is stronger than
critical; the thin black line in the picture indicates the potential support.

##  7 Concluding remarks

While the shape of the curved part in our result is quite general, modulo the
regularity requirement, the guide is supposed to be straight outside a
compact. A question arising naturally is what happens if $\Gamma$ approached
parallel lines only asymptotically; one expects that the general picture would
remain similar as here. Other type of asymptotic behaviors, on the other hand,
may lead to different spectral properties. One can ask, for instance, whether
there can be geometrically induced eigenfunctions supported away from the
‘bend’ of $\Gamma$; one expects this to happen, say, when the double guide is
locally bent or the gap between two channels is locally diminished at an
appropriate place.

### Data availability statement

Data are available in the article.

### Conflict of interest

The authors have no conflict of interest.

### Acknowledgements

The work was supported by the Czech Science Foundation within the project
21-07129S by the European Union’s Horizon 2020 research and innovation
programme under the Marie Skłodowska-Curie grant agreement No 873071.
Computational resources were provided by the e-INFRA CZ project (ID:90254),
supported by the Ministry of Education, Youth and Sports of the Czech
Republic.

## References

## References

  * [BEHL17] J. Behrndt, P. Exner, M. Holzmann, V. Lotoreichik: Approximation of Schrödinger operators with $\delta$-interactions supported on hypersurfaces, _Math. Nachr._ 290 (2017), 1215–1248.
  * [BEKŠ94] J.F. Brasche, P. Exner, Yu.A. Kuperin, P. Šeba: Schrödinger operators with singular interactions, _J. Math. Anal. Appl._ 184 (1994), 112–139.
  * [DH93] M. Dauge, B. Helffer: Eigenvalue variation, I. Neumann problem for Sturm-Liouville operators, _J. Diff. Eqs_ 104 (1993), 243–262.
  * [EKP20] S. Egger, J. Kerner, K. Pankrashkin: Bound states of a pair of particles on the half-line with a general interaction potential, _J. Spect. Theory_ 10 (2020), 1413–1444.
  * [Ex20] P. Exner: Spectral properties of soft quantum waveguides, _J. Phys. A: Math. Theor._ 53 (2020), 355302; corrigendum 54 (2021), 099501.
  * [Ex22] P. Exner: Soft quantum waveguides in three dimensions, _J. Math. Phys._ 63 (2022), 042103.
  * [Ex23] P. Exner: Geometry effects in quantum dot families, arXiv:2305.12748 [math.SP]
  * [EI01] P. Exner, T. Ichinose: Geometrically induced spectrum in curved leaky wires, _J. Phys. A: Math. Gen._ 34 (2001), 1439–1450.
  * [EKL22] P. Exner, S. Kondej, V. Lotoreichik: Bound states of weakly deformed soft waveguides, arxiv:2211.01989
  * [EK15] P. Exner, H. Kovařík: _Quantum Waveguides_ , Springer International, Heidelberg 2015.
  * [EL21] P. Exner, V. Lotoreichik: Optimization of the lowest eigenvalue of a soft quantum ring, Lett. Math. Phys. 111 (2021), 28.
  * [EV23] P. Exner, S. Vugalter: Bound states in bent soft waveguides, arXiv:2304.14776 [math.SP]
  * [EY01] P. Exner, K. Yoshitomi: Band gap of the Schrödinger operator with a strong $\delta$-interaction on a periodic curve, _Ann. Henri Poincaré_ 2 (2001), 1139–1158.
  * [GJ92] J. Goldstone, R.L. Jaffe: Bound states in twisting tubes, _Phys. Rev._ B45 (1992), 14100-14107.
  * [KKK21] S. Kondej, D. Krejčiřík, J. Kříž.: Soft quantum waveguides with a explicit cut locus, _J. Phys. A: Math. Theor._ 54, 30LT01 (2021).
  * [KL14] S. Kondej, V. Lotoreichik: Weakly coupled bound state of 2-D Schödinger operator with potential-measure, _J. Math. Anal. Appl._ 420 (2014), 1416–1438.
  * [KK23] D. Krejčiřík, J. Kříž: Bound states in soft quantum layers, _Proc. RIMS_ (2023), to appear; arXiv:2205.04919 [math-ph]
  * [KŠ12] D. Krejčiřík, H. Šediváková: The effective Hamiltonian in curved quantum waveguides under mild regularity assumptions, Rev. Math. Phys. 24 (2012), 1250018.
  * [RS78] M. Reed, B. Simon: _Methods of Modern Mathematical Physics, IV. Analysis of Operators_ , Academic Press, New York 1978.
  * [RS80] M. Reed, B. Simon: _Methods of Modern Mathematical Physics, I. Functional Analysis_ , 2nd edition, Academic Press, New York 1980.
  * [Si76] B. Simon: The bound state of weakly coupled Schrödinger operators in one and two dimensions, _Ann. Phys._ 97 (1976), 279–288.
  * [Si84] B. Simon: Semiclassical analysis of low lying eigenvalues, II. Tunneling, _Ann. Math._ 120 (1984), 89–118.
  * [SM90] F. Sols, M. Macucci: Circular bends in electron waveguides, _Phys. Rev._ B41 (1990), 11887–11891.
  * [T00] Lloyd N. Trefethen: Spectral Methods in MATLAB, SIAM, Philadelphia 2000.
  * [W00] J. Weidmann: _Lineare Operatoren in Hilberträumen. Teil I: Grundlagen_ , Teubner, Wiesbaden 2000.


# Learning Excavation of Rigid Objects with Offline Reinforcement Learning

Shiyu Jin1, Zhixian Ye1 and Liangjun Zhang1 1The authors are with Robotics and
Autonomous Driving Lab, Baidu Research, USA. {shiyujin, zhixianye,
<EMAIL_ADDRESS>
###### Abstract

Autonomous excavation is a challenging task. The unknown contact dynamics
between the excavator bucket and the terrain could easily result in large
contact forces and jamming problems during excavation. Traditional model-based
methods struggle to handle such problems due to complex dynamic modeling. In
this paper, we formulate the excavation skills with three novel manipulation
primitives. We propose to learn the manipulation primitives with offline
reinforcement learning (RL) to avoid large amounts of online robot
interactions. The proposed method can learn efficient penetration skills from
sub-optimal demonstrations, which contain sub-trajectories that can be
“stitched” together to formulate an optimal trajectory without causing
jamming. We evaluate the proposed method with extensive experiments on
excavating a variety of rigid objects and demonstrate that the learned policy
outperforms the demonstrations. We also show that the learned policy can
quickly adapt to unseen and challenging fragmented rocks with online fine-
tuning.

##  I INTRODUCTION

Excavators have been widely used in mining and construction environments for
many decades. The excavator operations usually rely on experienced operators
so that they can choose appropriate actions for different tasks. While we have
seen a lot of studies in the field of robotics and autonomous driving,
excavators can achieve only limited autonomy [1, 2, 3]. One of the main
difficulties lies in the fact that excavation tasks usually require rich
contact between the excavator bucket and the terrain. The problem becomes even
harder if the terrain being excavated contains irregular rigid objects, such
as fragmented rocks (Fig. 1), where the contact dynamics are complex and
extremely hard to model. In such scenarios, improperly applied forces could
easily result in jamming problems between the bucket and the terrain [4, 5,
6]. Handling rigid objects is crucial for excavation in mining and
construction environments, where the excavators often have to deal with large
rocks and stones. But there has been relatively little work examining how
excavators can achieve such objectives autonomously.

Many previous works have studied the planning [7, 8, 9, 10] and the control
[11, 12, 13, 14, 15, 16, 17] for the excavation of granular materials, such as
sands. The granular materials are relatively easy to be excavated because of
the large torques that can be generated from the hydraulic system in
excavators. In contrast, irregular rigid objects, such as fragmented rocks and
stones, which cannot be disintegrated during excavation, are extremely
challenging to be excavated. Only a few works have studied the autonomous
excavation of irregular rigid objects. Lu et al. [4, 5] learn the excavation
of wood blocks in simulation and then transfer it to the real world. Zhu et
al. [6] learn the dynamic model for wood blocks excavation. However, due to
the huge sim-to-real gap and the bias in the learned dynamic model, those
works can only achieve low excavation success rates on a single type of
terrain, wood blocks. A lot of jamming problems between the bucket and the
rigid objects cannot be properly handled by the previous works.

To the best of our knowledge, there is no previous work that can efficiently
avoid jamming during the excavation of rigid objects. And there are also no
methods that can generalize the excavation skills across different rigid
object types. We are the first to conduct a study on the autonomous excavation
of rigid objects covering a broad range of types, including red mulch, marble
chips, and fragmented rocks, etc. Although Egli et al. [18] address different
soil excavations through domain randomization using a soil analytical model,
the soil analytical model is only applicable to homogeneous soils instead of
irregular rigid objects, where the contact dynamics are much more complex.

Figure 1: Excavation of rigid objects. A robotic manipulator attempts to
penetrate the end-effector bucket into the fragmented rocks without causing
jamming.

In this paper, we propose to learn rigid object excavation with offline
reinforcement learning. Reinforcement learning algorithms have shown the
ability to learn an efficient policy for robotic manipulation tasks with good
generalization abilities. But a large number of on-robot trails make it
difficult to be deployed in real-world contact-rich robotic manipulation
tasks, such as excavations. In contrast, offline RL can avoid online robot
interaction by learning the policy with previously collected datasets. For the
excavation of rigid objects, the offline dataset is obtained from
teleoperation and scripted policy, which avoids dangerous actions during on-
robot RL training. To efficiently penetrate the terrain of the rigid object
while avoiding the jamming problems, we design three manipulation primitives,
sweep, rotate, and penetrate based on expert experiences. The sweep and rotate
primitives generate horizontal bucket motion and rotation in the pitch axis to
remove the small fragments that block the bucket penetration. The penetrate
primitive generates vertical bucket motion to penetrate into the terrain. The
parameters of the manipulation primitives are learned as the policy output
using offline RL. To generalize across different terrain configurations and
terrain types, we propose to utilize two long short-term memory (LSTM)
encoders. Experiments show that the proposed method is able to learn a better
excavation policy than the demonstrations, which contain sub-optimal
excavation trajectories. Our proposed framework also has the ability to
quickly adapt to unseen rigid objects with a few online fine-tuning.

The main contributions presented in this work are summarized below:

  * •

To the best of our knowledge, we are the first to study the jamming problems
for the excavation of a variety of rigid objects, which would be beneficial to
the robotics and autonomous construction community.

  * •

We proposed a framework for the excavation of rigid objects with offline
reinforcement learning. The proposed method can learn excavation policy from
sub-optimal demonstration trajectories.

  * •

We designed multiple manipulation primitives with continuous parameters which,
once trained, are able to generate robot commands to excavate rigid objects
without causing jamming problems.

  * •

We empirically evaluated the performance of our method by extensive
experiments. The learned policy outperforms the sub-optimal demonstrations. In
addition, the learned policy is able to quickly adapt to unseen and
challenging fragmented rocks with online fine-tuning.

##  II RELATED WORK

###  II-A Planning and Control for Autonomous Excavation

In recent years, a lot of works have been done in Autonomous Excavation. Some
works focus on building an integrated system. For example, Zhang et al. [1]
propose an autonomous excavator system for material loading tasks, which can
continuously operate for 24 hours without any human intervention; Jelavic et
al. [2] present an integrated system for performing precision harvesting
missions, which combines mapping, localization, planning, and control; Jud et
al. [3] achieve autonomous free-form trenching using a walking excavator. Many
other works focus on the task or manipulator trajectory planning for the
excavator using optimization-based or data-driven methods [7, 8, 9, 10]. Since
excavation is a contact-rich robotic manipulation task, simply executing a
planned trajectory could easily fail due to the unknown contact dynamics
between the terrain and the excavator bucket. So some works focus on
autonomous excavator control [11, 12, 13, 14, 15, 16, 17]. The target for most
of the excavation tasks is usually to collect the granular objects. Only a few
works study the excavation of irregular rigid objects, where the contact
dynamics are much more complex than the excavation of granular objects.
Sotiropoulos et al. [19, 20] utilize Gaussian Process and Unscented Kalman
Filter to capture a rock. The authors have to facilitate the rock collection
by putting sand underneath the rock. Lu and Zhang [4] train a classifier for
wood blocks excavation using voxel-based representation. But they achieve low
excavation success rates due to the jamming problems.

Reinforcement learning has demonstrated its ability to solve some contact-rich
robotic manipulation tasks [21, 22, 23]. Previous works also show the RL
applications in autonomous excavation. Hodel et al. [24] and Kurinov et al.
[25] learn to load an excavator in simulations using RL. But they do not
perform any experiments in the real world. Egli et al. [26, 27] learn a
tracking controller by collecting data from the real-world operation of a
hydraulic excavator arm. But the learned RL policy only works for free space
excavator operation without any interaction with the terrain. They [18] also
learn a controller for a full-sized hydraulic excavator that can adapt online
to different soil characteristics. The RL policy is trained in simulation and
deployed directly in the real world. But the method requires an analytical
soil model, which cannot be generalized well if the terrain type changes to
rigid objects where an analytical contact model is hard to obtain. Lu et al.
[5] and Zhu et al. [6] utilize RL to learn the excavation of wood blocks.
Although they achieve successful excavation of rigid objects with geometric
representation [5] and learned contact dynamic model [6], their methods still
suffer a lot of failure cases where jamming happens during bucket penetration.
The MPC controller in their method is also time-consuming to compute.

###  II-B Offline Reinforcement Learning in Robotic Manipulation

Different from the above, we directly learn the excavation policy from the
offline dataset, which does not require dangerous on-robot RL training.
Several prior works have explored offline RL methods for learning robotic
grasping and manipulation skills [28, 29, 30, 31, 32, 33]. Nair et al. [34]
learn a policy that maximizes reward while bounding the deviation from the
dataset. But the proposed method tends to overfit heavily with many offline
gradient steps. Kostrikov et al. [35] propose offline reinforcement learning
with Implicit Q-Learning (IQL) to estimate the maximum Q-value over actions
that are in support of the data distribution using the Expectile Regression.
Experiments have shown that IQL performs well on tasks that require multi-step
dynamic programming. This is a good fit for the excavation tasks, where we are
trying to find an optimal penetration policy by “stitching” several sub-
optimal trajectories from the demonstrations. In this work, we build on IQL to
learn a policy for the excavation of rigid objects.

##  III Background and Problem Formulation

###  III-A Offline Reinforcement Learning

The reinforcement learning problem is formulated using a Markov Decision
Process (MDP) associated with task $\mathcal{T}$ defined by
$\mathcal{M}_{\mathcal{T}}=\\{\mathcal{S},\mathcal{A},\mathcal{P},r,p_{0}(s),\gamma\\}$,
where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$\mathcal{P}$ is the environmental dynamics, $r$ is the reward function,
$p_{0}(s)$ is the initial state distribution, and $\gamma$ is the discount
factor. The agent interacts with the MDP according to a policy $\pi(a|s)$. The
final goal in an RL problem is to learn a policy that maximizes the cumulative
discounted rewards $R=\sum_{t=0}^{T}\gamma^{t}r(s_{t},a_{t})$.

Offline reinforcement learning problem can be defined as a data-driven
formulation of the reinforcement learning problem. The end goal is still to
optimize the RL objective. However, the agent no longer has the ability to
interact with the environment and collect additional transitions using the
behavior policy. Instead, the learning algorithm is provided with a static
dataset of transitions,
$\mathcal{D}=\\{(s_{t}^{i},a_{t}^{i},s_{t+1}^{i},r_{t}^{i})\\}$, and must
learn the best policy it can using this dataset [28].

###  III-B Problem Formulation

Figure 2: 4 phases in the excavation process: penetration, dragging, closing,
and lifting. The black arrows represent the bucket movement in each phase. In
this work, we only focus on the penetration phase, where the jamming problem
often happens. The brown line shows the terrain surface. Figure 3: Vertically
downward bucket movements will result in large contact forces and jamming. The
bucket cannot penetrate into the terrain. An optimal strategy should sweep the
surface of the terrain for downstream penetration.

In the autonomous excavation literature, the excavation process can usually be
divided into 4 phases: penetration, dragging, closing, and lifting (Fig. 2)
[6]. The jamming problem often happens at the penetration phase during the
excavation of irregular rigid objects. Compared to the granular material, such
as sand, irregular rigid objects are larger in size and non-deformable. The
contact force between the bucket and the terrain can easily become large and
result in jamming (Fig. 3), where the bucket cannot follow the planned path
due to the contact with the terrain. In this work, we focus on how to
penetrate the excavator bucket efficiently without causing jamming. The study
of the other three phases is left to future work.

We formulate a robotic excavation penetration of rigid objects problem.
Consider a flat terrain composed of irregular rigid objects, a robotic
manipulator attempts to penetrate the end-effector bucket into the terrain at
a certain depth for downstream excavation. Based on the bucket pose, the
bucket velocity, and the measured contact force, the agent needs to find the
appropriate robot command to penetrate into the terrain. We make the following
assumptions on the task: 1) we emulate an excavator using a $7DoF$ Franka
manipulator with a bucket as the end-effector; 2) the objects being excavated
are reachable by the bucket; 3) if the teeth of the bucket reach a certain
depth $d_{target}$ below the surface of the terrain, we assume that the
penetration phase is finished; 4) when a large contact force is detected, the
robot controller would take over and issue a halt to the robot, which we use
as a criterion for whether jamming happens or not throughout the paper; 5)
after the penetration phase, the rest of the trajectory including dragging,
closing, and lifting can be efficiently planned so that no jamming will happen
in those phases.

##  IV Approach

We propose a robotic excavation penetration framework for efficiently
penetrating different terrains. We design three manipulation primitives to
avoid jamming during the penetration. The parameters of the manipulation
primitives are learned using offline RL from a pre-collected dataset. The
dataset consists of demonstrated penetration trajectories collected using
teleoperation and scripted policy. With IQL, we can learn the penetration
policy even if the penetration trajectories in the offline dataset are sub-
optimal. Intuitively, IQL stitches several sub-optimal penetration
trajectories and finds an optimal penetration policy that is not only without
jamming, but also fast and with smaller contact forces. To generalize the
learned penetration policy, we introduce two LSTM auto-encoders to encode the
terrain configurations and terrain types using the current excavation
trajectories and the demonstrated excavation trajectories, respectively. Given
a new terrain type, the penetration policy can be fine-tuned with a few online
data collections.

###  IV-A Manipulation Primitives for Excavation Penetration

We define the manipulation primitives as the bucket’s desired movement. The
desired movement is a reference bucket velocity tracked by an impedance
controller. In [6], only discrete actions are empirically selected as the
primitives. The limited choices of 10 discrete primitives could result in sub-
optimal and less flexible policies. And the primitives that work for one rigid
object may not be able to generalize to other rigid objects.

Figure 4: 3 manipulation primitives: sweep, rotate, and penetration. In sweep,
the robot generates horizontal bucket motion in the $x$ axis; in rotate, the
robot generates rotational bucket motion in the pitch axis; in penetrate, the
robot generates vertical bucket motion in $z$ axis. sweep and rotate
manipulate the surface of the terrain to facilitate the penetration, and
penetrate selects proper vertical velocity to penetrate the terrain. The red
arrows represent the desired bucket movement in each primitive.

Different from the discrete actions in [6], we propose to use three primitives
with continuous parameters, sweep, rotate, and penetration (Fig. 4). With the
continuous parameters, we are able to select continuous bucket velocities and
force/displacement limits in order to avoid jamming. Each primitive generates
a bucket velocity in one dimension with the reference frame shown in Fig. 4.
Specifically, in sweep, the robot generates horizontal bucket motion in the
$x$ axis, so that the teeth of the bucket will sweep the surface of the
terrain and remove the fragments that cause jamming. In rotate, the robot
generates rotational bucket motion in the pitch axis to loose the surface of
the terrain. In penetrate, the robot generates vertical bucket motion in the
$z$ axis to penetrate the bucket into the terrain. The parameters of sweep and
rotate are $(v^{x},F_{lim}^{x},d_{lim}^{x})$ and
$(\omega^{pitch},M_{lim}^{pitch},\alpha_{lim}^{pitch})$, respectively, where
$v^{x}$ and $\omega^{pitch}$ are continuous reference velocities,
$F_{lim}^{x}$ and $M_{lim}^{pitch}$ are force/torque limits, and $d_{lim}^{x}$
and $\alpha_{lim}^{pitch}$ are displacement/angle limits. The limits are the
threshold where the primitives should change the moving direction. The
parameters of penetrate are $(v^{z},F_{lim}^{z})$. If the force limit
$F_{lim}^{z}$ is reached, the bucket will be lifted up to avoid jamming.

Intuitively, sweep and rotate manipulate the surface of the terrain to
facilitate the penetration, and penetrate selects proper vertical velocity to
penetrate the terrain. A small force may not generate enough force to
manipulate the terrain surface or penetrate the terrain, while a large force
could easily result in jamming.

###  IV-B Offline Reinforcement Learning for Excavation Penetration

Figure 5: The proposed framework to learn excavation policy using offline RL
followed by online fine-tuning.

In order to avoid the dangerous and lengthy on-robot learning, we propose to
learn the parameters of the manipulation primitives with offline RL from an
offline dataset (Fig. 5). The dataset consists of demonstrated penetration
trajectories collected using teleoperation and scripted policy. For the
teleoperation, we manually control the robot bucket using a remote controller.
For the scripted policy, we design a rule-based penetration strategy to select
the velocities and limits in the primitives. The demonstrated penetration
trajectories are saved for offline RL training.

An optimal penetration strategy should depend on not only the bucket pose and
contact force at the current time step, but also the terrain configurations
and terrain types. Thus our proposed method consists of two LSTM auto-encoders
to extract such features from penetration trajectories. The first encoder
network $q_{\phi_{1}}(z_{t}|c_{1:t})$ predicts $z_{t}$ from the current
penetration trajectory $c_{1:t}\in\mathbb{R}^{9\times t}$, where
$c_{i}=(x_{i},\dot{x}_{i},F_{i}^{ext})\in\mathbb{R}^{9}$ includes the relative
bucket pose $x_{i}=(x_{i}^{x},x_{i}^{z},\omega_{i}^{pitch})\in\mathbb{R}^{3}$
between the measured bucket pose and the initial bucket pose, bucket
velocities
$\dot{x}_{i}=(\dot{x}_{i}^{x},\dot{x}_{i}^{z},\dot{\omega}_{i}^{pitch})\in\mathbb{R}^{3}$,
and measured contact forces
$F_{i}^{ext}=(F_{i}^{x},F_{i}^{z},M_{i}^{pitch})\in\mathbb{R}^{3}$ at time
step $i$. The second encoder network $q_{\phi_{2}}(z_{demo}|c_{demo})$
predicts $z_{demo}$ from the demonstrated penetration trajectories
$c_{demo}\in\mathbb{R}^{9\times t_{demo}}$, where $t_{demo}$ is the length of
the demonstrated trajectory. Intuitively, with the LSTM auto-encoders, $z_{t}$
extracts the penetration progress and the current terrain configuration, and
$z_{demo}$ extracts the types of rigid objects and the information necessary
to excavate these rigid objects.

The observation space for the penetration policy $\pi(a|s)$ is defined as
$s=(x,\dot{x},F^{ext},z_{t},z_{demo})$. The action space is defined as the
continuous parameters in the manipulation primitives
$a=(v^{x},F_{lim}^{x},d_{lim}^{x},\omega^{pitch},M_{lim}^{pitch},\alpha_{lim}^{pitch},v^{z},F_{lim}^{z})$.
The reward function is defined as

$r=-w_{1}\times||d_{target}-d||_{2}^{2}-w_{2}\times||F^{ext}||_{2}^{2}$ (1)

where $w_{1}$ and $w_{2}$ are the weighted scalars, and $d$ is the measured
penetration depth. Essentially, this reward function encourages the bucket to
penetrate fast and with small contact forces.

To learn penetration policy from some sub-optimal demonstrations, we use IQL
[35], which has been shown to perform well on tasks that benefit from multi-
step dynamic programming by stitching several sub-optimal trajectories. IQL
tries to estimate the maximum Q-value over actions that are in support of the
data distribution. By utilizing the Expectile Regression, IQL does not need to
query the learned Q-function on out-of-sample actions. For our excavation
penetration task, the demonstration dataset contains several sub-optimal
penetration trajectories. We utilize IQL to stitch those trajectories and find
the best trajectory that can finish the task. In another word, the optimal
penetration trajectory is composed of several sub-trajectories from the
demonstrations. More details about IQL can be found in [35].

###  IV-C Online Fine-tuning and Different Terrains Generalization

Assume there are $N$ training terrains and $M$ unseen terrains. In our case,
$N=5$ and $M=1$ (fragmented rocks). We can obtain $N$ offline dataset
$(\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}_{j},...,\mathcal{D}_{N})$, where
$\mathcal{D}_{j}$ denotes the $jth$ dataset for the $jth$ terrain. The
penetration policy of the $jth$ terrain $\pi_{j}(a|s)$ can be obtained by
training IQL using $\mathcal{D}_{j}$. After training, $\pi_{j}(a|s)$ may
already find better penetration trajectories than the trajectories in
$\mathcal{D}_{j}$. In addition, $\pi_{j}(a|s)$ can be improved with a small
amount of online interaction. We continue to train IQL by initializing the
policy with $\pi_{j}(a|s)$. The newly collected online interactions are
appended to $\mathcal{D}_{j}$.

Since we include $z_{demo}$ into the observation to extract the types of rigid
objects, we can train a general penetration policy $\bar{\pi}(a|s)$ using the
entire dataset
$\mathcal{D}_{all}=(\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}_{j},...,\mathcal{D}_{N})$
[33]. The latent variable $z_{demo}$ performs as a classifier to distinguish
different terrains and it also captures the common features that can be
generalized to similar terrains. Given an unseen terrain, $z_{demo}$ can be
first inferred using $q_{\phi_{2}}(z_{demo}|c_{demo})$ by only collecting a
few demonstrated trajectories. We can then online fine-tune the policy with
$\bar{\pi}(a|s)$ as the initialization. This obviates the need for collecting
the entire dataset for a new terrain.

##  V Experiments

We aim to investigate three questions in our real-world experiments. First, we
examine if the proposed framework using offline RL can learn the efficient
bucket penetration policy without causing jamming. Second, we evaluate whether
the learned penetration policy outperforms the sub-optimal demonstrations in
the dataset. Third, we inspect whether one learned penetration policy can be
applied to different terrain types and handle unseen terrain.

###  V-A Experimental Setup

As shown in Fig. 1, our system includes one 7-DoF Franka robot manipulator,
one customized excavator bucket as the end-effector, and one tray containing
irregular rigid objects to be excavated. We assume that the objects being
excavated are uniformly distributed in the tray with a flat surface. We
conducted experiments on 5 different training terrains (sand, pea pebbles,
marble chips, red mulch, wood blocks), and one unseen terrain (fragmented
rocks), as shown in Fig. 6. To emulate the 4-DoF excavator, we only use the
shoulder panning, the shoulder lifting, the elbow lifting, and the wrist
lifting joints. The other 3 joints are fixed. The measured bucket-terrain
contact force is computed from the joint torques using the Jacobian matrix.
The contact force should be measured at the bucket teeth. In the experiments,
we use the force at the end-point of the $7th$ link to approximate it. Before
penetration, the bucket slowly moves downward to make contact with the
terrain. When the contact force is greater than $3N$, we assume that the
penetration phase starts. In the penetration phase, the agent controls the
bucket at $10Hz$, which is tracked and interpolated by a downstream impedance
controller at $1000Hz$. After a successful penetration, predefined dragging,
closing, and lifting trajectories can be executed without causing jamming.
Planning the entire excavation trajectory is left to future work.

Figure 6: Excavation penetration of 6 different terrains. (a) Sand; (b) Pea
Pebbles; (c) Marble Chips; (d) Red Mulch; (e) Wood Blocks; (f) Fragmented
Rocks.

###  V-B Implementation Details

####  V-B1 Demonstration Dataset Collection

We collect offline datasets on 5 training terrains except for the fragmented
rocks. For each of the 5 training terrains, 100 trajectories are collected
with the scripted policy, and 20 trajectories are collected using
teleoperation. In each demonstrated trajectory, there are about 75
$(s,a,r,s^{\prime})$ transitions. For the scripted policy, we uniformly sample
the parameters of the manipulation primitives from the parameter ranges (Table
I) as action $a$ . Those ranges are empirically selected to ensure safety for
the 5 training terrains and they are also utilized as the action space range
in offline RL training throughout the experiments. For the teleoperation, the
parameters of the manipulation primitives are extracted from the bucket
velocity and the states where the moving direction changes in a demonstrated
trajectory. We normalize both the observations and actions space to facilitate
the training.

To add variations in the training dataset, we also uniformly sample the
initial bucket-terrain contact position for each trajectory from a $10cm*15cm$
rectangular region on the surface of the terrain. After collecting each
trajectory, we manually reset the terrain to a flat surface. Note that the
distributions of the irregular rigid objects are different after each reset,
thus the contact dynamics are also different.

Table I: Parameters range of the manipulation primitives. Primitives | Parameters | Range
---|---|---
sweep | $v^{x}$ | $[-0.1,0.1]m/s$
$F_{lim}^{x}$ | $[25,40]N$
$d_{lim}^{x}$ | $[0.015,0.03]m$
rotate | $\omega^{pitch}$ | $[-0.5,0.5]rad/s$
$M_{lim}^{pitch}$ | $[40,70]N\cdot m$
$\alpha_{lim}^{pitch}$ | $[0.1,0.2]rad$
penetrate | $v^{z}$ | $[-0.03,0.03]m/s$
$F_{lim}^{z}$ | $[40,70]N$

####  V-B2 Behavioral Cloning Baseline

We implement behavioral cloning to learn a baseline policy from the offline
dataset to evaluate the learned offline RL policy. The network contains a
Multilayer Perceptron (MLP) with two fully connected layers. We use ReLU as
the activation function. And the learning rate is $3\times 10^{-4}$. The
inputs and outputs are the observations and actions in the offline dataset,
respectively. We use the same auto-encoders in the offline RL framework to
encode the trajectories.

####  V-B3 Offline Reinforcement Learning and Online Fine-tuning

We use the same training parameters as in the original IQL paper. We use Adam
optimizer with a learning rate $3\times 10^{-4}$, 2 layer MLP with ReLU
activations, and 256 hidden units for all networks. For the reward function,
we empirically find that $d_{target}=0.05m$ is deep enough for the downstream
excavation. The weighted scalars $w_{1}$ and $w_{2}$ are 400 and 0.0004,
respectively.

For the online fine-tuning, $z_{demo}$ is predicted using 10 trajectories from
the offline dataset. If the terrain has not been seen before, such as
fragmented rocks, we recollect 10 penetration trajectories using the scripted
policy to predict $z_{demo}$. (The reason that we recollect the trajectories
using the scripted policy instead of the learned policy is to avoid
distribution shifts. More details can be found in [36]). In the online fine-
tuning phase, we collect 20 trajectories, which is about $1500$ new online
interactions.

###  V-C Results and Discussions

Table II: Real-world experiments results of different penetration policies $\pi_{j}(a|s)$ learned on their own datasets $\mathcal{D}_{j}$. Terrains |  Sand |  Pea Pebbles |  Marble Chips |  Red Mulch |  Wood Blocks |  Fragmented Rocks
---|---|---|---|---|---|---
Vertically Downward | Jamming | Jamming | Jamming | Jamming | Jamming | Jamming
Behavioral Cloning | $-48.90\pm 11.06$ | $-35.40\pm 6.72$ | $-45.23\pm 10.08$ | $-30.20\pm 9.96$ | $-25.52\pm 7.82$ | $-$
Offline RL $\pi_{j}(a|s)$ | $-39.23\pm 9.30$ | $-30.63\pm 5.49$ | $-35.39\pm 15.12$ | $-26.84\pm 6.54$ | $-22.20\pm 7.94$ | $-$
Fine-tuned $\pi_{j}(a|s)$ | $-26.76\pm 4.60$ | $-28.83\pm 2.50$ | $-30.58\pm 3.30$ | $-27.03\pm 3.96$ | $-24.53\pm 7.09$ | $-$
Table III: Real-world experiments results of one general policy $\bar{\pi}(a|s)$ learned on the entire dataset $\mathcal{D}_{all}$.  Terrains |  Sand |  Pea Pebbles |  Marble Chips |  Red Mulch |  Wood Blocks |  Fragmented Rocks
---|---|---|---|---|---|---
General Policy $\bar{\pi}(a|s)$ | $-28.51\pm 7.02$ | $-32.37\pm 7.72$ | $-36.14\pm 9.46$ | $-57.90\pm 25.94$ | $-25.37\pm 8.32$ | $-40.17\pm 11.17$
Fine-tuned $\bar{\pi}(a|s)$ | $-26.07\pm 3.48$ | $-29.50\pm 4.01$ | $-31.11\pm 7.29$ | $-23.02\pm 5.81$ | $-23.98\pm 6.60$ | $-37.05\pm 10.81$

####  V-C1 Offline Reinforcement Learning Evaluation

Table II shows the performance of different penetration policies
$\pi_{j}(a|s)$ on different terrains. Note that the rewards are negative. A
larger reward means better penetration performance. There are 5 training
terrains with offline RL policies. To evaluate the performance of the
policies, we execute the policy on the terrain for 10 trials in each scenario.
We then record the means and standard deviations of the trajectory reward in
the table. No results are shown for the fragmented rocks using behavioral
cloning, offline RL $\pi_{j}(a|s)$, and fine-tuned $\pi_{j}(a|s)$, because no
offline datasets are collected on the fragmented rocks.

We also show the result of vertically downward trajectories, where the bucket
follows a fixed trajectory to move vertically downward. 5 out of 6 terrains
have jamming problems due to the large contact force. In contrast, the learned
offline RL policies do not cause jamming. We also compare the performance of
behavioral cloning and offline RL $\pi_{j}(a|s)$ in Table II. Offline RL
outperforms behavioral cloning on all 5 terrains. This shows the effectiveness
of the learn offline RL policies. By fine-tuning RL $\pi_{j}(a|s)$, we can see
that 3 out of 5 learned offline RL penetration policies are improved with a
few online interactions.

Figure 7: Average penetration duration and contact force for the
demonstrations and the RL policies.

Since the numbers in reward may not be intuitive to humans, we also report the
average penetration duration and contact force in a trajectory (Fig. 7). As we
can see that the learned offline RL policies have faster penetration and
smaller contact forces than the demonstration trajectories. This suggests that
the RL policies outperform the demonstrations in the dataset. There are some
exceptions, for example, the average contact force in fine-tuned RL for wood
blocks is larger than the average contact force in the demonstrations. The
possible reason is that the contact force for wood blocks penetration is
relatively small due to the light weight of wood blocks. So the contact force
would take smaller weights in the reward function, and the RL policies
sacrifice contact force for faster penetration.

To evaluate the generalization ability of the proposed method, we show the
performance of one general policy $\bar{\pi}(a|s)$ trained on the entire
dataset $\mathcal{D}_{all}$ (Table III). This suggests that one learned
penetration policy can be successfully applied to different rigid objects. The
last row in Table III shows that the general policy can also be improved with
online fine-tuning. There is one outlier data point for the red mulch. The
possible reason is that the red mulch is long and thin. One particle of red
mulch got stuck into the bucket teeth in the experiments, resulting in a large
contact force during the penetration. For the unseen fragmented rocks, we
consider it the hardest task. Because the particles of the fragmented rocks
are larger and sharper than the pea pebbles and the marble chips. As shown in
the last column in table III, the learned $\bar{\pi}(a|s)$ performs well on
the fragmented rocks without causing jamming, suggesting that $\bar{\pi}(a|s)$
is able to handle out-of-distribution rigid objects.

####  V-C2 A Case Study for Changing the Jamming Threshold

Figure 8: Jamming-Free Rate VS Maximum Contact Force Allowed

As mentioned in Section III-B, the Franka robot controller will issue a halt
to the robot if large contact forces are detected. This is utilized as a
criterion for whether jamming happens or not. However, different robots or
excavators could generate different maximum torques. A jamming-free scenario
for one robot may become a jamming problem if we only have the access to
another less powerful machine. If we redefine the jamming threshold as the
maximum contact force allowed in one penetration trajectory, we can plot the
jamming-free rate versus the maximum contact force allowed in Fig. 8 (We have
recorded the maximum contact force in a trajectory when we evaluate each
trajectory). In the figure, as the maximum contact force allowed increases,
the jamming-free rate also increases. We can see that the learned offline RL
policies have higher jamming-free rates than the demonstrations. This shows
the advantages of the learned offline RL policies when the jamming criteria
become more strict.

##  VI Conclusion and Future Work

This work proposes an offline reinforcement learning framework for the robotic
excavation of rigid objects. We learn the excavator bucket penetration
policies from the offline collected dataset. Three manipulation primitives are
proposed and the action parameters are learned with offline RL. Real-world
experiments show that the learned policy can successfully avoid jamming
problems and outperform the sub-optimal demonstrations in the dataset. One
learned policy can also be applied to different terrains and quickly adapt to
an unseen terrain with only a few online interactions. One of the limitations
of this work is the lack of visual perception. It will be helpful to utilize
the camera to find the gaps between rigid objects before penetration starts,
especially for large rocks and stones. For future work, it is noteworthy that
our framework only focuses on the penetration phase in the excavation process.
More general excavation policies are possible to be learned using this
framework if we provide a corresponding offline demonstration dataset for
other excavation phases.

## References

  * [1] L. Zhang, J. Zhao, P. Long, L. Wang, L. Qian, F. Lu, X. Song, and D. Manocha, “An autonomous excavator system for material loading tasks,” _Science Robotics_ , vol. 6, no. 55, p. eabc3164, 2021.
  * [2] E. Jelavic, D. Jud, P. Egli, and M. Hutter, “Towards autonomous robotic precision harvesting: Mapping, localization, planning and control for a legged tree harvester,” _arXiv preprint arXiv:2104.10110_ , 2021.
  * [3] D. Jud, P. Leemann, S. Kerscher, and M. Hutter, “Autonomous free-form trenching using a walking excavator,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 3208–3215, 2019.
  * [4] Q. Lu and L. Zhang, “Excavation learning for rigid objects in clutter,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 7373–7380, 2021\.
  * [5] Q. Lu, Y. Zhu, and L. Zhang, “Excavation reinforcement learning using geometric representation,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4472–4479, 2022.
  * [6] Y. Zhu, L. Wang, and L. Zhang, “Excavation of fragmented rocks with multi-modal model-based reinforcement learning,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 6523–6530.
  * [7] J. Zhao and L. Zhang, “Tasknet: A neural task planner for autonomous excavator,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 2220–2226.
  * [8] Y. Yang, P. Long, X. Song, J. Pan, and L. Zhang, “Optimization-based framework for excavation trajectory generation,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 1479–1486, 2021.
  * [9] E. Jelavic and M. Hutter, “Whole-body motion planning for walking excavators,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 2292–2299.
  * [10] E. Jelavic, Y. Berdou, D. Jud, S. Kerscher, and M. Hutter, “Terrain-adaptive planning and control of complex motions for walking excavators,” in _2020 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2020, pp. 2684–2691.
  * [11] H. A. Fernando, “Control and learning for robotic excavation,” Ph.D. dissertation, Queen’s University (Canada), 2021.
  * [12] D. Jud, G. Hottiger, P. Leemann, and M. Hutter, “Planning and control for autonomous excavation,” _IEEE Robotics and Automation Letters_ , vol. 2, no. 4, pp. 2151–2158, 2017.
  * [13] M. Lee, H. Choi, C. Kim, J. Moon, D. Kim, and D. Lee, “Precision motion control of robotized industrial hydraulic excavators via data-driven model inversion,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 1912–1919, 2022.
  * [14] J. Park, B. Lee, S. Kang, P. Y. Kim, and H. J. Kim, “Online learning control of hydraulic excavators based on echo-state networks,” _IEEE Transactions on Automation Science and Engineering_ , vol. 14, no. 1, pp. 249–259, 2016.
  * [15] F. E. Sotiropoulos and H. H. Asada, “Dynamic modeling of bucket-soil interactions using koopman-dfl lifting linearization for model predictive contouring control of autonomous excavators,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 1, pp. 151–158, 2021.
  * [16] F.E. Sotiropoulos and H.H. Asada, “A model-free extremum-seeking approach to autonomous excavator control based on output power maximization,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1005–1012, 2019.
  * [17] R. J. Sandzimier and H. H. Asada, “A data-driven approach to prediction and optimal bucket-filling control for autonomous excavators,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 2682–2689, 2020.
  * [18] P. Egli, D. Gaschen, S. Kerscher, D. Jud, and M. Hutter, “Soil-adaptive excavation using reinforcement learning,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 4, pp. 9778–9785, 2022.
  * [19] F. E. Sotiropoulos and H. H. Asada, “Autonomous excavation of rocks using a gaussian process model and unscented kalman filter,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 2491–2497, 2020.
  * [20] F. E. Sotiropoulos, “Methods for control in robotic excavation,” Ph.D. dissertation, Massachusetts Institute of Technology, 2021.
  * [21] S. Gu, E. Holly, T. Lillicrap, and S. Levine, “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 3389–3396.
  * [22] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine, “Learning complex dexterous manipulation with deep reinforcement learning and demonstrations,” _arXiv preprint arXiv:1709.10087_ , 2017.
  * [23] X. Zhang, S. Jin, C. Wang, X. Zhu, and M. Tomizuka, “Learning insertion primitives with discrete-continuous hybrid action space for robotic assembly tasks,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 9881–9887.
  * [24] B. J. Hodel, “Learning to operate an excavator via policy optimization,” _Procedia Computer Science_ , vol. 140, pp. 376–382, 2018.
  * [25] I. Kurinov, G. Orzechowski, P. Hämäläinen, and A. Mikkola, “Automated excavator based on reinforcement learning and multibody system dynamics,” _IEEE Access_ , vol. 8, pp. 213 998–214 006, 2020.
  * [26] P. Egli and M. Hutter, “Towards rl-based hydraulic excavator automation,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 2692–2697.
  * [27] P. Egli and M. Hutter, “A general approach for the automation of hydraulic excavator arms using reinforcement learning,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 5679–5686, 2022.
  * [28] S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” _arXiv preprint arXiv:2005.01643_ , 2020.
  * [29] L. Pinto and A. Gupta, “Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours,” in _2016 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2016, pp. 3406–3413.
  * [30] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn, “Robonet: Large-scale multi-robot learning,” _arXiv preprint arXiv:1910.11215_ , 2019.
  * [31] A. Mandlekar, D. Xu, R. Martín-Martín, S. Savarese, and L. Fei-Fei, “Learning to generalize across long-horizon tasks from human demonstrations,” _arXiv preprint arXiv:2003.06085_ , 2020.
  * [32] A. Mandlekar, F. Ramos, B. Boots, S. Savarese, L. Fei-Fei, A. Garg, and D. Fox, “Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 4414–4420.
  * [33] T. Z. Zhao, J. Luo, O. Sushkov, R. Pevceviciute, N. Heess, J. Scholz, S. Schaal, and S. Levine, “Offline meta-reinforcement learning for industrial insertion,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 6386–6393.
  * [34] A. Nair, A. Gupta, M. Dalal, and S. Levine, “Awac: Accelerating online reinforcement learning with offline datasets,” _arXiv preprint arXiv:2006.09359_ , 2020.
  * [35] I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with implicit q-learning,” _arXiv preprint arXiv:2110.06169_ , 2021.
  * [36] V. H. Pong, A. V. Nair, L. M. Smith, C. Huang, and S. Levine, “Offline meta-reinforcement learning with online self-supervision,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 17 811–17 829.