# Hydrodynamic Manipulation of Nano-Objects by Thermo-Osmotic Flows

Martin Fränzl  Frank Cichos<EMAIL_ADDRESS>Peter Debye
Institute for Soft Matter Physics, Universität Leipzig, Linnestr. 5, 04103
Leipzig, Germany

###### Abstract

The manipulation of micro- and nano-objects is of great technological
significance to construct new materials, manipulate tiny amounts of liquids in
fluidic systems, or detect minute concentrations of analytes. It is commonly
approached by the generation of potential energy landscapes, for example, with
optical fields. Here we show that strong hydrodynamic boundary flows enable
the trapping and manipulation of nano-objects near surfaces. These thermo-
osmotic flows are induced by modulating the van der Waals interaction at a
solid-liquid interface with optically induced temperature fields. We use a
thin gold film on a glass substrate to provide localized but reconfigurable
point-like optical heating. Convergent boundary flows with velocities of tens
of micrometres per second are observed and substantiated by a quantitative
physical model. The hydrodynamic forces acting on suspended nanoparticles and
attractive van der Waals or depletion induced forces enable precise
positioning and guiding of the nanoparticles. Fast multiplexing of flow fields
further provides the means for parallel manipulation of many nano-objects. Our
findings have direct consequences for the field of plasmonic nano-tweezers as
well as other thermo-plasmonic trapping schemes and pave the way for a general
scheme of nanoscopic manipulation with boundary flows.

##  I Introduction

The control and manipulation of nano-objects is a key element for future
nanophotonics [1, 2, 3, 4, 5], material science [6, 4, 7], biotechnology [2,
8, 9] or even quantum sensing [10]. Analytes dissolved in liquids, for
example, need to be delivered, concentrated, separated or locally confined for
further studies to become eventually processed and removed. Photonic elements
including plasmonic nano-structures require precise positioning or controlled
rearrangements to serve as adaptive functional structures. Key elements of the
control at the micro- and nanoscale are often pressure driven fluidics
transporting liquid volume and solutes as well as the generation of energy
landscapes or force fields. The latter is achieved with optical [3] and
plasmonic tweezers [11, 12], magnetic fields [13], or using electrokinetic
[14] or opto-electronic [15] effects. Especially in the field of plasmonic
tweezers and nanoantennas where light is used to excite collective electron
motion in noble-metals, the Joule losses lead to the unavoidable generation of
heat at boundaries as an unwanted side effect [16, 17]. Yet, such optically
generated temperature fields seem also suitable for the manipulation of nano-
objects in liquids, for example, for the trapping of nanoparticles [18] and
single molecules [19] or protein aggregates [20] as well as for manufacturing
active particles [21, 22, 23, 24]. Those techniques rely on a drift of
molecules and particles in optically generated temperature gradients termed
thermophoresis or suggest thermo-electric effects [25] relying on a thermally
induced charge separation. In addition, thermo-electrohydrodynamic effects
using time-varying electric fields have been proposed for rapid particle
transport [26, 27] and convective effects that arise from temperature-induced
density changes in the large liquid cells have been reported [28, 29, 30, 31].

Here we report on a fundamental physical process that is able to provide a
versatile trapping and manipulation of nano-objects near surfaces in the
simplest geometries. Contrary to most other techniques, our scheme is based on
hydrodynamic flows generated by thermo-osmosis. Thermo-osmosis relies on a
perturbation of the interfacial interactions at a solid-liquid boundary and is
present in all experiments involving temperature gradients in plasmonic
structures including plasmonic tweezers. We show that local temperature
gradients on a thin gold film induce strong interfacial flows of several 10 to
$100~{}\mathrm{\upmu m\,s^{-1}}$ in the direct vicinity (10 nm) of the gold
film that results in a flow pattern reminiscent of convection. Based on a
fully quantitative analysis of our experimental results we reveal that these
thermo-osmotic flows on gold-water interfaces are induced by a temperature-
induced perturbation of the van der Waals (vdW) interactions. Nano-objects
suspended in the liquid are therefore dragged by the hydrodynamic forces
originating from these flows. Utilizing attractive vdW interactions of the
nano-object with the gold surface or temperature-induced depletion, we are
able to trap and manipulate different types of nano-objects near the surface.
Due to the speed of heating at small scales, we are able to multiplex flow
fields to manipulate multiple objects with great precision. Our detailed
analysis of the flow fields, the localization accuracy of nano-objects, and a
comparison with numerical and theoretical predictions provide a quantitative
understanding of these effects and paves the way for controlling boundary
layer dynamics to manipulate objects at the smallest length scales in
solutions.

Figure 1: Thermo-hydrodynamic manipulation of Au NPs in NaCl solution. a,
Sketch of the sample design. It consists of two glass slides that confine a
$5~{}\mathrm{\upmu m}$ thin liquid film of gold nanoparticles (AuNPs)
dispersed in NaCl solution. The lower glass slide carries a 50 nm Au film that
is locally heated by optical absorption a focused laser. b, Schematic of the
experimental setup. It comprises an inverted optical microscope equipped with
an acusto-optical deflector controlled, steerable focused laser with a
wavelength of $\lambda=532~{}\mathrm{nm}$. The AuNPs are observed using
darkfield illumination with an oil-immersion dark-field condenser (NA 1.2) and
a 100$\times$ oil-immersion objective set to NA 0.6 and recorded with an EMCCD
camera.

##  II Results and Discussion

###  II.1 Experimental Configuration and Working Principle

Our experiments rely on a simple sample geometry with a gold film (50 nm) that
is deposited on a microscopy glass coverslip (Fig. 1a). The sample chamber
contains a suspension of gold nanoparticles (AuNPs) or other nano-objects
(polystyrene NPs and ellipsoids) with a controlled amount of salt (NaCl),
surfactants (SDS, …) or polymers (PEG). The gold film is heated locally in an
inverted microscopy setup by a highly focused laser (532 nm) using beam
steering optics (Acusto-Optic-Deflector, AOD). The nano-objects are observed
using darkfield illumination with an oil-immersion darkfield condenser (NA
1.2) and a 100$\times$ oil-immersion objective set to NA 0.6 (Fig. 1b).
Additional details of the experimental setup and sample preparation are
provided in the Methods section.

The trapping of nano-objects as detailed in the following is comprising two
effects. i) The vertical confinement of the suspended objects as achieved by
an attractive interaction of the suspended nano-objects with the gold surface,
which is found to be the vdW interaction for gold nanoparticles and can be
replaced by depletion forces for other materials. ii) The generation of
thermo-osmotic boundary flows that are induced by the local heating and the
corresponding perturbation of the liquid-solid interactions. This boundary
flow is directed radially inwards to the heated spot and provides a confining
hydrodynamic force on suspended objects at the heating spot.

Figure 2: DLVO potential, lateral diffusion analysis, temperature distribution
and thermo-osmotic flow field. a, Plot of the DLVO potential, equation (1),
between a 250 nm Au NP and a 50 nm Au film on a glass surface as function of
particle-surface distance $d$ for different NaCl concentrations $c_{0}$. b,
The measured diffusion coefficient $D_{\parallel}/D_{0}$ parallel to the Au
film with respect to the bulk diffusion coefficient $D_{0}$ as function of the
NaCl concentration. c, Relation between $\langle D_{\parallel}\rangle/D_{0}$
and the mean distance $\langle d\rangle$. d, Simulation of the relative
temperature increment in the $xz$-plane of the sample. e, Experimentally
obtained temperature increment $\Delta T_{\mathrm{max}}$ as a function of the
incident laser power $P_{0}$ (green data points) compared to the simulated
values (green curve). f, Measured thermo-osmotic flow field in the $xy$-plane
in close proximity to the gold film ($z<500~{}\mathrm{nm})$. g, Measured
thermo-osmotic flow field in the $xz$-plane. h, Illustration of the measured
flow field planes in f and g. i, j, The $x$\- and $z$-component of the
measured flow velocities compared to the simulation results in k and l.

###  II.2 Dynamics of AuNPs Close to a Au Film

Consider a single AuNP with a radius of $R=125~{}\mathrm{nm}$ that is
suspended in an aqueous solution of NaCl at a 10 mM concentration and
diffusing in a thin liquid film of about $3~{}\mathrm{\upmu m}$ thickness over
a 50 nm Au film (Fig. 1a). Exploring the diffusion of the particle we observe
a restriction of the $z$-positions to a thin layer close to the gold film. The
gold particle never defocuses under these conditions while it does in
deionized (DI) water (Supplementary Video 1). This restricted out-of-plane
motion is the result of interactions comprising an attractive vdW contribution
and a repulsion of the electrostatic double layers of the particle and surface
[32] as described by the DLVO theory and the gravitational potential (see
Supplementary Information for details).

$V(d,c_{0})=V_{\mathrm{E}}(d,c_{0})+V_{\mathrm{vdW}}(d)+V_{\mathrm{G}}(d)\ .$
(1)

The total potential is depicted in Figure 2a for different salt concentrations
(see SI for parameters) as a function of the surface-to-surface distance
$d=z-R$. The stronger screening of the surface charges at the gold film and
the AuNP at higher salt concentration increase the importance of attractive
vdW interactions to create this secondary minimum in the DLVO part of the
potential. This potential also influences the observed dynamics as the
particle couples with its hydrodynamic flow field to the solid boundary [33].
The in-plane $D_{\parallel}$ (equation 2) and out-of-plane $D_{\perp}$
diffusion coefficient (see Supplementary Information for details) are
modulated with the distance $z$ of the particle from the wall.

$\frac{D_{\parallel}(z)}{D_{0}}\approx
1-\frac{9}{16}\frac{R}{z}+\frac{1}{8}\left(\frac{R}{z}\right)^{3}\pm\;\dots=\gamma_{\parallel}^{-1}(z)$
(2)

Over the course of a diffusion trajectory, the particle samples different
diffusion coefficients according to its probability density
$p(d)\propto\exp(-V(d,c_{0})/(k_{\mathrm{B}}T))$ to be at a distance $d$ from
the surface (filled regions in Fig. 2a). The observed in-plane diffusion
coefficient is thus a weighted average of the diffusion coefficient over the
different vertical positions $d$. Using $p(d)$ we can calculate the
corresponding salt concentration dependence of the diffusion coefficient and
compare that to the experimental results. Figure 2b shows that the
experimentally observed $D_{\parallel}$ is decreasing with increasing salt
concentration due to the hydrodynamic coupling in fair agreement with the
theoretical predictions (see Supplementary Information for details).
Calculating in addition the mean distance $\langle d\rangle$ of the particle
from the surface reveals that only in the case of $c_{0}=10$ mM the particle
is confined in the DLVO potential well, while at lower salt concentrations an
enhanced probability of finding the particle near the surface is the cause of
the observed diffusion coefficient. These calculations help us to estimate the
mean distance $\langle d\rangle$ of the particle from the surface, which is
about $1.5~{}\mathrm{\upmu m}$ and $0.9~{}\mathrm{\upmu m}$ for the lowest
NaCl concentrations (Fig. 2c). At a concentration of $c_{0}=10$ mM the
particle is hovering at a distance of $\langle d\rangle=20$ nm surface. Note
that this corresponds to values of $z/(2R)\approx 0.58$, which is far below
the commonly explored region of the hydrodynamic coupling of colloids to walls
[33] allowing to experimentally explore new terrains also in the field of
hydrodynamic wall coupling of colloids.

###  II.3 Hydrodynamic Particle Confinement

When tightly focusing the light of 532 nm wavelength to the gold film, a part
of the incident energy (about 30 %) is absorbed and converted into heat that
perturbs the liquid-solid interactions. The temperature rise at the gold
surface can be determined using a thin nematic liquid crystal (5CB) film and
substantiated by finite element simulations with the complete three-
dimensional temperature profile in the solution (see Fig. 2d, e and
Supplementary Information for details).

These local temperature perturbations of the solid-liquid interactions at the
interface induce a thermo-osmotic flow [34, 35]. Taking a liquid volume
element close to the solid from the cold side and exchanging that with one at
the hot side would not only transport heat since the liquid volumes have
different temperatures, but also additional free energy as the liquid has a
different interaction with the solid in these regions. The flow is induced in
an ultrathin boundary layer corresponding in thickness to the length scale of
liquid-solid interactions. Since the interaction range of liquid-solid
interactions is only a few nanometers (the characteristic length of
interfacial interactions), the boundary flow is due to its dimension collapsed
into a quasi-slip hydrodynamic boundary condition:

$v_{\parallel}=-\frac{1}{\eta}\int\limits_{0}^{\infty}z\,h(z)\;\mathrm{d}z\;\frac{\nabla_{\parallel}T}{T}=\chi\,\frac{\nabla_{\parallel}T}{T}\
,$ (3)

where $h(z)$ is the excess enthalpy, $T$ the temperature and
$\nabla_{\parallel}T$ is the temperature gradient parallel to the surface. The
integral can be summarized to a thermo-osmotic coefficient $\chi$. The thermo-
osmotic coefficient $\chi$, therefore, contains all information about the
interfacial interaction between the liquid and the solid. If $\chi<0$ the
liquid is driven to the cold, whereas for $\chi>0$, the liquid is driven to
the hot. These boundary flows are present at all liquid-solid interfaces with
tangential temperature gradients, though, they are commonly overlooked. They
become particularly important for plasmonic and thermo-plasmonic trapping
[16], as those techniques rely on the dynamics of molecules and particles in
the direct vicinity of plasmonic nanostructures.

Figure 3: Forces on trapped NPs in 10 mM NaCl. a, The lateral trap stiffness
obtained from the experimental position histograms (blue data points) as
function the laser power $P_{0}$ compared to the simulation result (blue solid
line). b, The $z$-component of the thermo-osmotic drag force
$F^{\mathrm{TF}}_{z}$ (blue line), the optical force $F^{\mathrm{OF}}_{z}$
(green line) and the total force $F^{\mathrm{OF}}_{z}+F^{\mathrm{TF}}_{z}$
(black dashed line) as function the incident laser power $P_{0}$ for a NP
located at $x=0$, $d=30$ $(z=d+R)$. The attractive DLVO force
$F^{\mathrm{DLVO}}_{z}$ is independent of the incident laser power and
depicted as horizontal, red line. c, Trajectory of a AuNP for a heating laser
power of 1.25 mW (see Supplementary Video 2 for details). The inset shows the
corresponding lateral distribution histogram. d, Time traces of the
$z$-position for three different laser powers. e, Trajectory of a AuNP for a
heating laser power of 2.5 mW, this is, above the threshold power of 2.25 mW.

The boundary flow drives the flow field inside the fluid film. The resulting
volumetric flow field can be tracked experimentally by single AuNPs in DI
water, where the particles are not confined to a surface layer as reported
above. We analyze the in-plane ($xy$) position of the particle and its
$z$-position, where the latter is estimated from the radius $r_{0}$ of the
defocussed particle images (see Supplementary Video 2 and Supplementary
Information for details). The measured velocity distributions in the
$xy$-plane near the gold layer and in the $xz$-plane are shown in Fig. 2f and
g, respectively. The $x$\- and $z$-component of the measured flow velocities
are depicted in Fig. 2i, j and compare well to simulation results in Fig. 2k,
l. From these measurements, we extract a thermo-osmotic coefficient on the
order of $\chi\sim 10\cdot 10^{-10}~{}\mathrm{m^{2}\,s^{-1}}$ (see
Supplementary Information for details). We can break down the contributions to
this value with equations (4) and (5) to estimate the double layer and vdW
contributions using the experimental parameters. Note that AuNP do not show
thermophoresis due to their high thermal conductivity and thus isothermal
surface.

$\chi_{\mathrm{E}}=\frac{\varepsilon\zeta^{2}}{8\eta}\approx 0.8\cdot
10^{-10}~{}\mathrm{m^{2}/s^{-1}}\ .$ (4)

For the electrostatic contribution we used $\zeta=-30~{}\mathrm{mV}$ [36] and
$\varepsilon=80\,\varepsilon_{0}$ (see Methods section for details). An
estimate of the vdW contribution can be given by

$\chi_{\mathrm{vdW}}=\frac{A_{\mathrm{H}}\beta T}{3\pi\eta d_{0}}\approx
9.3\cdot 10^{-10}~{}\mathrm{m^{2}\,s^{-1}}\ ,$ (5)

with $\beta=0.2\cdot 10^{-3}~{}\mathrm{K}^{-1}$ being the thermal expansion
coefficient of water and $d_{0}=0.2~{}\mathrm{nm}$ for the cut-off parameter
[34] (see Supplementary Information for details). The sum of both
contributions $\chi=\chi_{\mathrm{E}}+\chi_{\mathrm{vdW}}=10.1\cdot
10^{-10}~{}\mathrm{m^{2}\,s^{-1}}$ matches well the experimental result and
suggests that thermo-osmosis at gold-water interfaces is governed by vdW
interactions. The obtained quasi slip velocities are ranging up to 80
$\upmu{\rm m/s}$ and provide, due to their omnipresence, a unique tool for
nanofluidics. These thermo-osmotic flows are induced without any external
pressure difference. They can be controlled by the light intensity heating
laser and are quickly switched due to the extremely fast heat conduction at
these length scales. Moreover the finding of the vdW dominated thermo-osmotic
flows suggest that such contributions must be present in any plasmonic
trapping experiment with extended gold structures [16, 17, 27, 12].

Using $F^{\mathrm{TF}}_{x}=6\pi\eta R\,\gamma_{\parallel}\,v_{x}$ and
$F^{\mathrm{TF}}_{z}=6\pi\eta R\,\gamma_{\perp}v_{z}$, where
$\gamma_{\parallel}$ and $\gamma_{\perp}$ are the correction factors for the
friction coefficient of a sphere close to a surface we are able to extract the
hydrodynamic forces that are exerted on the AuNP tracers (see equation (2) and
Supplementary Information for details). The lateral forces allow to confine
objects at the heating spot, yet the hydrodynamic force normal to the surface
($z$-direction) is repulsive without any additional interaction. Finally, such
boundary flows with substantial vertical velocity gradients also exhibit a
vorticity (see Supplementary Information for details) that generates a torque
on suspended objects causing them to rotate [37].

Figure 4: Manipulation of NPs over a Au film in NaCl and SDS solution. a, A
AuNP with 50 nm radius trapped at a NaCl of 30 mM (Supplementary Video 5). b,
Manipulation of two AuNPs by a multiplexed laser beam (Supplementary Video 6).
c, Control of three AuNPs (Supplementary Video 7). d, Actuation of a single
AuNP on a circular trajectory by a steerable laser beam (Supplementary Video
8). The green and white, dashed arrows denote the moving direction of the
laser focus and particle, respectively. e, Generation of thermo-viscous flows
by rotating the laser focus on a circle with a rotation frequency of
$f=500~{}\mathrm{Hz}$ at high laser powers (Supplementary Video 9). Note, that
laser movement (green arrow) and the thermo-viscous flow (white, dashed arrow)
and have opposite directions. f, Attraction of a AuNP and PS NPs in 5 mM SDS
due to depletion (Supplementary Video 10). g, A AuNP (125 nm radius) trapped
in an ensemble of polystyrene (PS) NPs of the same size (Supplementary Video
11). h, Attraction of PS ellipsoids ($2.39~{}\mathrm{\upmu m}$ major-axis
length, $0.34~{}\mathrm{\upmu m}$ minor-axis length) in 5 mM SDS
(Supplementary Video 12).

###  II.4 Single Particle Trapping and Flow Field Multiplexing

The repulsive normal component caused by the hydrodynamic drag is now
superimposed with an attractive force due to the DLVO potential when
increasing the NaCl concentration. The surface-to-surface distance between
AuNP and gold film and the depth of the appearing secondary DLVO potential
minimum can be controlled by the NaCl concentration. At a NaCl concentration
of about $c_{0}=10$ mM, the attractive potential has a depth of about
$10\,k_{\mathrm{B}}T$ (see Fig. 2a) and is strong enough to compete with the
vertical drag force and additional optical forces on the AuNP to trap the
particle above the heating spot.

Supplementary Video 3 demonstrates this trapping of an AuNP above the hot spot
on the Au surface. This is purely the result of the hydrodynamic drag forces
generated by the thermo-osmotic flow and the attractive vdW interaction
between the AuNP and the Au film. This observation is substantiated by a
quantitative evaluation of the lateral trap stiffness and vertical forces, as
depicted in Fig. 3a, b. The fluctuations of the particle in the hydrodynamic
flow arise from a balance of the restoring hydrodynamic currents and the
diffusive currents. Analysis of the lateral position histograms (inset in Fig.
3c for 1.25 mW) yields an effective stiffness of the trap (Fig. 3a) that well
matches the predictions based on the thermo-osmotic flow (Fig. 2i, j). The
hydrodynamic trapping stiffness increases linearly up to a heating power of
about $1.8$ mW. At this power, the vertical forces become strong enough to let
the particle escape the secondary DLVO minimum, which is visible from the
$z$-position time traces displayed in Fig. 3d. The AuNPs are then observed to
move vertically out of the DLVO potential to follow the flow inside the sample
and to eventually return to the boundary flow via sedimentation (Fig. 3e,
Supplementary Video 4). The forces which eject the particle from the potential
comprise the hydrodynamic and optical forces due to the radiation pressure
from the heating laser leaked through the film. We have evaluated the
individual contributions in simulations. They are shown together with the
hydrodynamic force and the total vertical force as compared to the attractive
force of the DLVO potential (Fig. 3b) and provide quantitative agreement
(threshold heating power of $2.25$ mW) with our experimental results. Note
that while the stationary distribution of particles in the vertical direction
is not influenced by the diffusive dynamics, the escape rate from the
potential well is heavily altered by the fact that the vertical diffusion
coefficient $D_{\perp}$ of the particle is decreasing to zero when approaching
the gold film. This is enhancing the trapping times considerably (see
Supplementary Information for details) but also increases the time required
for the particle to enter the DLVO minimum by diffusion.

The observed trapping is, hence, a vdW assisted thermo-hydrodynamic process.
Vertical confinement is achieved by vdW attraction and double layer repulsion,
while lateral confinement is the result of thermo-osmotic flows induced in an
ultra-thin sheet of liquid at the interface. No additional contributions, for
example, due to convective flows with similar flow patterns (see Supplementary
Information for details) or thermo-electric effects are required for a
quantitative description [38, 39, 25, 31].

Precise tuning of the DLVO potential enables the trapping of even smaller Au
NPs (Fig. 4a, Supplementary Video 5). The speed of heat diffusion, which is
about 4 orders of magnitude faster than the particle diffusion [40] allows us
to introduce a flow field multiplexing. Therefore, we switch the heating
location between different locations inducing flow thermo-osmotic flow fields
for time periods of about 100 $\upmu{\rm s}$. With the help of this
multiplexing, we are able to hold multiple $R=125~{}\mathrm{nm}$ AuNPs (Fig.
4b, c) at distances of less than 1 $\upmu{\rm m}$, which would not be possible
with continuous heating of close-by locations (Supplementary Videos 6 and 7).
A trapped AuNP can also be guided along the predefined path over the Au film
as fast as $10~{}\mathrm{\upmu m\,s^{-1}}$ (Fig. 4d, Supplementary Video 8).
At larger manipulation speeds ($f>100$ Hz) and higher heating power
($P_{0}>10$ mW) the thermo-osmotic attraction to the heating spot is combined
with thermo-viscous flows [41, 42]. These flows originate from the temperature
dependent viscosity $\eta(T)$ of the liquid and are directed opposite to the
scanning direction of the laser [42]. The result of this combination of
thermo-osmosis and thermo-viscous flows is a rotating ring-like particle
structure (Fig. 4e and Supplementary Video 9). These different effects that
can be exploited in a simple planar geometry give rise to numerous
applications including for example a freely configurable nanoparticle on
mirror geometry for plasmonic sensing [43, 5].

###  II.5 Beyond Thermo-Osmotic van der Waals Trapping

So far, the presented manipulation is based on thermo-osmotic flows that drive
the lateral motion of suspended colloids and a vertical confinement due to the
secondary minimum of the DLVO potential between AuNP and Au film. While the
thermo-osmotic flows are characteristic for all systems containing a heated
gold/water interface including all previous studies on thermo-plasmonic
trapping, the DLVO potential minimum is much weaker for other materials like
polymer colloids or macromolecules due to their smaller vdW attraction. Often,
those system even show a repulsion from the heat source due to thermophoresis,
which is not present for AuNP. A more generalized strategy therefore needs
additional attractive contributions, which confine suspended colloids or
molecules to regions close to the gold surface to take advantage of the
thermo-osmotic flow.

Such attractive contributions can arise from depletion interactions [44, 21].
Thereby a temperature gradient repels dissolved molecules from the heated
regions generating a concentration gradient that drives suspended nano-objects
to the heating spot. To demonstrate this effect we use the surfactant sodium
dodecyl sulfate (SDS) at a concentration of 5 mM well below the critical
micelle concentration (8.2 mM) to avoid complications of micelle formation. We
suspend additional polystyrene particles(PS) and AuNPs of the same size
($R=125~{}\mathrm{nm}$) in the solution and compare their dynamics to a
solution with AuNPs and PS particles without SDS but 10 mM NaCl. Remarkably,
the heated spot is attractive for both AuNPs and for PS NPs (Fig. 4f,
Supplementary Video 10) in the SDS solution showing even PS colloidal crystal
growth, while only the AuNP is trapped in the NaCl solution and the PS
particles are repelled by thermophoresis (Fig. 4g, Supplementary Video 11).

The observations in NaCl are readily explained by the fact that the AuNP is
confined in the DLVO minimum as demonstrated above but the PS particle is not
due to a 10 times lower Hamaker constant (see Supplementary Information for
details). The PS particle samples the whole liquid film thickness equally and
not preferentially the region close to the Au film and experiences an
additional thermophoretic drift velocity given by

$\vec{u}=-\frac{2}{3}\chi\frac{\nabla T}{T}=-D_{\mathrm{T}}\nabla T\ ,$ (6)

where $D_{\mathrm{T}}$ is the thermophoretic mobility [34] and $\nabla T$ the
temperature gradient (Fig. 4g, Supplementary Video 11). For $\chi>0$ the
particle is driven to the cold. From equation (4) we find
$\chi\approx\chi_{\mathrm{E}}=1.28\cdot 10^{-10}~{}\mathrm{m^{2}\,s^{-1}}$ and
$D_{\mathrm{T}}\approx 0.3~{}\mathrm{\upmu m^{2}\,K^{-1}\,s^{-1}}$, where we
have used a measured zeta potential of $\zeta\approx-38~{}\mathrm{mV}$. The
vdW contribution, $\chi_{\mathrm{vdW}}$ to either the thermophoretic drift or
the attraction to the gold surface can be neglected due to the smaller Hamaker
constant of PS. From the stationary probability distribution of the PS NP we
find a Soret coefficient of $S_{\mathrm{T}}\approx 0.24~{}\mathrm{K}^{-1}$
(see Supplementary Information for details) in agreement with our theoretical
prediction $S_{\mathrm{T}}=D_{\mathrm{T}}/D_{\parallel}\approx
0.21~{}\mathrm{K}^{-1}$.

In the SDS solution, the additional surfactant molecules now undergo
thermophoresis to yield a concentration gradient in which suspended colloidal
particles drift. The lower concentration in the heated regions promotes an
effective attractive interaction of suspended colloids with the gold surface
due to depletion forces. The drift velocity is described by an additional term
to the thermodiffusion coefficient $D_{\mathrm{T}}$, that is, the second term
in brackets in equation (7) [21, 34, 44].

$\vec{u}=-\left(D_{\mathrm{T}}-\frac{k_{\mathrm{B}}}{3\eta}R^{2}c_{0}N_{\mathrm{A}}\left(TS^{\mathrm{SDS}}_{\mathrm{T}}-1\right)\right)\nabla
T$ (7)

Here $R$ is the size of the SDS molecule, $c$ the concentration in units of
$\mathrm{mol/l}$ and $S^{\mathrm{SDS}}_{\mathrm{T}}$ the Soret coefficient of
SDS. For $R=2~{}\mathrm{nm}$ [45], $c_{0}=5~{}\mathrm{mM}$ and
$S^{\mathrm{SDS}}_{\mathrm{T}}=0.03~{}\mathrm{K^{-1}}$ [46] we find
$-0.43~{}\mathrm{\upmu m^{2}\,K^{-1}\,s^{-1}}$ for the additional depletion
contribution, which exceeds the thermophoretic mobility,
$D_{\mathrm{T}}\approx 0.3~{}\mathrm{\upmu m^{2}\,K^{-1}\,s^{-1}}$, rendering
the overall mobility negative. The PS NPs and the AuNPs are thus driven to the
the heated Au film surface(Fig. 4f, Supplementary Video 10) which allows for
further transport in the thermo-osmotic boundary flow. Additional
contributions, as for example thermo-electric fields may even enhance the
attractive components. Overall, this concept is readily transferred to other
objects as shown in Figure 4h and Supplementary Video 12, where we have
trapped ellipsoidal PS particles in a 5 mM solution of SDS.

##  III Conclusion

In conclusion, we have demonstrated that thermo-hydrodynamic boundary flows
can manipulate nano-objects with unprecedented flexibility in a very simple
sample geometry. These flows are the key for future thermo-optofluidic
implementations with an extensive range of applications in the fields of i)
nanoparticle sorting and separation [7]; ii) assembly of nanophotonic circuits
[47] and plasmonic quantum sensors [5, 4]; iii) biotechnology on-chip
laboratories [48] and iv) manufacturing of nanomaterials [6, 4] and functional
nanosurfaces [49, 50]. We have substantiated our experimental findings of
thermo-osmotic flow assisted trapping with a quantitative theoretical
description. A flow field multiplexing scheme has been further developed to
allow for the simultaneous manipulation of many individual nano-objects. Our
concept can be combined with other thermally induced effects such as
thermophoresis, depletion forces and thermoviscous flows to form a fully-
featured nanofluidic system-on-a-chip. Besides direct consequences for the
field of plasmonic nano-tweezers and other thermoplasmonic trapping schemes,
the use of thermo-hydrodynamic flows as a tool for nanofluidic applications
will extend the limits at the forefront of nanotechnology.

## References

  * Maragò  _et al._ [2013] O. M. Maragò, P. H. Jones, P. G. Gucciardi, G. Volpe, and A. C. Ferrari, Optical trapping and manipulation of nanostructures, Nat. Nanotechnol. 8, 807 (2013).
  * Gao  _et al._ [2017] D. Gao, W. Ding, M. Nieto-Vesperinas, X. Ding, M. Rahman, T. Zhang, C. Lim, and C.-W. Qiu, Optical manipulation from the microscale to the nanoscale: Fundamentals, advances and prospects, Light Sci. Appl. 6, 17039 (2017).
  * Bradac [2018] C. Bradac, Nanoscale Optical Trapping: A Review, Adv. Opt. Mat. 6, 1800005 (2018).
  * Xavier  _et al._ [2018] J. Xavier, S. Vincent, F. Meder, and F. Vollmer, Advances in optoplasmonic sensors – combining optical nano/microcavities and photonic crystals with plasmonic nanostructures and nanoparticles, Nanophotonics 7, 1 (2018).
  * Li  _et al._ [2018] G.-C. Li, Q. Zhang, S. A. Maier, and D. Lei, Plasmonic particle-on-film nanocavities: A versatile platform for plasmon-enhanced spectroscopy and photochemistry, Nanophotonics 7, 1865 (2018).
  * Lin  _et al._ [2017a] L. Lin, X. Peng, and Y. Zheng, Reconfigurable opto-thermoelectric printing of colloidal particles, Chem. Commun. 53, 7357 (2017a).
  * Xie  _et al._ [2020] Y. Xie, J. Rufo, R. Zhong, J. Rich, P. Li, K. W. Leong, and T. J. Huang, Microfluidic Isolation and Enrichment of Nanoparticles, ACS Nano 14, 16220 (2020).
  * Favre-Bulle  _et al._ [2019] I. A. Favre-Bulle, A. B. Stilgoe, E. K. Scott, and H. Rubinsztein-Dunlop, Optical trapping in vivo: Theory, practice, and applications, Nanophotonics 8, 1023 (2019).
  * Choudhary  _et al._ [2019] D. Choudhary, A. Mossa, M. Jadhav, and C. Cecconi, Bio-Molecular Applications of Recent Developments in Optical Tweezers, Biomolecules 9, 23 (2019).
  * Kayci  _et al._ [2014] M. Kayci, H.-C. Chang, and A. Radenovic, Electron Spin Resonance of Nitrogen-Vacancy Defects Embedded in Single Nanodiamonds in an ABEL Trap, Nano Lett. 14, 5335 (2014).
  * Juan  _et al._ [2011] M. L. Juan, M. Righini, and R. Quidant, Plasmon nano-optical tweezers, Nat. Photonics 5, 349 (2011).
  * Zhang  _et al._ [2021] Y. Zhang, C. Min, X. Dou, X. Wang, H. P. Urbach, M. G. Somekh, and X. Yuan, Plasmonic tweezers: For nanoscale optical trapping and beyond, Light Sci. Appl. 10, 59 (2021).
  * De Vlaminck and Dekker [2012] I. De Vlaminck and C. Dekker, Recent Advances in Magnetic Tweezers, Annu. Rev. Biophys. 41, 453 (2012).
  * Wang and Moerner [2011] Q. Wang and W. E. Moerner, An Adaptive Anti-Brownian Electrokinetic Trap with Real-Time Information on Single-Molecule Diffusivity and Mobility, ACS Nano 5, 5792 (2011).
  * Wu [2011] M. C. Wu, Optoelectronic tweezers, Nat. Photonics 5, 322 (2011).
  * Kotnala and Gordon [2014] A. Kotnala and R. Gordon, Quantification of High-Efficiency Trapping of Nanoparticles in a Double Nanohole Optical Tweezer, Nano Lett. 14, 853 (2014).
  * Jiang  _et al._ [2020] Q. Jiang, B. Rogez, J.-B. Claude, G. Baffou, and J. Wenger, Quantifying the Role of the Surfactant and the Thermophoretic Force in Plasmonic Nano-optical Trapping, Nano Lett. 20, 8811 (2020).
  * Braun and Cichos [2013] M. Braun and F. Cichos, Optically Controlled Thermophoretic Trapping of Single Nano-Objects, ACS Nano 7, 11200 (2013).
  * Braun  _et al._ [2015] M. Braun, A. P. Bregulla, K. Günther, M. Mertig, and F. Cichos, Single Molecules Trapped by Dynamic Inhomogeneous Temperature Fields, Nano Lett. 15, 5499 (2015).
  * Fränzl  _et al._ [2019] M. Fränzl, T. Thalheim, J. Adler, D. Huster, J. Posseckardt, M. Mertig, and F. Cichos, Thermophoretic trap for single amyloid fibril and protein aggregation studies, Nat. Methods 16, 611 (2019).
  * Jiang  _et al._ [2009] H.-R. Jiang, H. Wada, N. Yoshinaga, and M. Sano, Manipulation of Colloids by a Nonequilibrium Depletion Force in a Temperature Gradient, Phys. Rev. Lett. 102, 208301 (2009).
  * Bregulla  _et al._ [2014] A. P. Bregulla, H. Yang, and F. Cichos, Stochastic Localization of Microswimmers by Photon Nudging, ACS Nano 8, 6542 (2014).
  * Khadka  _et al._ [2018] U. Khadka, V. Holubec, H. Yang, and F. Cichos, Active particles bound by information flows, Nat. Commun. 9, 3864 (2018).
  * Fränzl  _et al._ [2021] M. Fränzl, S. Muiños-Landin, V. Holubec, and F. Cichos, Fully Steerable Symmetric Thermoplasmonic Microswimmers, ACS Nano 15, 3434 (2021).
  * Lin  _et al._ [2018] L. Lin, M. Wang, X. Peng, E. N. Lissek, Z. Mao, L. Scarabelli, E. Adkins, S. Coskun, H. E. Unalan, B. A. Korgel, L. M. Liz-Marzán, E.-L. Florin, and Y. Zheng, Opto-thermoelectric nanotweezers, Nat. Photonics 12, 195 (2018).
  * Ndukaife  _et al._ [2016] J. C. Ndukaife, A. V. Kildishev, A. G. A. Nnanna, V. M. Shalaev, S. T. Wereley, and A. Boltasseva, Long-range and rapid transport of individual nano-objects by a hybrid electrothermoplasmonic nanotweezer, Nat. Nanotechnol. 11, 53 (2016).
  * Hong  _et al._ [2020] C. Hong, S. Yang, and J. C. Ndukaife, Stand-off trapping and manipulation of sub-10 nm objects and biomolecules using opto-thermo-electrohydrodynamic tweezers, Nat. Nanotechnol. 15, 908 (2020).
  * Donner  _et al._ [2011] J. S. Donner, G. Baffou, D. McCloskey, and R. Quidant, Plasmon-Assisted Optofluidics, ACS Nano 5, 5457 (2011).
  * Roxworthy  _et al._ [2014] B. J. Roxworthy, A. M. Bhuiya, S. P. Vanka, and K. C. Toussaint, Understanding and controlling plasmon-induced convection, Nat. Commun. 5, 3173 (2014).
  * Chen  _et al._ [2020] J. Chen, J. F.-C. Loo, D. Wang, Y. Zhang, S.-K. Kong, and H.-P. Ho, Thermal Optofluidics: Principles and Applications, Adv. Opt. Mater. 8, 1900829 (2020).
  * Ciraulo  _et al._ [2021] B. Ciraulo, J. Garcia-Guirado, I. de Miguel, J. Ortega Arroyo, and R. Quidant, Long-range optofluidic control with plasmon heating, Nat. Commun. 12, 2001 (2021).
  * Israelachvili [2011] J. N. Israelachvili,  _Intermolecular and Surface Forces_ , 3rd ed. (Elsevier, Academic Press, Amsterdam, 2011).
  * Lin  _et al._ [2000] B. Lin, J. Yu, and S. A. Rice, Direct measurements of constrained Brownian motion of an isolated sphere between two walls, Phys. Rev. E 62, 3909 (2000).
  * Würger [2010] A. Würger, Thermal non-equilibrium transport in colloids, Rep. Prog. Phys. 73, 126601 (2010).
  * Bregulla  _et al._ [2016] A. P. Bregulla, A. Würger, K. Günther, M. Mertig, and F. Cichos, Thermo-Osmotic Flow in Thin Films, Phys. Rev. Lett. 116, 188303 (2016).
  * Giesbers  _et al._ [2002] M. Giesbers, J. Kleijn, and M. A. Cohen Stuart, The Electrical Double Layer on Gold Probed by Electrokinetic and Surface Force Measurements, J. Colloid Interface Sci. 248, 88 (2002).
  * Bluemink  _et al._ [2008] J. J. Bluemink, D. Lohse, A. Prosperetti, and L. Van Wijngaarden, A sphere in a uniformly rotating or shearing flow, J. Fluid Mech. 600, 201 (2008).
  * Lin  _et al._ [2016] L. Lin, X. Peng, M. Wang, L. Scarabelli, Z. Mao, L. M. Liz-Marzán, M. F. Becker, and Y. Zheng, Light-Directed Reversible Assembly of Plasmonic Nanoparticles Using Plasmon-Enhanced Thermophoresis, ACS Nano 10, 9659 (2016).
  * Lin  _et al._ [2017b] L. Lin, J. Zhang, X. Peng, Z. Wu, A. C. H. Coughlan, Z. Mao, M. A. Bevan, and Y. Zheng, Opto-thermophoretic assembly of colloidal matter, Sci. Adv. 3, 1700458 (2017b).
  * Baffou  _et al._ [2020] G. Baffou, F. Cichos, and R. Quidant, Applications and challenges of thermoplasmonics, Nat. Mater. 19, 946 (2020).
  * Weinert  _et al._ [2008] F. M. Weinert, J. A. Kraus, T. Franosch, and D. Braun, Microscale Fluid Flow Induced by Thermoviscous Expansion Along a Traveling Wave, Phys. Rev. Lett. 100, 164501 (2008).
  * Weinert  _et al._ [2011] F. M. Weinert, C. B. Mast, and D. Braun, Optical fluid and biomolecule transport with thermal fields, Phys. Chem. Chem. Phys. 13, 9918 (2011).
  * Chikkaraddy  _et al._ [2016] R. Chikkaraddy, B. de Nijs, F. Benz, S. J. Barrow, O. A. Scherman, E. Rosta, A. Demetriadou, P. Fox, O. Hess, and J. J. Baumberg, Single-molecule strong coupling at room temperature in plasmonic nanocavities, Nature 535, 127 (2016).
  * Maeda  _et al._ [2011] Y. T. Maeda, A. Buguin, and A. Libchaber, Thermal Separation: Interplay between the Soret Effect and Entropic Force Gradient, Phys. Rev. Lett. 107, 038301 (2011).
  * Syshchyk  _et al._ [2016] O. Syshchyk, D. Afanasenkau, Z. Wang, H. Kriegs, J. Buitenhuis, and S. Wiegand, Influence of temperature and charge effects on thermophoresis of polystyrene beads, Eur. Phys. J. E 39, 129 (2016).
  * Vigolo  _et al._ [2010] D. Vigolo, S. Buzzaccaro, and R. Piazza, Thermophoresis and Thermoelectricity in Surfactant Solutions, Langmuir 26, 7792 (2010).
  * Zhang  _et al._ [2019] Q. Zhang, H. Yu, M. Barbiero, B. Wang, and M. Gu, Artificial neural networks enabled by nanophotonics, Light Sci Appl 8, 42 (2019).
  * Xavier  _et al._ [2021] J. Xavier, D. Yu, C. Jones, E. Zossimova, and F. Vollmer, Quantum nanophotonic and nanoplasmonic sensing: Towards quantum optical bioscience laboratories on chip, Nanophotonics 10, 1387 (2021).
  * Peng  _et al._ [2019] J. Peng, H.-H. Jeong, Q. Lin, S. Cormier, H.-L. Liang, M. F. L. De Volder, S. Vignolini, and J. J. Baumberg, Scalable electrochromic nanopixels using plasmonics, Sci. Adv. 5, 10.1126/sciadv.aaw2205 (2019).
  * Mohammadi  _et al._ [2021] E. Mohammadi, A. Tittl, K. L. Tsakmakidis, T. V. Raziman, and A. G. Curto, Dual Nanoresonators for Ultrasensitive Chiral Detection, ACS Photonics 8, 1754 (2021).


# Localization and Tracking of User-Defined Points on Deformable Objects for
Robotic Manipulation

Sven Dittus, Benjamin Alt, Andreas Hermann, Darko Katic and Rainer Jäkel
Jürgen Fleischer This work was partially supported by the German Federal
Ministry of Education and Research (BMBF) under the grant no. 16SV8406.
ArtiMinds Robotics, Karlsruhe, Germany
{sven.dittus | benjamin.alt | andreas.hermann | darko.katic |<EMAIL_ADDRESS>Institute of Production Science, Karlsruhe Institute of Technology, Germany
<EMAIL_ADDRESS>
###### Abstract

This paper introduces an efficient procedure to localize user-defined points
on the surface of deformable objects and track their positions in 3D space
over time. To cope with a deformable object’s infinite number of DOF, we
propose a discretized deformation field, which is estimated during runtime
using a multi-step non-linear solver pipeline. The resulting high-dimensional
energy minimization problem describes the deviation between an offline-defined
reference model and a pre-processed camera image. An additional regularization
term allows for assumptions about the object’s hidden areas and increases the
solver’s numerical stability. Our approach is capable of solving the
localization problem online in a data-parallel manner, making it ideally
suitable for the perception of non-rigid objects in industrial manufacturing
processes.

##  I Introduction

Many manufacturing processes rely on image processing to enable industrial
robots to manipulate objects. Whereas many sophisticated camera systems meet
the need for localizing and tracking user-defined Points of Interest on rigid
objects, there is still no sufficiently accurate solution for coping with this
problem for deformable objects yet. Moreover, existing approaches reconstruct
the deformable object‘s model online, requiring POIs to be defined at runtime
and thus being unsuitable for fully automated processes. This paper proposes a
solution for defining POIs on an offline model and then localizing and
tracking these points on a deformable object in an online detection pipeline.

Figure 1: Overview of the localization and tracking pipeline

##  II Related Work

Existing approaches for localizing and tracking POIs are only applicable to
specific object categories (e.g. linear [1] or planar [2, 3]), assume speficic
deformation models (e.g. articulated models [4] or skeletons [5]) or
particular materials (e.g. textiles [6, 7]) or are restricted to detecting
specific features (e.g. points on corners or edges [8, 9]). To reach the
precision required for sophisticated manipulation tasks, prior work requires
external markers [10, 11] or elaborate physics models [12, 13, 14, 3, 15]. We
combine several SotA-solutions for rigid object state estimation (such as SHOT
descriptors), physical modelling (such as deformation fields) and computer
graphics (such as projection) into a processing pipeline which permits the
tracking and localization of user-defined points on arbitrary deformable
objects using only a single depth camera without markers or prior physics
modeling.

##  III Method

Our algorithm consists of three distinct phases: (1) Demonstration of the
reference model and the POIs; (2) an iterative localization and tracking
process consisting of observing a new point cloud, identification of
correspondences between the observation and the deformed reference model of
the previous timestep and estimation of the deformation; and (3) a coordinate
transformation of the localized POIs into the robot end-effector coordinate
system for subsequent manipulation.

###  III-A Surface and deformation model

To efficiently perform computations, we model object surfaces as triangle
meshes, while deformations are modelled via a deformation grid [16]. Unlike
[16], we use a highly detailed mesh as a surface representation which is
independent of the deformation model’s resolution. This allows to increase
computation performance while maintaining a highly detailed surface. Our
deformation model consists of two data structures, both containing $\lvert
G\rvert$ grid points. Whereas the equally spaced static grid $G$ describes the
undeformed reference model, the deformation field $\mathcal{V}$ represents the
object’s deformed state at the current timestep $t$. Each gridpoint $i$ is
defined in $G$ by a position vector $\hat{\boldsymbol{t}}_{i}$, allowing to
express the position $\hat{\boldsymbol{p}}$ of an undeformed vertex within $G$
as $\hat{\boldsymbol{p}}=\sum^{\lvert
G\rvert}_{i=1}\alpha_{i}\hat{\boldsymbol{t}}_{i}$ with trilinear weights
$\alpha_{i}\in[0,1]$ (cf. fig. 2). The position $\boldsymbol{p}$ of the same
vertex in the deformation field $\mathcal{V}$ can be described analogously by
a weighted sum of deformed gridpoint positions $\boldsymbol{t}_{i}$ as
$\boldsymbol{p}=\sum^{\lvert G\rvert}_{i=1}\alpha_{i}\boldsymbol{t}_{i}$. We
define an observation $\mathcal{D}$ as an organized point cloud of the
deformed object.

Figure 2: Trilinear weights (l.); static and deformed grid cells (r.)

###  III-B Demonstration

Most SotA approaches use either a low-resolution reference model of the
deformable object [17] or none at all [18, 16]. To allow for user
demonstrations of POIs, our algorithm requires a high-detail reference model
to be created offline. To improve the stability of our solver, we limit the
object’s initial deformation with respect to its reference model by
demonstrating a library $\mathcal{L}$ of reference models in an offline step.
Each model in $\mathcal{L}$ is a triangle mesh of the object in a distinct
deformation state. The models are generated by fusing several depth images
into a Truncated Signed Distance Field (TSDF) and then extracting the
triangles via the MarchingCubes algorithm [19]. After the demonstration of the
reference models, the user can select relevant POIs on the meshed surface via
a graphical user interface. At runtime, after the first observation
$\mathcal{D}$, the model in $\mathcal{L}$ most similar to $\mathcal{D}$ is
used to initialize $G$.

###  III-C Correspondence identification

The definition and identification of correspondences links the current
observation and the deformed reference model of the previous timestep $(t-1)$
and forms the basis of the deformation estimation: The estimation of a
deformation is equivalent to the minimization of the distances between all
correspondences. We define three correspondence types:

####  p2p

Result from projecting each surface point $\boldsymbol{p}_{c}$ of the deformed
reference model into the image plane and comparing it to the corresponding
point $\boldsymbol{p}_{c}^{a}$ that has been measured. The quality of a point-
to-point (P2P)-correspondence can be described by a weight
$w_{c}=(\frac{w_{d}+w_{n}+w_{v}}{3})^{2}$ where $w_{d}$ denotes the distance
between projected point $\boldsymbol{p}_{c}$ and its correspondent
$\boldsymbol{p}^{a}_{c}$, $w_{n}$ the distance between $\boldsymbol{p}_{c}$’s
normal $\boldsymbol{n}_{c}$ and its correspondent $\boldsymbol{n}^{a}_{c}$,
and $\boldsymbol{w}_{v}$ the angle between the camera view direction
$\boldsymbol{v}$ and $\boldsymbol{n}_{c}$.

####  p2s

Projective correspondences such as P2P typically only yield approximate, not
exact, correspondences. p2s-correspondences add another degree of freedom by
associating a point in the model to a plane in the observation, defined by the
P2P-correspondence $\boldsymbol{p}^{a}_{c}$ and its normal
$\boldsymbol{n}^{a}_{c}$. During deformation estimation, this allows to only
minimize the distance $d_{c}$ along the normal.

#### Feature correspondences

Unlike projective correspondences, correspondences based on feature matching
can detect large deformations, tangential movements and rotations of the
object out of the image plane. Prior work [20, 21] and our own experiments
have found the PFH and FPFH descriptors to be highly sensitive and specific
but to scale poorly with the size of the point cloud, while SHOT descriptors
scale linearly and are robust against outliers. We implement feature
correspondences using SHOT, as its sensitivity suffices for most real-world
applications.

###  III-D Deformation estimation

The deformation of the reference object can be estimated by formulating an
optimization problem to estimate their degrees of freedom and thus the
deformation of the reference object. Using the notation introduced in III-A,
the deformation of a single grid point $i$ can be expressed as
$\mathcal{V}_{i}=\hat{\boldsymbol{t}}_{i}-\boldsymbol{t}_{i}$. For estimating
the deformation field, we split up all unknows into a single global rigid
transformation $(\boldsymbol{t},R)$ and many local transformations
$(\boldsymbol{t}_{i},R_{i})$ and combine them in a vector $\mathcal{X}$:

$\mathcal{X}=\left(\boldsymbol{t},~{}R,\underbrace{\quad\ldots,\boldsymbol{t}_{i}^{T},\cdots\quad}_{3~{}|G|~{}translations}|\underbrace{\quad\cdots,R_{i},\cdots\quad}_{3~{}|G|~{}rotations}\right)^{T}$
(1)

The interpretation of correspondences as error terms $E$ allows to formulate
the deformation estimation of $\mathcal{X}$ as an energy minimization problem,
which is also suggested by [16, 17, 18]. This optimization can be regarded as
a model regression problem and solved by existing solvers:

$\displaystyle E(\mathcal{X})={}$
$\displaystyle\omega_{p}E_{P\mathit{2}P}(\mathcal{X})+\omega_{s}E_{P\mathit{2}S}(\mathcal{X})$
(2) $\displaystyle+{}$
$\displaystyle\omega_{f}E_{F}(\mathcal{X})+\omega_{r}E_{Reg}(\mathcal{X})$

[17] and [16] solve a similar high-dimensional nonlinear optimization problem
by linearizing the model and using the Gauss-Newton method, incurring a
significant overhead for the computation of the Jacobian $J$. [16] splits the
optimization into a two-stage process composed of a fixed registration
followed by a deformation estimation. We leverage the fact observed in [22]
that the deformation estimation can again be split into two independent sub-
problems, which allows to solve for nonlinear rotations and linear
translations using iterative Gauss-Newton on each subproblem in turn (“flip-
flop” strategy). We perform fixed registration, the estimation of a global
transformation $(\boldsymbol{t},R)$, via Prerejective RANSAC (PSC). For the
deformation estimation, setting up the Jacobian for the error terms of the
three correspondence types is straightforward:

$E_{P\mathit{2}P}=\sum^{|C|}_{c=1}\omega_{c}~{}\Biggl{|}\Biggl{|}\underbrace{\underbrace{R\left[\sum^{|G|}_{i=1}\alpha_{i}(\hat{\boldsymbol{p}}_{c})~{}\boldsymbol{t}_{i}\right]+\boldsymbol{t}}_{\mathcal{V}(\hat{\boldsymbol{p}}_{c})}-\boldsymbol{p}_{c}^{a}}_{\boldsymbol{r}_{P\mathit{2}P,C}(\mathcal{X})}\Biggl{|}\Biggl{|}^{2}_{2}$
(3)

Figure 3: Precision for different deformation types (l.); estimated
deformation grid and ground-truth point cloud (blue) before and after one
deformation cycle (r.)

$J_{P\mathit{2}P,ci}=\dfrac{\partial\left(\omega_{p}~{}\omega_{c}~{}\boldsymbol{r}_{P\mathit{2}P,c}(\mathcal{X})\right)}{\partial\boldsymbol{t}_{i}}=\omega_{p}~{}\omega_{c}~{}\alpha_{i}(\hat{\boldsymbol{p}}_{c})$
(4)

where $\lvert C\rvert$ is the number of correspondences and
$J_{P\mathit{2}P,ci}$ is the entry at the $c^{th}$ row and $i^{th}$ column of
the Jacobian $J_{P\mathit{2}P}$. The Jacobians for $E_{P\mathit{2}S}$ and
$E_{F}$ can be found analogously.

#### Regularization

With a single camera’s perspective, it is impossible to observe the complete
surface of an object. The spatial lack of correspondences implies an
underdetermined equation system and $E$ being ill-conditioned. To alleviate
this problem, we use an ARAP regularizer [22], where non-observable surface
points are deformed such that the total deformation of the body is as rigid as
possible. Unlike prior work [22, 23], we estimate the deformation in terms of
$\mathcal{V}$ instead of the mesh, leading to the adapted ARAP term

$E_{Reg}=\sum^{|G|}_{i=1}\sum_{j\in\mathcal{N}_{i}}\bigl{|}\bigl{|}\left(\boldsymbol{t}_{i}-\boldsymbol{t}_{j}\right)-R_{i}\left(\hat{\boldsymbol{t}}_{i}-\hat{\boldsymbol{t}}_{j}\right)\bigl{|}\bigl{|}^{2}_{2},$
(5)

where $\mathcal{N}_{i}$ denotes the neighborhood (6 surrounding grid points)
of grid point $i\in[1,\lvert G\rvert]$. For each grid point $i$, our solver
must solve for 6 unknowns describing its pose $(R_{i}|\boldsymbol{t}_{i})$. As
shown in [23], the (non-linear) estimation of $R_{i}$ can be solved in closed
form given $\boldsymbol{t}_{i}$. For $\boldsymbol{t}_{i}$, we obtain

$\frac{\partial
E_{Reg,i}}{\partial\boldsymbol{t}_{i}}\overset{!}{=}0\Leftrightarrow\underbrace{\sum_{j\in\mathcal{N}_{i}}(\boldsymbol{t}_{i}-\boldsymbol{t}_{j})}_{L~{}\cdot~{}\mathcal{X}_{i}}=\sum_{j\in\mathcal{N}_{i}}\dfrac{R_{i}+R_{j}}{2}(\hat{\boldsymbol{t}}_{i}-\hat{\boldsymbol{t}}_{j})$
(6)

where the left-hand side is the product of the Laplace matrix $L$ with the
vector of all unknowns $\mathcal{X}_{i}$.

#### Flip-flop solver

We iteratively estimate $R_{i}$ and $\boldsymbol{t}_{i}$ in turn by closed-
form solving for $R_{i}$ via singular value decomposition (see [23] for
details) and approximating $\boldsymbol{t}_{i}$ via Gauss-Newton, where the
update step $\Delta\mathcal{X}$ is obtained via preconditioned conjugate
gradients (PCG). Using the Jacobians derived above, we can obtain the
deformation $\mathcal{X}_{t+1}$ after an update step via

$\displaystyle J^{T}J$
$\displaystyle:=J^{T}_{P\mathit{2}P}J_{P\mathit{2}P}+J^{T}_{P\mathit{2}S}J_{P\mathit{2}S}+J^{T}_{F}J_{F}+L^{T}L$
(7) $\displaystyle J^{T}\boldsymbol{r}$
$\displaystyle:=J^{T}_{P\mathit{2}P}\boldsymbol{r}_{P\mathit{2}P}+J^{T}_{P\mathit{2}S}\boldsymbol{r}_{P\mathit{2}S}+J^{T}_{F}\boldsymbol{r}_{F}+L^{T}\boldsymbol{r}_{Reg}$
(8) $\displaystyle J^{T}J\,\ \boldsymbol{\Delta}\mathcal{X}$
$\displaystyle=J^{T}\boldsymbol{r}$ (9) $\displaystyle\mathcal{X}_{t+1}$
$\displaystyle=\mathcal{X}_{t}+\boldsymbol{\Delta}\mathcal{X}$ (10)

Figure 4: POI tracking on a deformable tripod (l.); rubber seal assembly (r.)

##  IV Results

#### Precision

In a first set of experiments, we assess the precision of our approach by
comparing tracking results versus manually labeled correspondences. 10 POIs on
a deformable tripod were considered, with each POI also fitted with a color-
coded marker to facilitate manual labeling.111Since our algorithm only
considers geometric features, the presence of the markers neither helped nor
hurt the algorithm. The tripod was repeatedly deformed and the poses of the
POIs were estimated by our algorithm as well as via the markers (cf. fig. 3).
Our approach was capable of localizing all POIs with sub-millimeter accuracy,
and tracking all POIs with errors between 0.6 and 2.2 mm. Unlike feature-
matching based approaches, we always estimate the deformation of the complete
surface and thereby avoid “mismatching” POIs by design.

#### Performance

A significant advantage of our approach is that each step of the solver
pipeline can be efficiently parallelized. We benchmarked our algorithm using
reference and deformation models at fine222Reference model: 30000 vertices,
$\mathcal{V}$: 3250 grid points and coarse333Reference model: 15000 vertices,
$\mathcal{V}$: 700 grid points resolutions. A parallelized CPU implementation
of our algorithm localized all POIs in under $1.5s$ in both cases on consumer
hardware.

#### Robot experiments

In a first robot experiment, we track a point on the surface of a tripod
subjected to several deformations of up to 20% of the tripod’s arm length, or
ca. 2.5 cm. We use a UR5 robot equipped with a measuring tip to visualize
tracking results (cf. fig. 4 (l.)), confirming precision within 2 mm. In a
second experiment, we use our approach to position a flat rubber seal on a
housing, illustrating its potential for real-world industrial applications
(cf. fig. 4 (r.)).

##  V Discussion and Outlook

Our approach and solver pipeline allows efficient tracking and localization of
POIs on deformable objects. Where prior work requires markers, explicit
modelling or does not allow for offline POI definition, our approach achieves
sub-millimeter precision localization and millimeter-precision tracking
without these drawbacks. This makes it particularly suitable for applications
in industrial robotics and flexible, quickly reconfigurable assembly or
surface treatment tasks. We are working on integrating our solution into an
industrial robot manipulation framework, a more efficient GPU implementation
and a more extensive evaluation on a wider set of benchmarks.

## References

  * [1] T. Tang, C. Wang, and M. Tomizuka, “A framework for manipulating deformable linear objects by coherent point drift,” _IEEE Robot. Autom. Lett._ , vol. 3, no. 4, pp. 3426–3433, 2018.
  * [2] M. Tang, R. Tong, R. Narain, C. Meng, and D. Manocha, “A GPU-based streaming algorithm for high-resolution cloth simulation,” _Comput. Graph. Forum_ , vol. 32, no. 7, pp. 21–30, 2013.
  * [3] J. Schulman, A. Lee, J. Ho, and P. Abbeel, “Tracking deformable objects with point clouds,” in _2013 IEEE International Conference on Robotics and Automation_. IEEE, 2013, pp. 1130–1137.
  * [4] T. Schmidt, R. Newcombe, and D. Fox, “DART: Dense articulated real-time tracking with consumer depth cameras,” _Auton. Robots_ , vol. 39, no. 3, pp. 239–258, 2015.
  * [5] J. Gall, C. Stoll, E. de Aguiar, C. Theobalt, B. Rosenhahn, and H.-P. Seidel, “Motion capture using joint skeleton tracking and surface estimation,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2009, pp. 1746–1753.
  * [6] Y. Li, Y. Wang, M. Case, S.-F. Chang, and P. K. Allen, “Real-time pose estimation of deformable objects using a volumetric approach,” in _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2014, pp. 1046–1052.
  * [7] Y. Li, Y. Wang, Y. Yue, D. Xu, M. Case, S.-F. Chang, E. Grinspun, and P. K. Allen, “Model-driven feedforward prediction for manipulation of deformable objects,” _IEEE Trans. Autom. Sci. Eng._ , vol. 15, no. 4, pp. 1621–1638, 2018.
  * [8] K. Yamazaki, “Grasping point selection on an item of crumpled clothing based on relational shape description,” in _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2014, pp. 3123–3128.
  * [9] A. Ramisa, G. Alenya, F. Moreno-Noguer, and C. Torras, “Using depth and appearance features for informed robot grasping of highly wrinkled clothes,” in _2012 IEEE International Conference on Robotics and Automation_. IEEE, 2012, pp. 1703–1708.
  * [10] J. Finnegan and J. Dorsey, Eds., _ACM SIGGRAPH 2006 Papers on - SIGGRAPH ’06_. New York, New York, USA: ACM Press, 2006.
  * [11] M. Trumble, A. Gilbert, C. Malleson, A. Hilton, and J. Collomosse, “Total capture: 3D human pose estimation fusing video and inertial sensors,” in _Procedings of the British Machine Vision Conference 2017_ , T.-K. Kim, S. Zafeiriou, G. Brostow, and K. Mikolajczyk, Eds. British Machine Vision Association, 2017.
  * [12] H. Bay, T. Tuytelaars, and L. van Gool, “SURF: Speeded up robust features,” in _Computer Vision – ECCV 2006_ , ser. Lecture Notes in Computer Science, A. Leonardis, H. Bischof, and A. Pinz, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, vol. 3951, pp. 404–417.
  * [13] J. Tian and Y.-B. Jia, “Modeling deformations of general parametric shells grasped by a robot hand,” _IEEE Trans. Robot._ , vol. 26, no. 5, pp. 837–852, 2010.
  * [14] H. Lang, J. Linn, and M. Arnold, “Multi-body dynamics simulation of geometrically exact Cosserat rods,” _Multibody Syst. Dyn._ , vol. 25, no. 3, pp. 285–312, 2011.
  * [15] I. Leizea, H. Alvarez, I. Aguinaga, and D. Borro, “Real-time deformation, registration and tracking of solids based on physical simulation,” in _2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)_. IEEE, 2014, pp. 165–170.
  * [16] M. Innmann, M. Zollhöfer, M. Nießner, C. Theobalt, and M. Stamminger, “VolumeDeform: Real-time volumetric non-rigid reconstruction,” in _Computer Vision – ECCV 2016_ , ser. Lecture Notes in Computer Science, B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International Publishing, 2016, vol. 9912, pp. 362–379.
  * [17] M. Zollhöfer, C. Theobalt, M. Stamminger, M. Nießner, S. Izadi, C. Rehmann, C. Zach, M. Fisher, C. Wu, A. Fitzgibbon, and C. Loop, “Real-time non-rigid reconstruction using an RGB-D camera,” _ACM Trans. Graph._ , vol. 33, no. 4, pp. 1–12, 2014.
  * [18] R. A. Newcombe, D. Fox, and S. M. Seitz, “DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time,” _IEEE Conf. Comput. Vis. Pattern Recognit. CVPR_ , pp. 343–352, 2015.
  * [19] W. E. Lorensen and H. E. Cline, “Marching cubes: A high resolution 3D surface construction algorithm,” _ACM SIGGRAPH Comput. Graph._ , vol. 21, no. 4, pp. 163–169, 1987.
  * [20] Y. Guo, M. Bennamoun, F. Sohel, M. Lu, J. Wan, and N. M. Kwok, “A comprehensive performance evaluation of 3D local feature descriptors,” _Int. J. Comput. Vis._ , vol. 116, no. 1, pp. 66–89, 2016.
  * [21] R. Hänsch, T. Weber, and O. Hellwich, “Comparison of 3D interest point detectors and descriptors for point cloud fusion,” _ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci._ , vol. II-3, pp. 57–64, 2014.
  * [22] O. Sorkine and M. Alexa, “As-rigid-as-possible surface modeling,” in _Proceedings of the Fifth Eurographics Symposium on Geometry Processing_ , ser. SGP ’07. Aire-la-Ville, Switzerland, Switzerland: Eurographics Association, 2007, pp. 109–116.
  * [23] ——, “Least-squares rigid motion using SVD,” Zürich, 2017.

  *[POIs]: Point of Interest
  *[TSDF]: Truncated Signed Distance Field
  *[P2P]: point-to-point
  *[PCG]: preconditioned conjugate gradients
  *[POI]: Point of Interest


$a_{0}^{\prime\prime}=\frac{\tau_{1}}{R^{\prime\prime}},\delta_{0}^{\prime\prime}=\frac{\epsilon^{\prime\prime}}{3}$
imply that it is sufficient to estimate

$\underbrace{\int_{a^{\prime}\geq
a_{0}^{\prime\prime}}\int_{\mathbb{R}^{d-1}}\Theta_{r,s}(h^{-1}h(t^{\prime},a^{\prime}))\text{\,d}(h(t^{\prime},a^{\prime}))}_{I_{1}}+\underbrace{\int_{a^{\prime}\leq
a_{0}^{\prime\prime}}\int_{|t^{\prime}|\geq\delta_{0}^{\prime\prime}}\Theta_{r,s}(h^{-1}h(t^{\prime},a^{\prime}))\text{\,d}(h(t^{\prime},a^{\prime}))}_{I_{2}}$

against a suitable power of $\|h\|$, where $h=h(t,a)$, and
$|t|<\delta_{1},a<a_{1}$ with $\delta_{1}=3\epsilon$ and
$a_{1}=\frac{2\tau_{2}}{R}$. Here $\text{\,d}(h(t^{\prime},a^{\prime}))$
denotes the left Haar integration over $H$, as computed in Lemma 6.8

For the estimate of $I_{1}$, we observe that $a^{\prime}\geq
a_{0}^{\prime\prime}$ implies via Lemma 6.10(b) that

$\|h(t,a)^{-1}h(t^{\prime},a^{\prime})\|\geq a^{-1}a_{0}^{\prime\prime}~{},$

hence

$(1+\|h(t,a)^{-1}h(t^{\prime},a^{\prime})\|)^{-1}\leq\frac{a}{a_{0}^{\prime\prime}}$

and thus for every $0<\alpha<r$

$\Theta_{r,s}(h(t,a)^{-1}h(t^{\prime},a^{\prime}))\leq
Ca^{\alpha}\Theta_{r-\alpha,s}(h(t,a)^{-1}h(t^{\prime},a^{\prime}))~{}.$

Therefore, as soon as $r-\alpha\geq\dim(H)+d+1=2d+1$ and $s\geq\dim(H)+d=2d$,
which guarantees the integrability of $\Theta_{r-\alpha,s}$ via Lemma 4.9, we
have that

$\displaystyle I_{1}$ $\displaystyle\leq$ $\displaystyle
Ca^{\alpha}\int_{H}\Theta_{r-\alpha,s}(h(t,a)^{-1}g)dg$ $\displaystyle\leq$
$\displaystyle C^{\prime}\|h(t,a)\|^{\alpha/\lambda_{\min}}~{},$

with the last inequality due to Lemma 6.10(a). Here, $C^{\prime}$ only depends
on $\tau_{1}$, $R^{\prime\prime}$, $\alpha,r,s$, and not on $a,t$.

For the estimate of $I_{2}$, note that $K_{o}(W,V,R)\subseteq
K_{o}(W^{\prime},V,R^{\prime})$ and conditions of Lemma 6.9 for
$W^{\prime},V,R^{\prime}$ are satisfied. So, the assumption $h(t,a)\in
K_{o}(W,V,R)$ and the integral bound
$|t^{\prime}|\geq\delta_{0}^{\prime\prime}$ entail

$|t^{\prime}|\geq\frac{\epsilon^{\prime\prime}}{3}~{},~{}|t|\leq\delta_{1}=3\epsilon<\frac{\epsilon^{\prime\prime}}{7C_{\mathfrak{s}}}~{}.$

It follows successively from (37) that

$1-C_{M}\delta_{1}\geq\frac{1}{2}~{},~{}\left(1-C_{M}\delta_{1}\right)\frac{|t^{\prime}|}{C_{\mathfrak{s}}}\geq\frac{\epsilon^{\prime\prime}}{6C_{\mathfrak{s}}}~{},$

and finally, via Lemma 6.10(c),

$\|h(t,a)^{-1}h(t^{\prime},a^{\prime})\|\geq
a^{-1}(a^{\prime})^{\lambda_{\max}}C^{\prime}$

with a global constant
$C^{\prime}=\frac{\epsilon^{\prime\prime}}{42C_{\mathfrak{s}}}$. It follows
for $0<\beta<\min(r,s)$ that

$\displaystyle I_{2}$ $\displaystyle\leq$
$\displaystyle(C^{\prime})^{-\beta}\int_{a^{\prime}\leq
a_{0}^{\prime\prime}}\int_{|t^{\prime}|\geq\delta_{0}^{\prime\prime}}a^{\beta}(a^{\prime})^{-\beta\lambda_{\max}}\Theta_{r-\beta,s}(h^{-1}h(t^{\prime},a^{\prime}))\text{\,d}(h(t^{\prime},a^{\prime}))$
$\displaystyle=$
$\displaystyle(C^{\prime})^{-\beta}a^{\beta(1-\lambda_{\max})}\int_{a^{\prime}\leq
a_{0}^{\prime\prime}}\int_{|t^{\prime}|\geq\delta_{0}^{\prime\prime}}a^{\beta\lambda_{\max}}(a^{\prime})^{-\beta\lambda_{\max}}\Theta_{r-\beta,s}(h^{-1}h(t^{\prime},a^{\prime}))\text{\,d}(h(t^{\prime},a^{\prime}))$
$\displaystyle\leq$ $\displaystyle
C^{\prime\prime}a^{\beta(1-\lambda_{\max})}\int_{H}\Theta_{r-\beta,s-\beta\lambda_{\max}}(h^{-1}g)\text{\,d}g~{},$

where the last inequality used

$|a||a^{\prime}|^{-1}\leq\|h(t^{\prime},a^{\prime})^{-1}h(t,a)\|~{},$

see Lemma 6.10(b), as well as $\Theta_{r-\beta,s-\beta\lambda_{\max}}\geq 0$.
Thus, again by Lemma 4.9 and Lemma 6.10(a),

$I_{2}\leq
C^{\prime\prime\prime}\|h\|^{\beta(1-\lambda_{\max})/\lambda_{\min}}$

with a finite constant $C^{\prime\prime\prime}$ as soon as $r\geq 2d+1+\beta$
and $s\geq 2d+\beta\lambda_{\max}$. Here, $C^{\prime\prime\prime}$ depends
only on $\epsilon^{\prime\prime}$, $C_{\mathfrak{s}},\beta,r,s,H$, and is
independent of $h$.

For a given $L>0$, substituting $\alpha=L\lambda_{\min}$ and
$\beta=L\frac{\lambda_{\min}}{1-\lambda_{\max}}$, we find the desired estimate

$\int_{H_{0}\setminus
K_{i}(W^{\prime\prime},V,R^{\prime\prime})}\Theta_{r,s}(h^{-1}g)\text{\,d}g\leq\tilde{C}\|h\|^{L}~{},$

for all $h\in K_{o}(W,V,R)$, as soon as

$r\geq 2d+1+L\frac{\lambda_{\min}}{1-\lambda_{\max}}~{},~{}s\geq
2d+L\frac{\lambda_{\min}\lambda_{\max}}{1-\lambda_{\max}}~{}.$

∎

With all conditions for the applicability of our main results to shearlet
dilation groups being checked, the following theorem spells out the resulting
version of Theorem 5.8 for this class of groups.

######  Theorem 6.12.

Let $H$ denote a shearlet dilation group. Let $Y$ denote the infinitesimal
generator of the scaling subgroup of $H$, with eigenvalues
$\lambda_{1},\ldots,\lambda_{d}$, normalized such that $\lambda_{1}=1$. Let
$\lambda_{\min}=\min\\{\lambda_{2},\ldots,\lambda_{d}\\}$,
$\lambda_{\max}=\max\\{\lambda_{2},\ldots,\lambda_{d}\\}$, and assume that
$0<\lambda_{\min}\leq\lambda_{\max}<1$ and $\lambda_{\min}+\lambda_{\max}\geq
1$. Let $n_{0}$ denote the nilpotency degree of the shearing subgroup of $H$.

Define

$\displaystyle s_{0}$ $\displaystyle=$ $\displaystyle
1+\left(\frac{3}{2}+\frac{1}{2\lambda_{\min}}\right)d+(n_{0}+1)\left(\frac{13}{2}d+(\frac{1}{2\lambda_{\min}}+\frac{3+3\lambda_{\max}}{2-2\lambda_{\max}})d+3\right)$
(38) $\displaystyle s_{1}$ $\displaystyle=$ $\displaystyle
1+(n_{0}+1)\left(1+\frac{\lambda_{\min}}{1-\lambda_{\max}}+\frac{\lambda_{\min}\lambda_{\max}}{1-\lambda_{\max}}\right)$
(39) $\displaystyle s_{2}$ $\displaystyle=$ $\displaystyle
1+\frac{1}{\lambda_{\min}}+(n_{0}+1)\left(2+\frac{1}{\lambda_{\min}}+\frac{2}{1-\lambda_{\max}}\right)~{}.$
(40)

Let $u\in\mathcal{S}^{\prime}(\mathbb{R}^{d})$ denote a tempered distribution
of order $N(u)$. Let $\psi\in\mathcal{S}(\mathbb{R}^{d})$ denote a real-valued
wavelet with vanishing moments in $\mathbb{R}\times\\{0\\}^{d-1}$ of order

$r>s_{0}+s_{1}N+s_{2}N(u)~{}.$ (41)

  1. (a)

If $(x,\pm\xi)$ is an unsigned $N$-regular directed point of $u$, then there
exists a neighborhood $U$ of $x$, some $R>0$ and a $\xi$-neighborhood
$W\subset S^{d-1}$ such that the following estimate holds:

$\exists\,C>0\,\forall\,y\in U\,\forall\,h\in K_{o}(W,V,R)\cup
K_{o}(-W,V,R)~{}:~{}|{W}_{\psi}u(y,h)|\leq
C\cdot\|h\|^{N-\frac{d}{2\lambda_{\min}}}\,.$

  2. (b)

Assume that $U$ is a neighborhood of $x$, and that there are $R>0$ and a
$\xi$-neighborhood $W\subset S^{d-1}$ such that

$\exists\,C>0\,\forall\,y\in U\,\forall\,h\in K_{o}(W,V,R)\cup
K_{o}(-W,V,R)~{}:~{}|{W}_{\psi}u(y,h)|\leq
C\cdot\left\|h\right\|^{\frac{N}{\lambda_{\min}}+\frac{3d}{2\lambda_{\min}}+\alpha_{2}}\,.$

Then $(x,\pm\xi)$ is an unsigned $N$-regular directed point of $u$.

###### Proof.

We first collect the various constants required for the application of Theorem
5.8, starting with $\dim(H)=d$. Recall from Lemma 6.4 that $H$ is microlocally
admissible, with constant

$\alpha_{1}=\frac{1}{\lambda_{\min}}>1~{}\mbox{ and
}\alpha_{2}=\frac{d}{\lambda_{\min}}+\epsilon^{*},\mbox{ for any
}\epsilon^{*}>0.$

Lemma 6.6 contributes the constant $\ell_{1}=n_{0}+1$ for which inequality
(18) holds. By Lemma 6.11, $H$ fulfills the overlap control condition with
constants

$\gamma_{0}=2d+1~{},~{}\gamma_{1}=\frac{\lambda_{\min}}{1-\lambda_{\max}}~{},~{}\gamma_{2}=\frac{\lambda_{\min}\lambda_{\max}}{1-\lambda_{\max}}.$

Clearly, $\gamma_{1}\alpha_{1}\geq 1$. Moreover, the assumption
$\lambda_{\min}+\lambda_{\max}\geq 1$ implies that

$\gamma_{1}\geq 1~{},\gamma_{2}\alpha_{1}\geq 1~{}.$ (42)

As a consequence, the sharpest lower bound for $\beta_{1}$ contained in the
conditions (i) through (iv) of Theorem 5.4 is provided by condition (i),
whereas the relevant lower bounds for $\beta_{2},\beta_{3}$ are provided by
(iv), as a consequence of (42). Thus the conditions of Theorem 5.4 (i) through
(iv) are fulfilled by

$\displaystyle\beta_{1}$ $\displaystyle=$ $\displaystyle
1+\left(1+\frac{1}{2\lambda_{\min}}\right)d+N+\left(1+\frac{1}{\lambda_{\min}}\right)N(u)~{}$
$\displaystyle\beta_{2}$ $\displaystyle=$ $\displaystyle
1+\left(2+\frac{3}{2-2\lambda_{\max}}\right)d+\frac{\lambda_{\min}}{1-\lambda_{\max}}N+\left(1+\frac{1}{1-\lambda_{\max}}\right)N(u)$
$\displaystyle\beta_{3}$ $\displaystyle=$ $\displaystyle
1+\left(\frac{7}{2}+\frac{3\lambda_{\max}}{2-2\lambda_{\max}}\right)d+\frac{\lambda_{\min}\lambda_{\max}}{1-\lambda_{\max}}N+\frac{1}{1-\lambda_{\max}}N(u)~{}.$

By Theorem 5.4, the relevant lower bound for the number of vanishing moments
is given by the inequality

$r>\frac{d}{2}+\ell_{1}(\beta_{1}+\beta_{2}+\beta_{3})+\beta_{1}~{}.$

Plugging in the values for $\beta_{1},\beta_{2}$ and $\beta_{3}$ that we just
obtained, and sorting terms, results in the equivalent inequality

$r>s_{0}+s_{1}N+s_{2}N(u)~{},$

with $s_{0},s_{1},s_{2}$ given by equations (38) through (40). ∎

Note that Theorem 5.8 is applicable to the shearlet dilation group $H$ as soon
as $\lambda_{\min}+\lambda_{\max}\geq 1$ and
$0<\lambda_{\min}\leq\lambda_{\max}<1$ hold. It is worthwhile noting that
these conditions are also sufficient to guarantee that $H$ characterizes the
wavefront set of infinite order, by Theorem 28 of [1]. Furthermore, Corollary
5.9 also applies under these conditions.

We note that both the standard and the Toeplitz shearlet groups from Remark
6.3 can fulfill these conditions, when suitable parameters are chosen. For the
standard case this is obvious, for the Toeplitz case any parameter
$0<\delta\leq 1/d$ will work.

###  6.1. Discussion/comparison/future directions for results on generalized
shearlets

It is interesting to note that there is essentially only one condition
required to make a shearlet dilation group a useful tool for the
characterization of the Sobolev wavefront set. In particular, the structure of
the shearing subgroup only influences the involved constants.

Theorem 6.12 can be briefly summarized by the statement that there is an
easily computed lower bound on the number $r$ of vanishing moments that allows
a near characterization of elements of the unsigned wavefront set, valid for
any Schwartz wavelet with $r$ vanishing moments in
$\mathbb{R}\times\\{0\\}^{d-1}$. It establishes an affine dependence of the
required number of vanishing moments on the degree of smoothness and/or the
order of the distribution under consideration. The main information provided
by Theorem 6.12, on top of guaranteeing the applicability of Theorem 5.8, is
the concrete expressions for the involved constants, and their dependence on
the structure of the underlying group.

Ultimately only very limited information on the group is required. Note that
$s_{0},s_{1},s_{2}$ only need to be determined once for each group, and they
depend only on the eigenvalue range for the infinitesimal generator of the
scaling subgroup and the nilpotency degree $n_{0}$ of the shearing subgroup.
As the dimension $d$ increases, the number of relevant groups grows rather
quickly. For example, it is known that for dimensions $d\geq 7$, there exists
an uncountably infinite family of shearing subgroups that are not related by
conjugation (see e.g. Remark 15 of [1]), and we expect that many of these
subgroups can be combined with a scaling subgroup fulfilling the eigenvalue
condition of Theorem 6.12. These observations emphasize that Theorem 6.12
applies to a large variety of groups.

It is reasonable to expect that the generality and scope of these results
comes at the price of suboptimal estimates and constants. Hence for specific
choices of the dilation group $H$ Theorem 6.12 should rather be seen as a
proof of principle, with considerable room for improvement.

One instance where currently known results indicate such a possibility
concerns the gap between the exponents in the decay conditions of the direct
and inverse theorems. In the shearlet case the requirement
$0<\lambda_{\min}<1$ implies that $\alpha_{1}=\frac{1}{\lambda_{\min}}>1$.
This entails that the gap between the exponents, namely $N-\alpha_{1}d/2$ and
$\alpha_{1}N+\frac{3}{2}\alpha_{1}d+\alpha_{2}$ grows linearly with $N$.

This can be contrasted to the results in [11]. The setting considered there is
essentially that of a shearlet dilation group in dimension two, with
$\lambda_{\min}=1/2$ and $n_{0}=1$. A detailed comparison of our results with
the direct and inverse theorems (i.e., 3.1 and 5.5) of [11] cannot be easily
performed, due to the various differences of notations and definitions. In
spirit however, the results are very similar, relying on lower bounds for
numbers of vanishing moments that are affine functions of the smoothness
degree $N$. A closer inspection indicates that the gap between the decay
orders in the direct and inverse theorems, which grows linearly with $N$ in
our setting, can be kept under control in [11]. It is currently unclear
whether this difference in the results can be addressed within our framework
by the use of more refined and possibly more specific estimates than the ones
that we are currently using.

In other respects, our results are stronger and/or more widely applicable than
the ones in [11]. To begin with, the results in the cited paper only apply to
shearlets in dimension two. As a further important difference we note that
[11] imposes the additional constraint $u\in{L}^{2}(\mathbb{R}^{d})$. This
assumption implies that the order $N(u)$ is bounded by $d/2$, which is the
main reason why $N(u)$ systematically features in our assumptions, but not in
[11]. As a consequence of its focus on ${L}^{2}$, [11] can work with a
somewhat extended reservoir of admissible wavelets, which explains additional
assumptions in the direct theorem of [11] on the decay properties of $\psi$,
that are trivially fulfilled by Schwartz functions, and hence do not appear in
our results.

A final difference between our paper and the predecessor papers [15, 11]
concerns the role of the unsigned wavefront set. While both [15, 11] formulate
their results for regular directed points, in both sources the definition of
such a point actually refers to Fourier decay inside $C\cup-C$, for a suitably
chosen cone $C\subset\mathbb{R}^{2}$ (see beginning of Section 5 in [15], and
[11, Definition 2.1]). Hence they effectively refer to unsigned regular
directed points in the terminology of our Definition 4.1, without mentioning
this fact explicitly. And this restriction is not coincidental; it results
from the fact that the shearlet groups in [15, 11] only use positive scalings
$a>0$. In the terminology of our paper, this corresponds precisely to
replacing $H$ by the half-space adapted subgroup $H_{0}$. This choice requires
a more restrictive admissibility condition being used in [15, 11], which is
ultimately responsible for the inability of the wavelet transform to
distinguish directions $\xi$ and $-\xi$ in the wavefront set analysis.

We emphasize that, in contrast to the mentioned sources, our Theorem 3.4 does
apply to regular directed points proper, as do the results of the precursor
paper [5].

## Acknowledgements

This paper was initiated during a visit of the second author to the Department
of Mathematics at RWTH Aachen University in 2018. The authors thank the
department for the support and hospitality during this visit. The second
author also acknowledges support from National Science Foundation Grant
DMS-1902301 during the preparation of this article.

## References

  * [1] G. S. Alberti, S. Dahlke, F. De Mari, E. De Vito, and H. Führ.  Recent progress in shearlet theory: systematic construction of shearlet dilation groups, characterization of wavefront sets, and new embeddings.  In Frames and other bases in abstract and function spaces, Appl. Numer. Harmon. Anal., pages 127–160. Birkhäuser/Springer, Cham, 2017\.
  * [2] H. G. Feichtinger and K. Gröchenig.  A unified approach to atomic decompositions via integrable group representations.  In Function spaces and applications (Lund, 1986), volume 1302 of Lecture Notes in Math., pages 52–73. Springer, Berlin, 1988.
  * [3] H. G. Feichtinger and K. Gröchenig.  Banach spaces related to integrable group representations and their atomic decompositions. I.  J. Funct. Anal., 86(2):307–340, 1989.
  * [4] H. G. Feichtinger and K. Gröchenig.  Banach spaces related to integrable group representations and their atomic decompositions. II.  Monatsh. Math., 108(2-3):129–148, 1989.
  * [5] J. Fell, H. Führ, and F. Voigtlaender.  Resolution of the wavefront set using general continuous wavelet transforms.  J. Fourier Anal. Appl., 22(5):997–1058, 2016.
  * [6] H. Führ.  Generalized Calderón conditions and regular orbit spaces.  Colloq. Math., 120(1):103–126, 2010.
  * [7] H. Führ.  Coorbit spaces and wavelet coefficient decay over general dilation groups.  Trans. Amer. Math. Soc., 367(10):7373–7401, 2015.
  * [8] H. Führ.  Vanishing moment conditions for wavelet atoms in higher dimensions.  Adv. Comput. Math., 42(1):127–153, 2016.
  * [9] H. Führ and M. Mayer.  Continuous wavelet transforms from semidirect products: cyclic representations and Plancherel measure.  J. Fourier Anal. Appl., 8(4):375–397, 2002.
  * [10] H. Führ and R. R. Tousi.  Simplified vanishing moment criteria for wavelets over general dilation groups, with applications to abelian and shearlet dilation groups.  Appl. Comput. Harmon. Anal., 43(3):449–481, 2017.
  * [11] P. Grohs.  Continuous shearlet frames and resolution of the wavefront set.  Monatsh. Math., 164(4):393–426, 2011.
  * [12] P. Grohs and v. Kereta.  Analysis of edge and corner points using parabolic dictionaries.  Appl. Comput. Harmon. Anal., 48(2):655–681, 2020.
  * [13] K. Guo and D. Labate.  Characterization and analysis of edges in piecewise smooth functions.  Appl. Comput. Harmon. Anal., 41(1):139–163, 2016.
  * [14] E. Hewitt and K. A. Ross.  Abstract harmonic analysis. Vol. I, volume 115 of Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences].  Springer-Verlag, Berlin-New York, second edition, 1979.  Structure of topological groups, integration theory, group representations.
  * [15] G. Kutyniok and D. Labate.  Resolution of the wavefront set using continuous shearlets.  Trans. Amer. Math. Soc., 361(5):2719–2754, 2009.
  * [16] G. Kutyniok and P. Petersen.  Classification of edges using compactly supported shearlets.  Appl. Comput. Harmon. Anal., 42(2):245–293, 2017.