# Quartz Crystal Microbalance frequency response to finite-size adsorbents in
liquid

Alexander M. Leshansky1<EMAIL_ADDRESS>Itzhak Fouxon1 Boris Y.
Rubinstein2 1Department of Chemical Engineering, Technion, Haifa 32000, Israel
3Stowers Institute for Medical Research, 1000 E 50th st., Kansas City, MO
64110, USA

###### Abstract

Quartz Crystal Microbalance with Dissipation monitoring (QCM-D) has become a
major tool in the analysis of adsorption of nanometric objects, such as
proteins, viruses, liposomes and inorganic particles from the solution. While
in vacuum extremely accurate mass measurements are possible, in a liquid phase
the quantitative analysis is intricate due to the complex interplay of
hydrodynamic and adhesion forces, varying with the physicochemical properties
of adsorbent and the quartz resonator surfaces. In the present paper we
dissect the role of hydrodynamics for the analytically tractable scenario of a
_stiff_ contact, whereas the adsorbed particles oscillate with the resonator
as a whole without rotation. Under the assumption of the low surface coverage,
we theoretically study the _excess_ shear force exerted on the resonator due
to presence of a single adsorbed particle. The excess shear force has two
contributions: (i) the _fluid-mediated_ force due to flow disturbance created
by the particle and (ii) the viscous force exerted on the particle by the
fluid and transmitted to the resonator _via contact_. We found that for small
enough particles there is a mutual cancellation of the above dual components
of the net excess shear force, reducing the overall effect of the
hydrodynamics to that comparable in magnitude to the inertial force. These
findings indicate that the accurate account of hydrodynamics in the analysis
of QCM-D response is as important as the inertial mass of the adsorbents
(determining the frequency shift in the Sauerbrey equation). The resulting
dimensionless frequency and dissipation shifts and the corresponding acoustic
ratio computed numerically, showing a fair agreement with previously published
experimental results at low oscillation frequencies.

Introduction. Quartz crystal microbalance (QCM) technique [1, 2] relies on the
fact that matter adsorbed on the surface of the fast oscillating crystal,
changes the frequency of the oscillations. In vacuum, the shift in the
resonant frequency of the crystal is linearly proportional to the mass of the
adsorbed film via the seminal Sauerbrey equation [3], allowing extremely
accurate measurements down to nanograms [2]. The quantitative interpretation
of the QCM-D measurement in liquids [4, 5] (where “D” stands for dissipation
monitoring via measuring the decay rate of the oscillations) is also well-
established for planar (including viscoelastic films [2]) adsorbed films.
However, interpreting the QCM-D measurements due to _discrete_ adsorbents
(such as, e.g., nanoparticles, liposomes, viruses, proteins, etc.) in liquids
remains a challenge mainly due to the interplay of complex hydrodynamics,
which has not yet been yet fully resolved and _a priori_ unknown viscoelastic
contact dynamics, which depends on physicochemical properties of the surfaces
(i.e., the adsorbent and the resonator) [2].

The impedance $\mathcal{Z}$ probed by the QCM-D is the ratio
$\overline{\sigma}/v_{c}$, where $\overline{\sigma}$ is the area-averaged
tangential stress (i.e. the net shear force $\mathcal{F}$ exerted on the
surface of the oscillating quartz resonator divided by its surface area) and
$v_{c}$ is the velocity of the crystal oscillations. Here $\mathcal{F}$ and
$v_{c}$ and, therefore, $\mathcal{Z}$ are all complex quantities characterized
by the amplitude and phase. In the framework of the _small load approximation_
the shift in oscillation frequency, $\Delta f$, and in half-bandwidth,
$\Delta\Gamma$ (related to a dissipation factor $\Delta\mathcal{D}$), are
linearly proportional to the impedance, $\Delta
f-\mathrm{i}\Delta\Gamma=\mathrm{i}f\mathcal{Z}/(\pi\mathcal{Z}_{q})$, where
$f$ stands for the oscillation frequency (typically in MHz range) and the
resonator’s shear-wave impedance $\mathcal{Z}_{q}$ is a known quantity [1, 2].
The small load approximation holds given that $\Delta f\ll f$. In liquids, in
contrast to vacuum where the adsorbed particles only alter the mass (i.e.,
solid inertia) of the resonator contributing to the frequency shift, $\Delta
f$, according to the Sauerbrey equation, the adsorbed particles modify the
viscous _shear force_ exerted onto the resonator, contributing to the shifts
in the resonant frequency, $\Delta f$, and the bandwidth, $\Delta\Gamma$
(absent in vacuum).

In the absence of particles, the horizontal small-amplitude time-periodic
oscillations of the resonator at $z\\!=\\!0$ with velocity
$v_{0}\hat{\bm{x}}\cos{\omega t}$ create unidirectional oscillatory flow of
the viscous liquid of viscosity $\eta$ and density $\rho$ occupying the upper
half-space $z\\!>\\!0$ with velocity given by the real part of
$v_{0}\hat{\bm{x}}\mathrm{e}^{-z/\delta}\mathrm{e}^{-\mathrm{i}(\omega
t-z/\delta)}$ [6]. The flow disturbance propagates upward as the transverse
wave attenuated by the exponential factor with $\delta=(2\nu/\omega)^{1/2}$
known as _viscous penetration depth_ , where $\nu\\!=\\!\eta/\rho$ stands for
the kinematic viscosity of the fluid (see Fig. 1). Computing the shear stress
at the resonator, $\sigma_{xz}=\eta\,(\partial u_{x}/\partial z)_{z=0}$, and
dividing by the resonator velocity readily yields the impedance
${\mathcal{Z}}\\!=\\!(\mathrm{i}-1)\,\eta v_{0}/\delta$ [5], corresponding to
a negative frequency shift and positive dissipation factor (as compared to the
unloaded resonator oscillating in vacuum). Obviously, the particles located
above the resonator would perturb this flow and modify the shear stress
exerted onto the resonator. The contribution to impedance due to the flow
disturbance is entirely _fluid-mediated_ , i.e., it takes place for both
adsorbed and freely suspended particles, as it does not require a physical
contact between the particle and the resonator. For the adsorbed particle,
however, there is another contribution to impedance due to the force exerted
on its surface by the perturbed flow and transmitted to the resonator via
contact.

The prior works applied a variety of numerical methods to account for the
hydrodynamics and compute the perturbed viscous stress viscous stress exerted
on resonator due to an adsorbed particle. Various factors, such as particle
size, surface coverage, particle mobility (e.g., rocking vs. sliding motion),
deviation from sphericity and other factors were considered using Finite
Element method (FEM) in the early works [7, 8], and later with Lattice
Boltmann method [9, 10, 11] and the Immersed Boundary method [12]. Although
the numerical methods are very powerful, the complex interplay of various
factors and uncertainty of physicochemical properties and/or parameters
governing the contact dynamics, call for a more analytical approach able to
dissect the role of the hydrodynamic forces in QCM-D analysis of finite-size
adsorbents. In Ref. [13] the hydrodynamic contribution to the impedance due to
an adsorbed particle was approximated by the analytical result for the force
exerted on a rigid sphere oscillating in an _unbounded_ viscous liquid (see,
e.g., [6]). One may expect such approximation to hold for a relatively large
(i.e., with respect to the penetration depth $\delta$) particle, as most of
its surface is in contact with otherwise quiescent fluid located above the
viscous penetration layer. Such assumption, however, requires justification
since the unsteady viscous flow in a wall-bounded domain could be quite
different from the unbounded flow (e.g., [14]). Obviously, for particle of the
size comparable to or smaller than the viscous penetration depth, this
approximation would not apply. Moreover, the above approximation implicitly
assumed that the hydrodynamic contribution to the _contact_ force dominates
over its fluid-mediated counterpart, which was entirely neglected.

Figure 1: Schematic illustration of the problem. A spherical particle of
radius $a$ immersed in an incompressible viscous liquid of density $\rho$ and
viscosity $\eta$ is rigidly attached to an infinite horizontal plane at
$z\\!=\\!0$ oscillating at MHz frequency with velocity
$\bm{v}=v_{0}\hat{\bm{x}}\cos{\omega t}$. The undisturbed (i.e., in the
absence of the particle) velocity profiles,
$\bm{u}_{0}=v_{0}\hat{\bm{x}}\mathrm{Re}[\mathrm{e}^{-z/\delta}\mathrm{e}^{-\mathrm{i}(\omega
t-z/\delta)}]$, are shown at two time instants $\omega t\\!=\\!0$ (solid, red)
and $\omega t\\!=\\!\pi/2$ (dashed, blue) vs. the scaled vertical distance
$z/\delta$. The short-dashed vertical line stands for the zero value of the
velocity.

The fluid-mediated contribution to the QCM-D impedance due to adsorbed
particles was recently studied in Ref. [15] using point-like particle
approximation assiming adhesion due to strong lubrication forces. This theory
was later revisited in Ref. [16] where the _excess_ shear force (or impedance)
due to presence of either freely suspended or well adhered (i.e., oscillating
as a while with a resonator) finite-size particles was determined analytically
using a distant-particle asymptotic theory. The derived in Ref. [16] closed-
form expressions for the impedance and the velocity (linear and angular) of
the freely suspended particle show a very close agreement with the numerical
(FEM) computations down to a rather close proximity of less than a particle
radius. It was found, in particular, that for some realistic experimental
conditions the flow disturbance due to a layer of freely suspended particles
located above the resonator, produces the common (“inertial loading”) response
with $\Delta f<0$ and $\Delta\Gamma>0$ of a magnitude of a few Hz’s (at
resonant frequency $f\\!=\\!5$ MHz). The same layer of adsorbed particles,
however, results in the _positive_ frequency shift and unorthodox _negative_
bandwidth shift of some hundreds of Hz’s. Notice the positive frequency shift
(which is typically associated with non-hydrodynamic effects, such as contact
viscoelasticity), while $\Delta\Gamma\\!<\\!0$, implies _reduced dissipation_
due to presence of the adsorbed particles. The reason for the seemingly
unphysical (sign- and magnitude-wise) response, is that the analysis only
concerned only the excess shear due to the flow disturbance, whereas an
adsorbed particle oscillating with a resonator as a whole excludes a fluid
volume above it and also shields the resonator from the transverse shear wave
that persists in the particle absence. The _net_ excess shear force due to
adsorbed particles should, however, combine the fluid-mediated and the contact
force. In the present paper we provide a detailed theoretical study of the net
excess shear force (impedance) due to finite-size adsorbents at low surface
coverage in the analytically tractable limit of a stiff contact, which allows
to decouple and analyze the role hydrodynamics independently from other
physical phenomena.

Problem formulation. The viscous incompressible liquid in the half-infinite
space $z\\!>\\!0$ is set into motion by the time-periodic horizontal
oscillations of the infinite plane at $z\\!=\\!0$ along the $x$-axis with
frequency $\omega$ and amplitude $v_{0}$ (see Fig. 1). We further assume that
a spherical particle of radius $a$ firmly adheres to the plane and, therefore,
oscillates with it in-sync without rotation. Assuming small amplitude of the
oscillations, $v_{0}/\omega\ll a$, to the leading approximation the flow
velocity ${\bm{V}}$ satisfies the unsteady Stokes equations

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\partial_{t}{\bm{V}}\\!=\\!-\rho^{-1}\nabla
P\\!+\\!\nu\nabla^{2}{\bm{V}},\ \ \nabla\cdot{\bm{V}}\\!=\\!0,$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!{\bm{V}}(z\\!=\\!0)={\bm{V}}(r\\!=\\!a,t)=v_{0}\hat{\bm{x}}\cos(\omega
t)\,.$ (1)

where $P$ is the pressure, $\rho$ and $\nu\\!=\\!\eta/\rho$ are the density
and the kinematic viscosity of the fluid, respectively, and the spherical
distance $r=|\bm{x}-\bm{x}_{c}|$ is measured from the particle center located
at $\bm{x}_{c}=(0,0,h)$. Although the particle adhesion corresponds to a
vanishing separation distance, $h\\!\approx\\!a$, we follow the general
formulation [15, 16] and keep an arbitrary proximity $h\geq a$ in the analysis
below. We introduce dimensionless variables by normalizing fluid velocity with
$v_{0}$, pressure with $\eta v_{0}/a$, time with $\omega^{-1}$ and distance
with $a$. Thus the dimensionless (complex) flow field $\bm{v}$ and pressure
$p$ defined via ${\bm{V}}=v_{0}\mathrm{Re}[\mathrm{e}^{-\mathrm{i}\omega
t}\bm{v}]$ and $P=\eta v_{0}\mathrm{Re}[\mathrm{e}^{-\mathrm{i}\omega t}p]/a$,
where $\mathrm{Re}$ stands for the real part, satisfy

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\lambda^{2}\bm{v}\\!=\\!-\nabla
p\\!+\\!\nabla^{2}\bm{v},\ \ \nabla\cdot\bm{v}\\!=\\!0,$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\bm{v}(z\\!=\\!0)=\hat{\bm{x}},\
\ \bm{v}(r\\!=\\!1)=\hat{\bm{x}}\,.$ (2)

Here
$\lambda^{2}\\!=\\!-\mathrm{i}a^{2}\omega/\nu=-2\mathrm{i}(a/\delta)^{2}$. In
the absence of a particle, the solution of Eqs. (2) is given by
$\bm{u}_{0}=\mathrm{e}^{-\lambda z}\hat{\bm{x}}$, where
$\lambda\\!=\\!(1-\mathrm{i})\,(a/\delta)$, and $p_{0}\\!=\\!0$.

When the particle is present, no analytical solution of Eqs. (2) is readily
available, however some analytical progress is possible, e.g., for a distant
particle (see [16]). The major aim of this paper is determining the
$x$-component of the complex _excess_ shear force (i.e., in excess to the
shear force applied by the particle-free background flow), $F$ exerted on the
oscillating plate in the incompressible viscous liquid due to an adsorbed
particle.

For low values of the particle surface number density, $\tilde{n}$, when
mutual hydrodynamic interactions between particles can be neglected, the
dimensionless excess shear force $F/\eta av_{0}$ is equivalent to the
dimensionless impedance, ${\\!\mathcal{Z}}/(\eta a\tilde{n})$ probed by the
QCM-D device. The _net_ excess shear force $F$ has two contributions: (i) the
fluid-mediated contribution (screening or shielding force) due to presence of
the particle and (ii) the direct force the particle exerts on the surface _via
contact_.

Fluid-mediated force. The dimensionless stress tensor corresponding to
$\\{\bm{v},p\\}$ in Eqs. (2) is defined by
$\sigma_{ik}\\!\equiv\\!-p\delta_{ik}\\!+\\!\partial_{k}v_{i}\\!+\\!\partial_{i}v_{k}$.
In absence of the particle, $\sigma_{ik}$ has only $xz$ and $zx$ components,
which at the plane $z\\!=\\!0$ equal to $-\lambda$. If the particle is
present, it modifies the stress exerted on the resonator by the fluid in the
vicinity of the contact, however, far from the particle we shall still have
$\sigma_{xz}\\!\simeq\\!-\lambda$. Therefore, the net fluid-mediated _excess_
shear force $F_{a}$ (i.e., in excess of $-\lambda$ times the surface of the
resonator) exerted on the oscillating plate due to presence of an adsorbed
particle is defined by

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!F_{a}\\!=\\!\int_{z=0}\\!\left(\sigma_{xz}\\!+\\!\lambda\right)dxdy,$
(3)

The flow perturbation, $\bm{u}=\bm{v}-\mathrm{e}^{-\lambda z}\hat{\bm{x}}$, is
governed by:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\lambda^{2}\bm{u}\\!=\\!-\nabla
p\\!+\\!\nabla^{2}\bm{u},\ \ \nabla\cdot\bm{u}\\!=\\!0,$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\bm{u}(z\\!=\\!0)=0,\ \
\bm{u}(r\\!=\\!1)=\left(1-\mathrm{e}^{-\lambda z}\right)\hat{\bm{x}}.$ (4)

The stress tensor
$\sigma_{ik}^{\prime}\\!=\\!-p\delta_{ik}\\!+\\!\partial_{k}u_{i}\\!+\\!\partial_{i}u_{k}$
corresponding to $\\{\bm{u},p\\}$ in Eq. (4) obeys
$\lambda^{2}u_{i}\\!=\\!\partial_{k}\sigma^{\prime}_{ik}$ and can be written
via $\sigma_{ik}$ as

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\sigma_{ik}^{\prime}\\!=\\!\sigma_{ik}+\left(\delta_{ix}\delta_{kz}+\delta_{iz}\delta_{kx}\right)\lambda\mathrm{e}^{-\lambda
z}\,.$ (5)

Thus $F_{a}$ in (3) can then be written as

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!F_{a}\\!=\\!\int_{z=0}\sigma^{\prime}_{xz}dxdy=\\!\int_{z=0}\partial_{z}u_{x}dxdy.$
(6)

The direct numerical study of the force using Eq. (6) is problematic. The
general structure of unsteady Stokes flows generated at the particle surface
indicates that, at distances from the boundary greater than the viscous
penetration depth $\delta/a\\!\propto\\!|\lambda|^{-1}$, the flow $\bm{u}$ a
is a superposition of a potential (inviscid) flow and exponential correction,
see, e.g. [6]. However the contribution of the dominant potential flow
component into the integral in Eq. (6) vanishes identically. Hence $F_{a}$ is
controlled entirely by the exponentially small correction to the potential
flow. This renders accurate numerical computation of $F_{a}$ over infinite
plate in Eqs. (6) challenging.

We rewrite $F_{a}$ in the form which is more suitable for the numerical study
by using the Lorentz reciprocity [17]. For an arbitrary incompressible dual
flow satisfying $\lambda^{2}{\hat{v}}_{i}=\partial_{k}{\hat{\sigma}}_{ik}$ we
have:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\frac{\partial({\hat{v}}_{i}\sigma^{\prime}_{ik})}{\partial
x_{k}}=\frac{\partial(u_{i}{\hat{\sigma}}_{ik})}{\partial x_{k}}.$ (7)

Integrating Eq. (7) over the fluid volume in the semi-infinite domain,
applying the divergence therem and using the original flow field $\bm{v}$
satisfying Eqs. (2) as the dual flow, we find that:

$\displaystyle\\!\\!\\!\\!F_{a}\\!$ $\displaystyle=$
$\displaystyle\\!-\oint_{r=1}\\!\\!\mathrm{e}^{-\lambda
z}\,\sigma^{\prime}_{xk}n_{k}dS\\!-\\!\frac{4\pi\mathrm{e}^{-\lambda
h}(\sinh{\lambda}\\!-\\!\lambda\cosh{\lambda})}{\lambda}\,$ (8)
$\displaystyle\\!\\!\\!\\!+\frac{\pi\mathrm{e}^{-2\lambda
h}(\sinh{2\lambda}-2\lambda\cosh{2\lambda})}{\lambda},$

where we made use of Eq. (5), giving the traction at the particle surface as
$\sigma_{xk}n_{k}\\!=\\!-\lambda\mathrm{e}^{-\lambda
z}\cos{\theta}+\sigma^{\prime}_{xk}n_{k}$, where $\theta$ is the polar
spherical angle. Thus, instead of integration over the infinite plane at
$z\\!=\\!0$ in Eq. (6), the excess shear force $F_{a}$ can be alternatively
evaluated by integrating the traction $\sigma^{\prime}_{xk}n_{k}$ over the
particle surface at $r\\!=\\!1$. Notice also that the last two (analytical)
terms in the RHS of Eq. (8) comprise (up to a factor of $\pi$) the net
hydrodynamic contribution to the impedance due to an adsorbed particle
reported in Ref. [15]. The numerical results indicate that the 1st (integral)
term is usually dominant over the last two (analytical) terms.

Contact force and torque. For _freely suspended_ particles the excess shear
force exerted on the resonator is mediated solely by the suspending fluid
[16]. The adsorbed particle not only modifies the flow above the resonator
(i.e., via $F_{a}$), but also applies a force via _contact_. We assume that
the contact force the rigidly attached particle exerts on the plane, $F_{c}$,
is equal in magnitude and opposite in sign to the force that the plane exerts
on the particle. The contact force $F_{c}$ is determined from the Newton’s
force balance:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\lambda^{2}\xi
U\\!=\\!\oint_{r=1}\\!\\!\sigma_{xk}n_{k}dS-F_{c},$ (9)

where for a particle moving with a plane as a whole its dimensionless
translation velocity $U\\!=\\!1$ and the traction $\sigma_{ik}n_{k}$
corresponds to the original flow in Eqs. (2). Here the parameter $\xi=m/\rho
a^{3}$, where $m$ stands for the particle’s mass, characterizes the solid
inertia.

Substituting the traction at the particle surface
$\sigma_{xk}n_{k}\\!=\\!-\lambda\mathrm{e}^{-\lambda
z}\cos{\theta}+\sigma^{\prime}_{xk}n_{k}$ into Eq. (9) yields the following
result:

$\displaystyle F_{c}$ $\displaystyle=$
$\displaystyle-\frac{4\pi\,\mathrm{e}^{-\lambda
h}(\sinh{\lambda}-\lambda\cosh{\lambda})}{\lambda}+$ (10)
$\displaystyle+\oint_{r=1}\\!\\!\sigma^{\prime}_{xk}n_{k}dS-\lambda^{2}\xi=F_{c}^{\prime}-\lambda^{2}\xi\,,$

where $F^{\prime}_{c}$ is the hydrodynamic part of the contact force. The net
excess shear force due to an adsorbed particle can now be found as
$F=F_{a}+F_{c}$. Notice that upon neglecting the hydrodynamics entirely, the
contact force (and so the net excess force) reduces, as expected, to the
Sauerbrey equation,
$F\\!=\\!-\lambda^{2}\xi\\!=\\!-(4\pi/3)(\rho_{s}/\rho)\lambda^{2}\\!=\\!\mathrm{i}m\omega/(\eta\tilde{n}a)$.

The contact _torque_ $L_{c}$ (the $y$-component, scaled with $\eta
a^{2}v_{0}$) the adsorbed particle exerts on the resonator could also be of
interest towards estimating the stiffness of the contact and it is given by
(with respect to the particle center at $z\\!=\\!h$):

$\displaystyle\frac{2}{5}\lambda^{2}\xi\Omega$ $\displaystyle=$
$\displaystyle\oint_{r=1}\\!\\!\left[(z-h)\sigma_{xk}\\!-\\!x\sigma_{zk}\right]n_{k}dS-
L_{c}\,,$ (11)

where $\Omega$ is the dimensionless angular velocity of the particle scaled
with $v_{0}/a$. For an adsorbed particle with a stiff contact (i.e., without
rotation, $\Omega\\!=\\!0$) there is no contribution of the solid inertia in
the LHS of Eq. (11) and the contact torque reduces to

$\displaystyle
L_{c}=\oint_{r=1}\\!\\!\left[(z-h)\sigma_{xk}\\!-\\!x\sigma_{zk}\right]n_{k}dS\,.$
(12)

The contact torque in Eq. (12) be rewritten as an integral over the perturbed
traction $\sigma^{\prime}_{ik}n_{k}$ using Eq. 5 as (cf. Eq. (22) for
$\mathcal{B}$ in [16]):

$\displaystyle L_{c}$ $\displaystyle=$
$\displaystyle\\!\oint_{r=1}\\!\\!\left[\cos{\theta}\,\sigma^{\prime}_{xk}\\!-\\!\sin{\theta}\cos{\phi}\,\sigma^{\prime}_{zk}\right]n_{k}dS$
(13) $\displaystyle\\!-4\pi\mathrm{e}^{-\lambda
h}\left[\sinh{\lambda}+\frac{3\left(\sinh{\lambda}-\lambda\cosh{\lambda}\right)}{\lambda^{2}}\right].$

If contact torque with respect to the point of contact (at $z\\!=\\!0$) is
considered, then we readily have:

$\displaystyle L^{(c)}_{c}$ $\displaystyle=$
$\displaystyle\\!\oint_{r=1}\\!\\!\left(z\sigma_{xk}\\!-\\!x\sigma_{zk}\right)n_{k}dS=$
(14) $\displaystyle
L_{c}+h\oint_{r=1}\\!\\!\sigma_{xk}n_{k}dS=L_{c}+hF^{\prime}_{c}\,,$

where $F_{c}^{\prime}$ is the hydrodynamic part of the contact force in Eq.
(10).

Notice that the above derivations of $F_{a}$ and $F_{c}$ are rigorous and do
not involve any approximation, besides from the assumption of small-amplitude
oscillations that allowed to neglect the nonlinear inertia terms in the flow
equations. The resulting expressions involve integrals of the traction
associated with the perturbed flow, $\sigma^{\prime}_{ik}n_{k}$, over the
particle surface at $r\\!=\\!1$ which can be performed numerically.

Small-particle limit. Let us consider the small-particle (or low-frequency)
limit, $|\lambda|\ll 1$, for which the steady Stokes equations hold to the
first approximation, as the unsteady term $\lambda^{2}\bm{u}$ in Eqs. (4)
produces $o(|\lambda^{2}|)$ corrections in the solution [19]. We next expand
the perturbed flow $\bm{u}$ in Eqs. (4) as
$\bm{u}=\lambda\bm{u}_{1}+\lambda^{2}\bm{u}_{2}+\ldots$. At the leading order
we have:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!0\\!=\\!-\nabla
p_{1}\\!+\\!\nabla^{2}\bm{u}_{1},\ \ \nabla\cdot\bm{u}_{1}\\!=\\!0,$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\bm{u}_{1}(z\\!=\\!0)=0,\
\ \bm{u}_{1}(r\\!=\\!1)=z\hat{\bm{x}},$ (15)

where $p_{1}$ stands for pressure to order $\lambda$. Notice that the
analytical terms in the r.h.s. of Eqs. (8) and (10) are all
$\mathcal{O}(|\lambda|^{2})$, meaning that at the leading order
$\mathcal{O}(|\lambda|)$ we have:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!F_{c}^{(1)}\\!=\\!-F_{a}^{(1)}\\!=\\!\oint_{r=1}\sigma^{\prime}_{xk}n_{k}dS\,.$
(16)

In other words, for small particles, the fluid-mediated contribution is
compensated by the (hydrodynamic part of) contact force at the leading
approximation, such that the net excess force due to an adsorbed particle
$F\\!=\\!F_{a}+F_{c}$ reduces to $\mathcal{O}(|\lambda|^{2})$. The Eqs. (15)
govern the problem of a steady linear shear flow past a fixed sphere in
contact with a plane wall, and its exact solution using special “touching
sphere” coordinates is given in [18]. In particular, the dimensionless contact
force in (16) is given by $F^{(1)}_{c}\\!=\\!-F^{(1)}_{a}\\!=\\!\\!-6\pi
f\lambda$, where the constant $f\\!\simeq\\!1.701$.

Analogously, at $|\lambda|\ll 1$, the torque applied on the adsorbed particle
can be estimated: the 2nd (analytical) term in (13) is of
${\mathcal{O}}(|\lambda|^{3})$ and the integral term to the leading
approximation contributes $L_{c}\\!\approx\\!-4\pi g\lambda$ [20], where the
constant $g\\!\simeq\\!0.944$ [18]. Given the asymptotic behavior of $F_{c}$
above we readily find that at contact ($h\\!=\\!1$) the torque with respect to
the point of contact to the leading approximation redas
$L^{(c)}_{c}\approx-(6f+4g)\pi\lambda\\!=\\!-13.981\pi\lambda$.

Thus, finding the net excess force $F$ at the lowest non-trivial order demands
the solution of the following problem at $\mathcal{O}(|\lambda|^{2})$:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!0\\!=\\!-\nabla
p_{2}\\!+\\!\nabla^{2}\bm{u}_{2},\ \ \nabla\cdot\bm{u}_{2}\\!=\\!0,$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\bm{u}_{2}(z\\!=\\!0)=0,\
\ \bm{u}_{2}(r\\!=\\!1)=-z^{2}\hat{\bm{x}}/2,$ (17)

The analytical solution of Eqs. (17), that would allow determining the
subleading corrections to $F_{a}^{(2)}$ and $F_{c}^{(2)}$, is possible
following the analysis in [18], and shall be performed elsewhere. However,
since at this order the correction $\propto\\!\\!\lambda^{2}$ is purely
imaginary, the subleading contribution to the real part of $F$ is limited to
${\mathcal{O}}(|\lambda|^{3})$, implying that for small particles
$\Delta\Gamma$ due to hydrodynamics is expected to be smaller than $\Delta f$
(see Figs. 3c and 4a below).

Numerical computations. The numerical solution of Eqs. (4) is performed in the
dimensionless cylindrical coordinates $\\{\varrho,\phi,z\\}$ (all distances
scaled with $a$), such that $x\\!=\\!\varrho\cos\phi$,
$y\\!=\\!\varrho\sin\phi$, with its origin at the plate at $z=0$ and the
$z$-axis passing through the center of the adsorbed spherical particle.

Figure 2: The perturbed flow (streamlines) and pressure (color map) fields in
Eq. (2) due to an adsorbed particle for $\delta/a\\!=\\!1$ in $xz$-plane (for
$\phi\\!=\\!0$) at two different time instances: (a) velocity
$\\{\mathrm{Re}[\mathcal{U}],\mathrm{Re}[\mathcal{W}]\\}$ and pressure
$\mathrm{Re}[\mathcal{P}]$ corresponding to $\omega t=0$; (b) velocity
$\\{\mathrm{Im}[\mathcal{U}],\mathrm{Im}[\mathcal{W}]\\}$ and
$\mathrm{Im}[\mathcal{P}]$ corresponding to $\omega t=\pi/2$.

We use the following ansatz admitting simple dependence on the azimuthal
angle: $v_{\varrho}\\!=\\!{\mathcal{U}}(\varrho,z)\cos{\phi}$,
$v_{\phi}\\!=\\!{\mathcal{V}}(\varrho,z)\sin{\phi}$,
$v_{z}\\!=\\!{\mathcal{W}}(\varrho,z)\cos{\phi}$ and
$p={\mathcal{P}}(\varrho,z)\cos{\phi}$ which reduces the solution to two
dimensions [22, 16]. The corresponding problem for $\mathcal{U}$,
$\mathcal{V}$, $\mathcal{W}$ and $\mathcal{P}$ is defined in the rectangular
domain $0\leq\varrho\leq\varrho_{m},\;0\leq z\leq z_{m}$ with an exclusion of
the half unit disk centered at $(0,h)$ representing the adsorbed particle. The
pressure $\mathcal{P}$ is set to a fixed (zero) value far from the particle at
$z\\!=\\!z_{\mathrm{max}},\ \varrho\\!=\\!\varrho_{\mathrm{max}}$. The
boundary condition $\bm{u}\\!=\\!0$ is applied at
$\varrho\\!=\\!\varrho_{\mathrm{max}}$, $z\\!=\\!0$ and
$z\\!=\\!z_{\mathrm{max}}$. We set no-flux boundary condition at
$\varrho\\!=\\!0$, while at the boundary of half-circle we specify
$\mathcal{U}\\!=\\!-\mathcal{V}\\!=\\!1-\mathrm{e}^{-\lambda z}$ and
$\mathcal{W}\\!=\\!0$. We then apply the Finite Element Method (FEM)
implemented in Mathematica 12.0 to solve the Eqs. (4). A typical mesh size is
selected to be $0.05$ within the domain and $0.025$ along the boundaries.
Notice that for stiff contact, the particle is oscillating in-sync with the
resonator and there is no relative shearing (or sliding) motion between the
two. In Ref. [16] the fluid-mediated part of the excess shear force ($F_{a}$)
for an adsorbed particle was determined via the numerical solution of the
auxiliary problem corresponding to a stationary (heavy inertial) particle
located above the resonator, and this resulted in numerical difficulties at
close proximity owing to strong lubrication forces. The direct formulation of
the problem in Eqs. 4 circumvents these complications, allowing for accurate
numerical solution near contact, $h\\!\rightarrow\\!1$.

Numerical computation shows that the flow $\bm{u}$ converges at
$\varrho_{\mathrm{max}}\\!\simeq\\!9$, $z_{\mathrm{max}}\\!\simeq\\!9+h$. The
typical flow and pressure disturbance due to an adsorbed particle for
$\delta\\!=\\!1$ and $h\\!=\\!1.001$ in meridional plane $xz$–plane (for
$\phi\\!=\\!0$) are shown in Figs. 2a,b at two instances, $\omega t\\!=\\!0$
and $\omega t\\!=\\!\pi/2$, respectively. It can be readily seen, that the
interaction of the transverse wave originated at the oscillating plate (see
the undisturbed velocity in Fig. 1) with the particle creates a rather complex
flow pattern with transient recirculations.

Results and discussion The numerical results for the real and imaginary part
of the excess shear force due to an adsorbed particle at contact ($h\\!=\\!a$)
are presented in Figs. 3a-d (solid curves).

|
---|---
|
Figure 3: Excess shear force (real and imaginary part) due to adsorbed
particle ($h\\!=\\!a$) vs. $a/\delta$. a) Fluid-mediated contribution $F_{a}$:
the solid (black) lines stand for the numerical results, short-dashed (gray)
lines for the small-$\lambda$ asymptote, $F^{(1)}_{a}$ and long-dashed (red)
lines correspond to the distant-particle prediction $F_{a}^{\mathrm{asym}}$ at
$h\\!=\\!a$ in Eq. (18); the blue curves stand to the analytic part (last two
terms) of $F_{a}$ in Eq. (8); b) Hydrodynamic part of the contact force
$F^{\prime}_{c}$ (black, gray); the short dashed lines for the small-$\lambda$
asymptote $F^{(1)}_{c}$ and long-dashed (blue) line for imaginary part of the
net contact force, $\mathrm{Im}[F_{c}]$, (the real part unchanged) for
neutrally buoyant particle with $\xi=4\pi/3$; c) Various components of the
excess force for $a/\delta\\!\lesssim\\!0.5$: $F_{c}$ (blue, for
$\xi=4\pi/3$), $F_{a}$ (red) and $F$ (black); solid and long-dashed lines
stand for real and imaginary parts of different terms, respectively. d)
Comparison of the net excess force $F^{\prime}$ (without solid inertia) vs.
the analytical result $F_{0}$ [6] for a sphere oscillating in an unbounded
liquid (long-dashed lines); short-dashed (blue) curve stands for
$\mathrm{Re}[F_{0}]$ upon subtraction of the pseudo-Stokes drag $6\pi$.

The fluid-mediated contribution $F_{a}$ in Eq. (8) is depicted in Fig. 3a vs.
$a/\delta$ together with the linear small-$\lambda$ asymptotes $F^{(1)}_{a}$
(short-dashed lines) and the prediction of the the distant-particle theory
(long-dashed, red curves) that assumes $h\\!\gg\\!\mathrm{max}(a,\delta)$,
while the ratio $a/\delta$ is not constrained [16]:

$\displaystyle F_{a}^{\mathrm{asym}}$ $\displaystyle=$ $\displaystyle
6\pi\mathrm{e}^{\lambda(1-h)}-\frac{\pi^{2}\mathrm{e}^{-2\lambda
h}}{\lambda}\times$ (18)
$\displaystyle\left[\frac{3(\mathrm{e}^{2\lambda}-1)}{\pi}+\sum_{l=1}^{\infty}\frac{4(l\\!+\\!1)I_{l+1/2}(\lambda)}{K_{l+1/2}(\lambda)}\right]\,.$

Here $I_{\nu}(\lambda)$ and $K_{\nu}(\lambda)$ are the modified Bessel
functions of the 1st and 2nd kind, respectively. It can be readily seen that
the numerical results show an excellent agreement with $F^{(1)}_{a}$ at low
values of $a/\delta$. The agreement with the theoretical prediction in Eq.
(18) is only qualitative. Recall that starting from relatively small
separations, $h\gtrsim 1.5a$, a surprisingly close agreement between the
numerical results and Eq. (18) was found [16], while at contact ($h\\!=\\!a$)
the theory considerably underestimates the fluid-mediated contribution,
$F_{a}$ (i.e., both the real and the imaginary parts, see red long-dashed
curves in Fig. 3a). Another observation is that the relative weight of the
analytical (the last two) terms in Eq. (8) to $F_{a}$ is small for all values
of $a/\delta$ (see the blue curves in Fig. 3a).

Notice that $\mathrm{Re}[F_{a}]>0$ while $\mathrm{Im}[F_{a}]<0$, which implies
positive frequency shift (which is typically associated with non-hydrodynamic
effects, such as contact viscoelasticity), and $\Delta\Gamma\\!<\\!0$,
indicating _reduced dissipation_. The reason for seemingly unorthodox result,
is that the adsorbed particle excludes a fluid volume above the resonator and
in the same time shields the resonator from the shear wave that would
otherwise persist in its absence. One might expect, that adding the contact
force would flip the signs of the net excess force (see below).

The numerical results for the hydrodynamic part of the contact force
(excluding solid inertia), $F^{\prime}_{c}$ [the sum of the first two terms in
Eq. (10)], are depicted in Fig. 3b vs. $a/\delta$ (solid curves). The linear
small-$\lambda$ asymptotes $F^{(1)}_{c}$ (short-dashed lines) approximate
$F_{c}^{\prime}$ very well up to $a/\delta\approx 1$. The long-dashed (blue)
line stands for the net contact force $F_{c}$ in Eq. (10) for neutrally
buoyant particle with $\xi=4\pi/3$. It can be readily seen, that for
$a\gtrsim\delta$ the excess force is dominated by the contact force, as
$F_{c}\gg F_{a}$, while for $a/\delta\lesssim 0.5$, the two terms are
comparable. Moreover, since $F^{(1)}_{a}=-F^{(1)}_{c}$, their contributions
compensate each other and the net effect is $\mathcal{O}(|\lambda|^{2})$. This
notion is illustrated in Fig. 3c, where we plot $F_{a}$, $F_{c}$ (for
neutrally buoyant particle, $\xi=4\pi/3$) and the resulting net excess force
$F$ vs. $a/\delta<0.5$. The small-$\lambda$ linear asymptotes are shown as
short dashed lines. The exact cancelation of the fluid-mediated and contact
forces at the leading order in $\lambda$ result in rather low values of $F$
for small particle, in particular its real part of
${\mathcal{O}}(|\lambda|^{3})$, while the imaginary part is of
${\mathcal{O}}(|\lambda|^{2})$ (see the analysis above). For example, for 50
nm ($a/\delta\\!=\\!0.1$) neutrally buoyant particles in water for the
fundamental frequency of $f_{0}\\!=\\!\omega/2\pi\\!=\\!5$ MHz, giving
$\delta\approx 252$ nm), yields ${\mathcal{Z}}/(\eta
a\tilde{n})\\!\approx\\!-0.10+0.78\mathrm{i}$. Using the small-load
approximation [2], the shift in oscillation frequency, $\Delta f$, and in its
half-bandwidth, $\Delta\Gamma$ (related to the dissipation factor,
$\Delta{\mathcal{D}}\\!=\\!2\Delta\Gamma/f$), can be found from $\Delta
f-\mathrm{i}\Delta\Gamma\\!=\\!\mathrm{i}f{\mathcal{Z}}/(\pi\mathcal{Z}_{q})$,
where the quartz resonator’s shear wave impedance
$\mathcal{Z}_{q}\\!=\\!8.8\cdot 10^{6}$ $\mathrm{kg}/\mathrm{m}^{2}\mathrm{s}$
and the oscillation frequency $f\\!=\\!nf_{0}$ where $n\\!=\\!1,3,5,\dots$ is
the overtone number. Assuming the particle number density at the surface of
the resonator $\tilde{n}\\!=\\!0.01a^{-2}$ (i.e., one nanoparticle per
$100a^{2}$ surface area), the small-load approximation at the fundamental
frequency $f_{0}$ yields $\Delta f\\!\approx\\!-56.0$ Hz and
$\Delta\Gamma\\!\approx\\!7.4$ Hz only.

In Fig. 3d we compare the hydrodynamic part of the net excess shear force,
$F^{\prime}$ (excluding the solid inertia term, solid curves) vs. the
classical result for the force exerted on an rigid sphere oscillating with
velocity $\bm{u}_{0}=v_{0}\hat{\bm{x}}\mathrm{e}^{-\mathrm{i}\omega t}$ in an
_unbounded_ viscous liquid, quiescent at infinity (long-dashed lines). This
force can be written in the dimensionless form (scaled with $\eta av_{0}$) as
(see, e.g., [6]):

$F_{0}=-6\pi\left(1+\frac{a}{\delta}\right)+6\pi\mathrm{i}\left(\frac{a}{\delta}\right)\left(1+\frac{2}{9}\frac{a}{\delta}\right)\,.$
(19)

It was previously proposed [13], that for large enough particles
($a\\!\gg\\!\delta$), the hydrodynamic contribution to the impedance can be
closely approximated by $F_{0}$, as most of the particle surface oscillates in
almost quiescent liquid located above the penetration depth $\delta$. It can
be seen that the agreement between numerical result for
$\mathrm{Im}[F^{\prime}]$ (dashed line) and the 2nd (“added mass”) term in Eq.
(19) is quite close and the relative error (which increases with $a/\delta$)
is $\sim\\!16$ % for $a/\delta\\!=\\!4$. For the same value of $a/\delta$, the
real part, $\mathrm{Re}[F^{\prime}]$ deviates from the 1st (“drag”) term in
Eq. (19) by $\sim\\!22$%, while this error becomes larger for smaller
particles, e.g., it is already $\sim\\!68$% for $a/\delta\\!=\\!1$. It appears
that subtracting the zero-frequency pseudo-Stokes drag term $-6\pi$ from
$\mathrm{Re}[F_{0}]$ yields much closer agreement (see the short-dashed line
in Fig. 3d), in particular for large values of $a/\delta$. For instance, for
$a/\delta\\!=\\!4$ the error is only $\sim\\!2.7$%, while for
$a/\delta\\!=\\!1$ the error is $\sim\\!35$%.

The dimensionless frequency shift, $-\Delta f/(f\alpha)$ and half bandwidth
shift, $\Delta\Gamma/(f\alpha)$ vs. $a/\delta$ for neutrally buoyant particles
(i.e., $\rho_{s}/\rho\\!=\\!1$) particles are shown as log-log plot in Fig. 4a
(see the black solid and dashed curves). Here $\alpha\\!=\\!\eta
a{\tilde{n}}/\mathcal{Z}_{q}$ is the dimensionless (viscous-to-solid)
impedance ratio. For example, for $50$ nm-diameter particles in water and
particle surface density $\tilde{n}\\!=\\!0.01a^{-2}$, we find that
$\alpha\\!=\\!4.55\cdot 10^{-5}$. Both shifts are monotonically increasing
functions of $a/\delta$, while for small values of $a/\delta$ we have $-\Delta
f\\!\propto\\!(a/\delta)^{2}$ and $\Delta\Gamma\\!\propto\\!(a/\delta)^{3}$.
The scaled frequency shift due to the Sauerbrey equation, $-\Delta
f_{S}/(\alpha f)\\!=\\!\frac{8}{3}(\rho_{s}/\rho)(a/\delta)^{2}$ for neutrally
buoyant particles is depicted for comparison (solid gray line). It can be
readily seen that the Sauerbrey equation significantly _underestimates_ the
mass of finite size adsorbents. The dimensionless acoustic ratio,
$\Delta\Gamma/(-\Delta f)$ is independent of the surface coverage $\tilde{n}$
(provided it is low enough so that hydrodynamic interaction between distinct
adsorbed particles can be neglected) and oscillation frequency; it is shown
vs. $a/\delta$ in Fig. 4b (solid line) together with some published
experimental results (symbols). Notice that for adsorbed particles with stiff
contact the theory predicts that the acoustic ratio is bounded, e.g., for
neutrally buoyant particles $\Delta\Gamma/(-\Delta f)\lesssim 0.38$. Heavier
particles are expected to yield even smaller values of the acoustic ratio at
the peak, as the inertial (Sauerbrey) term $-\lambda^{2}\xi$ in (10)
contributes to the imaginary part of $F$ therefore increases the (negative)
frequency shift, $(-\Delta f)$, while $\Delta\Gamma$ remains unchanged. For
instance, for silica nanoparticles $\rho_{s}\\!=\\!1.93$ g/cm3 suspended in
ethanol ($\rho\\!=\\!0.79$ g/cm3) [23] we have $\Delta\Gamma/(-\Delta
f)\lesssim 0.28$.

Figure 4: a) The dimensionless frequency shift $-\Delta f/(f\alpha)$ (solid
black curve) and the half-bandwidth $\Delta\Gamma/(f\alpha)$ (dashed curve)
shift due to adsorbed neutrally buoyant ($\rho_{s}/\rho\\!=\\!1$) particles
vs. $a/\delta$ (double-log plot); the red lines designate the asymptotic
behavior at $a/\delta\ll 1$; the solid blue line is the Sauerbrey frequency
shift, $-\Delta f_{S}/(f\alpha)$; b) The dimensionless acoustic ratio
$\Delta\Gamma/(-\Delta f)$ vs. $a/\delta$ for neutrally buoyant (solid line)
and non-buoyant (dashed line) particles with $\rho_{s}/\rho\\!=\\!2.44$ (e..g,
silica nanoparticles in ethanol [23]); empty squares ($\square$) are the
results for $26$ nm and $73$ nm diameter polystyrene nanoparticles at
fundamental frequency[24], circles ($\circ$) are the results for 30 nm CPMV
particles and $86$ nm and $114$ nm liposomes (at the 3rd overtone)[8] and
$\vartriangle$ stand for the 137 nm silica nanoparticles adsorbing on gold
from ethanol (at the 3rd overtone)[23].

Finally, in Fig. 5 the real and imaginary parts of the contact torque
$L_{c}/\eta a^{2}v_{0}$ (with respect to the particle center at $z\\!=\\!h$ in
Eq. 13) is plotted vs. $a/\delta$. The small-$\lambda$ asymptotic (short-
dashed lines) show an excellent agreement with the numerical results (black
solid and gray long-dashed curves).

Figure 5: The contact torque $L_{c}/\eta a^{2}v_{0}$ with respect to the
particle center vs. $a/\delta$. Solid (black) and long-dashed (gray) curves
stand for the numerical results for the real and the imaginary parts of
$L_{c}$; the short-dashed (gray) lines stands for the small-$\lambda$
asymptotics.

Notice that the torque $L_{c}^{(c)}$ with respect to the _point of contact_ in
Eq. (14) would be much higher owing to the large contact force, since
$|aF_{c}^{\prime}|\\!\gg\\!|L_{c}|$.

Conclusions and perspectives. Fluid-mediated contribution to the excess shear
force and the hydrodynamic part of the contact force are in competition for
$a/\delta\lesssim 0.5$, while the net effect of the viscous stresses reduces
to ${\mathcal{O}}(|\lambda|^{2})$ (or to ${\mathcal{O}}(|\lambda|^{3})$ for
the dissipation factor) due to the mutual cancellation of the linear in
$\lambda$ terms in $F_{a}$ and $F_{c}^{\prime}$. Since the Sauerbrey
contribution due to particle solid inertia is also of $O(|\lambda|^{2})$, it
implies that accurate account of hydrodynamics in the analysis of QCM-D
response is equally important.

We have previously shown that in the limit of vanishing proximity,
$\epsilon\\!=\\!h/a-1\\!\to\\!0$, the translation and rotation velocities of a
freely suspended spherical particle to the leading approximation in $\epsilon$
tend to that of the rigidly attached particle, i.e.,
$V-1,\Omega\sim|\ln{\epsilon}|^{-1}$ solely due to the lubrication forces (see
Sec. V and Eq. 90 in [16]). However, despite this fact, the fluid-mediated
contribution to the excess shear force due to freely suspended particle, that
can be written as $F_{f}\\!=\\!F_{a}+(V-1){\mathcal{A}}+\Omega{\mathcal{B}}$,
is different from the corresponding contribution due to an adsorbed particle,
$F_{f}\\!\neq\\!F_{a}$, due to divergence of the corresponding resistance
functions, $\mathcal{A}$ and $\mathcal{B}$ at $\epsilon\\!\to\\!0$. This
argument explains the apparent disagreement between the prediction of the
impedance due to the freely suspended particles _near contact_ and the
adsorbed particle _at contact_. Similarly, arguments concern the contact force
$F_{c}$ in Eq. (9), which is zero for a freely suspended particle and takes a
finite value (i.e., due to the hydrodynamics) for a particle forming a stiff
contact with the resonator.

In the limit of low surface coverage and stiff contact both signals, $-\Delta
f/n$ and $\Delta\Gamma/n$, are expected to increase with the oscillation
frequency (i.e., the overtone number) as can be seen from Fig. 4a. Obviously,
for a fixed $a$, the penetration depth decreases as
$\delta\\!\sim\\!1/\sqrt{f}$, resulting in some increase of $a/\delta$, and
since $-\Delta f/(f\alpha)$ is a monotonically increasing function of
$a/\delta$, some increase in $-\Delta f/n$ is expected theoretically. The same
concerns the shift in the half bandwidth, $\Delta\Gamma/n$. However,
experimental results in [23, 24] suggest that $-\Delta f/n$ decreases with the
overtone number (while $\Delta\Gamma/n$ increases), suggesting nontrivial
contact dynamics, due to, e.g., “elastic loading” and/or particle deformation.
While the opposite trend in the dependence of $-\Delta f/n$ on frequency, the
reasonable agreement for the acoustic ratio $\Delta\Gamma/(-\Delta f)$ vs.
$a/\delta$ in Fig. 4b is only obtained at low frequencies. At
$a/\delta\\!\sim\\!1$ the theoretical prediction of the acoustic ratio due to
low coverage of adsorbed particle with stiff contact reaches maximum, while in
the experiments it grows monotonically upon increasing the oscillation
frequency.

The idea to use QCM-D as a tool for probing the rheological (viscoelastic)
properties of the contact sounds attractive, but perhaps it is not very
practical. The accurate account of hydrodynamics in such case would be
difficult, as, subtle differences in particle mobility produces large
differences in the excess shear force, e.g., notice the difference in
impedance due to freely suspended (near contact, see [16]) and rigidly
attached particles. Moreover, for compliant contact yields the particle’s
motion with respect to the resonator (i.e., rocking or sliding [7]) which is
determined by the interplay of adhesive and hydrodynamic forces, rendering the
accurate quantitative analysis of the QCM-D signal extremely difficult.

Apparently in the experiments the contact elasticity and/or finite particle
deformability affects the signal at higher overtones, suggesting that perhaps
accurate gravimetric measurements are possible at lower frequencies. Since
fundamental frequency of AT cut quartz is inversely proportional to its
thickness, it is theoretically possible to build a device with a thicker
crystal operating at lower resonant frequency. Such modification would only be
useful provided that at lower frequencies the adhesive contact remains stiff.

This work was supported, in part, by the Israel Science Foundation (ISF) via
the grant No. 2899/21. A.M.L. also acknowledges the support of the David T.
Siegel Chair in Fluid Mechanics.

## References

  * [1] D. Johannsmann, _The quartz crystal microbalance in soft matter research_ (Springer, Switzerland, 2015).
  * [2] D. Johannsmann, A. Langhoff and C. Leppin, Studying Soft Interfaces with Shear Waves: Principles and Applications of the Quartz Crystal Microbalance (QCM), Sensors 21, 3490 (2021).
  * [3] G. H. Sauerbrey, Verwendung von Schwingquarzen zur Wägung dünner Schichten und zur Mikrowägung, Z. Phys. 155, 206 (1959).
  * [4] T. Nomura and A. Minemura, Behavior of a Piezoelectric Quartz Crystal in an Aqueous Solution and the Application to the Determination of Minute Amount of Cyanide, Nippon Kagaku Kaishi 10, 1621 (1980).
  * [5] K. K. Kanazawa and J. G. Gordon, Frequency of a quartz microbalance in contact with liquid, Anal. Chem. 57, 1770 (1985)
  * [6] L. D. Landau and E. M. Lifshitz, _Fluid Mechanics_ , 3rd ed. (Pergamon Press, Oxford, 1976).
  * [7] D. Johannsmann, I. Reviakine, R. P. Richter, Dissipation in films of adsorbed nanospheres studied by quartz crystal microbalance (QCM). Anal Chem. 81, 8167 (2009).
  * [8] E. Tellechea, D. Johannsmann, N. F. Steinmetz, R. P. Richter and I. Reviakine, Model-independent analysis of QCM data on colloidal particle adsorption, Langmuir 25, 5177 (2009).
  * [9] J. J. J. Gillissen, J. A. Jackman, S. R. Tabaei, B. K. Yoon, and N.-J. Cho, Quartz Crystal Microbalance Model for Quantitatively Probing the Deformation of Adsorbed Particles at Low Surface Coverage, Anal. Chem. 89 11711 (2017)
  * [10] J. J. J. Gillissen, J. A. Jackman, S. R. Tabaei, and N.-J. Cho, A numerical study on the effect of particle surface coverage on the Quartz Crystal Microbalance Response, Anal. Chem. 90, 2238 (2018).
  * [11] S. Gopalakrishna, A. Langhoff, G. Brenner and D. Johannsmann, Soft Viscoelastic Particles in Contact with a Quartz Crystal Microbalance (QCM): A Frequency-Domain Lattice Boltzmann Simulation. Anal Chem. 93, 10229 (2021).
  * [12] A. Vázquez-Quesada, M. Meléndez-Schofield, A. Tsortos, P. Mateos-Gil, D. Milioni, E. Gizeli, and R. Delgado-Buscalioni, Hydrodynamics of Quartz-Crystal-Microbalance DNA Sensors Based on Liposome Amplifiers, Phys. Rev. Applied 13, 64059 (2020).
  * [13] A. Tarnapolsky and V. Freger, Modelling QCM-D response to deposition and attachment of microparticles and living cells, Anal. Chem. 90, 13960 (2018).
  * [14] I. Fouxon and A. Leshansky, Fundamental solution of unsteady Stokes equations and force on an oscillating sphere near a wall, Phys. Rev. E 98, 063108 (2018).
  * [15] M. M. Schofield and R. Delgado-Buscalioni, Quantitative description of the response of finite size adsorbates on quartz crystal microbalance in liquids using analytical hydrodynamics. Soft Matter 17, 8160 (2021).
  * [16] I. Fouxon, B. Rubinstein, and A. Leshansky, Excess shear force exerted on an oscillating plate due to a nearby particle, Phys. Rev. Fluids 8, 054104 (2022).
  * [17] S. Kim and S. J. Karrila, Microhydrodynamics: principles and selected applications, (Butterworth–Heinemann, Boston, 1991).
  * [18] M. E. O’Neill, A sphere in contact with a plane wall in a slow linear shear flow, Chem. Eng. Sci. 23, 1293 (1968).
  * [19] This is different in the unbounded fluid, where $\mathcal{O}(1)$ velocity at the particle surface produces linear in $\lambda$ terms in the solution of Eqs. 2. However, in the semi-infinite domain bounded by the no-slip wall at $z=0$, the unsteady term produdes terms of lower order, see the detailed discussion in [14].
  * [20] Notice that in [18] the torque is showing as $L_{y}\\!=\\!-8\pi\mu ua^{2}g$, while it should be $-4\pi\mu ua^{2}g$; the correct coefficient ($-4\pi$) is provided in Ref. [21].
  * [21] A. J. Goldman, R. G. Cox and H. Brenner, Slow viscous motion of a sphere parallel to a plane wall – II Couette flow, Chem. Eng. Sci., 22, 653 (1967).
  * [22] I. Fouxon, B. Rubinstein, O. Weinstein and A. Leshansky, Fluid-Mediated Force on a Particle Due to an Oscillating Plate and Its Effect on Deposition Measurements by a Quartz Crystal Microbalance, Phys. Rev. Lett. 125, 144501 (2020).
  * [23] C. Grunewald, M. Schmudde, C. N. Noufele, C. Graf, and T. Risse, Ordered Structures of Functionalized Silica Nanoparticles on Gold Surfaces: Correlation of Quartz Crystal Microbalance with Structural Characterization. Anal Chem. 87, 10642 (2015).
  * [24] Z. Adamczyk, and M. Sadowska, Hydrodynamic Solvent Coupling Effects in Quartz Crystal Microbalance Measurements of Nanoparticle Deposition Kinetics, Anal. Chem. 92, 3896 (2020).


# Text to Point Cloud Localization with Relation-Enhanced Transformer

Guangzhi Wang1, Hehe Fan2, Mohan Kankanhalli2

###### Abstract

Automatically localizing a position based on a few natural language
instructions is essential for future robots to communicate and collaborate
with humans. To approach this goal, we focus on the text-to-point-cloud cross-
modal localization problem. Given a textual query, it aims to identify the
described location from city-scale point clouds. The task involves two
challenges. 1) In city-scale point clouds, similar ambient instances may exist
in several locations. Searching each location in a huge point cloud with only
instances as guidance may lead to less discriminative signals and incorrect
results. 2) In textual descriptions, the hints are provided separately. In
this case, the relations among those hints are not explicitly described,
leading to the difficulties of learning relations. To overcome these two
challenges, we propose a unified Relation-Enhanced Transformer (RET) to
improve representation discriminability for both point cloud and natural
language queries. The core of the proposed RET is a novel Relation-enhanced
Self-Attention (RSA) mechanism, which explicitly encodes instance (hint)-wise
relations for the two modalities. Moreover, we propose a fine-grained cross-
modal matching method to further refine the location predictions in a
subsequent instance-hint matching stage. Experimental results on the
KITTI360Pose dataset demonstrate that our approach surpasses the previous
state-of-the-art method by large margins.

## Introduction

Understanding natural language instructions in the 3D real world is a
fundamental skill for future artificial intelligence assistants to collaborate
with humans. In this paper, we focus on the outdoor environment and study the
task of natural language-based localization from city-scale point clouds. As
shown in Figure 1, given a linguistic description of a position, which
contains several hints, the goal of the task is to find out the target
location from a large-scale point cloud. This task can effectively help mobile
robots, such as self-driving cars and autonomous drones, cooperate with humans
to coordinate actions and plan their trajectories. By understanding the
destination from natural language instructions, it reduces the human effort
required for manual operation.

Figure 1: Illustration of the text to point cloud localization task. Given a
textual query, which usually contains several independent hints, the goal is
to localize the point of interest in a huge city-scale point cloud.

However, this task is intrinsically challenging. Precise localization requires
both correct language interpretation and effective large-scale point cloud
understanding. Considering the difficulties, an existing method (Kolmet et al.
2022) first divides a city-wide point cloud into several cells, and then
solves this task in a Coarse-to-Fine manner.

The goal of the ‘coarse’ stage is to find out the target cell that contains
the queried location according to the given natural language descriptions. In
this stage, the instances included in point cloud cells and those mentioned in
language descriptions are mainly used for text-to-point-cloud retrieval based
on their types, without considering their relations. In the ‘fine’ stage, each
object in the textual query is matched with an in-cell point cloud instance,
whereby a target location will be predicted from each hint. This pioneering
method sets up a significant starting point for tackling the challenging task.
However, it fails to consider the intrinsic relations in both stages,
resulting in sub-optimal performance.

For the coarse stage, because similar ambient instances may exist in several
cells, performing retrieval based on only the cell-contained and query-related
instance types without considering their relations may lead to low
discriminability for both cell and query representations, which inevitably
leads to ambiguity. Based on those low-discriminability representations, it is
difficult to find out the correct cell. In the fine stage, we observe that
insufficient cross-modal collaboration leads to difficulties in location
refinement. Given the retrieved cell, precise location prediction requires
joint understanding of both point clouds and textual queries. However, in the
previous method (Kolmet et al. 2022), the cross-modal collaboration is only
performed from textual queries to point clouds in a single step, which results
in optimization difficulty for multi-task learning.

In this work, we aim to solve the aforementioned shortcomings in both stages.
For the coarse stage, we propose to encode pairwise instance relations to
improve representation discriminability for both modalities, which is achieved
through a novel Relation-Enhanced Transformer (RET) architecture. In
particular, the in-cell point cloud instance relations are modeled as their
geometric displacements, while computed as the fusion of hint representations
in the linguistic domain. These relations from two modalities are respectively
incorporated into their representation in a unified manner, which is achieved
through the proposed Relation-enhanced Self-Attention (RSA) mechanism. For the
fine stage, we perform Cascaded Matching and Refinement (CMR) to enhance
cross-modal collaboration. In particular, different from (Kolmet et al. 2022)
which achieves this objective in a single step, we perform description-
instance matching and position refinement in two sequential steps. Such
formulation allows us to minimize the optimization difficulty of multi-
objective learning and noisy intermediate results, thereby improving cross-
modal collaboration.

We validated the effectiveness of our method on the KITTI360Pose benchmark
(Kolmet et al. 2022). Extensive experiments demonstrate that the proposed
method can surpass the previous approach by a large margin, leading to new
state-of-the-art results. Our contributions are three-fold:

  * •

We propose a novel Relation-Enhanced Transformer (RET) to improve
representation discriminability for both point clouds and textual queries. The
core component of RET is the Relation-enhanced Self-Attention (RSA) mechanism,
which encodes instance (hint) relations for the two modalities in a unified
manner.

  * •

We propose to perform cross-modal instance matching and position refinement in
two sequential steps. This formulation allows us to minimize the optimization
difficulty of multi-task learning and the influence of noisy intermediate
results, thereby improving cross-modal collaboration for fine-grained location
prediction.

  * •

We perform extensive experiments on the KITTI360Pose dataset (Kolmet et al.
2022). The results show that our approach can surpass previous method by a
large margin, resulting in new state-of-the-art performance. Additional
ablation studies further demonstrate the effectiveness of each component in
the proposed method.

## Related Work

Transformer and Attention Mechanism. Transformer and self-attention mechanism
(Vaswani et al. 2017; Fan, Yang, and Kankanhalli 2021) has become increasingly
popular in recent years. Although first proposed for natural language
processing, with architectural adaptation, Transformer has been widely applied
to many vision tasks including visual recognition (Dosovitskiy et al. 2020;
Liu et al. 2021), object detection (Carion et al. 2020; Zhu et al. 2020) and
semantic segmentation (Cheng, Schwing, and Kirillov 2021). Besides, the
transformer-based architectures are also utilized to model cross-modal (e.g.,
vision and language) relations (Tan and Bansal 2019; Lu et al. 2019; Li et al.
2019; Zhang et al. 2021; Li et al. 2022). In these architectures, the
attention mechanism is widely employed to implicitly learn relations among the
input tokens. Nevertheless, without explicit relation encoding, the vanilla
Transformer can only encode relations implicitly with the help of positional
encoding (Dosovitskiy et al. 2020). To facilitate better relation modeling,
some works modulate the attention computation process by explicitly
incorporating element relations. For example, (Wu et al. 2021) modified the
attention mechanism via unified relative position bias to improve visual
recognition. For object detection, spatial relations between bounding boxes
are introduced to modulate the attention weights (Liu et al. 2022; Gao et al.
2021). For dynamic point cloud analysis, displacement between points (Fan,
Yang, and Kankanhalli 2022) is utilized for point-specific attention
computation. In this work, we propose to model relations for both point clouds
and language queries by explicitly incorporating intra-modality relations in a
unified manner.

Visual Localization. The task that is most related to ours is vision-based
localization (Arandjelovic et al. 2016; Brachmann et al. 2017; Hausler et al.
2021), which is to estimate a pose based on an image or image sequence.
Existing methods mostly solve this task in two stages (Sarlin et al. 2019;
Sattler, Leibe, and Kobbelt 2016; Zhou et al. 2020). The first stage finds a
subset of all images using image retrieval-based techniques (Arandjelovic et
al. 2016; Hausler et al. 2021; Torii et al. 2015), while the second stage
establishes pixel-wise correspondence between the query image and the
retrieved one to predict the precise pose. In this work, we also study the
task of localization in a coarse-to-fine manner, but differ from visual
localization in that: 1) we try to infer the location from city-wide point
clouds instead of images. 2) we try to estimate the pose from textual query
rather than images. Compared to visual localization, our task requires multi-
modal understanding and is more challenging to solve.

Figure 2: Framework of the proposed method. The city-scale point cloud is
first divided into individual cells. Then, in the coarse stage, the cells and
the textual query are respectively encoded with the proposed Relation-Enhanced
Transformer (RET), which are later used for query-cell matching. In the fine
stage, each hint is matched with an in-cell instance. Then, cross-modal fusion
dynamically aggregates hints and instance representations for offset
prediction. The target location is predicted based on matching results and
offset predictions.

3D Language Grounding. As we humans live in a 3D world and communicate through
natural language, recent work has begun to investigate the tasks on the cross-
modal understanding of 3D vision and natural language. Among these tasks, the
one that is most related to ours is 3D language grounding, which aims at
localizing an object in point clouds from a given natural language query. For
example, ScanRefer (Chen, Chang, and Nießner 2020) studies 3D language
grounding from real-life in-door scenes. ReferIt3D (Achlioptas et al. 2020)
studies a related task under a simpler setting, which assumes the object
instances are segmented in advance. InstanceRefer (Yuan et al. 2021) improves
previous methods by adopting a 3D panoptic segmentation backbone, utilizing
multi-level visual context. Recently, graph structure (Feng et al. 2021) is
also utilized to improve the representation learning qualities.

## Methodology

### Preliminaries

Given a textual query, our goal is to identify the position it describes from
a city-scale point cloud. To handle the large-scale point cloud, we divide
each scene into a set of cubic cells of fixed size by a preset stride. Each
cell $\mathcal{C}$ contains a set of $p$ point cloud instances, which are
encoded by PointNet++ (Qi et al. 2017) into vector representations
$\\{{\boldsymbol{p}}_{i}\\}_{i=1}^{p}$. Following (Kolmet et al. 2022), the
textual query $\mathcal{T}$ is represented as a set of hints
$\\{{\boldsymbol{h}}_{j}\\}_{j=1}^{h}$, each encoding the direction relation
between the target location and an instance.

Inspired by the existing work (Kolmet et al. 2022), given the cell splits, we
solve this task in a coarse-to-fine manner with two stages. The coarse stage
is formulated as textual query based cell retrieval. The goal of this stage is
to train a model that encodes $\mathcal{C}$ and $\mathcal{T}$ into a joint
embedding space whereby matched query-cell pairs are close while those
unmatched are pulled apart (Kiros, Salakhutdinov, and Zemel 2014). In the fine
stage, given a retrieved cell, we aim to refine the position prediction by
utilizing fine-grained cross-modal information. In particular, we first match
each hint in the query with an in-cell instance by formulating it as an
optimal transport problem (Liu et al. 2020). After that, with the matching
results, we predict the target location through a cross-modal fusion of point
cloud instance and hint representations. Based on the fused representation, we
predict the target location for each matched instance. Finally, we obtain the
target location prediction based on a weighted combination of the matching and
location prediction results. The framework of our method is shown in Figure 2.
In the following of this section, we will explain the proposed method for
coarse stage and fine stage. After that, our training and inference procedure
will be detailed.

### Coarse Stage: Relation-Enhanced Transformer

After the cell split, the goal of the coarse stage is to successfully retrieve
the cell $\mathcal{C}$ given a textual query $\mathcal{T}$. To approach this
objective, we need to encode $\mathcal{C}$ and $\mathcal{T}$ into a joint
embedding space. An intuitive solution is to encode both $\mathcal{C}$ and
$\mathcal{T}$ based on the instances they contained as is done in (Kolmet et
al. 2022). However, with such representations, the low discriminability for
cells and textual queries results in poor retrieval performance. We argue that
this can be attributed to the following two reasons. On the one hand, the
outdoor scenes are often of low diversity, whereby a group of mentioned
instances can appear at multiple different locations. Thus, simply describing
a cell with its contained instances can result in less discriminative
representations. On the other hand, the textual queries often contain limited
clues compared to the point clouds, making this cross-modality retrieval
especially challenging. To this end, we propose to explicitly encode instance-
relations to provide more discriminative representations for both modalities.

Figure 3: Illustration of the proposed Relation-enhanced Self-Attention (RSA)
mechanism. Pairwise relations are explicitly encoded into the value
computation process.

The Transformer (Vaswani et al. 2017) has been widely utilized for relation-
based representation learning in various tasks (Hu et al. 2018; Liu et al.
2021; Fan, Yang, and Kankanhalli 2022). The key component of the Transformer
is the Self-Attention (SA) operation:

$\texttt{Attn}({\boldsymbol{Q}},{\boldsymbol{K}},{\boldsymbol{V}})=\texttt{Softmax}({\boldsymbol{Q}}{\boldsymbol{K}}^{T}/\sqrt{d}){\boldsymbol{V}},$
(1)

where $d$ is the representation dimension and
${\boldsymbol{Q}},{\boldsymbol{K}},{\boldsymbol{V}}\in\mathbb{R}^{N\times d}$
are the query, key and value matrices by transforming in-cell instances (or
hints for textual queries) with corresponding linear transformations:

${\boldsymbol{Q}}={\boldsymbol{W}}^{Q}{\boldsymbol{X}},{\boldsymbol{K}}={\boldsymbol{W}}^{K}{\boldsymbol{X}},{\boldsymbol{V}}={\boldsymbol{W}}^{V}{\boldsymbol{X}},$
(2)

with ${\boldsymbol{W}}^{*}\in\mathbb{R}^{d\times d}$ are learnable matrices
and ${\boldsymbol{X}}={\boldsymbol{P}}\in\mathbb{R}^{p\times d}$ or
${\boldsymbol{H}}\in\mathbb{R}^{h\times d}$ represents stacked
instances111Note that the attention operation is often performed in different
subspaces with multiple heads, which is omitted for simplicity..

Despite its generality, the vanilla SA lacks explicit relations in both
modalities, thus is less informative to represent the cell and query. To this
end, we propose a novel Relation-Enhanced Transformer (RET) to model explicit
instance relations in both point clouds and textual descriptions. Our RET is a
stack of multiple Transformer encoder layers, except that, in place of SA, we
propose a Relation-enhanced Self-Attention (RSA) to explicitly incorporate
relation information into value computation. The computation process is shown
as follows and illustrated in Figure 3.

$\texttt{RSA}({\boldsymbol{Q}},{\boldsymbol{K}},{\boldsymbol{V}},{\boldsymbol{R}})=\\\
\texttt{Softmax}({\boldsymbol{Q}}{\boldsymbol{K}}^{T}/\sqrt{d})({\boldsymbol{V}}+\texttt{Pool}({\boldsymbol{R}},1)),$
(3)

where ${\boldsymbol{R}}\in\mathbb{R}^{N\times N\times d}$ captures pairwise
relations with ${\boldsymbol{R}}_{ij}\in\mathbb{R}^{d}$ representing the
relation between the $i$-th and $j$-th instance (hint).
$\texttt{Pool}({\boldsymbol{R}},1)$ indicates pooling tensor
${\boldsymbol{R}}$ along dimension $1$. In this way, our model can explicitly
encode instance relations through this computation process, leading to more
informative representations.

The definition of relation varies flexibly with task objective and input
modality. For point cloud data, we take the geometric displacement of two
instances as their relations, as direction is often mentioned in textual
queries and thus informative for retrieval:222We have also tried other
features such as number of points and bounding boxes of instances but didn’t
observe performance improvement.

${\boldsymbol{R}}_{ij}^{V}={\boldsymbol{W}}^{V}({\boldsymbol{c}}_{i}-{\boldsymbol{c}}_{j}),$
(4)

where ${\boldsymbol{c}}_{i}\in\mathbb{R}^{3}$ represents the center coordinate
of the $i$-th instance and ${\boldsymbol{W}}^{v}\in\mathbb{R}^{d\times 3}$
transforms the displacement into embedding space. For the linguistic
description, we compute the hint relation as the concatenation of their
embeddings:

${\boldsymbol{R}}^{L}_{ij}={\boldsymbol{W}}^{L}[{\boldsymbol{h}}_{i};{\boldsymbol{h}}_{j}],$
(5)

where ${\boldsymbol{W}}^{L}\in\mathbb{R}^{d\times 2d}$ transforms the
linguistic feature into representation space. With the computation of RSA, the
instance-wise relations for different modalities can be uniformly incorporated
into query or cell representations

Finally, the cell (description) representations $\mathcal{C}_{m}$
($\mathcal{T}_{m}$) are obtained via a pooling operation over all instances
(hints) output from the RET for cross-modal retrieval.

Table 1: Performance comparison on the KITTI360Pose. Method | Localization Recall ($\epsilon<5/10/15m$) $\uparrow$
---|---
Validation Set | Test Set
$k=1$ | $k=5$ | $k=10$ | $k=1$ | $k=5$ | $k=10$
Text2Pos (Kolmet et al. 2022) | 0.14/0.25/0.31 | 0.36/0.55/0.61 | 0.48/0.68/0.74 | 0.13/0.21/0.25 | 0.33/0.48/0.52 | 0.43/0.61/0.65
RET (Ours) |  0.19/0.30/0.37 |  0.44/0.62/0.67 |  0.52/0.72/0.78 |  0.16/0.25/0.29 |  0.35/0.51/0.56 |  0.46/0.65/0.71

### Fine Stage: Cascaded Matching and Refinement

Following the coarse stage, we aim to refine the location prediction within
the retrieved cell in the fine stage. Inspired by (Kolmet et al. 2022), we
perform instance matching and location refinement to utilize the fine-grained
visual and linguistic information, which involves the following two
objectives: (1) For each hint, we find the in-cell instance it refers to via a
matching process. (2) For each matched pair $(i,j)$, a regressor predicts an
offset ${\boldsymbol{\hat{t}}}_{i}\in\mathbb{R}^{2}$ for each matched hint
${\boldsymbol{h}}_{j}$, which represents the offset from the instance center
${\boldsymbol{c}}_{i}$ to the target location.333For position prediction, we
ignore the height information and considers 2D coordinates only.

Previous method (Kolmet et al. 2022) achieves the two objectives within a
single step. However, given the objective of both hint-instance matching and
offset prediction, the multi-task learning process introduces optimization
difficulty. Furthermore, in the early training steps, the matcher is only
partially trained, which produces noisy matching results. The regressor learns
and makes predictions based on this noisy results, leading to unstable
learning process and sub-optimal performance.

To this end, we propose a Cascaded Matching and Refinement (CMR) strategy for
the fine stage, where hint-instance matching and offset regression are
sequentially performed. Specifically, following (Kolmet et al. 2022), we first
train the SuperGlue (Sarlin et al. 2020) matcher for hint-instance matching,
which is formulated as an optimal-transport problem. Given the trained
matcher, we obtain a set of hint-instance matching results
$\\{{\boldsymbol{p}}_{i},{\boldsymbol{h}}_{j},w_{i}\\}_{j=1}^{h}$, where
$w_{i}$ represents the confidence of the match. Then, to reduce the noise for
regression, we predict the target location according to matched instances
only.

Precise location prediction requires proper understanding on both point cloud
(what and where the referred instance is, e.g., dark-green terrain) and
language description (what is the relation between the matched instance and
the target location, e.g., east of). For this, we propose to facilitate cross-
modal collaboration via the Cross-Attention (CA) mechanism, which is commonly
used for cross-modality information fusion.

$\texttt{{CA}}({\boldsymbol{H}},{\boldsymbol{P}})=\texttt{Attn}({\boldsymbol{W}}^{Q}{\boldsymbol{H}},{\boldsymbol{W}}^{K}{\boldsymbol{P}},{\boldsymbol{W}}^{V}{\boldsymbol{P}}),$
(6)

where ${\boldsymbol{H}}$, ${\boldsymbol{P}}$ represent hints and instances,
respectively, and ${\boldsymbol{W}}^{*}$ are learnable transformation
matrices. Shortcut connection and layer normalization (Ba, Kiros, and Hinton
2016) follows the cross-attention operation. With these operations, the hint
representation ${\boldsymbol{h}}_{i}$ is accordingly updated to
${\boldsymbol{\tilde{h}}}_{i}$ by dynamically fusing visual information. As
such, the information in the two modalities are joint utilized with the help
of cross-modal collaboration.

Then, we predict the offset (the direction vector from instance center to
target location) from the updated hint:

${\boldsymbol{\hat{t}}}_{i}=\texttt{MLP}({\boldsymbol{\tilde{h}}}_{j}).$ (7)

To utilize the matching results, the final prediction is obtained via a
weighted combination of each hint’s prediction:

${{\boldsymbol{\hat{g}}}}=\sum_{i}\frac{w_{i}}{\sum_{m}w_{m}}({\boldsymbol{c}}_{i}+{\boldsymbol{\hat{t}}}_{i}),$
(8)

where $w_{i}\in[0,1]$ is the confidence score of the match
$({\boldsymbol{p}}_{i},{\boldsymbol{h}}_{j},w_{i})$ and is set to $0$ for non-
matched instances. To filter out noisy matches, we consider only matches with
confidence score greater than 0.2.

### Training and Inference

Training. For the coarse stage, we train the proposed RET for cross-modal
retrieval with pairwise ranking loss (Kiros, Salakhutdinov, and Zemel 2014):

$\begin{split}\mathcal{L}_{coarse}&=\sum_{m=1}^{N_{b}}\sum_{n\neq
m}^{N_{b}}[\alpha-\langle\mathcal{C}_{m},\mathcal{T}_{m}\rangle+\langle\mathcal{C}_{m},\mathcal{T}_{n}\rangle]_{+}\\\
&+\sum_{m=1}^{N_{b}}\sum_{n\neq
m}^{N_{b}}[\alpha-\langle\mathcal{T}_{m},\mathcal{C}_{m}\rangle+\langle\mathcal{T}_{m},\mathcal{C}_{n}\rangle]_{+},\end{split}$
(9)

where $N_{b}$ is the batch size, $\alpha$ is a hyper-parameter to control the
separation strength and $\langle\cdot,\cdot\rangle$ represents inner product
between vectors. This loss function encourages the representation of matched
description-cell pair to be by a margin $\alpha$ closer than those unmatched.
For the fine stage, we employ the loss in (Sarlin et al. 2020) to train the
matcher, while $L_{2}$ loss is applied to train the offset regressor.

Inference. We first encode all cells and queries into a joint embedding space
with the proposed Relation-Enhanced Transformer. Then, for each query
representation, we retrieve top-$k$ cells with highest similarity. For each
retrieved cell, we use the SuperGlue matcher trained in the fine stage to
match each hint with an in-cell instance, which is followed by offset
prediction based on the fused representations. Finally, the position
prediction is given by Eq. 8.

## Experiments

### Dataset and Implementation Details

Dataset Details. We evaluate our method on the recently proposed KITTI360Pose
dataset (Kolmet et al. 2022), which is built upon the KITTI360 dataset (Liao,
Xie, and Geiger 2021) with sampled locations and generated hints. It contains
point clouds of a total of 9 scenes, covering 14,934 positions with a total
area of 15.51$km^{2}$. We follow (Kolmet et al. 2022) to use five scenes for
training, one for validation, and the remaining three for testing. We sample
the cells of size 30m with a stride of 10m. For more details on the dataset
preprocessing, please refer to our supplementary material.

Implementation Details For the coarse stage, we trained the model with AdamW
optimizer (Loshchilov and Hutter 2018) with a learning rate of 2e-4. The
models are trained for a total of 18 epochs while the learning rate is decayed
by 10 at the 9-th epoch. The $\alpha$ is set to 0.35. For the fine stage, we
first train the matcher with a learning rate of 5e-4 for a total of 16 epochs.
Afterwards, we fix the matcher and train the regressor based on the matching
results for 10 epochs with a learning rate of 1e-4. The regressor is
formulated as a 3 layer Multi-Layer Perceptron. Both of the two steps adopt an
Adam (Kingma and Ba 2014) optimizer. The RET has 2 encoder layers for both
point cloud part and linguistic part, each utilizing the Relation-enhanced
Attention (RSA) mechanism with 4 heads and hidden dimension 2048. For the two
stages, we encode each instance in the cell with PointNet++ (Qi et al. 2017)
provided by Text2Pos (Kolmet et al. 2022) for a fair comparison. The hint
representations are obtained by concatenating learned word embeddings. More
details are provided in our appendix.444Code available at:
https://github.com/daoyuan98/text2pos-ret

### Comparison with the State-of-the-art

We compared our method with Text2Pos (Kolmet et al. 2022) on the KITTI360Pose
dataset. Following (Kolmet et al. 2022), we report top-$k$ ($k=1/5/10$) recall
rate of different error ranges $\epsilon<5/10/15m$ for comprehensive
comparison. The results are shown in Table 1. Text2Pos gives a recall of 0.14
when $k=1$ and $\epsilon<5m$. In contrast, our method can significantly
improve the recall rate to 0.19, which amounts to $35.7\%$ relative
improvement upon the baseline. Furthermore, when we relax the localization
error constraints or increase $k$, consistent improvements upon the baseline
can also be observed. For example, with $\epsilon<5m$, our method achieves
top-5 recall rate of $0.44$, which is $8\%$ higher than previous state-of-the-
art. Similar improvements can also be seen on the test set, showing our method
is superior to the baseline method.

### Ablation Studies

In this section, we perform ablation studies for both stages to investigate
the effectiveness of each proposed component in our method. The ablation
studies for coarse stage and fine stage are provided separately for clear
investigation.

Coarse Stage. We study the importance of explicit relation incorporation in
the coarse stage. Since the coarse stage is formulated as a retrieval task, we
use top-1/3/5 recall rate as evaluation metric, whereby the cell that contains
the ground truth location is defined as positive.

Relation Incorporation. We first study the necessity of explicit relation
modeling for both point cloud and textual queries. The results are shown in
Table 2. It can be observed that relation modeling contributes significantly
to successful retrieval. In particular, without any relation incorporation,
the top-5 recall rate is 0.32. With the explicit fusion of linguistic
relation, we observe an increase of 0.05 recall rate under same condition.
Besides, with the incorporation of visual (point cloud instance) relations
only, the top-5 recall rate can be improved by 0.08, indicating explicit
relations in the point clouds play a more important role. Finally, with both
relations, we achieve an improvement of 0.12 at top-5 recall rate upon that
without any relation, showing that both visual and linguistic relations are
necessary and complementary to improve the cell retrieval performance.

Table 2: Ablation study of the Relation-Enhanced Transformer (RET) on KITTI360Pose validation set. ”wo X relation” indicates replacing the proposed RSA with the vanilla Self-Attention in corresponding modality. Method | $k=1\uparrow$ | $k=3\uparrow$ | $k=5\uparrow$
---|---|---|---
w/o both relations | 0.11 | 0.24 | 0.32
w/o linguistic relation | 0.14 | 0.28 | 0.37
w/o visual relation | 0.16 | 0.30 | 0.40
Full (Ours) | 0.18 | 0.34 | 0.44

RET Hyper-parameters. We also studied the importance of the hyper-parameters
involved in RET, namely the number of layers of RET and the number of heads of
RSA. The results are shown in Table 3. It can be observed that, thanks to the
strong relation modeling capacity of the proposed RET, we can obtain the best
performance with 2 layers and 4 heads in the RSA. Decreasing and increasing
the number of layers both lead to worse performance, which may be attributed
to underfitting and overfitting, respectively.

Table 3: The effects of #layers of RET and #heads of RSA. #Layers | #Heads | $k=1\uparrow$ | $k=3\uparrow$ | $k=5\uparrow$
---|---|---|---|---
1 | 4 | 0.16 | 0.31 | 0.40
1 | 8 | 0.16 | 0.30 | 0.40
2 | 2 | 0.17 | 0.32 | 0.42
2 | 4 | 0.18 | 0.34 | 0.44
2 | 8 | 0.16 | 0.31 | 0.40
3 | 4 | 0.16 | 0.32 | 0.39
3 | 8 | 0.15 | 0.29 | 0.37

Fine Stage. The objective of the fine stage is to correctly match linguistic
hints and point cloud instances and regress the target location. Thus, we
study the performance of the matcher and regressor, respectively.

Table 4: Comparison of training strategy and matcher performance on the KITTI360Pose dataset. Strategy | Train | Validation
---|---|---
Precision $\uparrow$ | Recall $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$
joint | 98.12 | 98.16 | 86.67 | 87.59
cascade(ours) | 98.89 | 99.04 | 92.18 | 93.01
Table 5: Ablation study on the regression error of the fine-stage on the KITTI360Pose dataset. Method | Train Error $\downarrow$ | Validation Error $\downarrow$
---|---|---
w/o cascade training | 10.24 (+1.72) | 10.01 (+0.86)
w/o cross-attention | 9.57 (+1.05) | 9.56 (+0.41)
w/o confidence weighting | 9.02 (+0.50) | 9.23 (+0.08)
Ours | 8.52 | 9.15

Matcher. Following (Sarlin et al. 2020), we take precision and recall as the
the evaluation metric of the matcher. With an identical matcher architecture,
we investigate the impact of training strategy on the matcher performance. The
results are shown in Table 4. It can be seen that compared with joint training
(Kolmet et al. 2022), our cascaded training achieves not only high precision
and recall in the training set, but also stronger generalization on the
validation set. The results demonstrate that the cascade training strategy is
able to mitigate the multi-task optimization difficulty.

Figure 4: Qualitative retrieval results on KITTI360Pose validation set. The
red dot in the ground truth cell indicates the target location. In each
retrieved cell, the number in the lower right indicates the center distance
between this cell and the ground truth. Green box indicates positive cell
which contains the target location, while red box indicates negative cells.

Regressor. The regressor predicts the target location based on the the
matching results. We study the effects of cascaded training, cross-attention
based cross-modal fusion and confidence weighting for final location
prediction. We use regression error as evaluation metric and compare different
versions on both KITTI360Pose training and validation set. The results are
shown in Table. 5. Without cascaded training strategy, the regressor achieves
an error of 10.24 and 10.01 on the training and validation set, respectively,
which is 1.72 and 0.86 higher than that with cascaded training. This result
suggests that our cascaded training strategy also alleviates the optimization
difficulty of the regressor, which was caused by the noisy intermediate
results. Furthermore, without cross-attention mechanism, the regression error
also increases by a considerable margin, showing that cross-modal
collaboration is important for precise location prediction. Finally, with
confidence-based weighting, we can further reduce the regression error on both
the training and validation set, suggesting this information from the trained
matcher can be further utilized to improve performance.

### Visualizations

Embedding Space Visualization. We visualize the learned embedding space via
T-SNE (Van der Maaten and Hinton 2008) in Figure 5. It can be observed that
the baseline method Text2Pos (Kolmet et al. 2022) results in a less
discriminative space, where positive cells are relatively far away from the
query and sometimes separated across the embedding space. In contrast, our
method draw positive cell and query representations closer in the embedding
space, resulting in a more informative embedding space for retrieval.

Figure 5: T-SNE visualization of embedding space for the coarse stage. A cell
is considered as positive if it contains the location described by the query.
Compared with baseline method (Kolmet et al. 2022), our method can produce
better representation where positive cells are closer to the target.

Qualitative Cell Retrieval Results. We show some example text to point cloud
retrieval results in Figure. 4. For a given query, we visualize the top-3
retrieved cells. A retrieved cell is defined as positive if it contains the
target location. It can be observed that, our method can retrieve the ground
truth cell or those close in most cases. Sometimes, negative cells can also be
retrieved, e.g., top-1 in (a) and top-3 in (e). It can be seen that these
retrieved negative cells exhibit high semantic similarity with the ground
truth cell, even though far away from it. We also show a failure case (f),
where the retrieved cells are all negative. It can be seen that even though
far away from the target location, all these negative cells have instances
similar to the ground truth. These observations suggest that outdoor scenes
are indeed of low diversity, indicating that successful retrieval requires
highly discriminative representations to disambiguate the cells.

## Conclusion

In this work, we proposed a novel method for precise text-based localization
from large-scale point clouds. Our method employs a coarse-to-fine principle
and pipelines this process into two stages. For the coarse stage which is
formulated as a textual query based cell retrieval task, we aim to improve
representation discriminability for both point cloud and query
representations. This is achieved through explicit modeling of instance
relations and implemented via a newly proposed Relation-Enhanced Transformer
(RET). The core of RET is a novel Relation-enhanced Self-Attention (RSA)
mechanism, whereby the instance relations for the two modalities are
explicitly incorporated into the value computation process in a unified
manner. For the fine stage, our method performs description-instance matching
and position refinement in a cascaded way, whereby cross-modal information
collaboration is enhanced through the cross-attention mechanism. Extensive
experiments on the KITTI360Pose dataset validated the effectiveness of the
proposed method, which achieves new state-of-the-art performance. Additional
ablation studies further corroborate the effectiveness of each component in
the proposed method.

## Acknowledgement

This research is supported by the National Research Foundation, Singapore
under its Strategic Capability Research Centres Funding Initiative. Any
opinions, findings and conclusions or recommendations expressed in this
material are those of the author(s) and do not reflect the views of National
Research Foundation, Singapore.

## References

  * Achlioptas et al. (2020) Achlioptas, P.; Abdelreheem, A.; Xia, F.; Elhoseiny, M.; and Guibas, L. 2020.  Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes.  In _ECCV_. Springer.
  * Arandjelovic et al. (2016) Arandjelovic, R.; Gronat, P.; Torii, A.; Pajdla, T.; and Sivic, J. 2016.  NetVLAD: CNN architecture for weakly supervised place recognition.  In _CVPR_.
  * Ba, Kiros, and Hinton (2016) Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Brachmann et al. (2017) Brachmann, E.; Krull, A.; Nowozin, S.; Shotton, J.; Michel, F.; Gumhold, S.; and Rother, C. 2017.  Dsac-differentiable ransac for camera localization.  In _CVPR_.
  * Carion et al. (2020) Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020.  End-to-end object detection with transformers.  In _ECCV_. Springer.
  * Chen, Chang, and Nießner (2020) Chen, D. Z.; Chang, A. X.; and Nießner, M. 2020.  Scanrefer: 3d object localization in rgb-d scans using natural language.  In _ECCV_.
  * Cheng, Schwing, and Kirillov (2021) Cheng, B.; Schwing, A.; and Kirillov, A. 2021.  Per-pixel classification is not all you need for semantic segmentation.  _NeurIPS_.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _ICLR_.
  * Fan, Yang, and Kankanhalli (2022) Fan, H.; Yang, Y.; and Kankanhalli, M. 2022.  Point spatio-temporal transformer networks for point cloud video modeling.  _TPAMI_.
  * Fan, Yang, and Kankanhalli (2021) Fan, H.; Yang, Y.; and Kankanhalli, M. S. 2021.  Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos.  In _CVPR_.
  * Feng et al. (2021) Feng, M.; Li, Z.; Li, Q.; Zhang, L.; Zhang, X.; Zhu, G.; Zhang, H.; Wang, Y.; and Mian, A. 2021.  Free-form description guided 3d visual graph network for object grounding in point cloud.  In _ICCV_.
  * Gao et al. (2021) Gao, P.; Zheng, M.; Wang, X.; Dai, J.; and Li, H. 2021.  Fast Convergence of DETR With Spatially Modulated Co-Attention.  In _ICCV_.
  * Hausler et al. (2021) Hausler, S.; Garg, S.; Xu, M.; Milford, M.; and Fischer, T. 2021.  Patch-netvlad: Multi-scale fusion of locally-global descriptors for place recognition.  In _CVPR_.
  * Hu et al. (2018) Hu, H.; Gu, J.; Zhang, Z.; Dai, J.; and Wei, Y. 2018.  Relation Networks for Object Detection.  In _CVPR_.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kiros, Salakhutdinov, and Zemel (2014) Kiros, R.; Salakhutdinov, R.; and Zemel, R. S. 2014.  Unifying visual-semantic embeddings with multimodal neural language models.  _arXiv preprint arXiv:1411.2539_.
  * Kolmet et al. (2022) Kolmet, M.; Zhou, Q.; Osep, A.; and Leal-Taixe, L. 2022.  Text2Pos: Text-to-Point-Cloud Cross-Modal Localization.  In _CVPR_.
  * Li et al. (2019) Li, G.; Zhu, L.; Liu, P.; and Yang, Y. 2019.  Entangled Transformer for Image Captioning.  In _ICCV_.
  * Li et al. (2022) Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022.  BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.  In _ICML_.
  * Liao, Xie, and Geiger (2021) Liao, Y.; Xie, J.; and Geiger, A. 2021.  KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D.  _arXiv preprint arXiv:2109.13410_.
  * Liu et al. (2022) Liu, S.; Li, F.; Zhang, H.; Yang, X.; Qi, X.; Su, H.; Zhu, J.; and Zhang, L. 2022\.  DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR.  In _ICLR_.
  * Liu et al. (2020) Liu, Y.; Zhu, L.; Yamada, M.; and Yang, Y. 2020.  Semantic Correspondence as an Optimal Transport Problem.  In _CVPR_.
  * Liu et al. (2021) Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021\.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_.
  * Loshchilov and Hutter (2018) Loshchilov, I.; and Hutter, F. 2018.  Decoupled Weight Decay Regularization.  In _ICLR_.
  * Lu et al. (2019) Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _NeurIPS_.
  * Qi et al. (2017) Qi, C. R.; Yi, L.; Su, H.; and Guibas, L. J. 2017.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _NeurIPS_.
  * Sarlin et al. (2019) Sarlin, P.-E.; Cadena, C.; Siegwart, R.; and Dymczyk, M. 2019.  From coarse to fine: Robust hierarchical localization at large scale.  In _CVPR_.
  * Sarlin et al. (2020) Sarlin, P.-E.; DeTone, D.; Malisiewicz, T.; and Rabinovich, A. 2020.  Superglue: Learning feature matching with graph neural networks.  In _CVPR_.
  * Sattler, Leibe, and Kobbelt (2016) Sattler, T.; Leibe, B.; and Kobbelt, L. 2016.  Efficient & effective prioritized matching for large-scale image-based localization.  _TPAMI_.
  * Tan and Bansal (2019) Tan, H.; and Bansal, M. 2019.  LXMERT: Learning Cross-Modality Encoder Representations from Transformers.  In _EMNLP-IJCNLP_.
  * Torii et al. (2015) Torii, A.; Arandjelovic, R.; Sivic, J.; Okutomi, M.; and Pajdla, T. 2015.  24/7 place recognition by view synthesis.  In _CVPR_.
  * Van der Maaten and Hinton (2008) Van der Maaten, L.; and Hinton, G. 2008.  Visualizing data using t-SNE.  _JMLR_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _NeurIPS_.
  * Wu et al. (2021) Wu, K.; Peng, H.; Chen, M.; Fu, J.; and Chao, H. 2021.  Rethinking and improving relative position encoding for vision transformer.  In _ICCV_.
  * Yuan et al. (2021) Yuan, Z.; Yan, X.; Liao, Y.; Zhang, R.; Wang, S.; Li, Z.; and Cui, S. 2021.  Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring.  In _ICCV_.
  * Zhang et al. (2021) Zhang, H.; Sun, A.; Jing, W.; Nan, G.; Zhen, L.; Zhou, J. T.; and Goh, R. S. M. 2021\.  Video Corpus Moment Retrieval with Contrastive Learning.  In _SIGIR_.
  * Zhou et al. (2020) Zhou, Q.; Sattler, T.; Pollefeys, M.; and Leal-Taixe, L. 2020.  To learn or not to learn: Visual localization from essential matrices.  In _ICRA_.
  * Zhu et al. (2020) Zhu, X.; Su, W.; Lu, L.; Li, B.; Wang, X.; and Dai, J. 2020.  Deformable DETR: Deformable Transformers for End-to-End Object Detection.  In _ICLR_.