††thanks: ORCID ID: 0000-0001-7989-9271

# Symmetry-breaking, motion and bistability of active drops through
polarization-surface coupling

Fenna Stegemerten  Institut für Theoretische Physik, Westfälische Wilhelms-
Universität Münster, Wilhelm-Klemm-Str. 9, 48149 Münster, Germany  Karin John
Université Grenoble-Alpes, CNRS, Laboratoire Interdisciplinaire de Physique
38000 Grenoble, France  Uwe Thiele<EMAIL_ADDRESS>http://www.uwethiele.de Institut für Theoretische Physik, Westfälische
Wilhelms-Universität Münster, Wilhelm-Klemm-Str. 9, 48149 Münster, Germany
Center for Nonlinear Science (CeNoS), Westfälische Wilhelms-Universität
Münster, Corrensstr. 2, 48149 Münster, Germany

###### Abstract

Drops of active liquids are often considered as a simple realization of living
cells adhering to a solid substrate. Here we present a model for shallow
three-dimensional (3D) active free-surface drops with a mean polar order
parallel to the substrate and a polarization-surface (PS) coupling. This
allows us to identify a mechanism of motility in which active stresses drive
drop motion through spatial variations of polarization strength. It induces
motility even for liquid ridges (2D drops), in contrast to reported mechanisms
based on splay and bend configurations. Intriguingly, stable polar moving and
axisymmetric resting states may coexist, reminiscent of the interconversion of
moving and resting keratocytes by external stimuli. The newly identified
motility mode originates from a competition between the elastic bulk energy
and the orientational control exerted by the drop surface and breaks parity-
symmetry already for passive drops. This back-forth asymmetry enables active
stresses to effectively pump liquid and drop motion ensues.

Figure 1: Sketch of passive and active drops on a solid substrate. The height-
averaged polarization is shown as red arrows. The strengths of active stress
$c_{\text{a}}$ and the PS coupling $c_{\text{hpa}}$ allow one to distinguish:
(a) uniform polarization in a passive drop without PS coupling; (b) boundary
aster in a passive drop with weak PS coupling; (c) moving active drop of back-
forth asymmetric shape for contractility and PS coupling. The direction of
motion is indicated by the black arrow.

Active media far from thermodynamic equilibrium display rich dynamic phenomena
generated by active stress and self-propulsion Gompper  _et al._ (2020);
Marchetti  _et al._ (2013). For instance, bacterial colonies, aggregates of
epithelial cells or the cytoplasm of eukaryotic cells Wallmeyer  _et al._
(2018); Douezan  _et al._ (2011, 2012); Meacock  _et al._ (2021); Verkhovsky
_et al._ (1999) represent active free-surface drops on solid substrates. For
this system class active forces in the bulk related to nematic or polar order
compete with interfacial forces acting at the free surface and the solid
substrate. This results in active spreading phenomena and drop motility Pérez-
González  _et al._ (2019); Morita  _et al._ (2017); Sanchez  _et al._ (2012).
Often, such active drops are studied as a planar phenomenon Ziebert  _et al._
(2012); Giomi and DeSimone (2014); Ziebert and Aranson (2016); Marth  _et al._
(2015); Tjhung  _et al._ (2012), neglecting the direction perpendicular to the
substrate and, in consequence, the physics of the curved free surface. In the
context of cell crawling, a fully three-dimensional (3D) hydrodynamic model is
developed in Tjhung  _et al._ (2015). Alternatively, long-wave approximations
Oron  _et al._ (1997) are employed to derive thin-film models for shallow
drops of active polar liquids Sankararaman and Ramaswamy (2009); Joanny and
Ramaswamy (2012); Khoromskaia and Alexander (2015); Kitavtsev  _et al._
(2018); Loisy  _et al._ (2019, 2020); Wang  _et al._ (2021); Khoromskaia and
Alexander (2015) and of passive nematic liquid crystals Amar and Cummings
(2001); Lin  _et al._ (2013a, b). The latter approach may be extended to the
active case by endowing the nematic order with an active stress and including
evolution equations for the nematic order parameter Kitavtsev  _et al._
(2018).

Figure 2: (a) Morphological phase diagram in the plane spanned by the
strengths of active stress $c_{\text{a}}$ and PS coupling $c_{\text{hpa}}$.
(b-f) Snapshots of moving and resting drops [height profile (blue) with
overlayed polarization (black) and contact lines (red), gray arrows indicate
the direction of drop motion] at parameters indicated by symbols in (a). Shown
is only part of the full computational domain $\Omega$ [(b,c) $400\times 400$,
(d-f) $160\times 160$)]. The volume $V_{3\mathrm{D}}=70400$ is fixed in all
simulations. (g,h) Velocity dependencies for stationary moving drops at (g)
fixed $c_{\text{hpa}}=0.035$ and (h) $c_{\text{a}}=-0.01$, at parameters
indicated by respective solid and dashed black lines in (a). In (h) hysteresis
is shown by black dots [gray crosses] obtained when successively decreasing
[increasing] $c_{\text{hpa}}$, as indicated by the arrows. A positive
[negative] velocity indicates motion parallel [antiparallel] to the drop’s
mean polarization.

One striking result is that active contractile stresses related to nematic
order are sufficient to induce surface waves Sankararaman and Ramaswamy (2009)
and drop motion Tjhung  _et al._ (2012); Khoromskaia and Alexander (2015);
Loisy  _et al._ (2019). Indeed, fluid motion can be induced by spatial
variations in the director field either normal Loisy  _et al._ (2019) or
tangential to the substrate Tjhung  _et al._ (2012); Khoromskaia and Alexander
(2015); Sankararaman and Ramaswamy (2009). In the latter case, drop motion
requires a polar coupling between the liquid surface and the director field.
Most importantly, self-propulsion is not required for persistent motion. For
passive drops, their shape, spreading behavior and motion are crucially
determined by their wetting behavior on the substrate De Gennes (1985)
neglected by most models for active drops. In contrast, a recent generic thin-
film model describes the dynamics of free-surface drops of partially wetting
polar active liquids within a framework of fully nonlinear, coupled evolution
equations for film height and height-averaged polarization profiles. It
naturally accounts for moving three-phase contact lines and enables a study of
the role of defects in the polar order on the shape and motility of active
drops with self-propulsion Trinschek  _et al._ (2020). Here, we explicitly
include a coupling between the bulk polarization and the drop’s free surface
shape. The simple framework then captures drop motion solely induced by active
stress and polarization-surface (PS) coupling (Fig. 1). Motion even emerges
for liquid ridges (2D drops), i.e. it does not depend on splay configurations
in the bulk liquid, but rather on the spatial variation of the polarization
strength. The onset of motion shows hysteresis, i.e., for a finite parameter
range, resting and moving states are both stable. Strikingly, this bistability
is reminiscent of the conversion between moving and resting keratocytes that
can be triggered by mechanical or temperature stimuli Verkhovsky  _et al._
(1999); Barnhart  _et al._ (2015). At a further transition drops destabilize
and split.

We consider a liquid drop composed of active polar constituents, e.g.
actin/myosin assemblies. The drop height profile couples to the resulting
polarization field which is assumed to be mainly parallel to the substrate and
directed outward at the cell periphery, similar to polarized actin filaments
in real cells. The coupled dynamics of film height profile $h(\mathbf{x},t)$
and height-integrated polarization $\mathbf{P}(\mathbf{x},t)$ (Fig. 1) is
given by Trinschek  _et al._ (2020)

$\displaystyle\partial_{t}h$ $\displaystyle=$
$\displaystyle-\nabla\cdot\left(\mathbf{j}^{\mathrm{Cv}}+\mathbf{j}^{\mathrm{Ca}}\right)$
(1) $\displaystyle\partial_{t}P_{i}$ $\displaystyle=$
$\displaystyle-\nabla\cdot\left[p_{i}\left(\mathbf{j}^{\mathrm{Cv}}+\mathbf{j}^{\mathrm{Ca}}\right)+\mathbf{j}^{\mathrm{D}P_{i}}\right]+j^{\mathrm{R}}_{i}\,,$
(2)

with the passive fluxes $\mathbf{j}^{\mathrm{C_{\text{v}}}}$,
$\mathbf{j}^{\mathrm{D}P_{i}}$ and $j^{\mathrm{R}}_{i}$ derived from a
variational principle. The active flux is
$\mathbf{j}^{\mathrm{Ca}}={h^{3}\over 3\eta}\nabla\cdot\bm{\sigma}^{a}$ with
viscosity $\eta$ and active stress tensor
$\bm{\sigma}^{\text{\tiny{a}}}=-c_{\mathrm{a}}\mathbf{p}\mathbf{p}$. Here,
$c_{\text{a}}>0$ is extensile and $c_{\text{a}}<0$ contractile stress.
$\mathbf{p}=\mathbf{P}/h$ denotes the height-averaged polarization. The energy
functional underlying the variational part of the dynamics is

$\displaystyle\mathcal{F}[h,\mathbf{p}]=$
$\displaystyle\int\Big{[}\tfrac{\gamma}{2}(\nabla
h)^{2}+f_{\text{w}}(h)+hf_{\text{spo}}\left(h,\mathbf{p}^{2}\right)$
$\displaystyle+hf_{\text{el}}(\nabla\mathbf{p})+f_{\text{coupl}}(\nabla
h,\mathbf{p})\Big{]}\mathrm{d}\mathbf{x}\,.$ (3)

It contains capillarity, wettability, spontaneous polarization and elasticity
of the polarization field. The final term captures the coupling of bulk
polarization and drop surface shape

$f_{\text{coupl}}(\nabla h,\mathbf{p})={c_{\text{hpa}}\over
2}\mathbf{p}\cdot\nabla h\,,$ (4)

and favors at $c_{\text{hpa}}>0$ an aster-type polarization pattern.

Note, that the bulk liquid behavior is governed by nematic order. Only the
surface is sensitive to polar order. The complete theoretical framework
including self-propulsion and alternative PS couplings is detailed in the
Supplementary Material (SM).

Figure 3: (a) Drop states are characterized by a bifurcation diagram giving
the $L^{2}$-norm of $P$ over the strength of PS coupling $c_{\text{hpa}}$ for
polarized ridges with a boundary defect (blue) and with a central defect (red
and gray). The curves in light [strong] colors correspond to the active case
at $c_{\text{a}}=-0.005$ [passive case, $c_{\text{a}}=0$]. The pitchfork
[saddle-node] bifurcation is marked $c_{\text{hpa}}^{\text{\tiny{p}}}$
[$c_{\text{hpa}}^{\text{\tiny{sn}}}$]. For details see SM. Dashed [solid]
lines correspond to unstable [stable] solutions. (b-f) Height (top) and
polarization (bottom) profiles at $c_{\text{hpa}}$ values indicated in (a).
The fixed drop volume is $V_{2\mathrm{D}}=8880$.

Families of resting and moving drops are numerically analyzed using time
simulations (oomph-lib Heil and Hazel (2006)) and path continuation (auto07p
Dijkstra  _et al._ (2014), pde2path Uecker  _et al._ (2014)) - see SM.

Fig. 2(a) summarizes the role of active stress and PS coupling for drop
motility and morphology. Without PS coupling, uniformly polarized drops remain
at rest at any active stress, however, deform according to the stress-type
[orange region in Fig. 2(a)]. Passive drops ($c_{\text{a}}=0$) always rest and
change their polarization state with increasing PS coupling: From uniformly
polarized, to a boundary aster state with a half aster-like defect at the
contact line [light blue region in Fig. 2(a)] to a radial-symmetric central
aster state with outward pointing polarization [red region in Fig. 2(a) and
(f)]. The transition results from an increasing dominance of the energy gain
due to PS coupling over the energy loss due to a nematic defect in the bulk,
i.e. $f_{\text{coupl}}$ outweights $f_{\text{el}}$.

This sequence is also essential in the active case ($c_{\text{a}}\neq 0$). At
weak PS coupling a back-forth symmetry breaking in drop shape and polarization
occurs and drops steadily move [dark blue region in Fig. 2(a)].

Figure 4: (a) Velocities $v$ of active polar ridges at $c_{\text{hpa}}=0.17$
with a boundary defect (dark and bright blue) and a central defect (red) as a
function of active stress $c_{\text{a}}$. (b) Bifurcation diagram related to
the hysteretic transition between resting and moving drops and drop splitting.
Shown is the $L_{\text{2}}$-norm of $P$ over $c_{\text{a}}$. Labels
$c_{\text{a}}^{\text{\tiny{s1}}}$ to $c_{\text{a}}^{\text{\tiny{s3}}}$,
$c_{\text{a}}^{\text{\tiny{p}}}$ are explained in the main text [details in
SM]. Dashed [solid] lines correspond to unstable [stable] solutions. (c-h)
Film height (top) and polarization profiles (bottom) at $c_{\text{a}}$ values
as indicated by symbols in (a,b). The drop volume is fixed at
$V_{2\mathrm{D}}=8880$.

The mechanism of motion is detailed in Fig. 1 for contractile active stress.
For passive drops, the PS coupling destabilizes the uniform polarization of
Fig. 1(a) triggering a symmetry breaking that results in the asymmetric
boundary aster of Fig. 1(b) and radially symmetric drop shapes. With active
contractility the drop contracts along the direction of its net polarization
and the back-forth asymmetry in the polarization translates into steady drop
motion. The defect is located at the trailing edge of the drop [Figs. 1(c) and
2(d)], consistent with results of full 3D hydrodynamic models Tjhung  _et al._
(2015) and the planar case Marth  _et al._ (2015); Tjhung  _et al._ (2012). In
contrast, an active extensile stress elongates the drop in the direction of
its net polarization. Transiently, liquid is locally accumulated near the
defect before the energetic cost of surface and elastic energy reorients the
polarization field and forms the steadily moving drop in Fig. 2(e). Here the
advancing drop edge coincides with the location of the defect [Fig. 2(e)].

However, when strong PS coupling and large active stress overcome the
stabilizing surface tension large drops split into smaller ones [yellow region
in Fig. 2(a)]. Subsequent splitting continues until surface tension dominates
and results in ensembles of small steadily moving drops. Fig. 2(g) shows how
drop speed monotonically changes with active stress $c_{\text{a}}$ before
drops split at stronger PS coupling (where the line ends). Strikingly, a close
inspection of the low contractile stress range reveals that hysteresis between
moving and resting drops occurs when changing the strength of PS coupling
$c_{\text{hpa}}$ [Fig. 2(h)]. Starting with an axisymmetric resting drop at
$c_{\text{hpa}}=0.19$, a successive decrease in $c_{\text{hpa}}$ (black dots)
results in the transition to a moving polar drop at $c_{\text{hpa}}\approx
0.07$. However, starting with a moving drop at small $c_{\text{hpa}}=0.03$, a
successive increase in $c_{\text{hpa}}$ (gray crosses) only stops the drop at
$c_{\text{hpa}}\approx 0.15$. Such a coexistence of stable resting and stable
moving states is well experimentally documented for keratocytes Verkhovsky
_et al._ (1999); Barnhart  _et al._ (2015), but a conclusive mechanistic
picture has still been missing. A deeper understanding of the underlying
mechanism of motion and bistability can be gained considering simpler
transversally invariant liquid ridges (2D drops). By employing continuation
techniques Dijkstra  _et al._ (2014); Uecker  _et al._ (2014) that cover the
experimentally relevant range of contractility, we can capture stable and
unstable droplet states. Assuming that splay and bend instabilities in the
polarization field are needed to drive drop motion with active stresses as
argued in Marth  _et al._ (2015); Tjhung  _et al._ (2012, 2015), liquid ridges
with a polarization mainly parallel to the 1D substrate, i.e., without splay
or bend, should not move. In contrast, our computations for 2D drops reveal
qualitatively identical behavior as described above for 3D drops enabling us
to further scrutinize the transitions between resting and moving drops. A
detailed description of the encountered bifurcations in Fig. 3(a) and 4(a,b)
can be found in the SM. Fig. 3(a) shows on the one hand the bifurcation
behavior of passive ridges ($c_{\text{a}}=0$) in dependence of the PS coupling
$c_{\text{hpa}}$. At small $c_{\text{hpa}}$ polarized drops with a boundary
defect [Fig. 3(b)] are stable. They loose stability at large $c_{\text{hpa}}$
where symmetric 111In 2D ’symmetric’ refers to the parity symmetry
$(h(x),P(x))=(h(-x),-P(-x))$ drops with one central defect are stable [Figs.
3(c), outward pointing polarization], equivalent to the aster structure for
the 3D drops. Note that the transition between the two stable states shows a
hysteresis between $c_{\text{hpa}}^{\text{\tiny{p}}}=0.131$ and
$c_{\text{hpa}}^{\text{\tiny{sn}}}=0.175$ [Fig. 3(a)]. On the other hand, in
the active case [$c_{\text{a}}=-0.005$] the lightly colored lines in Fig. 3(a)
show a similar scenario. However, now activity implies that the light blue
branches represent moving ridges. As a result regions of multistability of
moving and resting ridges exist. For the stable drops with a boundary defect
[at $c_{\text{hpa}}=0.17$, Fig. 3(a,b)], the solid blue curve in Fig. 4(a)
shows that an increase in activity from zero induces a steady ridge motion
without critical stress. As in the 3D case, the direction of motion depends on
the type of active stress and the direction of net polarization. The
occurrence of moving ridges clearly evidences that liquid motion results from
a spatial modulation of the strength and not the orientation of polarization.
Although splay may contribute to motion in 3D drops, the described moving
ridges directly point to the polarization strength as primary origin of
motility. The similarity of velocities in both cases [Figs. 2(g), 4(a)]
suggests little importance of splay. A central feature of the complex
bifurcation diagram in Fig. 4(a,b) is the bistability of (i) a polar moving
state (solid dark blue branch) and a symmetric resting state (red solid
branch) for
$c_{\text{a}}^{\text{\tiny{p}}}=-0.0105<c_{\text{a}}<c_{\text{a}}^{\text{\tiny{s1}}}=0.0068$,
and (ii) of two different polar moving states (solid dark and bright blue
branches) for
$c_{\text{a}}^{\text{\tiny{s2}}}=-0.0177<c_{\text{a}}<c_{\text{a}}^{\text{\tiny{p}}}$,
which move at different speed. In particular, the first-mentioned bistability
qualitatively corresponds to the hysteresis behavior for the transition of a
resting aster state to a moving boundary aster state in 3D. In the biological
context, this means that at a given contractile stress (e.g.
$c_{\text{a}}=-0.005$) drops (cells) may coexist in a stable moving state with
a single polarization direction and a stable resting state with two oppositely
polarized domains. The two states are separated by an unstable moving
threshold state, i.e. the interconversion between resting and moving states
requires a finite perturbation. For example, insufficiently strong
perturbation of the resting state only results in transient motion. Increasing
the contractile stress beyond a critical value the resting state loses
stability and only moving states exists.

To conclude, we have modeled shallow three-dimensional (3D) active free-
surface drops with a mean polar order parallel to the substrate. Incorporating
a polarization-surface coupling has allowed us to identify a mechanism of
motility in which active stresses drive drop motion through spatial variations
of polarization strength. Motility is induced even for liquid ridges, implying
that splay in the polarization field is not essential for the mechanism. This
provides an alternative to previous proposals based on splay or bend Tjhung
_et al._ (2012, 2015); Marth  _et al._ (2015). Living cells exert a tight
control over the cytoskeleton via signaling pathways which are partially
localized in the cell membrane, i.e. the machinery necessary for the growth
and architecture of polar Arp2/3 cross-linked actin networks and linear
filopodia structures induced by formins is localized at the plasma membrane
Blanchoin  _et al._ (2014). Within the context of active drops as minimal
models for living cells a fine tuning of the PS coupling by the cell is a
reasonable assumption. Thus, the physico-chemical and mechanical properties of
active polar filament assemblies may play an important role in the robustness
of cell migration in addition to complex biochemical signaling pathways.

Furthermore, our model provides a possible mechanism of bistability in the
keratocyte system based on well-identified physical components (encoded in a
variational model) and active stress, without the need for complex biochemical
feedback mechanisms Barnhart  _et al._ (2015); Ron  _et al._ (2020); Ziebert
_et al._ (2012).

We acknowledge the doctoral school “Active living fluids” of the German French
University (Grant No. CDFA-01-14), and the DAAD & Campus France (PROCOPE 2020
Grant No. 57511756). FS thanks the “Studienstiftung des deutschen Volkes” for
financial support.

## References

  * Gompper  _et al._ (2020) G. Gompper, R. G. Winkler, T. Speck, A. Solon, C. Nardini, F. Peruani, H. Löwen, R. Golestanian, U. B. Kaupp, L. Alvarez, _et al._ , J. Phys.: Condens. Matter 32, 193001 (2020).
  * Marchetti  _et al._ (2013) M. C. Marchetti, J.-F. Joanny, S. Ramaswamy, T. B. Liverpool, J. Prost, M. Rao, and R. A. Simha, Rev. Mod. Phys. 85, 1143 (2013).
  * Wallmeyer  _et al._ (2018) B. Wallmeyer, S. Trinschek, S. Yigit, U. Thiele, and T. Betz, Biophys. J. 114, 213 (2018).
  * Douezan  _et al._ (2011) S. Douezan, K. Guevorkian, R. Naouar, S. Dufour, D. Cuvelier, and F. Brochard-Wyart, Proc. Natl. Acad. Sci. U.S.A 108, 7315 (2011).
  * Douezan  _et al._ (2012) S. Douezan, J. Dumond, and F. Brochard-Wyart, Soft Matter 8, 4578 (2012).
  * Meacock  _et al._ (2021) O. J. Meacock, A. Doostmohammadi, K. R. Foster, J. M. Yeomans, and W. M. Durham, Nat. Phys. 17, 205 (2021).
  * Verkhovsky  _et al._ (1999) A. Verkhovsky, T. Svitkina, and G. Borisy, Curr. Biol. 9, 11 (1999).
  * Pérez-González  _et al._ (2019) C. Pérez-González, R. Alert, C. Blanch-Mercader, M. Gómez-González, T. Kolodziej, E. Bazellieres, J. Casademunt, and X. Trepat, Nat. Phys. 15, 79 (2019).
  * Morita  _et al._ (2017) H. Morita, S. Grigolon, M. Bock, S. G. Krens, G. Salbreux, and C.-P. Heisenberg, Dev. Cell 40, 354 (2017).
  * Sanchez  _et al._ (2012) T. Sanchez, D. T. N. Chen, S. J. DeCamp, M. Heymann, and Z. Dogic, Nature 491, 431 (2012).
  * Ziebert  _et al._ (2012) F. Ziebert, S. Swaminathan, and I. S. Aranson, J. R. Soc. Interface 9, 1084 (2012).
  * Giomi and DeSimone (2014) L. Giomi and A. DeSimone, Phys. Rev. Lett. 112, 147802 (2014).
  * Ziebert and Aranson (2016) F. Ziebert and I. S. Aranson, npj Comput. Mater. 2, 16019 (2016).
  * Marth  _et al._ (2015) S. Marth, V. Wieland, and A. Praetorius, J. Royal Soc. Interface 12, 20150161 (2015).
  * Tjhung  _et al._ (2012) E. Tjhung, D. Marenduzzo, and M. E. Cates, Proc. Natl. Acad. Sci. U. S. A 109, 12381 (2012).
  * Tjhung  _et al._ (2015) E. Tjhung, A. Tiribocchi, D. Marenduzzo, and M. E. Cates, Nat. Commun. 6, 5420 (2015).
  * Oron  _et al._ (1997) A. Oron, S. Davis, and S. Bankoff, Rev. Mod. Phys. 69, 931 (1997).
  * Sankararaman and Ramaswamy (2009) S. Sankararaman and S. Ramaswamy, Phys. Rev. Lett. 102, 118107 (2009).
  * Joanny and Ramaswamy (2012) J. F. Joanny and S. Ramaswamy, J. Fluid Mech. 705, 46 (2012).
  * Khoromskaia and Alexander (2015) D. Khoromskaia and G. P. Alexander, Phys. Rev. E 92, 062311 (2015).
  * Kitavtsev  _et al._ (2018) G. Kitavtsev, A. Münch, and B. Wagner, Proc. R. Soc. A. 474, 20170828 (2018).
  * Loisy  _et al._ (2019) A. Loisy, J. Eggers, and T. B. Liverpool, Phys. Rev. Lett. 123, 248006 (2019).
  * Loisy  _et al._ (2020) A. Loisy, J. Eggers, and T. B. Liverpool, Soft Matter 16, 3106 (2020).
  * Wang  _et al._ (2021) H. Wang, T. Qian, and X. Xu, Soft Matter 17, 3634 (2021).
  * Amar and Cummings (2001) B. M. Amar and L. J. Cummings, Phys. Fluids 13, 1160 (2001).
  * Lin  _et al._ (2013a) T.-S. Lin, L. J. Cummings, A. J. Archer, L. Kondic, and U. Thiele, Phys. Fluids 25, 082102 (2013a).
  * Lin  _et al._ (2013b) T.-S. Lin, L. Kondic, U. Thiele, and L. J. Cummings, J. Fluid Mech. 729, 214 (2013b).
  * De Gennes (1985) P.-G. De Gennes, Rev. Mod. Phys. 57, 827 (1985).
  * Trinschek  _et al._ (2020) S. Trinschek, F. Stegemerten, K. John, and U. Thiele, Phys. Rev. E 101, 062802 (2020).
  * Barnhart  _et al._ (2015) E. Barnhart, K.-C. Lee, G. M. Allen, J. A. Theriot, and A. Mogilner, Proc. Natl. Acad. Sci U.S.A. 112, 5045 (2015).
  * Heil and Hazel (2006) M. Heil and A. L. Hazel, in  _Fluid-Structure Interaction: Modelling, Simulation, Optimisation_, edited by H.-J. Bungartz and M. Schäfer (Springer, Berlin, Heidelberg, 2006) pp. 19–49.
  * Dijkstra  _et al._ (2014) H. A. Dijkstra, F. W. Wubs, A. K. Cliffe, E. Doedel, I. F. Dragomirescu, B. Eckhardt, A. Y. Gelfgat, A. Hazel, V. Lucarini, A. G. Salinger, E. T. Phipps, J. Sanchez-Umbria, H. Schuttelaars, L. S. Tuckerman, and U. Thiele, Commun. Comput. Phys. 15, 1 (2014).
  * Uecker  _et al._ (2014) H. Uecker, D. Wetzel, and J. Rademacher, Numer. Math.-Theory Methods Appl. 7, 58 (2014).
  * Note (1) In 2D ’symmetric’ refers to the parity symmetry $(h(x),P(x))=(h(-x),-P(-x))$.
  * Blanchoin  _et al._ (2014) L. Blanchoin, R. Boujemaa-Paterski, C. Sykes, and J. Plastino, Physiol. Rev. 94, 235 (2014).
  * Ron  _et al._ (2020) J. E. Ron, P. Monzo, N. C. Gauthier, R. Voituriez, and N. S. Gov, Phys. Rev. Res. 2, 033237 (2020).


#  ALT-Pilot: Autonomous navigation with Language augmented Topometric maps
https://navigate-anywhere.github.io/ALT-Pilot/

Mohammad Omama1 Pranav Inani∗2 Pranjal Paul∗2 Sarat Chandra Yellapragada2
Krishna Murthy Jatavallabhula†3 Sandeep Chinchali†1 and Madhava Krishna†2
1The University of Texas at Austin 2International Institute of Information
Technology, Hyderabad
3Massachusetts Institute of Technology *Co-second authors $\dagger$Equal
advising

###### Abstract

We present an autonomous navigation system that operates without assuming HD
LiDAR maps of the environment. Our system, ALT-Pilot, relies only on publicly
available road network information and a sparse (and noisy) set of
crowdsourced language landmarks. With the help of onboard sensors and a
language-augmented topometric map, ALT-Pilot autonomously pilots the vehicle
to any destination on the road network. We achieve this by leveraging vision-
language models pre-trained on web-scale data to identify potential landmarks
in a scene, incorporating vision-language features into the recursive Bayesian
state estimation stack to generate global (route) plans, and a reactive
trajectory planner and controller operating in the vehicle frame. We implement
and evaluate ALT-Pilot in simulation and on a real, full-scale autonomous
vehicle and report improvements over state-of-the-art topometric navigation
systems by a factor of 3$\times$ on localization accuracy and 5$\times$ on
goal reachability.

##  I Introduction

Existing industry-led autonomous driving (AD) systems rely crucially on high-
definition (HD) maps, meticulously constructed with centimeter-level
precision. These HD maps impose large compute and memory requirements for map
creation and maintenance [2], which has insofar limited the applicability of
such AD systems to urban areas where HD map generation and maintenance are
both computationally and economically justifiable. Moreover, the inherent
memory demands and human resource investments for HD map creation severely
limit the scalability of these systems.

###  I-A Related Work

Autonomous driving without HD maps. To eliminate the dependence on HD maps,
existing approaches [3, 1, 4, 5] use crowdsourced _topometric_ maps like
OpenStreetMap (OSM) – topologically accurate, but metrically imprecise road
networks. These approaches employ topometric maps for global route planning
and integrate them with onboard sensors for state estimation, trajectory
planning, and control. While these systems have enabled navigation to
destinations near road intersections, their localization accuracy deteriorates
as the vehicle ventures farther away (see Fig. LABEL:fig:splash).

Vision-language models. Recent advances in vision-language models [6, 7, 8, 9]
have demonstrated promising capabilities for both 2D and 3D perception [10,
11, 12, 13]. These approaches enable navigation to open-vocabulary goals,
usually specified in the form of text prompts, by aligning semantic concepts
across language and images. However, these approaches too, require
significantly large memory and/or compute footprints, owing to the storage of
high-dimensional neural embeddings [10, 7] or require a large number of
queries to a neural field [12, 13]; both impractical for AD.

###  I-B Summary of Our Contributions

In this paper, we address two primary questions:

  1. 1.

What kinds of map representations enable autonomous navigation to any
destination on a road network111Current best approach [1] only enables
reliable navigation to intersections, but not arbitrary destinations along a
road., without relying on HD lidar maps?

  2. 2.

How do we accurately localize on this map representation, while accounting for
errors arising due to the map representation, and due to imperfect perception
and control?

Figure 2: Pipeline: ALT-Pilot uses OSM maps augmented with language
descriptions of landmarks. We precompute the CLIP (language) descriptors for
each of these landmarks. At inference time, the LiDAR and Camera information
is used to generate per-pixel CLIP (image) descriptors used in our multimodal
localization algorithm (III-B). The LiDAR data is used to segment points on
the road and generate a costmap (III-A). The localization and perception
information then goes to a planning module which generates actuation commands.

To address the former question, we propose to use language-augmented
topometric maps – where we augment off-the-shelf OSM road networks with a
small number of language landmarks (such as street signs, traffic signals, and
other persistent features in the environment useful for localization). The
landmarks are _language-only_ , meaning we store only a language tag; but no
other information (such as appearance or size information) is used. This makes
the map extremely convenient to build, mirroring OSM’s annotation methodology,
where users simply click on a displayed road network and enter a brief
language description222We demonstrate robustness to errors in landmark
location, false positives/negatives, incorrect language descriptions in Sec.
IV..

Towards the latter, we propose ALT-Pilot, an autonomous navigation system
capable of piloting a vehicle to any destination over a language-augmented
topometric map. We develop a multimodal localization pipeline that efficiently
bridges multiple modalities such as LiDAR (for road perception), RGB stream
(for vision-language feature extraction), and language (for landmark
description) to safely and accurately navigate to any location on the road
network. Besides enabling localization and autonomous navigation, language
landmarks also enable open-vocabulary navigation based on goals described in
language.

To summarize, our key contributions are:

  1. 1.

ALT-Pilot: An autonomous navigation system that is completely devoid of HD
LiDAR maps, relying instead on language-augmented topometric maps; enabling
accurate localization and open-vocabulary navigation.

  2. 2.

A probabilistic multimodal localization system bridging vision, lidar, and
language to estimate a posterior over vehicle poses.

Our key results are as follows:

  1. 1.

ALT-Pilot localizes to a much higher degree of accuracy, _and_ over longer
stretches of the road network, compared to state-of-the-art topometric
localization approaches such as MapLite [1].

  2. 2.

Based on evaluations on a real-world, full-size autonomous car over 3 km of
driving data, and more elaborate evaluations on the CARLA [14] simulator, we
outperform current art by a factor of 3$\times$ on localization precision and
5$\times$ on goal reachability.

##  II Multimodal Topometric Maps

This work focuses on autonomous driving platforms equipped with a lidar sensor
and extrinsically calibrated color cameras. To enable our system to be as
general as possible, our multimodal topometric map is minimalist by design and
is only intended to be used as a _guide_ to aid global planning and
localization. Our map comprises two key elements: the road network, and
(approximately) geotagged language landmarks.

The road network is a graph where the presence of an edge between two nodes
$N_{A}$ and $N_{B}$ indicates that there exists a navigable path between the
two nodes. These road networks are already openly available for the entire
world. For instance, using the popular OpenStreetMap API, the entire road
network for the continental United States may be downloaded and stored in
about 1 GB of memory. While prior approaches have investigated using solely
the OSM’s road network[1, 4], such approaches are only suited to navigate to
intersections, owing to their localization algorithm only being able to
precisely localize at intersections.

In all of the real-world experiments in this work, we preprocess and extract
only the road network from the OpenStreetMap (OSM). We augment the road
network with approximately geotagged point _landmarks_. Each point landmark
has a geolocation that places it relative to the UTM coordinate frame for the
local UTM zone; and a short (1-3 word) language tag that briefly describes the
landmark. Ideal choices for such landmarks are traffic signs, roundabouts,
underpasses, benches, poles, or other stationary objects in the environment.
Using crowdsourced maps such as OSM enables these landmarks to be created by
any user via their web interface; resulting in desirable scaling properties
and minimal human intervention.

##  III Approach

Using the aforementioned multimodal topometric map representation, we design
an autonomous navigation system (see Fig-2) that can safely navigate to any
location on the road network. The system accepts a geolocation of the
destination as input, and navigates toward the goal with the help of the
following subsystems:

  1. 1.

the LiDAR Perception subsystem consumes a laser scan, segments the road plane,
and constructs a Euclidean Signed Distance Field (ESDF) for use in trajectory
planning and control.

  2. 2.

our novel Multimodal Localization subsystem extracts cues from the LiDAR and
Image data to localize against the road network and text landmarks encoded in
the topometric map.

  3. 3.

our Path Planning, Trajectory Planning, and Controllers are based on prior
work [1, 15] , with minor modifications necessary for integration with the
other subsystems.

###  III-A LiDAR Perception

Each input lidar scan is first segmented using the segmentation method
proposed in [16] to extract all points on the road plane. We then construct a
(binary) occupancy grid in a bird’s-eye view, by extracting a $30\ m\times 15\
m$ region centered around the vehicle. This is used to compute a distance
field $\mathcal{D}:\mathbb{R}^{2}\mapsto\mathbb{R}^{+}$, where for each grid
location $\mathit{x}\in\mathbb{R}^{2}$, $\mathcal{D}(\mathit{x})$ denotes the
distance from $\mathit{x}$ to the closest obstacle. For use in our
localization and variational trajectory planner, we compute a Euclidean signed
distance field $\mathcal{E}:\mathbb{R}^{2}\mapsto\mathbb{R}$, where
$\mathcal{E}(\mathit{x})=\mathcal{D}(\mathit{x})-\mathcal{D}(1-\mathit{x})$ is
the difference between the distance field and the _inverse_ distance field.
The ESDF $\mathcal{E}$ forms the basis for our probabilistic multimodal
localization subsystem.

###  III-B Multimodal Localization across LiDAR, Image, and Text

Accurately localizing the vehicle within the topometric map is critical to the
success of autonomous navigation. In our context, the localization subsystem
must bridge the gap across multiple modalities: the topometric map (a graph
augmented with landmarks and language tags), and onboard sensor data (lidar,
camera, IMU, and other odometry sources). We develop a probabilistic
multimodal localization approach based on particle filters. For a refresher on
particle filters, we refer the interested reader to [17, 18, 19].

We want to estimate the posterior distribution, at time step $t$, over the
vehicle state $\mathbf{x_{t}}=(x_{t},y_{t},\theta_{t})\in SE(2)$. This
posterior is a set of $N$ hypotheses (particles)
$\mathcal{X}_{t}=\mathbf{x}^{[1]}_{t},\mathbf{x}^{[2]}_{t},\cdots,\mathbf{x}^{[N]}_{t}$,
initialized uniformly and randomly over the entire map. At each time step,
given the control $u_{t}$, the set of particles is updated by resampling
according to a motion model $p$, derived from either the exteroceptive (lidar,
camera) or the proprioceptive (IMU, wheel encoders) sensors.

$\mathbf{x}^{[m]}_{t}\sim
p(\mathbf{x}_{t}^{[m]}|u_{t},\mathbf{x}^{[m]}_{t-1})$

In our experiments, motion model updates are performed solely from off-the-
shelf lidar odometry [20]. Sampling solely based on the motion model (dead
reckoning) results in the state estimation error increasing unboundedly over
time.

To mitigate this, we define an observation likelihood that considers all
sensing modalities and map context. Recall that CLIP features are aligned
across vision and lanugage, and may be compared across domains by means of a
cosine similarity measure. We precompute CLIP text features for all landmarks
in the map to create a set $K$ where each $k_{i}\in K$ has an associated
position $k_{i}^{pos}$ and a language feature $k_{i}^{feat_{l}}$.

At run time, we first process the incoming RGB image using the pixel-aligned
feature extraction approach proposed in ConceptFusion [10] to extract dense
CLIP features for each pixel in the image. For each particle denoted as
$\mathbf{x}_{t}^{[m]}$, we employ its pose hypothesis to extract a subset of
landmarks $\mathrm{\tilde{K}}\subseteq K$ from the topometric map that may be
visible in the image. Given our extrinsic calibration of the LiDAR-Camera
setup, we can determine the collection of LiDAR points that project to valid
image pixels. We create a set $L$ of such lidar points (projected to map frame
using the particle hypothesis) such that each $l_{i}\in L$ has a vision
feature $l_{i}^{feat_{v}}$ and a position $l_{i}^{pos}$ associated with it. We
project the ESDF computed by the perception system to the map frame to get
$\tilde{\mathcal{E}}$. Also, we create a set $R=(r_{1},r_{2},...,r_{v})$ of
$v$ closest road points to the particle hypothesis.

Then, for each particle $\mathbf{x}_{t}^{[m]}$, we calculate an importance
factor denoted as $w_{t}^{[m]}$ in accordance with Algorithm 1. This
importance factor is subsequently used in the particle resampling process. The
algorithm has the following key components:

Data: Landmarks $\mathrm{\tilde{K}}$, LiDAR points $L$, ESDF in map frame
$\mathcal{\tilde{E}}$, Road Points $R$

Result: Hypothesis Importance Factor $w^{[m]}_{t}$

1 Landmark Weight $w_{lm}\leftarrow 0$

2foreach  _landmark $\tilde{k}_{j}$ in $\mathrm{\tilde{K}}$_ do

3 $\text{Closest LiDAR Match}\ \tilde{l}\leftarrow$ null

4 $\text{Maximum Cosine Similarity c}\leftarrow 0$

5 foreach  _lidarPoint $l_{i}$ in $L$_ do

6

7 $\text{sim}\leftarrow$ CosineSim(_$k_{j}^{feat_{l}}$ , $l_{i}^{feat_{v}}$_)

8 if  _$\text{sim} >\text{c}$_ then

9

10 $\tilde{l}\leftarrow l_{i}$

11 $\text{c}\leftarrow\text{sim}$

12

13 Distance Reciprocal
$\tilde{d}\leftarrow\dfrac{1}{||\tilde{l}_{j}^{pos}-\tilde{k}_{j}^{pos}||+\epsilon}$

14 Distance Factor $d\leftarrow\alpha+\dfrac{1}{1+e^{-\beta\tilde{d}}}$

15 $w_{lm}:=w_{lm}+c\times d$

16Yaw Weight $w_{yaw}\leftarrow\sum_{j}\mathcal{\tilde{E}}(r_{j})$

$w^{[m]}_{t}=w_{lm}+\lambda w_{yaw}$

Algorithm 1 Calculation of the importance factor for a single particle
hypothesis

  * •

For each landmark we calculate the cosine similarity value $c\in[0,1]$ of the
closest matching lidar point $\tilde{l}$ (line 1-1). It ensures that the
observation model corrects itself only when pixels corresponding to the
appropriate landmark description are observed, thus mitigating the impact of
false-positive landmarks.

  * •

In line 1-1, we calculate the distance factor $d\in[0,1]$. We use generalized
logistic functions [21] with tunable parameters $\alpha$ and $\beta$ to
control the temperature of the distance term and ensure that it’s in the range
[0,1]. The purpose of $d$ is to activate the observation model only when the
landmark is actually located at that position, thereby reducing the influence
of false-positive CLIP matches.

  * •

The terms $c$ and $d$ are multiplied and summed for all landmarks on line 1 to
get the landmarks weight $w_{lm}$.

  * •

On line 1 we compare the ESDF projected on the map frame,
$\tilde{\mathcal{E}}$, and the road geometry expected around that location to
calculate how well the particle aligns with the road. If the yaw of the
particle hypothesis is close to the ground truth yaw, then the road geometry
$R$ and the orientation of the road observed in $\tilde{\mathcal{E}}$ will
align leading to a larger value of $w_{yaw}$ and vice-versa.

  * •

We sum $w_{lm}$ and $w_{yaw}$ with a tunable parameter $\lambda$ to get the
importance factor $w_{t}^{[m]}$ of the particle $\mathbf{x}_{t}^{[m]}$.

Sparse landmarks often prove insufficient for effective yaw correction,
necessitating the incorporation of an additional ESDF component. The impact of
$w_{yaw}$ is presented in section IV-B while the ablation on false positives
and negatives is studied in section IV-E.

###  III-C Planning and Control

Given a vehicle’s current pose estimate and a goal point in the topological
map, we employ the A* algorithm [22] to determine the shortest path and
extract waypoints for the next $15m$ stretch of road. To account for
uncertainty in the map and our state estimates, we employ a Frenet planner
[15] that samples trajectories given the waypoints and the precomputed ESDF
from the perception stage. Each trajectory is evaluated based on proximity to
waypoints and ESDF costs, ensuring collision-free and accurate tracking.
Concurrently, a Stanley controller [23] actively guides the vehicle along the
planned trajectory, while a PID controller manages dynamic speed adjustments.

##  IV Experiments, Results, and Analyses

In this section, we evaluate ALT-Pilot across simulated and real-world
topomeric navigation scenarios with respect to current art [1]. Our
experiments span topological, topometric, and metric localization, while also
analyzing convergence times and planner reachability. Specifically, our
experiments aim to address these primary questions:

  1. 1.

How accurately does ALT-Pilot localize over a topometric map? This is measured
by means of three metrics:

    1. (a)

Recall@K: Jaccard similarity of the K nearest landmarks of the ground truth
position and the estimated position. Measures topological localization
accuracy.

    2. (b)

Distance to Closest Landmark Region (DCLR): Measures the proximity of the
estimated position to a landmark region, defined by a radius $r$ from the
nearest actual landmark. If within $r$, the metric is zero; otherwise, it
measures the distance to the region. This combines topological and metric
aspects for a nuanced evaluation. Mathematically, it can be written as:
$max(0,\sqrt{(p_{x}-l_{x})^{2}+(p_{y}-l_{y})^{2}}-r)$, where $p_{x}$ and
$p_{y}$ represent the position estimate, $l_{x},l_{y}$ are the landmark
coordinates, and $r$ is the radius of the landmark region.

    3. (c)

Absolute Position Error (APE): It’s the Euclidean distance between the
estimated and actual positions. Measures metric accuracy.

  2. 2.

How quickly does the vehicle achieve convergence in a global location setting?
To quantify this, we measure the distance traversed until our localization
system (multimodal particle filter) has converged.

  3. 3.

How effectively does ALT-Pilot reach its intended destination in a closed-loop
setting? To assess this, we measure the distance from the vehicle’s final
location (where it declares task success/failure) to the true destiation.

Additionally, we present qualitative results and case studies underscoring the
benefits of augmenting OpenStreetMap (OSM) with language landmarks for
language-guided navigation. We also investigate the robustness of our approach
to noise in landmark locations, and false postives/negatives.

TABLE I: Absolute Position Error in Map Frame. Comparison of various baselines on multiple maps with varying levels of map noise. ALT-Pilot(L) is a subset of our system (no yaw correction), while ALT-Pilot, our full system, achieves superior performance. Scene | Noise | DeadReck | MapLite | ALT-Pilot(L) | ALT-Pilot
---|---|---|---|---|---
CARLA1 | 0% | 21.1 | 10.3 | 7.2 | 3.98
| 10% | 32.7 | 15.5 | 17.3 | 4.94
| 15% | 45.3 | 24.7 | 29.7 | 5.61
| 20% | 50.4 | 24.4 | 39.2 | 7.52
CARLA2 | 0% | 10.2 | 6.4 | 6.8 | 2.3
| 10% | 13.8 | 10.1 | 7.4 | 3.1
| 15% | 17.3 | 11.7 | 6.6 | 3.7
| 20% | 20.4 | 11.9 | 7.1 | 5.1
CARLA3 | 0% | 10.2 | 7.4 | 4.1 | 4.5
| 10% | 15.4 | 11.3 | 6.5 | 7.1
| 15% | 24.7 | 15.3 | 7.2 | 7.7
| 20% | 30.1 | 18.4 | 8.9 | 9
CARLA4 | 0% | 23.2 | 15.1 | 9.8 | 3.2
| 10% | 35.2 | 19.1 | 14.1 | 6.65
| 15% | 48.3 | 33.3 | 20.7 | 6.17
| 20% | 66.9 | 40.8 | 30.5 | 8.22
Real1 | 0% | 13.03 | 9.94 | 4.01 | 3.29
Real2 | 0% | 16.38 | 12.66 | 3.89 | 3.88

###  IV-A Experimental Setup and Baselines

We assess localization performance across four CARLA scenarios (spread across
three distinct town maps), averaging results across three start and end
points. Given our reliance on crowd-sourced OpenStreetMap (OSM) data, which
lacks HD map precision, to assess our system’s resilience to map inaccuracies,
we scale the road network by upto 20% of its original size (to induce errors
in road segment lengths, and exacerbate errors due to incorrectly positioned
landmarks). We also evaluate real-world performance on two maps collected at
the IIIT Hyderabad campus, over trajectories spanning 3 km with several
intersections and turns.

We compare the following approaches:

  1. 1.

Dead Reckoning: odometry is integrated over time (error grows unboundedly due
to the lack of feedback)

  2. 2.

MapLite: Using the observation likelihood outlined in Maplite[1] (the current-
best approach to topometric localization), and extending it to a particle
filter setting for fair evaluation.

  3. 3.

ALT-Pilot(L): Utilizing only language landmarks, without the yaw correction
term (i.e., no ESDF)

  4. 4.

ALT-Pilot: Using our full system (language landmarks and ESDF) outlined in
Algorithm 1

We evaluate closed-loop performance (i.e., the system navigating in
conjunction with state estimation, planning, and control) over two CARLA
scenarios.

For the data collection of Real1 and Real2 maps, and for the closed-loop
language-guided navigation experiments in Sec. IV-E, we employ our custom,
fullsize drive-by-wire electric vehicle, outfitted with an extrinsically
calibrated 16-channel Velodyne LiDAR and a RGB camera.

Figure 3: Topological and Topometric localization on two CARLA maps. ALT-Pilot
outperforms MapLite in both topological and topometric localization. Further,
it’s more robust to noise in the map while MapLite is adversely affected by
it. The impact of noise is more noticeable in the larger CARLA1 map (panels A,
C).

###  IV-B Localization Results

Figure 4: Particle Filter Localization. ALT-Pilot utilizes language
descriptions of landmarks (red bounding boxes) for state estimation while
MapLite only uses intersection information. ALT-Pilot is more accurate and
certain throughout the run. MapLite shows comparable performance only near the
intersection (top row).

In Fig-4, we show a qualitative analysis of our approach against MapLite. ALT-
Pilot utilizes language landmarks along the way, resulting in better state
estimation while MapLite, which only uses intersection information, incurs
substantial drift. Further ALT-Pilot has lower uncertainty throughout the run
while MapLite only achieves low uncertainty near an intersection.

Table I compares the APE of our method, ALT-Pilot, with other baselines. ALT-
Pilot consistently excels in APE. In large maps like ‘CARLA1’ and ‘CARLA4’,
yaw correction notably boosts localization – without the yaw correction,
landmark sparsity in these maps will have resulted in poor localization
performance (as can be seen in ALT-PilotL).

We further evaluate our performance by examining the topological performance
using the ‘Recall@K’ metric and the topometric performance with the ‘DCLR’
metric. In Fig-3[A, B] and Fig-3[C, D], we present the results for ‘Recall@K’
and ‘DCLR’ on two CARLA maps, respectively. The metrics are important for
downstream reachability tasks. Notably, as the size of the topological region
increases, whether by enlarging K or R, both ALT-Pilot and MapLite displays
comparable results. However, in smaller topological regions, ALT-Pilot
outperforms, showcasing its relevance in addressing fine-grained reachability
challenges.

Further ALT-Pilot is quite robust to map noise. By inducing 20% map noise,
Recall@K scores for both ALT-Pilotand MapLite drop by 0.1 and 0.23 in CARLA1
(3[A]) and by 0.12 and 0.17 CARLA2 (3[B]) respectively. Similarly, the DCLR
values increase by 3.8m and 11m in CARLA1 (3[C]) and by 3m and 5m in CARLA2
(3[D]) respectively. The impact of noise is more observable in larger maps.
Notice that ALT-Pilot degrades more gracefully compared to MapLite, which
incurs more significant errors.

###  IV-C Global Localization: Distance traveled until Convergence

In Table II, we present the distance traveled by the vehicle until the
particle filter _converges_. The particle filter is deemed converged when the
median of all particles is within 5m ground-truth location, and their variance
is less than 10m. In all these scenarios, the initial particle distribution
was spread across regions with similar road geometries. This posed a
considerable challenge for MapLite in achieving convergence, as it primarily
depends on road intersections for localization. Notably, in maps such as
‘CARLA4’ and ‘Real2’ where the road geometry remains consistently similar over
extended durations, the MapLite system experiences substantial delays in
achieving convergence. In contrast, ALT-Pilot, which relies on language
descriptions of landmarks along the way, has a higher likelihood of
encountering a unique set of landmarks, consequently leading to faster global
convergence. This observation highlights the significance of our approach,
particularly in urban environments with Manhattan’s grid-like road layout.
Further, the performance of both ALT-Pilot and ALT-Pilot(L) is comparable.

TABLE II: Distance(m) to Converge (lower = better) Scene | CARLA2 | CARLA4 | Real1 | Real2
---|---|---|---|---
MapLite | 52.45 | 120.45 | 46.99 | 73.1
ALT-Pilot(L) | 29.87 | 34.9 | 24.9 | 23.65
ALT-Pilot | 27.79 | 32.21 | 25.82 | 23.32

###  IV-D Closed Loop Reachability Results

Table III evaluates performance in closed-loop goal reachability experiments
across two CARLA maps. We classify goals as either ‘near intersection’ (within
30m of an intersection) or ‘not near intersection’. Using five random goals
for each category, we measure the distance between the vehicle’s final
position and the goal.

Capitalizing on language landmark descriptions, ALT-Pilot achieves superior
reachability reachability. For ‘near intersection’ goals, it’s on average
twice as accurate as MapLite. For ‘not near intersection’ goals, it surpasses
MapLite by six times in the ‘CARLA2’ map and ten times in the larger ‘CARLA1’
map.

TABLE III: Closed Loop Reachability. Distance(m) between the goal and the final vehicle position. Goal Category | CARLA1 | CARLA2
---|---|---
MapLite | ALT-Pilot | MapLite | ALT-Pilot
Near Intersection | 3.22 | 1.53 | 3.896 | 1.49
Not Near Intersection | 14.37 | 1.5675 | 11.24 | 1.76

###  IV-E Case Studies

False Positive and False Negative Landmarks: Table IV analyzes our system’s
resilience to erroneous landmarks, measured by false negatives (FN) and false
positives (FP) percentages. For x% FN, we simulate the omission of x% of
landmark descriptions from OSM, leaving the physical landmarks unchanged. For
x% FP, we modify x% of landmark descriptions, e.g., changing ”signboard” to
”fountain”. Conducted on the ‘CARLA1’ map (where MapLite had an APE of 10.3 as
in Table I), our system remains robust to 40%-50% erroneous landmarks. Beyond
this threshold, our performance begins to degrade. At around 80% erroneous
landmarks, our system’s performance is similar to MapLite, indicating our
robustness to gross landmark errors.

TABLE IV: False Positive and False Negative Landmarks. APE and Variance of ALT-Pilot for various levels of erroneous landmarks compared to MapLite on the same map. Erroneous Landmarks Type | False Negative | False Positive
---|---|---
| APE | Variance | APE | Variance
% of Erroneous Landmarks |  |  |  |
ALT-Pilot | 0 | 3.72 | 16.51 | 3.72 | 16.51
10 | 3.64 | 18.05 | 3.42 | 17.64
20 | 3.8 | 24.72 | 3.96 | 18.96
30 | 3.65 | 26.01 | 3.54 | 19.91
40 | 4.23 | 31.33 | 3.7 | 20.77
50 | 5.89 | 47.4 | 4.33 | 29.32
60 | 7.31 | 65.22 | 5.46 | 37.14
70 | 8.71 | 79.38 | 9.11 | 55.86
80 | 9.58 | 97.62 | 9.93 | 71.68
MapLite |  | APE | Variance
| 10.3 | 120.44

Real-world Language-goal Navigation: Topometric maps augmented with language
landmarks introduce a novel application of language-driven navigation.
Following a similar approach to [10], we built a goal selection function,
denoted as goal(‘description’), capable of planning a path to a goal based on
an open-ended goal description. An off-the-shelf LLM (GPT-4) is employed to
parse natural language goal descriptions into an appropriate function call.
For instance, a natural language request such as “Take me to a place where I
can sit” will be transformed into goal(“place where I can sit”). The goal
selection function will then identify the closest match among landmarks
(possibly a bench) corresponding to the description and plan a path to it.
Subsequently, the ALT-Pilot system will autonomously navigate to the specified
goal, as demonstrated in Figure 5.

Figure 5: Language Guided Navigation in Real World. Openset language goals can
be interpreted and correlated with the language landmarks to find a goal. The
ALT-Pilot system navigates to the goal autonomously.

##  V Conclusion

This paper presents a method for robust navigation and localization using
multi-modal maps. We demonstrated that off-the-shelf vision-language models
may be used to construct language-enhanced topometric maps, which can then be
consumed by ALT-Pilot for navigating to arbitrary destinations without the
need for HD LiDAR maps. We implement this on a fullsize real-world autonomous
car, demonstrating that our approach is both practical and performant. Future
work may focus on improving the efficiency of the navigation system, and also
automating map construction and updates so that all processing is carried out
on the vehicle with minimal human intervention. See our project page for more
details.

## References

  * [1] T. Ort, K. Murthy, R. Banerjee, S. K. Gottipati, D. Bhatt, I. Gilitschenski, L. Paull, and D. Rus, “Maplite: Autonomous intersection navigation without a detailed prior map,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 556–563, 2020.
  * [2] D. Pannen, M. Liebner, W. Hempel, and W. Burgard, “How to keep hd maps for automated driving up to date,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 2288–2294.
  * [3] T. Ort, L. Paull, and D. Rus, “Autonomous vehicle navigation in rural environments without detailed prior maps,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 2040–2047.
  * [4] M. Elhousni, Z. Zhang, and X. Huang, “Lidar-osm-based vehicle localization in gps-denied environments by using constrained particle filter,” _Sensors_ , vol. 22, no. 14, 2022. [Online]. Available: https://www.mdpi.com/1424-8220/22/14/5206
  * [5] S. Ninan and S. Rathinam, “Road descriptors for fast global localization on rural roads using openstreetmaps,” 2023.
  * [6] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Clip: Connecting vision and language with localized narratives,” 2021.
  * [7] B. Li, K. Q. Weinberger, S. J. Belongie, V. Koltun, and R. Ranftl, “Language-driven semantic segmentation,” _ArXiv_ , vol. abs/2201.03546, 2022\. [Online]. Available: https://api.semanticscholar.org/CorpusID:245836975
  * [8] G. Ghiasi, X. Gu, Y. Cui, and T.-Y. Lin, “Scaling open-vocabulary image segmentation with image-level labels,” in _European Conference on Computer Vision_. Springer, 2022, pp. 540–557.
  * [9] X. Dong, J. Bao, Y. Zheng, T. Zhang, D. Chen, H. Yang, M. Zeng, W. Zhang, L. Yuan, D. Chen _et al._ , “Maskclip: Masked self-distillation advances contrastive language-image pretraining,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 10 995–11 005.
  * [10] K. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, S. Li, G. Iyer, S. Saryazdi, N. Keetha, A. Tewari, J. Tenenbaum, C. de Melo, M. Krishna, L. Paull, F. Shkurti, and A. Torralba, “Conceptfusion: Open-set multimodal 3d mapping,” _RSS_ , 2023.
  * [11] C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , London, UK, 2023.
  * [12] N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam, “Clip-fields: Weakly supervised semantic fields for robotic memory,” _RSS_ , 2022.
  * [13] J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik, “Lerf: Language embedded radiance fields,” _ICCV_ , 2023.
  * [14] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in _Proceedings of the 1st Annual Conference on Robot Learning_ , 2017, pp. 1–16.
  * [15] M. Werling, J. Ziegler, S. Kammel, and S. Thrun, “Optimal trajectory generation for dynamic street scenarios in a frenét frame,” in _2010 IEEE International Conference on Robotics and Automation_ , 2010, pp. 987–993.
  * [16] E. Horváth, C. Pozna, and M. Unger, “Real-time lidar-based urban road and sidewalk detection for autonomous vehicles,” _Sensors_ , vol. 22, no. 1, 2022\. [Online]. Available: https://www.mdpi.com/1424-8220/22/1/194
  * [17] S. Thrun, “Probabilistic robotics,” _Communications of the ACM_ , vol. 45, no. 3, pp. 52–57, 2002.
  * [18] D. Fox, W. Burgard, F. Dellaert, and S. Thrun, “Monte carlo localization: Efficient position estimation for mobile robots,” in _Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI’99)_ , 1999, pp. 343–349.
  * [19] D. Fox, “Adapting the sample size in particle filters through kld-sampling,” _International Journal of Robotics Research_ , vol. 22, no. 12, pp. 985–1003, 2003.
  * [20] T. Shan and B. Englot, “Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 4758–4765.
  * [21] F. J. Richards, “A flexible growth function for empirical use,” _Journal of experimental Botany_ , vol. 10, no. 2, pp. 290–301, 1959.
  * [22] P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic determination of minimum cost paths,” _IEEE Transactions on Systems Science and Cybernetics_ , vol. 4, no. 2, pp. 100–107, 1968.
  * [23] G. M. Hoffmann, C. J. Tomlin, M. Montemerlo, and S. Thrun, “Autonomous automobile trajectory tracking for off-road driving: Controller design, experimental validation and racing,” in _2007 American Control Conference_ , 2007, pp. 2296–2301.