#  ${}_{\\!\\!\\!\\!\\!\\!}$Lana: A Language-Capable Navigator for Instruction
Following and Generation${}_{\\!\\!\\!\\!\\!\\!}$

Xiaohan Wang Wenguan Wang Jiayi Shao Yi Yang
ReLER, CCAI, Zhejiang University
https://github.com/wxh1996/LANA-VLN Corresponding author: Yi Yang.

###### Abstract

Recently, visual-language navigation (VLN) – entailing robot agents to follow
navigation instructions – has shown great${}_{\\!}$ advance.${}_{\\!}$
However,${}_{\\!}$ existing${}_{\\!}$ literature${}_{\\!}$ put${}_{\\!}$
most${}_{\\!}$ empha- sis on interpreting instructions into actions, only
delivering${}_{\\!}$ “dumb”${}_{\\!}$ wayfinding${}_{\\!}$ agents.${}_{\\!}$
In${}_{\\!}$ this${}_{\\!}$ article,${}_{\\!}$ we${}_{\\!}$ devise${}_{\\!}$
Lana, a${}_{\\!}$ language-capable${}_{\\!}$ navigation${}_{\\!}$
agent${}_{\\!}$ which${}_{\\!}$ is${}_{\\!}$ able${}_{\\!}$ to${}_{\\!}$ not
only execute human-written navigation commands, but also provide${}_{\\!}$
route${}_{\\!}$ descriptions${}_{\\!}$ to${}_{\\!}$ humans.${}_{\\!}$
This${}_{\\!}$ is${}_{\\!}$ achieved${}_{\\!}$ by${}_{\\!}$ si-
multaneously${}_{\\!}$ learning${}_{\\!}$ instruction${}_{\\!}$
following${}_{\\!}$ and${}_{\\!}$ generation with${}_{\\!}$ only${}_{\\!}$
one${}_{\\!}$ single${}_{\\!}$ model.${}_{\\!}$ More${}_{\\!}$
specifically,${}_{\\!}$ two${}_{\\!}$ encoders, respectively${}_{\\!}$
for${}_{\\!}$ route${}_{\\!}$ and${}_{\\!}$ language${}_{\\!}$
encoding,${}_{\\!}$ are${}_{\\!}$ built${}_{\\!}$ and shared${}_{\\!}$
by${}_{\\!}$ two${}_{\\!}$ decoders,${}_{\\!}$ respectively${}_{\\!}$
for${}_{\\!}$ action${}_{\\!}$ prediction and${}_{\\!}$ instruction${}_{\\!}$
generation,${}_{\\!}$ so${}_{\\!}$ as${}_{\\!}$ to${}_{\\!}$ exploit${}_{\\!}$
cross-task${}_{\\!}$ know- ledge${}_{\\!}$ and${}_{\\!}$ capture${}_{\\!}$
task-specific${}_{\\!}$ characteristics.${}_{\\!}$ Throughout pretraining and
fine-tuning, both instruction following and generation${}_{\\!}$ are${}_{\\!}$
set${}_{\\!}$ as${}_{\\!}$ optimization${}_{\\!}$ objectives.${}_{\\!}$
We${}_{\\!}$ empirically verify that, compared with recent advanced task-
specific solutions,${}_{\\!}$ Lana${}_{\\!}$ attains${}_{\\!}$
better${}_{\\!}$ performances on both instruction following and route
description, with nearly half complexity. In addition, endowed with language
generation capability,${}_{\\!}$ Lana${}_{\\!}$ can${}_{\\!}$
explain${}_{\\!}$ to${}_{\\!}$ human${}_{\\!}$ its${}_{\\!}$
behaviours${}_{\\!}$ and${}_{\\!}$ as- sist${}_{\\!}$ human’s${}_{\\!}$
wayfinding.${}_{\\!}$ This${}_{\\!}$ work${}_{\\!}$ is${}_{\\!}$
expected${}_{\\!}$ to${}_{\\!}$ foster${}_{\\!}$ fu- ture${}_{\\!}$
efforts${}_{\\!}$ towards${}_{\\!}$ building${}_{\\!}$ more${}_{\\!}$
trustworthy${}_{\\!}$ and${}_{\\!}$ socially-intelligent navigation robots.

##  1 Introduction

Developing agents that can interact with humans in natural language while
perceiving and taking actions in their environments is one of the fundamental
goals in artificial intel- ligence.${}_{\\!}$ As${}_{\\!}$ a${}_{\\!}$
small${}_{\\!}$ step${}_{\\!}$ towards${}_{\\!}$ this${}_{\\!}$
target,${}_{\\!}$ visual-language navigation${}_{\\!}$ (VLN)${}_{\\!}$ [4] –
endowing agents to execute natural language navigation commands – recently
received signifi- cant attention. In VLN space, much work has been done on
language grounding – teaching agents how to relate human instructions with
actions associated with perceptions. How- ever,${}_{\\!}$ there${}_{\\!}$
has${}_{\\!}$ been${}_{\\!}$ far${}_{\\!}$ little${}_{\\!}$ work${}_{\\!}$
[27, 71, 1, 78, 23] on${}_{\\!}$ the${}_{\\!}$ reverse${}_{\\!}$
side${}_{\\!}$ –${}_{\\!}$ language${}_{\\!}$ generation${}_{\\!}$ –${}_{\\!}$
teaching${}_{\\!}$ agents how${}_{\\!}$ to verbalize${}_{\\!}$ a${}_{\\!}$
vivid${}_{\\!}$ description${}_{\\!}$ of${}_{\\!}$ navigation${}_{\\!}$
routes.${}_{\\!}$ More${}_{\\!}$ criti- cally, existing VLN literature
separately train agents that are specialized${}_{\\!}$ for${}_{\\!}$
each${}_{\\!}$ single${}_{\\!}$ task.${}_{\\!}$ As${}_{\\!}$ a${}_{\\!}$
result,${}_{\\!}$ the${}_{\\!}$ delivered agents${}_{\\!}$ are either strong
wayfinding actors but never talking, or conversable route instructors but
never walking.

Figure 1: Lana is capable of both instruction following and generation. Its
written report benefits human-robot collaboration, and, to some extend, can
explain its behavior: it takes a wrong action at step 2 as it mistakes the
dining room for bedroom. After gathering more information at step 3, it
changes to the correct direction.

This${}_{\\!}$ article${}_{\\!}$ underlines${}_{\\!}$ a${}_{\\!}$
fundamental${}_{\\!}$ challenge${}_{\\!}$ in${}_{\\!\\!}$ VLN: Can we learn a
single agent that is capable of both navigation instruction following and
route description creation?

We${}_{\\!}$ propose${}_{\\!}$ Lana,${}_{\\!}$ a${}_{\\!}$ language-
capable${}_{\\!}$ navigation${}_{\\!}$ agent, that${}_{\\!}$ is${}_{\\!}$
fully${}_{\\!}$ aware${}_{\\!}$ of${}_{\\!}$ such${}_{\\!}$
challenge${}_{\\!}$ (Fig.${}_{\\!\\!}$ 1).${}_{\\!}$ By${}_{\\!}$ simultane-
ously${}_{\\!}$ learning${}_{\\!}$ instruction${}_{\\!}$ grounding${}_{\\!}$
and${}_{\\!}$ generation,${}_{\\!}$ Lana formalises human-to-robot${}_{\\!}$
and${}_{\\!}$ robot-to-human${}_{\\!}$ communi- cation,${}_{\\!}$
conveyed${}_{\\!}$ using${}_{\\!}$ navigation-oriented${}_{\\!}$
natural${}_{\\!}$ language,${}_{\\!\\!}$ in${}_{\\!}$ a unified${}_{\\!}$
framework.${}_{\\!}$ This${}_{\\!}$ is${}_{\\!}$ of${}_{\\!}$ great${}_{\\!}$
importance,${}_{\\!}$ because: i)${}_{\\!}$ It completes${}_{\\!}$
the${}_{\\!}$ necessary${}_{\\!}$ communication${}_{\\!}$ cycle${}_{\\!}$
between human${}_{\\!}$ and${}_{\\!}$ agents,${}_{\\!}$ and${}_{\\!}$
promotes${}_{\\!}$ VLN${}_{\\!}$ agent’s real-world utility${}_{\\!}$ [59].
For instance, when an agent takes long time to execute a navigation command,
during which sustained human attention is infeasible and undesirable, the
agent should report${}_{\\!}$ its${}_{\\!}$ progress${}_{\\!}$ [73].${}_{\\!}$
Also,${}_{\\!}$ agents${}_{\\!}$ are${}_{\\!}$ expected${}_{\\!}$ to${}_{\\!}$
direct human${}_{\\!}$ in${}_{\\!}$ agents’${}_{\\!}$ explored${}_{\\!}$
areas${}_{\\!}$ [82], which is relevant for search and rescue robots in
disaster regions${}_{\\!}$ [72, 19], guide robots in public spaces${}_{\\!}$
[78], and navigation devices for the visually impaired${}_{\\!}$ [36]. ii)
Two-way communication is integral to tight human-robot coordination (_i.e_.,
“I will continue this way $\cdots$”) [7], and boosts human trust in robot [6,
24], hence${}_{\\!}$ increasing${}_{\\!}$ the${}_{\\!}$ acceptance${}_{\\!}$
of${}_{\\!}$ navigation${}_{\\!}$ robots.${}_{\\!}$ iii)${}_{\\!}$ Developing
the language generation skill makes for more explainable robots, which can
interpret their navigation beha- viors in a form of human-readable route
descriptions.

${}_{\\!}$Technically,${}_{\\!}$ Lana${}_{\\!}$ is${}_{\\!}$ built${}_{\\!}$
as${}_{\\!}$ a${}_{\\!}$ Transformer-based,${}_{\\!}$ multi-task${}_{\\!}$
learning${}_{\\!}$ framework.${}_{\\!}$ The${}_{\\!}$ network${}_{\\!}$
consists${}_{\\!}$ of${}_{\\!}$ two${}_{\\!}$ uni-modal${}_{\\!}$
encoders${}_{\\!}$ respectively${}_{\\!}$ for${}_{\\!}$ language${}_{\\!}$
and${}_{\\!}$ route${}_{\\!}$ encod- ing,${}_{\\!}$ and${}_{\\!}$
two${}_{\\!}$ multimodal${}_{\\!}$ decoders${}_{\\!}$ respectively${}_{\\!}$
for${}_{\\!}$ route-to-instruction and instruction-to-route translation, based
on the${}_{\\!}$ two${}_{\\!}$ encoders.${}_{\\!}$ The${}_{\\!}$
whole${}_{\\!}$ network${}_{\\!}$ is${}_{\\!}$ end-to-end${}_{\\!}$ learned
with the tasks of both instruction grounding and generation, during both
pretraining and fine-tuning phases. Taken all these together,${}_{\\!}$
Lana${}_{\\!}$ provides${}_{\\!}$ a${}_{\\!}$ unified,${}_{\\!}$
powerful${}_{\\!}$ framework${}_{\\!}$ that explores both${}_{\\!}$ task-
specific${}_{\\!}$ and${}_{\\!}$ cross-task${}_{\\!}$ knowledge${}_{\\!}$
at${}_{\\!}$ the heart of model design and network training. Lana${}_{\\!}$
thus${}_{\\!}$ can better${}_{\\!}$ comprehend${}_{\\!}$ linguistic${}_{\\!}$
cues${}_{\\!}$ (_e.g_.,${}_{\\!}$ words,${}_{\\!}$ phrases,${}_{\\!}$ and
sentences), visual perceptions, actions over long temporal horizons and their
relationships, even in the absence of ex- plicit${}_{\\!}$
supervision,${}_{\\!}$ and${}_{\\!}$ eventually${}_{\\!}$ benefits${}_{\\!}$
both${}_{\\!}$ the${}_{\\!}$ two${}_{\\!}$ tasks.

We conduct extensive experiments on three famous VLN datasets (_i.e_.,
R2R${}_{\\!}$ [4], R4R${}_{\\!}$ [38], REVERIE${}_{\\!}$ [63]), for both
instruction${}_{\\!}$ following${}_{\\!}$ and${}_{\\!}$ generation,${}_{\\!}$
giving${}_{\\!}$ a${}_{\\!}$ few${}_{\\!}$ intriguing points:${}_{\\!}$
First,${}_{\\!}$ Lana${}_{\\!}$ successfully${}_{\\!}$ solves${}_{\\!}$
the${}_{\\!}$ two${}_{\\!}$ tasks${}_{\\!}$ using only one single agent,
without switching between different models.${}_{\\!}$ Second,${}_{\\!}$
with${}_{\\!}$ an${}_{\\!}$ elegant${}_{\\!}$ and${}_{\\!}$
integrated${}_{\\!}$ architecture, Lana${}_{\\!}$ performs${}_{\\!}$
comparable,${}_{\\!}$ or even${}_{\\!}$ better${}_{\\!}$ than${}_{\\!}$
recent${}_{\\!}$ top-leading,${}_{\\!}$ task-specific${}_{\\!}$
alternatives.${}_{\\!}$ Third,${}_{\\!}$ compared${}_{\\!}$ to${}_{\\!}$
learn- ing each task individually, training Lana on the two tasks
jointly${}_{\\!}$ obtains${}_{\\!}$ better${}_{\\!}$ performance${}_{\\!}$
with${}_{\\!}$ reduced${}_{\\!}$ complexity and model size, confirming the
advantage of Lana in cross-task${}_{\\!}$ relatedness${}_{\\!}$
modeling${}_{\\!}$ and${}_{\\!}$ parameter${}_{\\!}$ efficiency.${}_{\\!}$
Forth, Lana${}_{\\!}$ can${}_{\\!}$ explain${}_{\\!}$ to${}_{\\!}$
human${}_{\\!}$ its${}_{\\!}$ behavior${}_{\\!}$ by${}_{\\!}$
verbally${}_{\\!}$ describing its navigation routes. Lana can be essentially
viewed as an explainable VLN robot, equipped with a self-adaptively trained
language explainer. Fifth, subjective analyses reveal our linguistic outputs
are of higher quality than the baselines but still lag behind human-
generated${}_{\\!}$ utterances.${}_{\\!}$ While${}_{\\!}$ there is${}_{\\!}$
still${}_{\\!}$ room${}_{\\!}$ for${}_{\\!}$ improvement,${}_{\\!}$
our${}_{\\!}$ results${}_{\\!}$ shed${}_{\\!}$ light${}_{\\!}$ on${}_{\\!}$
a${}_{\\!}$ pro- mising${}_{\\!}$ direction${}_{\\!}$ of${}_{\\!}$
future${}_{\\!}$ VLN${}_{\\!}$ research,${}_{\\!}$ with${}_{\\!}$
great${}_{\\!}$ poten- tial${}_{\\!}$ for${}_{\\!}$ explainable${}_{\\!}$
navigation${}_{\\!}$ agents${}_{\\!}$ and${}_{\\!}$ robot${}_{\\!}$
applications.${}_{\\!}$

##  2 Related Work

Navigation Instruction Following. Building autonomous, language-based
navigation agents is a long-standing target for natural language processing
and robotics communities. Rather than previous studies bounded to controlled
environ- mental${}_{\\!}$ context${}_{\\!}$ [55, 72, 10, 5, 57],${}_{\\!}$
Anderson${}_{\\!}$  _et al_. ${}_{\\!}$[4]${}_{\\!}$ lift such${}_{\\!}$
task${}_{\\!}$ to${}_{\\!}$ a${}_{\\!}$ photo-realistic${}_{\\!}$
setting${}_{\\!}$ –${}_{\\!}$ VLN,${}_{\\!}$ stimulating${}_{\\!}$ in-
creasing${}_{\\!}$ interest${}_{\\!}$ in${}_{\\!}$ computer${}_{\\!}$
vision${}_{\\!}$ field.${}_{\\!}$ Early${}_{\\!}$ efforts${}_{\\!}$ were
built${}_{\\!}$ upon${}_{\\!}$ recurrent${}_{\\!}$ neural${}_{\\!}$
networks.${}_{\\!}$ They${}_{\\!}$ explore${}_{\\!}$ diverse${}_{\\!}$
training${}_{\\!}$ strategies${}_{\\!}$ [84, 83],${}_{\\!}$ mine${}_{\\!}$
extra${}_{\\!}$ supervisory${}_{\\!}$ signals from synthesized
samples${}_{\\!}$ [27, 71, 28] or auxiliary tasks [83, 35, 53, 93,
78],${}_{\\!}$ and${}_{\\!}$ explore${}_{\\!}$ intelligent${}_{\\!}$
path${}_{\\!}$ planning${}_{\\!}$ [39, 54, 81].${}_{\\!}$ For${}_{\\!}$
structured${}_{\\!}$ and${}_{\\!}$ long-range${}_{\\!}$ context${}_{\\!}$
modeling, recent${}_{\\!}$ solutions${}_{\\!}$ were${}_{\\!}$
developed${}_{\\!}$ with${}_{\\!}$ environment map [92, 13, 21, 80],${}_{\\!}$
transformer${}_{\\!}$ architectures${}_{\\!}$ [33, 61, 48, 64, 11], and multi-
modal pretraining [56, 31, 30, 12].

Unlike existing VLN solutions that are all specialized for navigation
instruction following, we are ambitious to build a powerful agent that is able
to not only execute navigation in- structions but also describe its navigation
routes. We stick to this target throughout our algorithm – from network
design, to model pretraining, to fine-tuning. Through jointly learning
instruction execution and generation, our agent can bet- ter${}_{\\!}$
ground${}_{\\!}$ instructions${}_{\\!}$ into${}_{\\!}$ perception${}_{\\!}$
and${}_{\\!}$ action,${}_{\\!}$ and,${}_{\\!}$ to${}_{\\!}$ certain degree,
interpret its behavior and foster human trust. Our target and visual-dialog
navigation [73] are different (yet complementary), as the latter only focuses
on the situation where agents use language to ask for human assistance.

Navigation${}_{\\!}$ Instruction${}_{\\!}$ Generation.${}_{\\!}$ The${}_{\\!}$
study${}_{\\!}$ of${}_{\\!}$ instruc- tion${}_{\\!}$ creation${}_{\\!}$
[17]${}_{\\!}$ can${}_{\\!}$ date${}_{\\!}$ back${}_{\\!}$ to${}_{\\!}$
the${}_{\\!}$ 1960s${}_{\\!}$ [52].${}_{\\!}$ Early${}_{\\!}$ work [88, 2,
51]${}_{\\!}$ found${}_{\\!}$ human${}_{\\!}$ route${}_{\\!}$
direction${}_{\\!}$ is${}_{\\!}$ tied${}_{\\!}$ to${}_{\\!}$
cognitive${}_{\\!}$ map${}_{\\!}$ [42], and impacted${}_{\\!}$ by${}_{\\!}$
many${}_{\\!}$ factors,${}_{\\!}$ _e.g_.,${}_{\\!}$ cultural${}_{\\!}$ back-
ground${}_{\\!}$ [74]${}_{\\!}$ and${}_{\\!}$ gender${}_{\\!}$ [37].${}_{\\!}$
They${}_{\\!}$ also${}_{\\!}$ reached${}_{\\!}$ a${}_{\\!}$ consensus that
involving turn-by-turn directions and salient landmarks makes instructions
easier for human to follow [50, 77, 67]. Based on these efforts, a few
computational systems are developed using pre-built templates${}_{\\!}$ [50,
29], or hand-crafted rules${}_{\\!}$ [18]. Though providing high-quality
output in targeted scenarios, they${}_{\\!}$ require${}_{\\!}$
expertise${}_{\\!}$ of${}_{\\!}$ linguistic${}_{\\!}$ knowledge${}_{\\!}$ and
extensive${}_{\\!}$ effort${}_{\\!}$ for${}_{\\!}$ building${}_{\\!}$
the${}_{\\!}$ templates/rules.${}_{\\!}$ Some${}_{\\!}$ data-driven solutions
[16, 59, 19, 26] emerged later, yet confined to${}_{\\!}$ simplified${}_{\\!}$
grid-like${}_{\\!}$ or${}_{\\!}$ perception-poor${}_{\\!}$ environments.

Generating${}_{\\!}$ natural${}_{\\!}$ language${}_{\\!}$
instructions${}_{\\!}$ has${}_{\\!}$ long${}_{\\!}$ been viewed${}_{\\!}$ as a
core functionality of socially intelligent robots and${}_{\\!}$ been${}_{\\!}$
of${}_{\\!}$ great${}_{\\!}$ interest${}_{\\!}$ in${}_{\\!}$ many${}_{\\!}$
disciplines${}_{\\!}$ such${}_{\\!}$ as${}_{\\!}$ robo- tics${}_{\\!}$
[29],${}_{\\!}$ linguistics${}_{\\!}$ [69],${}_{\\!}$ cognition [42,
25],${}_{\\!}$ psychology${}_{\\!}$ [74],${}_{\\!}$ and${}_{\\!}$
geo${}_{\\!}$ science${}_{\\!}$ [20]. Surprisingly little has been done in the
field${}_{\\!}$ of${}_{\\!}$ embodied${}_{\\!}$ vision.${}_{\\!}$
For${}_{\\!}$ the${}_{\\!}$ rare${}_{\\!}$ exceptions${}_{\\!}$ [27, 71, 68,
1, 78, 23],${}_{\\!}$ [27, 71, 68]${}_{\\!}$ are${}_{\\!}$ only${}_{\\!}$
to${}_{\\!}$ augment${}_{\\!}$ the${}_{\\!}$ training${}_{\\!}$ data
for${}_{\\!}$ boosting${}_{\\!}$ wayfinding,${}_{\\!}$ and,${}_{\\!}$
all${}_{\\!}$ of${}_{\\!}$ them${}_{\\!}$ learn${}_{\\!}$ a${}_{\\!}$ single
agent specialized${}_{\\!}$ for${}_{\\!}$ instruction${}_{\\!}$
generation.${}_{\\!}$ Our${}_{\\!}$ idea${}_{\\!}$ is${}_{\\!}$ fundamen-
tally${}_{\\!}$ different.${}_{\\!}$ We${}_{\\!}$ are${}_{\\!}$ to${}_{\\!}$
build${}_{\\!}$ a${}_{\\!}$ language-capable${}_{\\!}$ navigation
agent${}_{\\!}$ that${}_{\\!}$ masters${}_{\\!}$ both${}_{\\!}$
instruction${}_{\\!}$ following${}_{\\!}$ and${}_{\\!}$ creation. As a result,
this work represents an early yet solid attempt towards socially intelligent,
embodied navigation robots.

$\mathcal{E}^{r}$$\mathcal{E}^{l}$$\mathcal{D}^{r}$$\mathcal{D}^{l}$

Figure 2: Architecture overview of Lana (§3.1).

Auxiliary${}_{\\!}$ Learning${}_{\\!}$ in${}_{\\!}$ VLN.${}_{\\!}$
There${}_{\\!}$ are${}_{\\!}$ several${}_{\\!}$ VLN${}_{\\!}$ solu-
tions${}_{\\!}$ [53, 93, 79]${}_{\\!}$ exploit${}_{\\!}$ extra${}_{\\!}$
supervision${}_{\\!}$ signals${}_{\\!}$ from${}_{\\!}$ auxi- liary${}_{\\!}$
tasks${}_{\\!}$ to${}_{\\!}$ aid${}_{\\!}$ navigation${}_{\\!}$
policy${}_{\\!}$ learning.${}_{\\!}$ For${}_{\\!}$ the${}_{\\!}$ auxiliary
tasks,${}_{\\!}$ representative${}_{\\!}$ ones${}_{\\!}$ include${}_{\\!}$
next-step orientation reg- ression${}_{\\!}$ [93],${}_{\\!}$
navigation${}_{\\!}$ progress${}_{\\!}$ estimation${}_{\\!}$ [53],${}_{\\!}$
path${}_{\\!}$ back-translation${}_{\\!}$ [93, 78], trajectory-
instruction${}_{\\!}$ compatibility${}_{\\!}$ prediction [93], as well as
final target localization${}_{\\!}$ [92].

These${}_{\\!}$ VLN solutions put focus on instruction following;
the${}_{\\!}$ auxiliary${}_{\\!}$ tasks${}_{\\!}$ are${}_{\\!}$ the${}_{\\!}$
means,${}_{\\!}$ not${}_{\\!}$ the${}_{\\!}$ end.${}_{\\!}$ By${}_{\\!}$
contrast, we aim to build one single agent that learns to master
both${}_{\\!}$ instruction${}_{\\!}$ following${}_{\\!}$ and${}_{\\!}$
creation${}_{\\!}$ well.${}_{\\!\\!}$ Although${}_{\\!}$ [78]${}_{\\!}$ pays
equal attention to instruction following and generation under a dual-task
learning scheme, it still learns two separate single-task agents. Moreover,
the aforementioned auxiliary tasks, in principle, can be utilized by our
agent.

Vision-Language${}_{\\!}$ Pretraining${}_{\\!}$ for${}_{\\!}$ VLN.${}_{\\!}$
Vision-language${}_{\\!}$ pretraining [65, 70, 14] on massive-scale image-text
pairs has${}_{\\!}$ recently${}_{\\!}$ witnessed${}_{\\!}$ rapid${}_{\\!}$
progress.${}_{\\!}$ It${}_{\\!}$ has${}_{\\!}$ been${}_{\\!}$ proven${}_{\\!}$
that transferable${}_{\\!}$ cross-modal${}_{\\!}$ representations${}_{\\!}$
can${}_{\\!}$ be${}_{\\!}$ delivered${}_{\\!}$ via such${}_{\\!}$
pretraining${}_{\\!}$ and${}_{\\!}$ facilitate${}_{\\!}$ downstream${}_{\\!}$
tasks [85, 47, 91, 65, 44, 46].${}_{\\!}$ Such${}_{\\!}$ training${}_{\\!}$
regime${}_{\\!}$ has${}_{\\!}$ become${}_{\\!}$ increasingly${}_{\\!}$
popular${}_{\\!}$ in${}_{\\!}$ VLN.${}_{\\!}$ In${}_{\\!}$
particular,${}_{\\!}$ a${}_{\\!}$ few${}_{\\!}$ early${}_{\\!}$
endeavors${}_{\\!}$ [45, 33] directly${}_{\\!}$ adopt general${}_{\\!}$
vision-language${}_{\\!}$ pretraining${}_{\\!}$ for${}_{\\!}$ VLN,${}_{\\!}$
without considering task-specific nature. Later, [30, 31, 12] conduct
pretraining on abundant web image-captions [30] or synthesized trajectory-
instruction pairs [31, 12] with different VLN-specific proxy tasks. [11, 64]
introduce history-aware proxy tasks for more VLN-aligned pretraining.

From the view of proxy task, existing VLN pretraining follows${}_{\\!}$
the${}_{\\!}$ masked${}_{\\!}$ language${}_{\\!}$ modeling${}_{\\!}$
regime${}_{\\!}$ [40].${}_{\\!}$ Differ- ently,${}_{\\!}$ our${}_{\\!}$
pretraining${}_{\\!}$ is${}_{\\!}$ based${}_{\\!}$ on${}_{\\!}$
language${}_{\\!}$ generation,${}_{\\!}$ which helps${}_{\\!}$ the${}_{\\!}$
agent${}_{\\!}$ to${}_{\\!}$ capture${}_{\\!}$ the${}_{\\!}$
linguistic${}_{\\!}$ structures${}_{\\!}$ so${}_{\\!}$ as${}_{\\!}$
to${}_{\\!}$ reach${}_{\\!}$ comprehensive${}_{\\!}$ understanding${}_{\\!}$
of${}_{\\!}$ language${}_{\\!}$ commands and${}_{\\!}$ boost${}_{\\!}$
instruction${}_{\\!}$ execution.${}_{\\!}$ Recent${}_{\\!}$ advance${}_{\\!}$
[87, 90, 22, 86] in${}_{\\!}$ general${}_{\\!}$ vision-language${}_{\\!}$
pretraining${}_{\\!}$ also${}_{\\!}$ confirm${}_{\\!}$ the${}_{\\!}$ value
of${}_{\\!}$ generative${}_{\\!}$ language${}_{\\!}$ modeling.${}_{\\!}$
Moreover,${}_{\\!}$ for${}_{\\!}$ Lana,${}_{\\!}$ instruction${}_{\\!}$
generation is not merely a proxy task that is often dropped after pretraining,
but also a main training target during fine-tuning, and the fundamental basis
for the ability of language-based route direction during deployment.

##  3 Methodology

Task${}_{\\!}$ Setup.${}_{\\!}$ Our target is to build a language-capable
navi- gation agent, which masters${}_{\\!}$ both${}_{\\!}$
instruction${}_{\\!}$ following${}_{\\!}$ and generation${}_{\\!}$
tasks,${}_{\\!}$ using${}_{\\!}$ one single model instance only.

  * •

Instruction following:${}_{\\!}$ the agent needs to find a route $R\\!=$
$\\!\\{r_{t}\\}^{T}_{t=1\\!\\!}$ of${}_{\\!}$ $T_{\\!}$ steps${}_{\\!}$
to${}_{\\!}$ a${}_{\\!}$ target${}_{\\!}$ location,${}_{\\!}$
following${}_{\\!}$ a${}_{\\!}$ human-written instruction
$X\\!=\\!\\{x_{l}\\}^{L}_{l=1\\!}$ of $L$ words. At each step $t$,${}_{\\!}$
the${}_{\\!}$ agent${}_{\\!}$ gets${}_{\\!}$ a${}_{\\!}$ panoramic${}_{\\!}$
RGB${}_{\\!}$ percept${}_{\\!}$ ${O}^{t}$,${}_{\\!}$ discretized
into${}_{\\!}$ $K_{\\!}\\!=\\!36_{\\!}$ views,${}_{\\!}$ _i.e_.,${}_{\\!}$
${O}_{t\\!}\\!=\\!\\{o_{t,k}\\!\in\\!\mathbb{R}^{3\\!\times\\!224\\!\times\\!224\\!}\\}^{K}_{k=1}$.${}_{\\!}$
The agent${}_{\\!}$ selects${}_{\\!}$ a${}_{\\!}$ navigation${}_{\\!}$
action${}_{\\!}$ $a_{t}$,${}_{\\!}$ _i.e_.,${}_{\\!}$ a${}_{\\!}$
navigable${}_{\\!}$ view,${}_{\\!}$ from${}_{\\!}$ $\\!\\{o_{t,k}\\}_{k\\!}$
to${}_{\\!}$ perform. Together, ${O}_{t}$ and $a_{t}$ determine the route
state $r_{t}$ at step $t$, _i.e_., $r_{t}\\!=\\!(O_{t},a_{t})$.

  * •

Instruction${}_{\\!}$ generation:${}_{\\!}$ the${}_{\\!}$ agent${}_{\\!}$
observes${}_{\\!}$ a${}_{\\!}$ navigation route${}_{\\!}$
$R_{\\!}\\!=_{\\!}\\!\\{r_{t}\\}^{T}_{t=1}$,${}_{\\!}$ _i.e_.,${}_{\\!}$
a${}_{\\!}$ sequence${}_{\\!}$ of${}_{\\!}$ actions${}_{\\!}$
$\\{a_{t}\\}^{T}_{t=1\\!}$ along with the panoramic percept
$\\{O_{t}\\}^{T}_{t=1}$, and must verbalize a grounded description
$X\\!=\\!\\{x_{l}\\}^{L}_{l=1\\!}$ of the route $R$.

Method${}_{\\!}$ Overview.${}_{\\!}$ To${}_{\\!}$ address${}_{\\!}$
our${}_{\\!}$ challenging${}_{\\!}$ setting,${}_{\\!}$ we devise Lana, a
Transformer-based, multi-task VLN agent (Fig. 2) that exploits cross-task
commonalities throughout architecture design (§3.1) and network training
(§3.2).

###  3.1 Model Architecture

${}_{\\!}$Lana${}_{\\!}$ can${}_{\\!}$ take${}_{\\!}$ the${}_{\\!}$
navigation${}_{\\!}$ route${}_{\\!}$ $R_{\\!}\\!=_{\\!}\\!\\{r_{t}\\}_{t\\!}$
as${}_{\\!}$ input${}_{\\!}$ and output${}_{\\!}$ corresponding${}_{\\!}$
description${}_{\\!}$ $X_{\\!}\\!=_{\\!}\\!\\{x_{l}\\}_{l}$,${}_{\\!}$
and${}_{\\!}$ vice${}_{\\!}$ versa. To${}_{\\!}$ achieve${}_{\\!}$
such${}_{\\!}$ bi-directional${}_{\\!}$ translation${}_{\\!}$
between${}_{\\!}$ navigation route${}_{\\!}$ $R_{\\!}$ and${}_{\\!}$
instruction${}_{\\!}$ $X$, and explore cross-task rela- tedness, Lana is
elaborately designed as a composition of:

  * •

Route${}_{\\!}$ Encoder${}_{\\!}$ $\mathcal{E}^{r\\!}$ and${}_{\\!}$
Language${}_{\\!}$ Encoder${}_{\\!}$ $\mathcal{E}^{l\\!}$ for${}_{\\!}$
unimoda- lity representation encoding based on self-attention; and

  * •

Language${}_{\\!}$ Decoder${}_{\\!}$ $\mathcal{D}^{l\\!}$ and${}_{\\!}$
Route${}_{\\!}$ Decoder${}_{\\!}$ $\mathcal{D}^{r\\!}$ for${}_{\\!}$ cross-
attention${}_{\\!}$ based${}_{\\!}$ route-language${}_{\\!}$ bi-
directional${}_{\\!}$ translation.

The${}_{\\!}$ two${}_{\\!}$ unimodal${}_{\\!}$ encoders${}_{\\!}$
are${}_{\\!}$ shared${}_{\\!}$ between${}_{\\!}$ and${}_{\\!}$ jointly
trained${}_{\\!}$ with${}_{\\!}$ the${}_{\\!}$ two${}_{\\!}$
multimodal${}_{\\!}$ decoders.${}_{\\!}$ They${}_{\\!}$ work${}_{\\!}$
closely:

  * •

Instruction${}_{\\!}$ Following:${}_{\\!}$ at${}_{\\!}$ navigation${}_{\\!}$
step${}_{\\!}$ $t$,${}_{\\!}$ Lana${}_{\\!}$ respectively feeds${}_{\\!}$
the${}_{\\!}$ entire${}_{\\!}$ instruction${}_{\\!}$ $X_{\\!}$ and${}_{\\!}$
the${}_{\\!}$ sequence${}_{\\!}$ of historical${}_{\\!}$ route${}_{\\!}$
states${}_{\\!}$ $\\{r_{1},\cdots_{\\!},r_{t-1}\\}_{\\!}$ and
current${}_{\\!}$ percept $O_{t}$ into${}_{\\!}$ the${}_{\\!}$
corresponding${}_{\\!}$ encoders,${}_{\\!}$ and${}_{\\!}$ utilizes${}_{\\!}$
the${}_{\\!}$ route${}_{\\!}$ decoder $\mathcal{D}^{r\\!}$ to predict the
navigation action $a_{t}$.

  * •

Instruction${}_{\\!}$ Generation:${}_{\\!}$ at${}_{\\!}$ generation${}_{\\!}$
step${}_{\\!}$ $l$, Lana${}_{\\!}$ respec- tively${}_{\\!}$ feeds${}_{\\!}$
the${}_{\\!}$ full${}_{\\!}$ route${}_{\\!}$ $R_{\\!}$ and${}_{\\!}$
prior${}_{\\!}$ predicted${}_{\\!}$ words${}_{\\!}$
$\\{x_{1},\cdots_{\\!},x_{l-1}\\}_{\\!}$ into${}_{\\!}$ the${}_{\\!}$
corresponding${}_{\\!}$ encoders,${}_{\\!}$ and${}_{\\!}$ uti- lizes the
language decoder $\mathcal{D}^{l\\!}$ to predict the next word $x_{l}$.

We remark that Lana conducts the instruction generation task in an
autoregressive manner. In this way, both the two decoders are conditioned on
both the two encoders, leading to extensive visual and linguistic knowledge
exchange.

Route${}_{\\!}$ Encoder${}_{\\!}$ $\mathcal{E}^{r\\!}$.${}_{\\!}$
$\mathcal{E}^{r\\!}$ takes${}_{\\!}$ input${}_{\\!}$ either${}_{\\!}$
the${}_{\\!}$ entire${}_{\\!}$ route,${}_{\\!}$ _i.e_.,${}_{\\!}$
$R\\!=\\!\\{r_{t}\\}^{T}_{t=1\\!}\\!=\\!\\{(O_{t},a_{t})\\}^{T}_{t=1}$,${}_{\\!}$
during${}_{\\!}$ instruction${}_{\\!}$ generation; or historical${}_{\\!}$
route${}_{\\!}$ states${}_{\\!}$ along with current observation, _i.e_.,
$\\{r_{1},\cdots_{\\!},r_{t-1},O_{t}\\}_{\\!}\\!=_{\\!}\\!\\{O_{1},a_{1},\cdots_{\\!},O_{t-1},a_{t-1},O_{t}\\}$,
during wayfinding. Hence it has two types of input tokens corresponding to the
panoramic observation $O_{t}$ and action $a_{t}$. In particular, the
observation token of $O_{t}$ is calculated by:

$\displaystyle\bm{O}_{t}$
$\displaystyle=[\bm{o}_{t,1},\bm{o}_{t,2},\cdots,\bm{o}_{t,K}]~{}~{}\in\mathbb{R}^{K\times
d},$ (1) $\displaystyle\bm{o}_{t,k}$
$\displaystyle=\mathcal{F}^{v}(\bm{v}_{t,k})+\mathcal{F}^{\theta}(\bm{\theta}_{t,k})+\bm{\tau}^{t}+\bm{\tau}^{O}~{}~{}\in\mathbb{R}^{d},$

where${}_{\\!}$ $\bm{v}_{t,k\\!}$ and${}_{\\!}$ $\bm{\theta}_{t,k\\!}$
are${}_{\\!}$ respectively${}_{\\!}$ the${}_{\\!}$ visual${}_{\\!}$
and${}_{\\!}$ orienta- tion${}_{\\!}$ embeddings of view ${o}_{t,k}$;
$\mathcal{F}^{v/\theta\\!}$ is linear projection for feature dimension
alignment; $\bm{\tau}^{t\\!}\\!\in\\!\mathbb{R}^{d}$ embeds the temporal order
– $t$; and $\bm{\tau}^{O\\!}\\!\in\\!\mathbb{R}^{d}$ is a learnable type
embedding which indicates $\bm{o}_{t,k}$ is an observation token.

Similarly, the action token of $a_{t}$ is given as:

$\displaystyle\bm{a}_{t}=\mathcal{F}^{v}(\bm{v}_{t,a_{t}})+\mathcal{F}^{\theta}(\bm{\theta}_{t,a_{t}})+\bm{\tau}^{t}+\bm{\tau}^{A}~{}~{}\in\mathbb{R}^{d},$
(2)

where $\bm{v}_{t,a_{t}\\!}$ and $\bm{\theta}_{t,a_{t}\\!}$ respectively embed
the visual view and turned angle that are associated with action $a_{t}$.
Analogous to Eq. 1, $\bm{\tau}^{A}\\!\in\\!\mathbb{R}^{d}$ encodes the action
token type.

Tokenizing all the $K_{\\!}$ subviews $\\{\bm{o}_{t,k}\\}_{k\\!}$ of each
panoramic percept $O_{t}$ allows Lana to access/memorize all the observations
along the navigation route. Unfortunately, consider- ing${}_{\\!}$
such${}_{\\!}$ many${}_{\\!}$ tokens${}_{\\!}$ causes${}_{\\!}$
unaffordable${}_{\\!}$ computation${}_{\\!}$ load for self-attention based
encoding. To pursue a good balance between computational cost and
representation ability, we first compute an action-attentive route state:

$\displaystyle\\!\\!\\!\bm{r}_{t}$
$\displaystyle\\!=\\!\bm{a}_{t}+\bm{c}_{t}~{}\in\mathbb{R}^{d},$ (3)
$\displaystyle\\!\\!\\!\bm{c}_{t}$
$\displaystyle\\!=\\!{cross\\_{att}}(\bm{a}_{t},\bm{O}_{t})\\!=\\!{cross\\_{att}}(\bm{a}_{t},[\bm{o}_{t,k}]^{K}_{k=1})~{}\in\mathbb{R}^{d}.\\!$

Through cross-attention, _i.e_., ${cross\\_{att}}(\cdot,\cdot)$, action-
related visual context $\bm{c}_{t}$ are gathered and compressed into a
$d$-dimensional vector. Then the output of $\mathcal{E}^{r\\!}$ is obtained
via:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\text{Ins.~{}following:}~{}[\bar{\bm{r}}_{1:t-1},\bar{\bm{O}}_{t}]$
$\displaystyle=\\!{self_{\\!}\\_{att}}([\bm{r}_{1:t-1},\bm{O}_{t}])~{}\\!\in\\!\mathbb{R}^{(t-1+K)\times
d},\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!$ (4)
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\text{Ins.~{}generation:}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}_{\\!}[\bar{\bm{r}}_{1:T}]$
$\displaystyle=\\!{self_{\\!}\\_{att}}([\bm{r}_{1:T}])~{}\\!\in\\!\mathbb{R}^{T\times
d}.\\!\\!$

Language${}_{\\!}$ Encoder${}_{\\!}$ $\mathcal{E}^{l}$.${}_{\\!}$
$\mathcal{E}^{l\\!}$ takes${}_{\\!}$ input${}_{\\!}$ either${}_{\\!}$
the${}_{\\!}$ complete instruction,${}_{\\!}$ _i.e_.,${}_{\\!}$
$X\\!=\\!\\{x_{l}\\}^{L}_{l=1}$, during${}_{\\!}$ wayfinding;${}_{\\!}$
or${}_{\\!}$ previously generated words, _i.e_.,
$\\{x_{1},\cdots_{\\!},x_{l-1}\\}$, during instruc- tion${}_{\\!}$
generation.${}_{\\!}$ It${}_{\\!}$ is${}_{\\!}$ built${}_{\\!}$ as${}_{\\!}$
a${}_{\\!}$ standard${}_{\\!}$ Transformer${}_{\\!}$ language${}_{\\!}$
encoder for contextualized linguistic feature extraction:

$\displaystyle\\!\\!\\!\\!\text{Ins.~{}following:}~{}~{}~{}~{}~{}[\bar{\bm{x}}_{1:L}]$
$\displaystyle\\!=\\!\mathcal{E}^{l}([x_{1:L}])~{}\\!\in\\!\mathbb{R}^{L\times
d},\\!\\!\\!$ (5)
$\displaystyle\\!\\!\\!\\!\text{Ins.~{}generation:}~{}[\bar{\bm{x}}_{1:l-1}]$
$\displaystyle\\!=\\!\mathcal{E}^{l}([x_{1:l-1}])~{}\\!\in\\!\mathbb{R}^{(l-1)\times
d},\\!\\!$

where $\mathcal{E}^{l}$ contains several blocks, each of which has a multi-
head self-attention layer and a feed-forward sub-layer [66]; position
embeddings are omitted for the sake of brevity.

We note that, during the training of the instruction generation task, causal
future mask [75] is applied to each self-attention layer, ensuring each word
token can only attend to the previous ones, and allowing our single model to
tackle both instruction following and generation simultaneously.

Route${}_{\\!}$ Decoder${}_{\\!}$ $\mathcal{D}^{r\\!}$.${}_{\\!}$
$\mathcal{D}^{r\\!}$ is${}_{\\!}$ for${}_{\\!}$ instruction-to-route${}_{\\!}$
translation.${}_{\\!}$ Concretely,${}_{\\!}$ at${}_{\\!}$ navigation${}_{\\!}$
step${}_{\\!}$ $t$,${}_{\\!}$ $\mathcal{D}^{r\\!}$ takes${}_{\\!}$
input${}_{\\!}$ the${}_{\\!}$ complete instruction${}_{\\!}$
embedding${}_{\\!}$ $\bar{\bm{x}}_{1:L}$,${}_{\\!}$ historical${}_{\\!}$
route${}_{\\!}$ states${}_{\\!}$ $\bar{\bm{r}}_{1:t-1}$,${}_{\\!}$ as well as
current observation feature $\bar{\bm{O}}_{t}\\!=\\!\bar{\bm{o}}_{t,1:K}$, and
outputs probability${}_{\\!}$ distribution${}_{\\!}$
$\bm{p}_{t\\!}\\!\in\\!\triangle^{\\!K\\!\\!}$ of${}_{\\!}$ action${}_{\\!}$
selection${}_{\\!}$ over current $K$ subviews ${{o}}_{t,1:K}$111Note${}_{\\!}$
that,${}_{\\!}$ in${}_{\\!}$ addition${}_{\\!}$ to${}_{\\!}$ the${}_{\\!}$
$K_{\\!}$ action${}_{\\!}$ subviews,${}_{\\!}$ STOP${}_{\\!}$ token${}_{\\!}$
is${}_{\\!}$ also${}_{\\!}$ con- sidered${}_{\\!}$ here,${}_{\\!}$
leading${}_{\\!}$ to${}_{\\!}$ $K\\!+\\!1_{\\!}$ decision${}_{\\!}$
choices.${}_{\\!}$ We${}_{\\!}$ omit${}_{\\!}$ STOP${}_{\\!}$ for${}_{\\!}$
simplicity.. More specifically, $\mathcal{D}^{r\\!}$ is built as a stack of
several cross-attention-based blocks for modeling cross-modal relationships.
For each block, we have:

$\displaystyle{\small{[\hat{\bm{r}}_{1:t-1},\hat{\bm{o}}_{t,1:K}]\\!=\\!~{}{cross\\_{att}}([\bar{\bm{r}}_{1:t-1},\bar{\bm{o}}_{t,1:K}],\bar{\bm{x}}_{1:L}),}}$
(6)
$\displaystyle{\small{[\bar{\bm{r}}_{1:t-1},\bar{\bm{o}}_{t,1:K}]\\!\leftarrow\\!{self_{\\!}\\_{att}}([\hat{\bm{r}}_{1:t-1},\hat{\bm{o}}_{t,1:K}]).}}$
(7)

Eq.${}_{\\!}$ 6${}_{\\!}$ obtains${}_{\\!}$ language-enhanced${}_{\\!}$
route${}_{\\!}$ and${}_{\\!}$ observation${}_{\\!}$ rep-${}_{\\!\\!}$
resentations, _i.e_., $\hat{\bm{r}}_{1:t-1}$ and $\hat{\bm{o}}_{t,1:K}$,
through cross-attention. Eq.${}_{\\!}$ 7${}_{\\!}$ adopts${}_{\\!}$ self-
attention${}_{\\!}$ to${}_{\\!}$ model${}_{\\!}$ temporal${}_{\\!}$
dependencies among historical route states $\hat{\bm{r}}_{1:t-1}$, and capture
the correlations between $\hat{\bm{r}}_{1:t-1}$ and current observation
$\hat{O}_{t}\\!=\\!\hat{\bm{o}}_{t,1:K}$.

After several $\mathcal{D}^{r\\!}$ decoder blocks, the action probability over
the $K$ subviews ${o}_{t,1:K}$ is given as:

$\displaystyle\bm{p}_{t}=softmax(\\{\mathcal{F}^{r}(\bar{\bm{o}}_{k})\\}^{K}_{k=1})~{}~{}\in\triangle^{\\!K},$
(8)

where${}_{\\!}$
$\mathcal{F}^{r}\\!\\!:{\\!}\mathbb{R}^{d}\\!\rightarrow\\!\mathbb{R}_{\\!}$
is${}_{\\!}$ a${}_{\\!}$ two-layer${}_{\\!}$ feed-forward${}_{\\!}$
network${}_{\\!}$ for action score mapping, as in [11, 92].

Language${}_{\\!}$ Decoder${}_{\\!}$ $\mathcal{D}^{l\\!}$.
$\mathcal{D}^{l\\!}$ is${}_{\\!}$ for${}_{\\!}$ route-to-instruction${}_{\\!}$
trans- lation.${}_{\\!}$ Concretely,${}_{\\!}$ at${}_{\\!}$
instruction${}_{\\!}$ generation${}_{\\!}$ step${}_{\\!}$ $l$,${}_{\\!}$
$\mathcal{D}^{l\\!}$ takes${}_{\\!}$ input${}_{\\!}$ the${}_{\\!}$
full${}_{\\!}$ route${}_{\\!}$ states${}_{\\!}$ $\bar{\bm{r}}_{1:T}$, and the
embeddings of pre- viously${}_{\\!}$ generated${}_{\\!}$ instruction${}_{\\!}$
words${}_{\\!}$ $\bar{\bm{x}}_{1:{l-1}}$,${}_{\\!}$ and${}_{\\!}$
outputs${}_{\\!}$ pro- bability${}_{\\!}$ distribution${}_{\\!}$
$\bm{q}_{l\\!}\\!\in\\!\\!\triangle^{\\!M\\!\\!}$ of${}_{\\!}$ word${}_{\\!}$
selection${}_{\\!}$ over${}_{\\!}$ a${}_{\\!}$ pre-defined${}_{\\!}$
vocabulary${}_{\\!}$ with${}_{\\!}$ $M_{\\!}$ words.${}_{\\!}$
Analogous${}_{\\!}$ to${}_{\\!}$ $\mathcal{D}^{r}$,${}_{\\!}$
$\mathcal{D}^{l\\!}$ has several${}_{\\!}$ cross-attention${}_{\\!}$
based${}_{\\!}$ blocks.${}_{\\!}$ Each${}_{\\!}$ block${}_{\\!}$ is${}_{\\!}$
given${}_{\\!}$ as:

$\displaystyle{\small{\hat{\bm{x}}_{1:l-1}\\!=\\!~{}{cross\\_{att}}(\bar{\bm{x}}_{1:l-1},\bar{\bm{r}}_{1:T}),}}$
(9)
$\displaystyle{\small{\bar{\bm{x}}_{1:l-1}\\!\leftarrow\\!{causal\\_self_{\\!}\\_{att}}(\hat{\bm{x}}_{1:l-1}).}}$
(10)

Eq. 9 lets the text attend to the route context. In Eq. 10, we adopt the
causally-masked self-attention, instead of normal, bi-directional self-
attention, to force $\mathcal{D}^{l\\!}$ to “attend-ahead”, which is needed
for autoregressive inference.

After several $\mathcal{D}^{l\\!}$ decoder blocks, the probability over the
$M$-word vocabulary is given as:

$\displaystyle\bm{q}_{l}=softmax(\mathcal{F}^{l}(\bar{\bm{x}}_{l-1}))~{}~{}\in\triangle^{\\!M},$
(11)

where${}_{\\!}$
$\mathcal{F}^{r}\\!\\!:{\\!}\mathbb{R}^{d}\\!\rightarrow\\!\mathbb{R}^{M\\!}$
is${}_{\\!}$ a${}_{\\!}$ two-layer${}_{\\!}$ feed-forward${}_{\\!}$
network${}_{\\!}$ for the prediction of the word score distribution.

###  3.2 Network Training

All the modules of Lana, _i.e_., two unimodal encoders $\mathcal{E}^{r\\!}$
and $\mathcal{E}^{l}$, as well as two multimodal decoders $\mathcal{D}^{r\\!}$
and $\mathcal{D}^{l\\!}$, are jointly end-to-end learned, by optimizing the
training objec- tives of instruction following and generation.

Instruction${}_{\\!}$ Generation.${}_{\\!}$ For${}_{\\!}$ each${}_{\\!}$
instruction-route${}_{\\!}$ training pair${}_{\\!}$ $(X,R)$,${}_{\\!}$
where${}_{\\!}$ $X\\!=\\!x_{1:L\\!}$ and${}_{\\!}$ $R\\!=\\!r_{1:T}$, Lana
learns instruction generation by predicting $x_{l}$ based on the full route
$R_{\\!}$ and${}_{\\!}$ proceeding${}_{\\!}$ reference${}_{\\!}$
words${}_{\\!}$ $x_{0:l-1\\!}$.${}_{\\!}$ We${}_{\\!}$ append${}_{\\!}$ two
special${}_{\\!}$ tokens${}_{\\!}$ to${}_{\\!}$ $X$,${}_{\\!}$
_i.e_.,${}_{\\!}$ $x_{0}\\!=\\![\texttt{BOS}]_{\\!}$ and${}_{\\!}$
$x_{L+1}\\!=\\![\texttt{EOS}]$, res- pectively${}_{\\!}$ indicating${}_{\\!}$
the${}_{\\!}$ start${}_{\\!}$ and${}_{\\!}$ end${}_{\\!}$ of${}_{\\!}$
the${}_{\\!}$ instruction sen- tence.${}_{\\!}$ To${}_{\\!}$
generate${}_{\\!}$ word${}_{\\!}$ $x_{l}$,${}_{\\!}$ Lana${}_{\\!}$
respectively${}_{\\!}$ feeds${}_{\\!}$ $R_{\\!}$ and${}_{\\!}$ $x_{0:l-1\\!}$
into${}_{\\!}$ $\mathcal{E}^{r\\!}$ and${}_{\\!}$ $\mathcal{E}^{l\\!}$
for${}_{\\!}$ unimodal${}_{\\!}$ encoding${}_{\\!}$ (cf.${}_{\\!}$
Eq.${}_{\\!}$ 4&5). Conditioned${}_{\\!}$ on${}_{\\!}$ the${}_{\\!}$
route${}_{\\!}$ and linguistic${}_{\\!}$ embeddings,${}_{\\!}$
_i.e_.,${}_{\\!}$ $\bar{\bm{r}}_{1:T}$ and${}_{\\!}$
$\bar{\bm{x}}_{1:l-1}$,${}_{\\!}$ $\mathcal{D}^{l\\!}$ gives the word
probability $\bm{q}_{l}$ (cf.${}_{\\!}$ Eq.${}_{\\!}$ 11). The training
objective of instruction generation, formulated as the language modeling loss,
can be written as:

$\displaystyle\mathcal{L}^{g}=-\sum_{l=1}^{L+1}{\log}(p(x_{l}|x_{0:l-1},R))=-\sum_{l=1}^{L+1}{\log}(\bm{q}_{l}(x_{l})),$
(12)

where $\bm{q}_{l}(x_{l})\\!\in\\![0,1]$ is the probability of word $x_{l}$.
Lana is

trained to minimize the negative log-likelihood of the reference
instruction${}_{\\!}$ words.${}_{\\!}$ Teacher-forcing${}_{\\!}$
[89]${}_{\\!}$ is${}_{\\!}$ used${}_{\\!}$ here${}_{\\!}$ to${}_{\\!}$
enable${}_{\\!}$ the parallel${}_{\\!}$ text${}_{\\!}$ input.${}_{\\!}$
Worth${}_{\\!}$ mentioning${}_{\\!}$ is${}_{\\!}$ that,${}_{\\!}$ existing VLN
pretraining methods [30, 31, 30, 11, 64] rely on the masked language modeling
(MLM) strategy. Since MLM only predicts a small portion (typically 15%) of
input words during each training iteration, it is less efficient for large-
scale pretraining data, as pointed out by many recent literature in general
vision-language pretraining [34, 9, 15].

Instruction${}_{\\!}$ Following.${}_{\\!}$ For${}_{\\!}$ each${}_{\\!}$
training pair${}_{\\!}$ $(X,R)$,${}_{\\!}$ where $X\\!=\\!x_{1:L\\!}$
and${}_{\\!}$ $R\\!=\\!r_{1:T}\\!=\\!(O_{t},a_{t})_{1:T}$,${}_{\\!}$
Lana${}_{\\!}$ concurrently learns instruction following by predicting $a_{t}$
based on the full${}_{\\!}$ instruction${}_{\\!}$ $X$,${}_{\\!}$
history${}_{\\!}$ from${}_{\\!}$ expert${}_{\\!}$ demonstration${}_{\\!}$
$r_{1:t-1}$, and the current percept $O_{t}$. Specifically, Lana${}_{\\!}$
respectively feeds${}_{\\!}$ $X_{\\!}$ and${}_{\\!}$
$\\{r_{1:t-1\\!},O_{t}\\}$ into${}_{\\!}$ $\mathcal{E}^{l\\!}$ and${}_{\\!}$
$\mathcal{E}^{r}$ (cf.${}_{\\!}$ Eq.${}_{\\!}$ 4&5). Conditioned on the output
unimodal encodings, _i.e_., $\bar{\bm{x}}_{1:L}$ and${}_{\\!}$
$[\bar{\bm{r}}_{1:t-1},\bar{\bm{O}}_{t}]$, $\mathcal{D}^{r\\!}$ gives the
action probability $\bm{p}_{t}$ (cf.${}_{\\!}$ Eq.${}_{\\!}$ 8). The training
objective of instruction following is to mini- mize${}_{\\!}$ the${}_{\\!}$
negative${}_{\\!}$ log-likelihood${}_{\\!}$ of${}_{\\!}$ the${}_{\\!}$
target${}_{\\!}$ view${}_{\\!}$ action${}_{\\!}$ $a_{t}$:

$\displaystyle\\!\\!\mathcal{L}^{f}\\!=\\!-\\!\sum_{t=1}^{T}\log(p(a_{t}|r_{0:t-1},O_{t},X))\\!=\\!-\\!\sum_{t=1}^{T}\log(\bm{p}_{t}(a_{t})).$
(13)

${}_{\\!}$Lana${}_{\\!}$ is${}_{\\!}$ end-to-end${}_{\\!}$ learned${}_{\\!}$
with${}_{\\!}$ the${}_{\\!}$ two${}_{\\!}$ training${}_{\\!}$ targets
(cf.${}_{\\!}$ Eq. 12&13)${}_{\\!}$ during${}_{\\!}$ both${}_{\\!}$
pretraing${}_{\\!}$ and${}_{\\!}$ fine-tuning phases. Note${}_{\\!}$
that${}_{\\!}$ the${}_{\\!}$ encoders${}_{\\!}$ $\mathcal{E}^{r\\!}$
and${}_{\\!}$ $\mathcal{E}^{l\\!}$ receive${}_{\\!}$ the${}_{\\!}$
supervision${}_{\\!}$ sig- nals${}_{\\!}$ from${}_{\\!}$ both${}_{\\!}$
instruction${}_{\\!}$ generation${}_{\\!}$ (cf.${}_{\\!}$ Eq.${}_{\\!}$
12)${}_{\\!}$ and following (cf.${}_{\\!}$ Eq.${}_{\\!}$ 13). Moreover, such a
joint learning framework grants Lana improved interpretability – Lana can be
viewed as a navigator born with a language explainer $\mathcal{D}^{l}$.

###  3.3 Implementation Details

Network${}_{\\!}$ Architecture.${}_{\\!}$ The${}_{\\!}$ route${}_{\\!}$
$\mathcal{E}^{r\\!}$ and${}_{\\!}$ language${}_{\\!}$ $\mathcal{E}^{l\\!}$
enco- ders${}_{\\!}$ respectively${}_{\\!}$ have${}_{\\!}$ one and nine
layers, and the decoders $\mathcal{D}^{r\\!\\!}$ and${}_{\\!}$
$\mathcal{D}^{l\\!}$ both${}_{\\!}$ have${}_{\\!}$ four${}_{\\!}$
blocks.${}_{\\!}$ The${}_{\\!}$ feature${}_{\\!}$ dimension${}_{\\!}$
is${}_{\\!}$ set${}_{\\!}$ as $d\\!=\\!768$. The orientation feature
$\bm{\theta}_{k}$ of view $o_{k}$ (cf.${}_{\\!}$ Eq.${}_{\\!}$ 1) is defined
as:
$\bm{\theta}_{k}\\!=\\!(\cos\varphi_{k},\sin\varphi_{k},\cos\phi_{k},\sin\phi_{k})$,
where $\varphi$ and $\phi$ are the angles of heading and elevation,
respectively.

Training. Following recent VLN practice [56, 31, 30, 11, 64], the pretraining
and fine-tuning paradigm is adopted:

  * •

Pretraining:${}_{\\!}$ With${}_{\\!}$ the${}_{\\!}$ two${}_{\\!}$
training${}_{\\!}$ objectives${}_{\\!}$ (cf.${}_{\\!}$ Eq.${}_{\\!}$ 12&
13),${}_{\\!}$ Lana${}_{\\!}$ is${}_{\\!}$ pretrained${}_{\\!}$ on${}_{\\!}$
offline-sampled${}_{\\!}$ instruction-route${}_{\\!}$ pairs${}_{\\!}$
from${}_{\\!}$ PREVALENT${}_{\\!}$ [31],${}_{\\!}$ including${}_{\\!}$
104K${}_{\\!}$ ori- ginal R2R samples and 6482K synthesized ones. Lana is
trained for 100K iterations, using Adam optimizer [41] with 1e-4 learning
rate, and${}_{\\!}$ $N\\!=\\!128_{\\!}$ batch${}_{\\!}$ size.${}_{\\!}$

  * •

Fine-tuning: Then we fine-tune Lana on different VLN datasets,${}_{\\!}$
still${}_{\\!}$ using${}_{\\!}$ our${}_{\\!}$ two${}_{\\!}$ training${}_{\\!}$
tasks${}_{\\!}$ (cf.${}_{\\!}$ Eq.${}_{\\!}$ 12&13). Following the standard
protocol [33, 78, 32, 83], the training of instruction following is based on
the mixture of imitation learning and reinforcement learning. In this stage,
we set the learning rate to 1e-5 and batch size to 8.

We use four NVIDIA Tesla A100 GPUs for network training, and sample only one
training task for each mini-batch.

Inference.${}_{\\!}$ Once${}_{\\!}$ trained,${}_{\\!}$ Lana is${}_{\\!}$
capable${}_{\\!}$ of${}_{\\!}$ both${}_{\\!}$ following and verbalizing
navigation instructions with only one single model instance, without any
architectural change. Specifically, for instruction following, greedy search,
_i.e_., selecting the action with the highest probability at each prediction
step, is adopted and terminated when STOP is chosen. For instruction
generation, the sentence is predicted in an autoregressive manner, _i.e_.,
generating one word at a time until EOS is chosen, conditioned on previous
generated ones.

|  | R2R val unseen | R2R test unseen
---|---|---|---
Methods |  SR $\uparrow$ |  SPL $\uparrow$ |  OR $\uparrow$ |  TL $\downarrow$ |  SR $\uparrow$ |  SPL $\uparrow$ |  OR $\uparrow$ |  TL $\downarrow$
BT-follower [27] | [NeurIPS2018] | 36 | - | 45 | - | 35 | 28 | 44 | 14.8
EDrop-follower [71] | [NAACL2019] | 52 | 48 | - | 10.7 | 51 | 47 | 59 | 11.7
AuxRN [93] | [CVPR2020] | 55 | 50 | 62 | - | 55 | 51 | 62 | -
PREVALENT [31] | [CVPR2020] | 58 | 53 | - | 10.2 | 54 | 51 | - | 10.5
VLN$\circlearrowright$BERT [33] | [CVPR2021] | 63 | 57 | - | 12.0 | 63 | 57 | - | 12.4
AirBERT [30] | [ICCV2021] | 62 | 56 | - | 11.8 | 62 | 57 | - | 12.4
HAMT [11] | [NeurIPS2021] | 65 | 59 | - | 11.9 | 63 | 58 | - | 12.7
HOP [64] | [CVPR2022] | 64 | 57 | - | 12.3 | 64 | 59 | - | 12.7
Lana${}_{st}~{}$ | (ours) | 66 | 60 | 73 | 11.9 | 64 | 59 | 71 | 12.4
Lanamt | (ours) | 68 | 62 | 76 | 12.0 | 65 | 60 | 71 | 12.6

Table 1: Quantitative comparison results (§4.1) for instruction following on
R2R${}_{\\!}$ [4]. ‘$-$’: unavailable statistics.

|  | R4R val unseen
---|---|---
Methods |  CLS $\uparrow$ |  nDTW $\uparrow$ |  SDTW $\uparrow$ |  SR $\uparrow$ |  TL $\downarrow$
BT-follower [27] | [NeurIPS2018] | 30 | - | - | 24 | 19.9
RCM [83] | [CVPR2019] | 35 | 30 | 13 | 26 | 28.5
PTA [43] | [NeurIPS2021] | 37 | 32 | 10 | 24 | 17.7
EDrop-follower [71] | [NAACL2019] | 34 | - | 9 | 29 | 27.0
OAAM [62] | [ECCV2020] | 40 | - | 11 | 31 | 13.8
ActiveVLN [81] | [ECCV2020] | 59 | 44 | 22 | 32 | 19.7
EGP [21] | [NeurIPS2020] | 44 | 37 | 18 | 30 | 18.3
HAMT [11] | [NeurIPS2021] | 57.7 | 50.3 | 31.8 | 44.6 | -
Lana${}_{st}~{}$ | (ours) | 58.6 | 51.9 | 31.4 | 43.0 | 22.7
Lanamt | (ours) | 59.7 | 52.3 | 31.7 | 43.2 | 22.1

Table 2: ${}_{\\!\\!}$Quantitative${}_{\\!}$ comparison${}_{\\!}$
results${}_{\\!}$ (§4.1)${}_{\\!}$ for${}_{\\!}$ instruction${}_{\\!}$ fol-
lowing on R4R [38].

|  | REVERIE val unseen | REVERIE test unseen
---|---|---|---
Methods |  SR${}_{\\!}$ $\uparrow$ |  SPL${}_{\\!}$ $\uparrow$ |  OR $\uparrow$ |  TL $\downarrow$ |  RGS${}_{\\!}$ $\uparrow$ |  RGSPL${}_{\\!}$ $\uparrow$ |  SR${}_{\\!}$ $\uparrow$ |  SPL${}_{\\!}$ $\uparrow$ |  OR${}_{\\!}$ $\uparrow$ |  TL $\downarrow$ |  RGS${}_{\\!}$ $\uparrow$ |  RGSPL${}_{\\!}$ $\uparrow$
RCM [83] | [CVPR2019] | 9.29 | 6.97 | 14.23 | 11.98 | 4.89 | 3.89 | 7.84 | 6.67 | 11.68 | 10.60 | 3.67 | 3.14
VLN$\circlearrowright$BERT [33] | [CVPR2021] | 30.67 | 24.90 | 35.02 | 16.78 | 18.77 | 15.27 | 29.61 | 23.99 | 32.91 | 15.86 | 16.50 | 13.51
AirBERT [30] | [ICCV2021] | 27.89 | 21.88 | 34.51 | 18.71 | 18.23 | 14.18 | 30.28 | 23.61 | 34.20 | 17.91 | 16.83 | 13.28
HAMT [11] | [NeurIPS2021] | 32.95 | 30.20 | 36.84 | 14.08 | 18.92 | 17.28 | 30.40 | 26.67 | 33.41 | 13.62 | 14.88 | 13.08
HOP [64] | [CVPR2022] | 30.39 | 25.10 | 35.30 | 17.16 | 18.23 | 15.31 | 29.12 | 23.37 | 32.26 | 17.05 | 17.13 | 13.90
Lana | (ours) | 34.00 | 29.26 | 38.54 | 16.28 | 19.03 | 16.18 | 33.50 | 26.89 | 36.41 | 16.75 | 17.53 | 14.25

Table 3: Quantitative comparison results (§4.1) for instruction following on
REVERIE${}_{\\!}$ [63].

|  | R2R val seen | R2R val unseen
---|---|---|---
Methods |  SPICE $\uparrow$ |  Bleu-1 $\uparrow$ |  Bleu-4 $\uparrow$ |  CIDEr $\uparrow$ |  Meteor $\uparrow$ |  Rouge $\uparrow$ |  SPICE $\uparrow$ |  Bleu-1 $\uparrow$ |  Bleu-4 $\uparrow$ |  CIDEr $\uparrow$ |  Meteor $\uparrow$ |  Rouge $\uparrow$
BT-speaker${}_{\\!}$ [27] | [NeurIPS2018] | 0.203 | 0.537 | 0.155 | 0.121 | 0.233 | 0.350 | 0.188 | 0.522 | 0.142 | 0.114 | 0.228 | 0.346
EDrop-speaker${}_{\\!}$ [71] | [NAACL2019] | 0.202 | - | 0.245 | 0.493 | 0.228 | 0.467 | 0.181 | - | 0.237 | 0.422 | 0.225 | 0.458
VLS [1] | [CVPRW2019] | 0.214 | 0.549 | 0.157 | 0.137 | 0.228 | 0.352 | 0.197 | 0.548 | 0.159 | 0.132 | 0.231 | 0.357
CCC-speaker${}_{\\!}$ [78] | [CVPR2022] | 0.231 | 0.728 | 0.287 | 0.543 | 0.236 | 0.493 | 0.214 | 0.708 | 0.272 | 0.461 | 0.231 | 0.477
Lana${}_{st}~{}$ | (ours) | 0.251 | 0.743 | 0.305 | 0.522 | 0.243 | 0.502 | 0.223 | 0.722 | 0.287 | 0.433 | 0.235 | 0.490
Lanamt | (ours) | 0.256 | 0.759 | 0.314 | 0.533 | 0.245 | 0.503 | 0.226 | 0.736 | 0.298 | 0.457 | 0.238 | 0.498

Table 4: Quantitative comparison results (§4.2) for instruction generation on
R2R [4].

##  4 Experiment

We${}_{\\!}$ evaluate${}_{\\!}$ Lana${}_{\\!}$ for${}_{\\!}$ both${}_{\\!}$
instruction${}_{\\!}$ following${}_{\\!}$ (§4.1) and${}_{\\!}$ generation
(§4.2) tasks, followed by a series of diagnostic experiments (§4.3) and
qualitative studies (§4.4).

For each task, we give scores of two versions of Lana:

  * •

Lanamt:${}_{\\!}$ jointly${}_{\\!}$ learn${}_{\\!}$ the${}_{\\!}$
two${}_{\\!}$ target${}_{\\!}$ tasks${}_{\\!}$ throughout${}_{\\!}$ pre-
training${}_{\\!}$ and${}_{\\!}$ fine-tuning.${}_{\\!}$ Thus,${}_{\\!}$
such${}_{\\!}$ multi-task${}_{\\!}$ version${}_{\\!}$ only${}_{\\!}$
has${}_{\\!}$ one single${}_{\\!}$ agent${}_{\\!}$ instance, tested on the two
tasks.

  * •

Lanast:${}_{\\!}$ jointly${}_{\\!}$ pre-train${}_{\\!}$ on${}_{\\!}$
the${}_{\\!}$ two${}_{\\!}$ tasks,${}_{\\!}$ but${}_{\\!}$ fine-tune${}_{\\!}$
on each${}_{\\!}$ task${}_{\\!}$ individually.${}_{\\!}$ There${}_{\\!}$
are${}_{\\!}$ two${}_{\\!}$ single-task${}_{\\!}$ agent${}_{\\!}$ instances;
each is only tested on the corresponding task.

|  | R4R val seen | R4R val unseen
---|---|---|---
Methods |  SPICE $\uparrow$ |  Bleu-1 $\uparrow$ |  Bleu-4 $\uparrow$ |  CIDEr $\uparrow$ |  Meteor $\uparrow$ |  Rouge $\uparrow$ |  SPICE $\uparrow$ |  Bleu-1 $\uparrow$ |  Bleu-4 $\uparrow$ |  CIDEr $\uparrow$ |  Meteor $\uparrow$ |  Rouge $\uparrow$
BT-speaker${}_{\\!}$ [27] | [NeurIPS2018] | 0.164 | 0.691 | 0.223 | 0.099 | 0.213 | 0.453 | 0.207 | 0.387 | 0.088 | 0.139 | 0.172 | 0.359
EDrop-speaker${}_{\\!}$ [71] | [NAACL2019] | 0.209 | 0.750 | 0.281 | 0.216 | 0.245 | 0.473 | 0.218 | 0.433 | 0.106 | 0.200 | 0.187 | 0.363
CCC-speaker${}_{\\!}$ [78] | [CVPR2022] | 0.219 | 0.758 | 0.312 | 0.245 | 0.252 | 0.480 | 0.233 | 0.403 | 0.115 | 0.206 | 0.193 | 0.365
Lana${}_{st}~{}$ | (ours) | 0.237 | 0.768 | 0.327 | 0.264 | 0.265 | 0.483 | 0.259 | 0.437 | 0.123 | 0.216 | 0.199 | 0.375
Lanamt | (ours) | 0.245 | 0.772 | 0.333 | 0.287 | 0.261 | 0.484 | 0.262 | 0.443 | 0.128 | 0.231 | 0.200 | 0.376

Table 5: Quantitative comparison results (§4.2) for instruction generation on
R4R [38].

# | Pretraining | Fine-tuning | Instruction Following | Instruction Generaion
---|---|---|---|---
| Instruction
---
Following
| Instruction
---
Generation
| Instruction
---
Following
| Instruction
---
Generation
SR $\uparrow$ |  SPL $\uparrow$ |  OR $\uparrow$ |  TL $\downarrow$ |  SPICE $\uparrow$ |  Bleu-1 $\uparrow$ |  Bleu-4 $\uparrow$ |  CIDEr $\uparrow$ |  Meteor $\uparrow$ |  Rouge $\uparrow$
1 |  |  | ✔ |  | 52.1 | 48.3 | 59.3 | 11.2 | - | - | - | - | - | -
2 |  |  |  | ✔ | - | - | - | - | 0.178 | 0.692 | 0.241 | 0.321 | 0.216 | 0.463
3 |  |  | ✔ | ✔ | 53.1 | 48.9 | 60.9 | 11.6 | 0.182 | 0.704 | 0.245 | 0.304 | 0.219 | 0.469
4 | ✔ |  | ✔ |  | 61.3 | 55.4 | 69.7 | 12.0 | - | - | - | - | - | -
5 |  | ✔ |  | ✔ | - | - | - | - | 0.215 | 0.718 | 0.255 | 0.378 | 0.230 | 0.472
6 | ✔ | ✔ | ✔ |  | 65.7 | 59.9 | 73.4 | 11.9 | - | - | - | - | - | -
7 | ✔ | ✔ |  | ✔ | - | - | - | - | 0.223 | 0.722 | 0.287 | 0.433 | 0.235 | 0.490
8 | ✔ | ✔ | ✔ | ✔ | 67.9 | 61.6 | 75.7 | 12.0 | 0.226 | 0.736 | 0.298 | 0.457 | 0.238 | 0.498

Table 6: Ablation study (§4.3) on R2R val unseen [4].

We${}_{\\!}$ collect${}_{\\!}$ here${}_{\\!}$ key${}_{\\!}$
observations${}_{\\!}$ from${}_{\\!}$ our${}_{\\!}$ subsequent${}_{\\!}$ expe-
riments:${}_{\\!}$ i)${}_{\\!}$ Lana${}_{\\!}$ performs${}_{\\!}$
comparable,${}_{\\!}$ or${}_{\\!}$ even${}_{\\!}$ better${}_{\\!}$ than
prior${}_{\\!}$ tasks-specific${}_{\\!}$ agents;${}_{\\!}$ ii)${}_{\\!}$
Lana${}_{mt\\!}$ outperforms${}_{\\!}$ Lana${}_{st\\!}$ with more
efficient${}_{\\!}$ parameter${}_{\\!}$ utilization;${}_{\\!}$ iii)${}_{\\!}$
Lana${}_{\\!}$ can${}_{\\!}$ pro- vide test-time behavioral interpretation by
verbalizing des- criptions of its navigation routes; and iv)${}_{\\!}$ our
model design and training targets indeed contribute to our strong results.

###  4.1 Performance on Instruction Following

Dataset. We conduct experiments on three VLN datasets:

  * •

R2R${}_{\\!}$ [4]:${}_{\\!}$ It${}_{\\!}$ has${}_{\\!}$ four${}_{\\!}$
splits,${}_{\\!}$ _i.e_.,${}_{\\!}$ train${}_{\\!}$ ($61_{\\!}$
scenes,${}_{\\!}$ $14,039$ instructions), val seen ($61$ scenes, $1,021$
instructions), val unseen ($11$ scenes, $2,349$ instructions), and test
unseen${}_{\\!}$ ($18_{\\!}$ scenes,${}_{\\!}$ $4,173_{\\!}$
instructions).${}_{\\!}$ There${}_{\\!}$ are${}_{\\!}$ no${}_{\\!}$
overlapping scenes between train and unseen splits.

  * •

R4R${}_{\\!}$ [38]:${}_{\\!}$ It${}_{\\!}$ extends${}_{\\!}$ R2R${}_{\\!}$
by${}_{\\!}$ connecting${}_{\\!}$ two${}_{\\!}$ close${}_{\\!}$ tail-to-
head${}_{\\!}$ trajectories${}_{\\!}$ and${}_{\\!}$ corresponding${}_{\\!}$
instructions${}_{\\!}$ in${}_{\\!}$ R2R.${}_{\\!}$ R4R${}_{\\!}$
contains${}_{\\!}$ three${}_{\\!}$ sets,${}_{\\!}$ _i.e_.,${}_{\\!}$
train${}_{\\!}$ ($61$ scenes, $233,613$ instructions), val seen ($61$ scenes,
$1,035$ instructions), and val unseen ($11$ scenes, $45,162$ instructions).

  * •

REVERIE${}_{\\!}$ [63]:${}_{\\!}$ It${}_{\\!}$ replaces${}_{\\!}$
detailed${}_{\\!}$ instructions${}_{\\!}$ in${}_{\\!}$ R2R with high-level
descriptions of target locations and ob- jects.${}_{\\!}$ It${}_{\\!}$
is${}_{\\!}$ composed${}_{\\!}$ of${}_{\\!}$ four${}_{\\!}$ sets,${}_{\\!}$
_i.e_.,${}_{\\!}$ train${}_{\\!}$ ($53_{\\!}$ scenes, $10,466$ instructions),
val seen ($61$ scenes, $1,371$ in- structions),${}_{\\!}$ val${}_{\\!}$
unseen${}_{\\!}$ ($10_{\\!}$ scenes,${}_{\\!}$ $3,753_{\\!}$ instructions),
and test unseen ($16$ scenes, $6,292$ instructions).

Evaluation${}_{\\!}$ Metric.${}_{\\!}$ For${}_{\\!}$ R2R,${}_{\\!}$
we${}_{\\!}$ follow${}_{\\!}$ conventions${}_{\\!}$ [4, 27] to report four
evaluation metrics: i) Success Rate (SR), ii) Trajectory Length (TL), iii)
Oracle success Rate (OR), and iv) Success rate weighted by Path Length (SPL),
where${}_{\\!}$ SR${}_{\\!}$ and${}_{\\!}$ SPL${}_{\\!}$ are${}_{\\!}$
of${}_{\\!}$ priority.${}_{\\!}$ For R4R, we further adopt v) Coverage
weighted by Length Score (CLS) [38], vi) normalized Dynamic Time Warping
(nDTW), and vii) Success weighted by nDTW (SDTW). For${}_{\\!}$
REVERIE,${}_{\\!}$ the${}_{\\!}$ first four metrics are also employed for its
navigation sub-task, and viii) Remote Grounding Success rate (RGS) and ix) RGS
weighted by Path Length (RGSPL) are additionally used for overall performance
evaluation.

Quantitative Result. Several famous and recent advanced solutions${}_{\\!}$
[27, 71, 93, 83, 81, 31, 33, 21, 30, 11, 78, 64] for instruction following are
involved in comparison. Note that we report the score of the single model
under the single run setup following the tradition [33, 78, 64, 11]. As shown
in Table 3.3, Lanast, which is only fine-tuned on wayfinding after multi-task
pretraining, demonstrates comparable, if not better, results than those
alternatives on R2R. Remarkably, Lanamt, which learns to interpret navigation
paths alongside following instructions, even yields better navigation
performance. For instance, Lanamt lifts Lanast by 2% and 1% SPL, on val unseen
and test respectively. This verifies the efficacy of our language-capable
navigation scheme and multi-task learning strategy. More significant
improvements can be observed on R4R (cf.${}_{\\!}$ Table${}_{\\!}$ 3.3) and
REVERIE (cf.${}_{\\!}$ Table${}_{\\!}$ 3), where the former focuses on long-
horizon navigation with longer instructions and trajectories, while the latter
gives abstract instructions only.${}_{\\!}$ These${}_{\\!}$ results${}_{\\!}$
confirm${}_{\\!}$ our${}_{\\!}$ generality${}_{\\!}$ and${}_{\\!}$
versatility.${}_{\\!}$ It${}_{\\!}$ is important to note that all the
competitors are only aware of wayfinding, while our agent can generate
grounded route descriptions for interpreting its navigation behaviors/plans.

###  4.2 Performance on Instruction Generation

Dataset. We compare machine generated route descriptions with the human-
written instructions, on two VLN datasets:

  * •

R2R${}_{\\!}$ [4]:${}_{\\!}$ As${}_{\\!}$ R2R${}_{\\!}$ test${}_{\\!}$
is${}_{\\!}$ preserved${}_{\\!}$ for${}_{\\!}$ benchmarking${}_{\\!}$
instruction following agents, we report the performance of instruction
generation on val sets. Each R2R navigation path is associated with three
ground-truth instructions.

  * •

R4R${}_{\\!}$ [38]:${}_{\\!}$ Performance${}_{\\!}$ is${}_{\\!}$
reported${}_{\\!}$ on${}_{\\!}$ R4R${}_{\\!}$ val${}_{\\!}$ sets,${}_{\\!}$
where each path corresponds to nine ground-truth instructions.

REVERIE${}_{\\!}$ [63]${}_{\\!}$ is${}_{\\!}$ not${}_{\\!}$ involved${}_{\\!}$
as${}_{\\!}$ its${}_{\\!}$ instructions${}_{\\!}$ are${}_{\\!}$ high-level,
concise descriptions of remote objects, which cannot serve our purpose of
grounded instruction generation.

Figure 3: (a-b) Visual comparison results between Lanamt and Lanast for (a)
instruction following and (b) instruction generation tasks. The start and end
points of a navigation route are respectively denoted by  and . (c) Lana is
able to interpret its navigation behavior using natural language. The
generated report keeps the monitor updated on the navigation process, and even
helps reveal the failure mode. For example, at step 2-3, Lanamt enters the
storeroom because Lanamt thought it is a bedroom. See §4.4 for more detailed
discussion.

Evaluation${}_{\\!}$ Metric.${}_{\\!}$ Following${}_{\\!}$ [1, 78],${}_{\\!}$
we${}_{\\!}$ opt${}_{\\!}$ for${}_{\\!}$ five${}_{\\!}$ text${}_{\\!}$
metrics:${}_{\\!}$ i)${}_{\\!}$ BLEU${}_{\\!}$ [60],${}_{\\!}$ ii)${}_{\\!}$
CIDEr${}_{\\!}$ [76],${}_{\\!}$ iii)${}_{\\!}$ METEOR${}_{\\!}$ [8],${}_{\\!}$
iv) ROUGE${}_{\\!}$ [49],${}_{\\!}$ and${}_{\\!}$ v)${}_{\\!}$ SPICE${}_{\\!}$
[3].${}_{\\!}$ For${}_{\\!}$ each${}_{\\!}$ navigation${}_{\\!}$
path,${}_{\\!}$ the${}_{\\!}$ metrics are averaged over all the corresponding
groundtruth instructions. SPICE is considered as the primary metric.

Quantitative Result. We compare Lana with four instruction generation
algorithms${}_{\\!}$ [27, 71, 1, 78]. Table 4 and Table 5 summarize our
comparison results. We can find that our${}_{\\!}$ task-specific${}_{\\!}$
agent,${}_{\\!}$ _i.e_.,${}_{\\!}$ Lanast,${}_{\\!}$ already${}_{\\!}$
outperforms${}_{\\!}$ all the${}_{\\!}$ competitors,${}_{\\!}$
across${}_{\\!}$ all${}_{\\!}$ the${}_{\\!}$ metrics${}_{\\!}$ and${}_{\\!}$
datasets.${}_{\\!}$ Note${}_{\\!}$ that, CCC${}_{\\!}$ [78], a current top-
leading solution, learns the instruction generation model with the aid of a
separate wayfinder. More impressively, our multi-task agent, _i.e_., Lanamt,
performs on par or even better than Lanast, demonstrating the algorithmic and
functional advantages of our approach.

User${}_{\\!}$ Study.${}_{\\!}$ To${}_{\\!}$ provide${}_{\\!}$ a${}_{\\!}$
complete${}_{\\!}$ measure${}_{\\!}$ of${}_{\\!}$ the${}_{\\!}$
quality${}_{\\!}$ of our created instructions, we conduct a set of human
evalua- tion${}_{\\!}$ experiments,${}_{\\!}$ based${}_{\\!}$ on${}_{\\!}$
pair-wise${}_{\\!}$ comparison.${}_{\\!}$ Concretely,${}_{\\!}$ 50${}_{\\!}$
college${}_{\\!}$ students${}_{\\!}$ are${}_{\\!}$ asked${}_{\\!}$
to${}_{\\!}$ respectively${}_{\\!}$ compare${}_{\\!}$ the${}_{\\!}$ ins-
tructions${}_{\\!}$ generated${}_{\\!}$ by${}_{\\!}$ Lana${}_{mt\\!}$
with${}_{\\!}$ those${}_{\\!}$ created${}_{\\!}$ by CCC, BT-Speaker, and
humans, for 100 paths in total. The paths are sampled${}_{\\!}$ from${}_{\\!}$
R2R${}_{\\!}$ val${}_{\\!}$ unseen.${}_{\\!}$ Finally,${}_{\\!}$ Lana receive
more preference votes, _i.e_., 63.4% vs CCC 36.6%, and${}_{\\!}$
75.1%${}_{\\!}$ vs${}_{\\!}$ BT-Speaker${}_{\\!}$ 24.9%.${}_{\\!}$
Yet,${}_{\\!}$ human-written${}_{\\!}$ instructions${}_{\\!}$ are${}_{\\!}$
far${}_{\\!}$ more${}_{\\!}$ favorable,${}_{\\!}$ _i.e_.,${}_{\\!}$
69.3%${}_{\\!}$ vs${}_{\\!}$ Lana${}_{\\!}$ 30.7%, demonstrating there remains
large room for improvement.

###  4.3 Diagnostic Experiment

To thoroughly study the effectiveness of our language-capable navigation
framework, we carry out a series of diagnostic experiments on val unseen set
of R2R [4], for both instruction following and generation tasks. The
experimental results are summarized in Table 6. More specifically, a total of
eight baselines are involved in our ablation study:

  1. 1.

fine-tune on instruction following only, w/o pretraining;

  2. 2.

fine-tune${}_{\\!}$ on${}_{\\!}$ instruction${}_{\\!}$ generation${}_{\\!}$
only,${}_{\\!}$ w/o${}_{\\!}$ pre-training;${}_{\\!\\!}$

  3. 3.

fine-tune on both instruction following and generation, w/o pretraining;

  4. 4.

pretrain and fine-tune on instruction following only;

  5. 5.

pretrain and fine-tune on instruction generation only;

  6. 6.

pretrain${}_{\\!}$ on${}_{\\!}$ both${}_{\\!}$ instruction${}_{\\!}$
following${}_{\\!}$ and${}_{\\!}$ generation,${}_{\\!}$ and fine-tune on
instruction following only;

  7. 7.

pretrain${}_{\\!}$ on${}_{\\!}$ both${}_{\\!}$ instruction${}_{\\!}$
following${}_{\\!}$ and${}_{\\!}$ generation,${}_{\\!}$ and fine-tune on
instruction generation only;

  8. 8.

pretrain and fine-tune on both instruction following and generation.

These baselines can be roughly grouped into three classes: i) baselines 1,2,
and 3 are all w/o pretraining, and fine-tune on each task either individually
or jointly; ii) baselines 4 and 5 pretrain and fine-tune on each task
individually; and iii) baselines 6, 7, and 8 are w/ joint-task pretraining,
and fine-tune on each task either individually or jointly. Baselines 6 and 7
are the two sing-task agents, _i.e_., Lanast, and baseline 8 are our finally
delivered agent Lanamt; their performance have been thoroughly reported in
§4.1 and §4.2.

Also, note that baselines 1, 4, and 6 only master wayfinding, while baselines
2, 5, and 7 can only undertake the route description task. Baselines 3 and 8
are capable of both.

Several essential conclusions can be drawn:

  * •

Joint-task fine-tuning can benefit the performance of both tasks (baseline 3
vs 1 vs 2);

  * •

Joint-task pretraining and fine-tuning can benefit the performance of both
tasks (baseline 8 vs 6 vs 7);

  * •

Pretraining can facilitate the final performance (baseline 4 vs 1, baseline 5
vs 2, baseline 6 vs 1, baseline 7 vs 2, and baseline 8 vs 3);

  * •

Joint-task pretraining is more favored than single-task pretraining (baseline
8 vs 4 vs 5);

  * •

Joint-task pretraining and fine-tuning is more favored than all the other
training strategies (baseline 8 vs 1-7).

Note that, joint-tasking pretraining and fine-tuning not only promotes the
performance, but increases parameter efficiency, _i.e_., baseline 8 (143 M) vs
6 + 7 (220 M = 123 M + 97 M). In a nutshell, our ablative experiments solidly
verify the power of our idea, the efficacy of our algorithmic design, and our
advantage in efficient-parameter utilization.

###  4.4 Qualitative Experiment

Fig. 3 depicts three exemplar navigation episodes from val unseen set of
R2R${}_{\\!}$ [4] . Fig.${}_{\\!}$ 3${}_{\\!}$ (a) compares Lanamt against
Lanast on the instruction following task. As seen, Lanamt performs robust in
this challenging case, while Lanast fails to reach the target location. As
both Lanamt and Lanast are built with similar network architectures and
pretraining protocol, we attribute this to the exploration of cross-task
knowledge${}_{\\!}$ during${}_{\\!}$ fine-tuning.${}_{\\!}$ Fig.${}_{\\!}$
3${}_{\\!}$ (b)${}_{\\!}$ visua- lizes comparison on instruction creation. As
seen, Lanamt outputs more grounded instructions that contain precise action
descriptions (_e.g_., turn left, walk down) as well as sa- lient landmarks
(_e.g_., double doors, dining table, refrigerator). These descriptions have
similar properties as human- generated${}_{\\!}$ texts,${}_{\\!}$
even${}_{\\!}$ involving${}_{\\!}$ some${}_{\\!}$ landmarks${}_{\\!}$
(_e.g_.,${}_{\\!}$ sofa) that${}_{\\!}$ are${}_{\\!}$ informative${}_{\\!}$
yet${}_{\\!}$ missed${}_{\\!}$ in${}_{\\!}$ human${}_{\\!}$
reference.${}_{\\!}$ Fig.${}_{\\!}$ 3${}_{\\!}$ (c) shows that, Lana can offer
real-time behavioral interpretation by showing human text report of its
navigation process. This not only eases human from consistent monitoring, but
also reveals its inner mode to some extent. For example, the report at step
1-3 informs that Lana wrongly recognizes the storeroom as the bedroom – this
is why Lana chooses to enter the storeroom at step 2. In short, as a language-
capable navigator, Lana shows advantages in (post-hoc) interpretability and
human-robot bi-directional communication, which are the basic premises of
human trust generated.

##  5 Conclusion and Discussion

This work calls for a paradigm shift from current VLN agents${}_{\\!}$
–${}_{\\!}$ strong${}_{\\!}$ language-aided${}_{\\!}$ wayfinders${}_{\\!}$
but${}_{\\!}$ without${}_{\\!}$ lan- guage generation ability – towards more
language-capable navigation robots that can not only execute navigation ins-
tructions but also verbally describe the navigation routes. We present Lana,
which learns to master both instruction following and${}_{\\!}$
generation${}_{\\!}$ with${}_{\\!}$ one${}_{\\!}$ single${}_{\\!}$
model.${}_{\\!}$ Lana${}_{\\!}$ performs on par or even better than previous
task-specific solutions in both tasks, with much reduced complexity.
Crucially, Lana can write high-quality route descriptions that are informative
to interpret its behavior and direct humans in${}_{\\!}$
collaboration.${}_{\\!}$ We${}_{\\!}$ believe${}_{\\!}$ Lana${}_{\\!}$
provides${}_{\\!}$ a${}_{\\!}$ solid${}_{\\!}$ basis${}_{\\!}$ for
the${}_{\\!}$ creation${}_{\\!}$ of${}_{\\!}$ language-capable${}_{\\!}$
robots${}_{\\!}$ and${}_{\\!}$ brings${}_{\\!}$ us${}_{\\!}$ clo- ser to the
ultimate goal of building socially-intelligent and trustworthy${}_{\\!}$
robots.${}_{\\!}$ Future${}_{\\!}$ work${}_{\\!}$ should${}_{\\!}$
reinforce${}_{\\!}$ Lana${}_{\\!}$ with the knowledge of large-scale
pretrained foundation models.

Lana: A Language-Capable Navigator for Instruction Following and
Generation${}_{\\!\\!\\!\\!\\!\\!}$
Supplementary Material

This document provides more details of our approach and additional
experimental results, organized as follows:

  * •

§6 Implementation Details of Lana.

  * •

§7 Additional Quantitative Results on REVERIE${}_{\\!}$ [63].

  * •

§8 More Ablative Study on Route Encoder.

  * •

§9 Additional Qualitative Results.

  * •

§10 Discussion about Social Impact and Limitations.

##  6 Implementation Details of Lana

We employ an additional Instruction Trajectory Matching (ITM) task following
previous efforts [11] during pretraining, which predicts whether a pair of
instruction and trajectory is aligned. The three tasks IF (Instruction
Following), IG (Instruction Generation) and ITM (Instruction Trajectory
Matching) are sampled with a ratio IG:IF:ITM=4:1:2. We present the pseudo-code
of the pretraining procedure in Algorithm 1 (ITM is omitted for simplicity).
For finetuning, the instruction following task is optimized with Reinforcement
Learning (RL) and Imitation Learning (IL). IL utilizes the same loss in Eq.13
while RL is implemented based on the Asynchronous Advantage Actor-Critic (A3C)
algorithm [58]. During finetuning, the sampling ratio for IG and IF is set to
IG:IF=2:5; the ITM task is abandoned. Following the common practice [11, 33,
64], we concatenate the object features with the panoramic features and add an
object grounding loss for the instruction following task on REVERIE [63]. The
detailed architecture of Lana is shown in Table 7.

Algorithm 1 The pseudo-code of pre-training for Lana.

Arguments: The labeled dataset $\mathcal{H}\\!=\\!\\{(R,X)\\}$, the maximum
iteration $N$, Route Encoder $\mathcal{E}^{r\\!}$, Language${}_{\\!}$
Encoder${}_{\\!}$ $\mathcal{E}^{l\\!}$, Language${}_{\\!}$ Decoder${}_{\\!}$,
$\mathcal{D}^{l\\!}$, and${}_{\\!}$ Route${}_{\\!}$ Decoder${}_{\\!}$
$\mathcal{D}^{r\\!}$.

1:Initialize $\mathcal{E}^{r\\!}$, $\mathcal{E}^{l\\!}$, $\mathcal{D}^{r\\!}$,
$\mathcal{D}^{l\\!}$

2:for iteration $i\in[1,\dots,N]$ do

3: Sample batch $B\subset\mathcal{H}$

4: Sample a pretraining task $\mathcal{T}$ from {IG, IF}

5: $\mathcal{L}\leftarrow 0$

6: if $\mathcal{T}$ is IG then

7: for $(R,X)\in B$ do

8: $[\bar{\bm{r}}_{1:T}]=\mathcal{E}^{r\\!}(R)$

9: $[\bar{\bm{x}}_{1:l-1}]=\\!\mathcal{E}^{l}([x_{1:l-1}])$

10:
$\bm{q}_{l}=\mathcal{D}^{l\\!}([\bar{\bm{x}}_{1:l-1}],[\bar{\bm{r}}_{1:T}])$

11: Estimate $\mathcal{L}^{g}$ $\triangleright$ Defined in Eq.12.

12: $\mathcal{L}\leftarrow\mathcal{L}+\mathcal{L}^{g}$

13: Calculate $\partial\mathcal{L}$

14: Update $\mathcal{E}^{r\\!}$, $\mathcal{E}^{l\\!}$, $\mathcal{D}^{l\\!}$

15: else if $\mathcal{T}$ is IF then

16: for $(R,X)\in B$ do

17:
$[\bar{\bm{r}}_{1:t-1},\bar{\bm{O}}_{t}]=\\!\mathcal{E}^{r\\!}([\bm{r}_{1:t-1},\bm{O}_{t}])$

18: $[\bar{\bm{x}}_{1:L}]=\\!\mathcal{E}^{l}(X)$

19:
$\bm{p}_{t}=\mathcal{D}^{r\\!}([\bar{\bm{r}}_{1:t-1},\bar{\bm{O}}_{t}],[\bar{\bm{x}}_{1:L}])$

20: Estimate $\mathcal{L}^{f}$ $\triangleright$ Defined in Eq.13.

21: $\mathcal{L}\leftarrow\mathcal{L}+\mathcal{L}^{f}$

22: Calculate $\partial\mathcal{L}$

23: Update $\mathcal{E}^{r\\!}$, $\mathcal{E}^{l\\!}$, $\mathcal{D}^{r\\!}$
return $\mathcal{E}^{r\\!}$, $\mathcal{E}^{l\\!}$, $\mathcal{D}^{r\\!}$,
$\mathcal{D}^{l\\!}$

|  Language Encoder $\mathcal{E}^{l\\!}$ |  Route Encoder $\mathcal{E}^{r\\!}$ |  Language Decoder $\mathcal{D}^{l\\!}$ |  Route Decoder $\mathcal{D}^{r\\!}$
---|---|---|---|---
Layer | $\left[\begin{array}[]{c}\text{self\\_att}\\\\[-0.90005pt] \text{feedforward}\\\\[-0.90005pt] \text{-}\end{array}\right]$$\times$9 | $\left[\begin{array}[]{c}\text{cross\\_att}\\\\[-0.90005pt] \text{self\\_att}\\\\[-0.90005pt] \text{feedforward}\end{array}\right]$$\times$1 | $\left[\begin{array}[]{c}\text{cross\\_att}\\\\[-0.90005pt] \text{self\\_att}\\\\[-0.90005pt] \text{feedforward}\end{array}\right]$$\times$4 | $\left[\begin{array}[]{c}\text{cross\\_att}\\\\[-0.90005pt] \text{self\\_att}\\\\[-0.90005pt] \text{feedforward}\end{array}\right]$$\times$4

Table 7: Detailed model architecture of Lana (§6).

##  7 ${}_{\\!}$Additional${}_{\\!}$ Quantitative${}_{\\!}$ Results${}_{\\!}$
on${}_{\\!}$ REVERIE${}_{\\!\\!\\!\\!\\!}$

The synthetic samples in the PREVALENT dataset are created with a speaker
trained on R2R [4]. A recent work DUET [13] collected a new augmented dataset
by synthesizing instructions with a speaker model trained on the REVERIE
dataset [63]. We report additional quantitative results of Lana trained with
this dataset in Table 8. Remarkably, this training strategy boosts the
performance by a large margin on REVERIE [63]. Lana achieves better navigation
performance than DUET [13] with the same training set, demonstrating the
algorithmic advantages of our approach.

|  | REVERIE val unseen | REVERIE test unseen
---|---|---|---
Methods |  SR${}_{\\!}$ $\uparrow$ |  SPL${}_{\\!}$ $\uparrow$ |  OR $\uparrow$ |  TL $\downarrow$ |  RGS${}_{\\!}$ $\uparrow$ |  RGSPL${}_{\\!}$ $\uparrow$ |  SR${}_{\\!}$ $\uparrow$ |  SPL${}_{\\!}$ $\uparrow$ |  OR${}_{\\!}$ $\uparrow$ |  TL $\downarrow$ |  RGS${}_{\\!}$ $\uparrow$ |  RGSPL${}_{\\!}$ $\uparrow$
RCM [83] | [CVPR2019] | 9.29 | 6.97 | 14.23 | 11.98 | 4.89 | 3.89 | 7.84 | 6.67 | 11.68 | 10.60 | 3.67 | 3.14
VLN$\circlearrowright$BERT [33] | [CVPR2021] | 30.67 | 24.90 | 35.02 | 16.78 | 18.77 | 15.27 | 29.61 | 23.99 | 32.91 | 15.86 | 16.50 | 13.51
AirBERT [30] | [ICCV2021] | 27.89 | 21.88 | 34.51 | 18.71 | 18.23 | 14.18 | 30.28 | 23.61 | 34.20 | 17.91 | 16.83 | 13.28
HAMT [11] | [NeurIPS2021] | 32.95 | 30.20 | 36.84 | 14.08 | 18.92 | 17.28 | 30.40 | 26.67 | 33.41 | 13.62 | 14.88 | 13.08
HOP [64] | [CVPR2022] | 30.39 | 25.10 | 35.30 | 17.16 | 18.23 | 15.31 | 29.12 | 23.37 | 32.26 | 17.05 | 17.13 | 13.90
DUET† [13] | [CVPR2022] | 46.98 | 33.73 | 51.07 | 22.11 | 32.15 | 23.03 | 52.51 | 36.06 | 56.91 | 21.30 | 31.88 | 22.06
Lana | (ours) | 34.00 | 29.26 | 38.54 | 16.28 | 19.03 | 16.18 | 33.50 | 26.89 | 36.41 | 16.75 | 17.53 | 14.25
Lana${}^{\dagger}~{}$ | (ours) | 48.31 | 33.86 | 52.97 | 23.18 | 32.86 | 22.77 | 51.72 | 36.45 | 57.20 | 18.83 | 32.95 | 22.85

Table 8: Additional quantitative results for instruction following on
REVERIE${}_{\\!}$ [63]. $\dagger$ indicates the model is trained on the DUET
dataset [13]. See §7 for details.

# | Route Encoder | Instruction Following | Instruction Generation
---|---|---|---
self_att | cross_att |  SR $\uparrow$ |  SPL $\uparrow$ |  OR $\uparrow$ |  TL $\downarrow$ |  SPICE $\uparrow$ |  Bleu-1 $\uparrow$ |  Bleu-4 $\uparrow$ |  CIDEr $\uparrow$ |  Meteor $\uparrow$ |  Rouge $\uparrow$
1 | ✔ |  | 65.6 | 60.1 | 72.7 | 11.7 | 0.202 | 0.703 | 0.262 | 0.375 | 0.224 | 0.476
2 |  | ✔ | 64.8 | 59.8 | 72.8 | 11.9 | 0.222 | 0.715 | 0.281 | 0.430 | 0.233 | 0.486
3 | ✔ | ✔ | 67.9 | 61.6 | 75.7 | 12.0 | 0.226 | 0.736 | 0.298 | 0.457 | 0.238 | 0.498

Table 9: Ablation study on R2R val unseen [4]. See §8 for details.

##  8 More Ablative Study on Route Encoder

In this section, we further study the efficacy of our route encoder design.
Our route encoder $\mathcal{E}^{r}$ considers both previous action tokens
$\\{\bm{a}_{t}\\}_{t}$ as well as past panoramic observations
$\\{\bm{O}_{t}\\}_{t}$ (see Eq. 3). We therefore report two variants, whose
route encoder 1) only performs the temporal self-attention over the previous
action tokens $\\{\bm{a}_{t}\\}_{t}$, and 2) only adopts the cross-attention
operation to perceive the historical panoramic observation
$\\{\bm{O}_{t}\\}_{t}$.

The results on R2R val unseen [4] are summarized in Table 9. As seen,
integrating both the historical panoramic observation and previous action
information yields the best performance.

##  9 Additional Qualitative Results

In this section, we provide more qualitative results for instruction following
and generation. Fig. 4 visualizes the comparison between Lanamt and Lanast on
instruction generation. We can observe that Lanamt generates more accurate and
vivid instructions. Concretely, Lanamt is able to not only describe precise
actions (_e.g_., turn left, walk through), but also highlight crucial landmark
(_e.g_., office, bathroom, toilet).

Fig. 5 compares Lanamt with Lanast on instruction following. Given the
challenging instruction “Leave the closet $\cdots$ on your left”, Lanamt
successfully take actions to reach the target location, while Lanast
terminates the navigation at a wrong position. This intuitively demonstrates
the effectiveness of the joint-training strategy.

Fig. 6 shows the real-time behavioral description provided by Lanamt. The
generated report keeps the monitor updated on the navigation process, and
reveals its inner decision mode. For examples, the route descriptions
generated for Step 1-3 and Step 3-8 can vividly explain to human how Lanamt
executes the complex command “Walk to the left of the table and chairs down
the hallway” – first “Walk into the room. Stop in front of the table,” then
“exit the room and walk through the hallway”. This case reveals the advantages
of Lana in interpretability and human-robot communication.

Figure 4: ${}_{\\!}$Visual${}_{\\!}$ comparison${}_{\\!}$ results${}_{\\!}$
between${}_{\\!}$ Lana${}_{mt\\!}$ and${}_{\\!}$ Lana${}_{st\\!}$ for the
instruction generation task (§9). The start and end points of a navigation
route are respectively denoted by  and . Figure 5: Visual comparison results
between Lanamt and Lanast for the instruction following task (§9). The start
and end points of a navigation route are respectively denoted by  and . Figure
6: Step-by-step navigation behavioral explanation (§9). The start and end
points of a navigation route are respectively denoted by  and . Lana is able
to interpret its navigation behavior using natural language. For example, at
step 2-4, Lanamt enters the room and then exits the room because Lanamt
intends to find the table mentioned in the instruction.

##  10 Discussion

Social Impact. A language-capable navigator can find much broader application
scenarios compared with previous “dumb” ones and can be more deeply involved
into human daily life. It can also serve as a guide robot to assist people who
are low-vision or blind.

Limitations. The agent is developed in virtual simulated environments. If the
algorithm is deployed on a real robot in a real dynamic environment, the
collisions during navigation can potentially cause damage to persons and
assets. More work should be done to practice real-world deployment, _e.g_.,
introducing hard constraints to the action space to avoid collisions, and
including additional experiments to study the risk of potential damage. In
addition, the generated route description, though informative and readable for
human, is a kind of post-hoc interpretation. It cannot perfectly and exactly
explain the inner decision mode of the agent.

Future work. In the future, in addition to investigating the efficacy of our
approach in other navigation tasks (_e.g_., object-goal navigation, audio-goal
navigation), we will design more compact architecture to jointly learn the two
tasks.

## References

  * [1] Sanyam Agarwal, Devi Parikh, Dhruv Batra, Peter Anderson, and Stefan Lee.  Visual landmark selection for generating grounded and interpretable navigation instructions.  In CVPR Workshop, 2019.
  * [2] Gary L Allen.  From knowledge to words to wayfinding: Issues in the production and comprehension of route directions.  In International Conference on Spatial Information Theory, 1997\.
  * [3] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  Spice: Semantic propositional image caption evaluation.  In ECCV, 2016.
  * [4] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  In CVPR, 2018.
  * [5] Jacob Andreas and Dan Klein.  Alignment-based compositional semantics for instruction following.  In EMNLP, 2015.
  * [6] Sean Andrist, Erin Spannan, and Bilge Mutlu.  Rhetorical robots: making robots more effective speakers using linguistic cues of expertise.  In International Conference on Human-Robot Interaction, 2013.
  * [7] Jacob Arkin, Daehyung Park, Subhro Roy, Matthew R Walter, Nicholas Roy, Thomas M Howard, and Rohan Paul.  Multimodal estimation and communication of latent semantic knowledge for robust execution of robot instructions.  The International Journal of Robotics Research, 39(10-11):1279–1304, 2020.
  * [8] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In ACL Workshop, 2005.
  * [9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In NeurIPS, 2020.
  * [10] David L Chen and Raymond J Mooney.  Learning to interpret natural language navigation instructions from observations.  In AAAI, 2011.
  * [11] Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev.  History aware multimodal transformer for vision-and-language navigation.  In NeurIPS, 2021.
  * [12] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev.  Learning from unlabeled 3d environments for vision-and-language navigation.  In ECCV, 2022.
  * [13] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev.  Think global, act local: Dual-scale graph transformer for vision-and-language navigation.  In CVPR, 2022.
  * [14] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In ECCV, 2020.
  * [15] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [16] Heriberto Cuayáhuitl, Nina Dethlefs, Lutz Frommberger, Kai-Florian Richter, and John Bateman.  Generating adaptive route instructions using hierarchical reinforcement learning.  In International Conference on Spatial Cognition, 2010.
  * [17] Amanda Cercas Curry, Dimitra Gkatzia, and Verena Rieser.  Generating and evaluating landmark-based navigation instructions in virtual environments.  In European Workshop on Natural Language Generation, 2015.
  * [18] Robert Dale, Sabine Geldof, and J Prost.  Using natural language generation in automatic route.  Journal of Research and Practice in Information Technology, 36(3):23, 2004.
  * [19] Andrea F Daniele, Mohit Bansal, and Matthew R Walter.  Navigational instruction generation as inverse reinforcement learning with neural machine translation.  In International Conference on Human-Robot Interaction, 2017.
  * [20] Laure De Cock, Kristien Ooms, Nico Van de Weghe, Nina Vanhaeren, Pieter Pauwels, and Philippe De Maeyer.  Identifying what constitutes complexity perception of decision points during indoor route guidance.  International Journal of Geographical Information Science, 35(6):1232–1250, 2021.
  * [21] Zhiwei Deng, Karthik Narasimhan, and Olga Russakovsky.  Evolving graphical planner: Contextual global planning for vision-and-language navigation.  In NeurIPS, 2020.
  * [22] Karan Desai and Justin Johnson.  Virtex: Learning visual representations from textual annotations.  In CVPR, 2021.
  * [23] Zi-Yi Dou and Nanyun Peng.  Foam: A follower-aware speaker model for vision-and-language navigation.  In NAACL, 2022.
  * [24] Mary T Dzindolet, Scott A Peterson, Regina A Pomranky, Linda G Pierce, and Hall P Beck.  The role of trust in automation reliance.  International Journal of Human-Computer Studies, 58(6):697–718, 2003.
  * [25] Gary W Evans, David G Marrero, and Patricia A Butler.  Environmental learning and cognitive mapping.  Environment and Behavior, 13(1):83–104, 1981.
  * [26] Daniel Fried, Jacob Andreas, and Dan Klein.  Unified pragmatic models for generating and following instructions.  arXiv preprint arXiv:1711.04987, 2017.
  * [27] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell.  Speaker-follower models for vision-and-language navigation.  In NeurIPS, 2018.
  * [28] Tsu-Jui Fu, Xin Wang, Matthew Peterson, Scott Grafton, Miguel Eckstein, and William Yang Wang.  Counterfactual vision-and-language navigation via adversarial path sampling.  In ECCV, 2020.
  * [29] Robert Goeddel and Edwin Olson.  Dart: A particle-based method for generating easy-to-follow directions.  In International Conference on Intelligent Robots and Systems, 2012\.
  * [30] Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid.  Airbert: In-domain pretraining for vision-and-language navigation.  In ICCV, 2021.
  * [31] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao.  Towards learning a generic agent for vision-and-language navigation via pre-training.  In CVPR, 2020.
  * [32] Yicong Hong, Cristian Rodriguez-Opazo, Yuankai Qi, Qi Wu, and Stephen Gould.  Language and visual entity relationship graph for agent navigation.  In NeurIPS, 2020.
  * [33] Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould.  A recurrent vision-and-language bert for navigation.  In CVPR, 2021.
  * [34] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Scaling up vision-language pre-training for image captioning.  In CVPR, 2022.
  * [35] Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhaes, Jason Baldridge, and Eugene Ie.  Transferable representation learning in vision-and-language navigation.  In ICCV, 2019.
  * [36] Zanming Huang, Zhongkai Shangguan, Jimuyang Zhang, Gilad Bar, Matthew Boyd, and Eshed Ohn-Bar.  Assister: Assistive navigation via conditional instruction generation.  In ECCV, 2022.
  * [37] Alycia M Hund and Jennifer L Minarik.  Getting from here to there: Spatial anxiety, wayfinding strategies, direction type, and wayfinding efficiency.  Spatial Cognition and Computation, 6(3):179–201, 2006.
  * [38] Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge.  Stay on the path: Instruction fidelity in vision-and-language navigation.  In ACL, 2019.
  * [39] Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa.  Tactical rewind: Self-correction via backtracking in vision-and-language navigation.  In CVPR, 2019.
  * [40] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL, 2019.
  * [41] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [42] Benjamin Kuipers.  Modeling spatial knowledge.  Cognitive Science, 2(2):129–153, 1978.
  * [43] Federico Landi, Lorenzo Baraldi, Marcella Cornia, Massimiliano Corsini, and Rita Cucchiara.  Perceive, transform, and act: Multi-modal attention networks for vision-and-language navigation.  arXiv preprint arXiv:1911.12377, 2020.
  * [44] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In ICML, 2022.
  * [45] Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah A Smith, and Yejin Choi.  Robust navigation with language pretraining and stochastic sampling.  In EMNLP, 2019.
  * [46] Chen Liang, Wenguan Wang, Tianfei Zhou, Jiaxu Miao, Yawei Luo, and Yi Yang.  Local-global context aware transformer for language-guided video.  PAMI, 2023.
  * [47] Chen Liang, Wenguan Wang, Tianfei Zhou, and Yi Yang.  Visual abductive reasoning.  In CVPR, 2022.
  * [48] Chuang Lin, Yi Jiang, Jianfei Cai, Lizhen Qu, Gholamreza Haffari, and Zehuan Yuan.  Multimodal transformer with variable-length memory for vision-and-language navigation.  In ECCV, 2022.
  * [49] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In Text Summarization Branches Out, pages 74–81, 2004.
  * [50] Gary Look, Buddhika Kottahachchi, Robert Laddaga, and Howard Shrobe.  A location representation for generating descriptive walking directions.  In Proceedings of the International Conference on Intelligent User Interfaces, 2005.
  * [51] Kristin L Lovelace, Mary Hegarty, and Daniel R Montello.  Elements of good route directions in familiar and unfamiliar environments.  In International Conference on Spatial Information Theory, 1999\.
  * [52] Kevin Lynch.  The Image of the City.  The MIT Press, 1960.
  * [53] Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong.  Self-monitoring navigation agent via auxiliary progress estimation.  In ICLR, 2019.
  * [54] Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira.  The regretful agent: Heuristic-aided navigation through progress estimation.  In CVPR, 2019.
  * [55] Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers.  Walk the talk: connecting language, knowledge, and action in route instructions.  In AAAI, 2006.
  * [56] Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra.  Improving vision-and-language navigation with image-text pairs from the web.  In ECCV, 2020.
  * [57] Hongyuan Mei, Mohit Bansal, and Matthew R Walter.  Listen, attend, and walk: Neural mapping of navigational instructions to action sequences.  In AAAI, 2016.
  * [58] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In ICML, 2016.
  * [59] Stefan Oßwald, Henrik Kretzschmar, Wolfram Burgard, and Cyrill Stachniss.  Learning to give route directions from human demonstrations.  In ICRA, 2014.
  * [60] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In ACL, 2002.
  * [61] Alexander Pashevich, Cordelia Schmid, and Chen Sun.  Episodic transformer for vision-and-language navigation.  In ICCV, 2021.
  * [62] Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, and Qi Wu.  Object-and-action aware model for visual language navigation.  In ECCV, 2020.
  * [63] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel.  Reverie: Remote embodied visual referring expression in real indoor environments.  In CVPR, 2020.
  * [64] Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu.  Hop: History-and-order aware pre-training for vision-and-language navigation.  In CVPR, 2022.
  * [65] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [66] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI Blog, 1(8):9, 2019.
  * [67] Kai-Florian Richter and Matt Duckham.  Simplest instructions: Finding easy-to-describe routes for navigation.  In International Conference on Geographic Information Science, 2008\.
  * [68] Raphael Schumann and Stefan Riezler.  Generating landmark navigation instructions from maps as a graph-to-text problem.  In ACL, 2021.
  * [69] Kristina Striegnitz, Alexandre Denis, Andrew Gargett, Konstantina Garoufi, Alexander Koller, and Mariët Theune.  Report on the second challenge on generating instructions in virtual environments (give-2.5).  In European Workshop on Natural Language Generation, 2011.
  * [70] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  In EMNLP, 2019.
  * [71] Hao Tan, Licheng Yu, and Mohit Bansal.  Learning to navigate unseen environments: Back translation with environmental dropout.  In NAACL, 2019.
  * [72] Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy.  Understanding natural language commands for robotic navigation and mobile manipulation.  In AAAI, 2011.
  * [73] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer.  Vision-and-dialog navigation.  In Conference on Robot Learning, 2020.
  * [74] Eric J Vanetti and Gary L Allen.  Communicating environmental knowledge: The impact of verbal and spatial abilities on the production and comprehension of route directions.  Environment and Behavior, 20(6):667–682, 1988.
  * [75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [76] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In CVPR, 2015.
  * [77] David Waller and Yvonne Lippa.  Landmarks as beacons and associative cues: their role in route learning.  Memory & Cognition, 35(5):910–924, 2007.
  * [78] Hanqing Wang, Wei Liang, Jianbing Shen, Luc Van Gool, and Wenguan Wang.  Counterfactual cycle-consistent learning for instruction following and generation in vision-language navigation.  In CVPR, 2022.
  * [79] Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang.  Towards versatile embodied navigation.  In NeurIPS, 2022.
  * [80] Hanqing Wang, Wenguan Wang, Wei Liang, Caiming Xiong, and Jianbing Shen.  Structured scene memory for vision-language navigation.  In CVPR, 2021.
  * [81] Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen.  Active visual information gathering for vision-language navigation.  In ECCV, 2020.
  * [82] Su Wang, Ceslee Montgomery, Jordi Orbay, Vighnesh Birodkar, Aleksandra Faust, Izzeddin Gur, Natasha Jaques, Austin Waters, Jason Baldridge, and Peter Anderson.  Less is more: Generating grounded navigation instructions from landmarks.  In CVPR, 2022.
  * [83] Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang.  Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation.  In CVPR, 2019.
  * [84] Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang.  Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation.  In ECCV, 2018.
  * [85] Xiaohan Wang, Linchao Zhu, and Yi Yang.  T2vlad: global-local sequence alignment for text-video retrieval.  In CVPR, 2021.
  * [86] Xiaohan Wang, Linchao Zhu, Zhedong Zheng, Mingliang Xu, and Yi Yang.  Align and tell: Boosting text-video retrieval with local alignment and fine-grained supervision.  IEEE TMM, 2022.
  * [87] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  Simvlm: Simple visual language model pretraining with weak supervision.  In ICLR, 2021.
  * [88] Shawn L Ward, Nora Newcombe, and Willis F Overton.  Turn left at the church, or three miles north: A study of direction giving and sex differences.  Environment and Behavior, 18(2):192–213, 1986.
  * [89] Ronald J Williams and David Zipser.  A learning algorithm for continually running fully recurrent neural networks.  Neural Computation, 1989.
  * [90] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  arXiv preprint arXiv:2205.01917, 2022.
  * [91] Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang.  Centerclip: Token clustering for efficient text-video retrieval.  In SIGIR, 2022.
  * [92] Yusheng Zhao, Jinyu Chen, Chen Gao, Wenguan Wang, Lirong Yang, Haibing Ren, Huaxia Xia, and Si Liu.  Target-driven structured transformer planner for vision-language navigation.  In ACMMM, 2022.
  * [93] Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang.  Vision-language navigation with self-supervised auxiliary reasoning tasks.  In CVPR, 2020.