# NaVid: Video-based VLM Plans the Next Step
for Vision-and-Language Navigation

Jiazhao Zhang1,2,∗ Kunyu Wang2,∗ Rongtao Xu2,3,∗ Gengze Zhou4 Yicong Hong5
Xiaomeng Fang2 Qi Wu4 Zhizheng Zhang6,† Wang He1,2,†
1 CFCS, School of Computer Science, Peking University 2 Beijing Academy of
Artificial Intelligence
3 CASIA 4 University of Adelaide 5 Australian National University 6 GalBot

###### Abstract

Vision-and-Language Navigation (VLN) stands as a key research problem of
Embodied AI, aiming at enabling agents to navigate in unseen environments
following linguistic instructions. In this field, generalization is a long-
standing challenge, either to out-of-distribution scenes or from Sim to Real.
In this paper, we propose NaVid, a video-based large vision language model
(VLM), to mitigate such a generalization gap. NaVid makes the first endeavour
to showcase the capability of VLMs to achieve state-of-the-art level
navigation performance without any maps, odometer and depth inputs. Following
human instruction, NaVid only requires an on-the-fly video stream from a
monocular RGB camera equipped on the robot to output the next-step action. Our
formulation mimics how humans navigate and naturally gets rid of the problems
introduced by odometer noises, and the Sim2Real gaps from map or depth inputs.
Moreover, our video-based approach can effectively encode the historical
observations of robots as spatio-temporal contexts for decision making and
instruction following. We train NaVid with 550k navigation samples collected
from VLN-CE trajectories, including action-planning and instruction-reasoning
samples, along with 665k large-scale web data. Extensive experiments show that
NaVid achieves SOTA performance in simulation environments and the real world,
demonstrating superior cross-dataset and Sim2Real transfer. We thus believe
our proposed VLM approach plans the next step for not only the navigation
agents but also this research field. We will release the code and data to
benefit the community.

00footnotetext: * Joint first authors (e-mail:
zhngjizh@gmail.com).00footnotetext: $\dagger$ Corresponding author (e-mail:
hewang@pku.edu.cn).

##  I Introduction

As a fundamental task of Embodied AI, vision-and-language navigation (VLN)
[30, 66] requires the agents to navigate in diverse and especially unseen
environments following free-form linguistic instructions. VLN requires robots
to understand complex and diverse visual observations and meanwhile interpret
fine-grained instructions [13, 90], “go up the stairs and stop in the
doorway”, and thus maintains to be a challenging task. To address this
challenging task, a large volume of research [78, 18, 93, 89, 65, 46, 4] in
this field is launched in the simplified setting, i.e., decision-making in
discrete environments (e.g., R2R [44] in MP3D simulator [12]). Specifically,
the real environments are abstracted as connectivity graphs, and the
navigation is cast as the teleportation over a waypoint set on such graphs.
Despite rapid improvements that have been made, this modeling presents Sim-to-
Real gaps as robots move in a continuous manner in practice.

The generalization issue presents a critical yet under-explored challenge in
large-scale real-world deployment, encompassing transitions from seen scenes
to novel environments and from Simulation to Real-world (Sim-to-Real). The
recent prosperity of large Vision Language Models (VLMs) has shown
unprecedented promises in a lot of research fields [98, 50, 3]. In this paper,
we explore whether large models can do the same in propelling generalizable
VLN.

Toward more realistic modeling, navigation in continuous environments, e.g.,
R2R-CE, RxR-CE, is bringing more and more attention. Lots of excellent
research efforts are devoted to reducing the Sim-to-Real gaps by studying the
transferring technologies from discrete modeling to continuous modeling [45,
35] or from Sim-to-Real [97, 9]. However, they still face severe challenges in
generalization due to data scarcity and domain gaps in their model inputs
including RGBD, odometer data or maps.

Figure 1: Real-world demo of our proposed video-based VLM, NaVid, for vision-
and-language navigation. Given the human instruction, NaVid only takes online
RGB video frames as input and outputs a language action for robotic execution.

Large models have exhibited impressive generalization capacities in a broad
range, covering AIGC [74, 10, 101], generalist chatbot [2], and autonomous
driving [27], etc. They have also been shaping the future of embodied AI. RT-2
[11] shows the promise of transferring web knowledge from VLMs to
generalizable robotic manipulation. Large Language Models (LLMs) have been
effective as the planners for VLN in discrete environments [60, 65, 104, 14,
70]. The recent advancements in VLMs are ushering the VLN research into an
exciting juncture. It is the time to study whether VLMs can boost the
generalization of VLN in continuous real-world environments.

In this paper, we make the first endeavor to capitalize on the power of
foundational VLMs for generalizing VLN to the real world and propose a video
VLM-based navigation agent, dubbed NaVid. It solely relies on videos captured
by a robot’s monocular camera and humane-issued instructions as inputs for
planning the next-step action in an end-to-end manner. We put the proposed
NaVid into perspective by comparing it with three categories of models. 1)
Compared with AGI models [98] or so-called navigation generalists [103] that
can perform coarse navigation planning, NaVid is a practical Vision-Language-
Action (VLA) model that can infer executable actions with quantitative
arguments, e.g., moving distances and rotation degrees. This allows NaVid to
be deployed in the real world. 2) Compared to the VLN models that employ LLMs
as planners, NaVid adopts a more realistic modeling for VLN. Especially, NaVid
directly deduces low-level executable actions in continuous environments and
encodes visual observations in the video form, unlike previous LLM-based
methods of modeling VLN in discrete spaces or encoding historical observations
with textual descriptions [65, 104, 60, 14]. 3) Distinct from existing
specialized VLN models, NaVid eliminates the reliance on odometer data, depths
and maps for action planning, thereby obviating the need for generalization
challenges posed by odometer noises or the domain discrepancies in depth
perception or navigation maps, making NaVid easy to be deployed. To the best
of our knowledge, the proposed NaVid is the first video-based VLM for VLN in
continuous environments, achieving RGB-only navigation akin to human
navigational behavior.

Our proposed NaVid employs a pre-trained vision encoder to encode visual
observations and a pre-trained LLM to reason navigation actions. In this way,
the general-purpose knowledge acquired in large-scale pretraining is
transferred to VLN tasks, facilitating the learning and boosting the
generalization. Drawing inspiration from the advanced video-based VLM, LLaMA-
VID [55], we represent each frame in robotic visual observations with two
kinds of tokens. The first kind consists of an instruction-queried token that
extracts visual features specifically relevant to given instructions. The
other kind comprises instruction-agnostic tokens that globally encode fine-
grained visual information, where the token number determines the granularity
of encoded features. The number of tokens for historical observations is
allowed to differ from that for current observations. As such, in NaVid,
robotic historical trajectories are encoded as visual tokens in the video
form, which provide more informative and adaptive context compared to prior
LLM-based VLN models that perform encoding in discrete spaces [18, 19] or
using textual descriptions [65, 104, 60, 14]. Such video-based modeling
imposes stringent constraints on the model inputs as it does not involve other
information, e.g., depth, odometer data or maps, except for the monocular
video. When harnessed in the right manner, it actually advances in mitigating
the generalization challenges resulting from odometer noises and domain
discrepancies in depth perception or navigation maps of previous VLN works.

We conduct extensive experiments for the evaluation of our proposed NaVid in
both simulated and real-world environments. Specifically, NaVid achieve SOTA-
level performance on VLN-CE R2R dataset and showcases a significant
improvement (e.g., from 12.1 to 21.2 in SPL) on cross-dataset evaluation
(R2R-RxR). Besides, our method demonstrates impressive robustness on Sim-to-
Real deployment, achieving about $66\%$ success rate on 200 instructions
across four diverse indoor scenes, leveraging only RGB videos as inputs.

##  II Related works

Vision-and-Language Navigation (VLN). A large endeavor of learning to navigate
in unvisited environments following human instructions is established over
discretized simulated scenes [7, 47, 68, 88], where agents teleport between
nodes on a pre-defined navigation graph by aligning language and visual
observations for decision-making [61, 91, 26, 87, 40, 28, 67, 33]. Despite
efficiency, directly transferring VLN models trained in discrete space to
real-world robot applications is impractical. As a result, the more realistic
VLN in continuous environments (VLN-CE) is proposed [43, 76], allowing agents
to navigate freely to any unobstructed space in the simulator either by
predicting low-level controls [73, 38, 15, 29, 16] or selecting from navigable
sub-goals estimated by waypoint predictors [35, 45, 42]. Meanwhile, following
the success of learning generic visual-linguistic representations from web-
scale image-text pairs [20, 54, 82, 51, 86], many VLN models benefited from
large vision-language models [53, 34, 18, 19] and VLN-specific pre-training
[32, 64, 31, 96, 69]. Very recently, by scaling up navigation training data,
the performance of VLN agents on the well-recognized R2R benchmark [44] is
approaching humans [93]. This significant development suggests that the
implementation of VLN techniques in real-world robotics is an increasingly
viable and timely consideration.

Figure 2: The overview of NaVid. The inputs of NaVid consist of the RGB frames
from the online video observation $\\{\mathbf{x}_{0},\cdots,\mathbf{x}_{t}\\}$
along with the human instruction $\mathcal{I}$. For each frame, we use an
observation encoder to extract the visual information with the instruction to
obtain observation tokens, including, instruction-queried tokens (orange
blocks) and instruction-agnostic tokens (blue blocks). At the current step
$t$, the history frames and current frame $\mathbf{x}_{t}$ are encoded as
observation tokens, with 4 and 64 instruction-agnostic tokens for history
frames and current frames, respectively. Besides, our method obtains language
tokens by a text encoder. Finally, split by the special tokens [HIS], [OBS],
and [NAV], we concatenate the observation tokens and language tokens and send
the tokens to the Vicuna-7B then obtain the next-step action.

Sim-to-Real Transfer for VLN. Despite great advances, existing VLN methods are
predominantly built and evaluated in simulation, which largely overlooks the
intricate and unpredictable nature of real-world conditions. Sim-to-real VLN
transfer is an under-studied topic; until today, the only literature that
systematically studies this problem was from Anderson et al. [8], which
justifies the performance gap (more than 50% drop in success rate) due to
action space and visual domain differences. In addition, we would like to
highlight the challenge of generalizing to free-form language instructions -
agents often fail to interpret different styles of instructions even when
trained on millions of in-domain visual data [93, 39]. In light of this, many
recent researches leverage the exceptional generalization abilities of the
Large (Vision-)Language Models to facilitate VLN generalization. Either
investigating the navigational reasoning capacity intrinsic to LLMs [104, 65,
103, 60, 14, 77, 56] or incorporating LLMs into navigation systems through
modular designs that facilitate instruction parsing [17] or through the
injection of commonsense knowledge [70]. We follow such a trend and further
explore how to utilize a unified large model for low-level action prediction
and its generalizability in real-world scenarios. This approach seeks not only
to advance the state of VLN by leveraging the comprehensive understanding and
versatile capabilities of LLMs [104, 65, 103, 60, 14, 70] but also to bridge
the gap between simulated environments and the multifaceted challenges
presented by real-world applications.

Large Models as Embodied Agents. Recently, researchers have been exploring the
integration of large models into diverse embodied domains [25, 104, 57, 79,
81, 77, 37]. PaLM-E [25], for instance, suggests the incorporation of tokens
from various modalities, alongside text tokens, into a large model. The model
then generates high-level robotics instructions for tasks such as mobile
manipulation, motion planning, and tabletop manipulation. Taking a step
further, RT-2 [11] generates low-level actions for robots, facilitating
closed-loop control. GR-1 [95] introduces a GPT-style model [71, 104, 60]
specifically designed for multi-task language-conditioned visual robot
manipulation [99]. This model predicts robot actions and future images based
on language instructions, observed images, and robot states. RoboFlamingo [52]
proposes a vision-language manipulation framework that utilizes pre-trained
vision-language models to formulate manipulation policies for robotics. It
aims to offer a cost-effective and high-performance solution for robot
manipulation [22, 85, 99, 80], allowing users to fine-tune their robots with
large models. Diving deeper into these works, this paper centers on another
pivotal embodied domain: vision-and-language navigation, which necessitates
robots to navigate in unseen environments following human instructions.

##  III Problem Formulation

The formulation of Vision-and-Language Navigation in Continuous Environments
(VLN-CE) in this work is as follows: At the time $t$, given a natural language
instruction $\mathcal{I}$ consisting of $l$ words and a video observation
$\mathcal{O}_{t}$ comprising a sequence of frames
$\\{\mathbf{x}_{0},\cdots,\mathbf{x}_{t}\\}$, the agent is required to plan a
low-level actions $\mathbf{a}_{t+1}\in\mathcal{A}$ for the next step. This
action will take the agent to the next state and the agent will receive a new
observation $\mathbf{x}_{t+1}$. Overall, we can formulate the decision-making
as a Partially Observable Markov Decision Process (POMDP), denoted by
$\\{\mathbf{x}_{0},\mathbf{a}_{1},\mathbf{x}_{1},\mathbf{a}_{2},\cdots,\mathbf{x}_{t}\\}$.
In this work, the observation space $\mathcal{O}$ corresponds to the videos
captured with a monocular RGB camera without other additional data involved,
and the action space incorporates qualitative action types alongside
quantitative action arguments, also known as low-level actions [43] in this
domain. This modeling enables a natural paradigm where observations are purely
vision-based and readily obtainable, while actions are directly executable,
akin to human navigational behavior.

##  IV The Proposed NaVid Agent

With the formulation in Sec.III, we devise a video VLM-based navigation agent,
named NaVid. The NaVid is the first of its kind to transfer general knowledge
of VLMs to a realistic VLN agent. We introduce its architecture in Sec. IV-A,
then elaborate on the detailed modeling for VLN inputs and outputs in Sec.
IV-B. The training strategy is detailed in Sec. IV-C, and the implementation
details are described in Sec. IV-D.

###  IV-A Overall Architecture

We build NaVid on top of a general-purpose video-based VLM named LLaMA-VID
[55]. For our proposed NaVid, we inherit the main architecture of LLaMA-VID
and incorporate the task-specific designs on top of it, to facilitate the
transfer of general knowledge to VLN-CE so as to make its generalization
challenges more readily solvable.

As illustrated in Fig. 2, NaVid consists of a vision encoder, a query
generator, a LLM and two cross-modality projectors. Given the observations up
to time $t$, i.e., a video sequence comprising $t$ frames, we encode this
video to a sequence of tokens via the vision encoder (EVA-CLIP [83] in
implementation) and project them to a space aligned with language tokens. For
brevity, we call the projected tokens as observation tokens. As common, the
instructions are also tokenized as a set of tokens, called instruction tokens.
We concatenate both observation tokens and instruction tokens and send them to
the LLM to infer the VLN actions in linguistic form. Note that our work
focuses on task-specific modeling rather than model architecture, as detailed
in the following.

###  IV-B The VLN-CE Modeling of NaVid

Observation encoding. Given the captured monocular video up to time $t$,
denoted by $\mathcal{O}_{t}=\\{\mathbf{x}_{0}.\cdots,\mathbf{x}_{t}\\}$, we
represent each frame with one instruction-queried visual token and several
instruction-agnostic visual tokens. The instruction-queried tokens extract
visual features specifically relevant to given instructions, and the
instruction-agnostic tokens globally encode fine-grained visual information.
For each frame $\mathbf{x}_{t}$, we first get its visual embedding
$\mathbf{X}_{t}\in\mathbb{R}^{N_{x}\times C}$ with a vision encoder, where
$N_{x}$ is the patch number ($N_{x}$ is set to 256) and $C$ is the embedding
dimension.

To get the instruction-queried tokens, we employ a Q-Former-based query
generator to generate the instruction-aware query
$\mathbf{Q}_{t}\in\mathbb{R}^{M\times C}$ with the query generator $G_{Q}$,
where $M$ denotes the number of queries per frame and $C$ is the dimension of
each query. The query generation can be formulated as:

$\mathbf{Q}_{t}=G_{Q}(\mathbf{X}_{t},\mathbf{I}),$ (1)

where $\mathbf{X}_{t}$ and $\mathbf{I}$ are the visual embeddings and the
textual embeddings of the frame $\mathbf{x}_{t}$ and the instruction
$\mathcal{I}$. $G_{Q}(\cdot)$ is a Q-Former-based transformer to learn
instruction-aware queries via a cross-modality interaction between
$\mathbf{X}_{t}$ and $\mathbf{I}$ as that in [55]. Similar to the operation in
Q-Former [50], the instruction-queried tokens $\mathbf{E}^{Q}_{t}$ are
obtained by a cross-attention between $\mathbf{X}_{t}$ and $\mathbf{Q}_{t}$,
formulated as below:

$\mathbf{E}^{Q}_{t}=P_{Q}(Pool(Softmax(\mathbf{Q}_{t}\mathbf{X}_{t}^{T})\mathbf{X}_{t})),$
(2)

where $P_{Q}(\cdot)$ represents the cross-modality projector for getting
instruction-quired tokens, and $Pool(\cdot)$ is an averaging operation along
the dimension of queries, making $\mathbf{E}^{Q}_{t}\in\mathbb{R}^{1\times
C}$.

For the instruction-agnostic tokens, we directly perform a grid pooling
operation and a cross-modality projection to get them, which can be formulated
as:

$\mathbf{E}^{V}_{t}=P_{V}(GridPool(\mathbf{X}_{t})),$ (3)

where $GridPool(\cdot)$ is a pooling operation adopted in spatial [55],
squeezing the tokens from $N_{x}$ to $N_{v}$, yielding
$\mathbf{E}^{V}_{t}\in\mathbb{R}^{N_{v}\times C}$. Representing each frame
with two tokens as in LLaMA-VID [55] does not meet the requirements of the
VLN-CE task, as experimentally evidenced in the following. This is because
LLaMA-VID is primarily designed for high-level question-answering tasks,
whereas NaVid needs to plan executable actions for robots. Thus, we adopt the
grid pooling here to enable the instruction-agnostic tokens to preserve
sufficient geometry information so that the LLM in NaVid have enough contexts
for reasoning the quantitative arguments of robotic actions.

For VLN-CE, the current frame serves as the primary basis for navigation
action reasoning, while historical frames provide important contexts for
tracing navigation progress. Considering their different requirements for
preserving geometry information, we adopt a varying number of instruction-
agnostic tokens in encoding the historical frames and the current frame. In
this work, unless otherwise specified, we set the number of instruction-
agnostic tokens to 64 for the current frame while 4 for each historical frame.
This not only facilitates the learning but also improves the efficiency. To
further facilitate the training of NaVid, we explicitly distinguish different
types of information with special tokens before sending them to the LLM within
NaVid.

Specifically, we adopt special tokens <HIS>, </HIS>, and <OBS>, </OBS> to
demarcate tokens encoded from historical frames and the current frame,
respectively. Here, <HIS> and <OBS> signify the beginning of the corresponding
token sequences while </HIS> and </OBS> mark their ends. Additionally, we use
another special token <NAV> to prompt the LLM to begin to process the textual
tokens of instructions and output the robotic actions in linguistic form. As a
result, the input of NaVid can be summarized as below.

Input:
$<\\!HIS\\!>\\{historical\\_frames\\}<\\!/HIS\\!><\\!OBS\\!>\\{current\\_frame\\}<\\!/OBS\\!>)<\\!NAV\\!>\\{instruction\\_content\\}$
Output: $\\{answer\\_content\\}$

In this format, $\\{historical\\_frames\\}$, $\\{current\\_frame\\}$,
$\\{instruction\\_content\\}$ and $\\{answer\\_content\\}$ are the
placeholders for the tokens of historical frames, current frame, instruction,
and reasoned actions, respectively.

Action planning. NaVid plans the next-step action for VLN-CE in linguistic
form. Each action of its outputs consists of two variables aligned with the
setting of VLN-CE. One of them is the action type, chosen from a discrete set
$\\{\texttt{FORWARD},\texttt{TURN-LEFT},\texttt{TURN-RIGHT},\texttt{STOP}\\}$.
The other is the quantitative arguments corresponding to different action
types. For FORWARD, NaVid further infers the specific moving distance. For
TURN-LEFT and TURN-RIGHT, NaVid also predicts specific rotation degrees. A
regular expression parser [41] is employed to extract the action types and
arguments for model evaluation and real-world deployment.

###  IV-C The Training of NaVid

The available navigational simulation data are still limited in their
diversity, authenticity and scale. We design a hybrid training strategy to
maximize the utilization of these data in enabling NaVid to generalize as
effectively as possible to novel scenes or the real world. To this end, two
key approaches are proposed in the hybrid training strategy. One is to collect
non-oracle navigation trajectories and incorporate them into the training
loop. The other is to design auxiliary tasks to enhance the capabilities of
NaVid on navigation scene understanding and instruction following. We
elaborate them separately below.

Figure 3: Examples of navigation data, comprising the action planning sample
(above) and the instruction reasoning sample (below).

Non-oracle navigation trajectories collection. Inspired by the Dagger
technique [75], we collect non-oracle navigation trajectories and incorporate
them into the training of NaVid. Without this approach, our NaVid would only
be exposed to oracle navigation trajectories during training, which diverges
from practical application conditions and diminishes the robustness of the
learned navigation strategy. To achieve this, we first collect oracle
navigation trajectories, including monocular video observations, instructions
and robotic actions, from the VLN-CE R2R dataset. Specifically, we gather data
from 61 MP3D indoor scenes [12], containing 320k step-wise samples in total.
Then, we train NaVid on these oracle trajectory data and deploy the obtained
agent in VLN-CE environments for further collecting the non-oracle navigation
trajectories. As a result, we get another 180k step-wise samples. The samples
from both oracle and non-oracle trajectories are combined for the final
training of NaVid, as shown in the above of Fig. 3.

Co-training of VLN-CE and auxiliary tasks. As a navigation agent, precisely
understanding environments and following given instructions are two
indispensable capabilities besides planning navigation actions. To facilitate
agent learning, we combine the VLN-CE action planning together with two
auxiliary tasks in a co-training paradigm. For the environment understanding,
we design an auxiliary task as that given the video of a complete navigation
trajectory, namely instruction reasoning, we ask NaVid to reason the
instruction corresponding trajectory. This auxiliary task can be easily
implemented with a shared data organization format introduced in Sec. IV-B, in
which the $\\{instruction\\_content\\}$ and $\\{answer\\_content\\}$ can be
instantiated as the prompts of requesting the descriptions of robotic
navigation trajectories and the human-labeled instructions provided in the
dataset. The instruction reasoning auxiliary task contains 10k trajectories,
an example is provided in the below of Fig. 3. Moreover, for the enhancement
of instruction following and the anti-forgetting of general knowledge acquired
in pre-training, we incorporate the video-based question-answering samples
into our co-training as well. Details can be found in [55]. For brevity, we do
not elaborate on them here.

###  IV-D Implementation details

Training configurations. NaVid is trained on a cluster server with 24 NVIDIA
A100 GPUs for approximately 28 hours, totaling 672 GPU hours. For video-
caption data, we sample frames at 1 FPS, to remove redundant information
between consecutive frames. We keep all the frames for navigation-action data,
mostly smaller than 300 frames. During training, all modules including EVA-
CLIP [83], QFormer [23], BERT [24] and Vicuna-7B [21] are loaded with default
pre-trained weight. Following the strategy in [55], we only optimize the
trainable parameters of LLaMA and text encoder for only 1 epoch.

Evaluation configurations. After NaVid predict language actions, we leverage
regular expressions matching [41] for expecting valid actions. We find this
simple algorithm achieves a $100\%$ success rate for obtaining valid actions
under the VLN-CE val-unseen R2R evaluation. For real-world navigation, we use
a remote server to run NaVid to receive observations (along with text
instructions) and command a local robot to execute the predicted actions.
During navigation, the agent requires about 1.2 to 1.5 seconds to output one
action per frame. This could be improved by using acceleration techniques,
such as quantization [59, 58].

##  V Experiments

###  V-A Experiment Setup

Simulated environments. We evaluate our method on the VLN-CE benchmarks, which
provide continuous environments for executing low-level actions on
reconstructed photorealistic indoor scenes [43]. We consider R2R [44] and RxR
[47] in VLN-CE, the two most recognized benchmarks of VLN. For a fair
comparison, all methods are trained on the 10,819 R2R train split, and
evaluated on the 1,839 R2R val-unseen split and 1,517 RxR val-unseen split to
evaluate cross-split and cross-datasets performance, respectively.

Real-world environments. To evaluate the performance of our method in real-
world environments, we follow the experiment setting of [8, 97] and design
comprehensive experiments which include different indoor scenes and different
difficulty instructions. We select four diverse indoor scenes including
Meetting_room, Office, Lab, and Lounge. For instruction, we design two types
of instruction: 1) Simple landmark instruction following task, which requires
the agent to understand the semantics of the target and move to the relative
location of the target. 2) Complex Composite instruction following Task, which
requires the agent to complete 2-5 combination instructions of landmark
following task. For each selected scene, we design 25 Simple landmark
instructions and 25 complex composite instructions, leading to a total of 200
instruction-following cases.

We conduct all real-world experiments using a Turtlebot4111Turtlebot4
overview: https://clearpathrobotics.com/turtlebot-4/. equipped with a Kinect
DK camera for capturing both depth and RGB images. For baselines that require
odometry, we use a built-in RPLIDAR A1M8 Lidar of the turtlbeot4 and leverage
the toolbox of Nav2 [62] for localization and mapping. The calibration of the
Lidar and camera follows the previous work [102]. Please refer to the
supplemental material for details.

Metrics. We follow the standard VLN evaluation metrics [7, 44, 48] to evaluate
the navigation performance, including success rate (SR), oracle success rate
(OS), success weighted by path length (SPL) [6], trajectory length (TL), and
navigation error from goal (NE). Among all, SPL is the primary metric as it
reflects both the accuracy and efficiency of navigation [5, 49]. Note that an
episode is considered successful if the agent calls the STOP action within 3 m
of the goal in the VLN-CE and 1.5 m in real-world environments. More details
about evaluation metrics can be found in the supplemental material.

Baselines. For a fair comparison with NaVid, we compare the performance of
methods that directly predict low-level action primitives in the VLN-CE
environments:

  * •

Seq2Seq [43] is a simple sequence-to-sequence baseline that uses a recurrent
policy to predict the action directly from the RGBD observations. RGB-Seq2Seq
indicates the use of RGB observations only.

  * •

CMA [43] utilizes cross-model attention between instruction and RGBD-
observations to predict action. RGB-CMA indicates the use of RGB observations
only.

  * •

WS-MGMap [16] leverages a multi-granularity map, which contains both object
geometry, texture, and semantic information.

###  V-B Comparison on Simulated Environment

VLN-CE R2R. We first conduct experiments on the VLN-CE R2R Val-Unseen dataset
to evaluate the cross-split generalizability of methods. The results can be
found in Table I. Here we mark the specific observations required by the
methods, whereas our method only uses RGB observations. Compared to the
related VLN-CE setting (low-level action space), our method achieves SOTA
performance in terms of SPL, NE, OS, and comparable performance on SR, without
using depth and odometry information. For the methods Seq2Seq-RGB and CMA-RGB
that share the same setting, our method significantly outperforms these
methods by improving the SR from $5.00\%$ to $37.4\%$, and SPL from $4.43\%$
to $35.9\%$. Our method outperforms the SOTA method WS-MGMap by $1.6\%$
regarding SPL. The results demonstrate the effectiveness of our method by
showing SOTA-level performance with only RGB observations.

TABLE I: Comparing on VLN-CE R2R Val-Unseen. ∗: Methods use high-level action
space. ${\dagger}$: Methods use the same waypoint predictor proposed in [35].
${\ddagger}$: Methods use additional visual data than MP3D scenes [12].

| Observation | VLN-CE R2R Val-Unseen
---|---|---
Pan. | S.RGB | Depth | Odo. | TL | NE$\downarrow$ | OS$\uparrow$ | SR$\uparrow$ | SPL$\uparrow$
HPN+DN∗ [45] | ✓ |  | ✓ | ✓ | 7.62 | 6.31 | 40.0 | 36.0 | 34.0
CMA∗${\dagger}$ [35] | ✓ |  | ✓ | ✓ | 10.90 | 6.20 | 52.0 | 41.0 | 36.0
VLN  $\circlearrowright$ BERT∗${\dagger}$ [35] | ✓ |  | ✓ | ✓ | 12.23 | 5.74 | 53.0 | 44.0 | 39.0
Sim2Sim∗ [42] | ✓ |  | ✓ | ✓ | 10.69 | 6.07 | 52.0 | 43.0 | 36.0
GridMM∗${\dagger}$ [92] | ✓ |  | ✓ | ✓ | 13.36 | 5.11 | 61.0 | 49.0 | 41.0
HAMT∗${\dagger}$${\ddagger}$ [93] | ✓ |  | ✓ | ✓ | – | 4.80 | – | 55.0 | 51.0
ETPNav∗ [4] | ✓ |  | ✓ | ✓ | 11.99 | 4.71 | 65.0 | 57.0 | 49.0
AG-CMTP [15] | ✓ |  | ✓ | ✓ | – | 7.90 | 39.2 | 23.1 | 19.1
R2R-CMTP [15] | ✓ |  | ✓ | ✓ | – | 7.90 | 38.0 | 26.4 | 22.7
LAW [73] |  | ✓ | ✓ | ✓ | 8.89 | 6.83 | 44.0 | 35.0 | 31.0
CM2 [29] |  | ✓ | ✓ | ✓ | 11.54 | 7.02 | 41.5 | 34.3 | 27.6
WS-MGMap [16] |  | ✓ | ✓ | ✓ | 10.00 | 6.28 | 47.6 | 38.9 | 34.3
Seq2Seq [43] |  | ✓ | ✓ |  | 9.30 | 7.77 | 37.0 | 25.0 | 22.0
CMA [43] |  | ✓ | ✓ |  | 8.64 | 7.37 | 40.0 | 32.0 | 30.0
RGB-Seq2Seq |  | ✓ |  |  | 4.86 | 10.1 | 8.10 | 0.00 | 0.00
RGB-CMA |  | ✓ |  |  | 6.28 | 9.55 | 10.8 | 5.00 | 4.43
Ours |  | ✓ |  |  | 7.63 | 5.47 | 49.1 | 37.4 | 35.9

TABLE II:  Comparing cross-dataset performance on VLN-CE RxR Val-Unseen. Note
that, $A^{2}$Nav is a zero-shot method which leverage GPT as the planner.

| Observation | VLN-CE RxR Val-Unseen
---|---|---
S.RGB | Depth | Odo. | TL | NE$\downarrow$ | OS$\uparrow$ | SR$\uparrow$ | SPL$\uparrow$
LAW [73] | ✓ | ✓ | ✓ | 4.01 | 10.87 | 21.0 | 8.0 | 8.0
CM2 [29] | ✓ | ✓ | ✓ | 12.29 | 8.98 | 25.3 | 14.4 | 9.2
WS-MGMap [16] | ✓ | ✓ | ✓ | 10.80 | 9.83 | 29.8 | 15.0 | 12.1
Seq2Seq [43] | ✓ | ✓ |  | 1.16 | 11.8 | 5.02 | 3.51 | 3.43
CMA [43] | ✓ | ✓ |  | 5.09 | 11.7 | 10.7 | 4.41 | 2.47
RGB-Seq2Seq | ✓ |  |  | 4.43 | 11.2 | 12.2 | 0.0 | 0.0
RGB-CMA | ✓ |  |  | 13.56 | 9.55 | 14.8 | 0.0 | 0.0
$A^{2}$Nav [17] | ✓ |  |  | – | – | – | 16.8 | 6.3
Ours | ✓ |  |  | 10.59 | 8.41 | 34.5 | 23.8 | 21.2

VLN-CE RxR. To evaluate the cross-dataset performance of methods, we train the
models on R2R trajectory-instruction samples and compare their zero-shot
performance on RxR Val-Unseen data split. RxR dataset contains instructions of
finer granularity that describe rich landmarks and longer trajectories, which
establishes a large language and visual gap that is suitable for evaluating
the generalization potential of the methods. As shown in Table II, we find
that our method outperforms existing methods by a large margin in terms of NE,
OS, SR and SPL. Note that the trajectory length (TL) of Seq2Seq is extremely
small because of its poor instruction-following ability ($3.51\%$ SR), leading
by early stopping. Compared with $A^{2}$Nav [17], the current SOTA method in
zero-shot VLN-CE RxR, our method achieves $41.7\%$ (from 16.8 to 23.8) and
$236.5\%$ (from 6.3 to 21.2) better results on SR and SPL. Besides, the
Seq2Seq-RGB and CMA-RGB baselines, which share the same setting, fail on all
instructions, demonstrating the challenges of RGB-only methods. This proves
the generalization potential of our method by incorporating the video-based
large vision-language model into VLN for addressing free-form language
navigation tasks.

Comparison with Large foundation models. We then want to evaluate the
performance of variant mainstream large foundation models on the vision
language navigation task. Here, GPT-4V [98] and Emu [84] support multi-
conversation with image inputs, so in each iteration of conversation we feed a
new observed RGB image. LLaVA [58] is a sing-conversation image model, so we
leverage an observation-to-history technique to encode history. LLaMA-VID [55]
is able to answer questions based on the video inputs, which can be replaced
by the navigation observations. For LLaVA and LLaMA-VID which released the
training dataset, we co-tune them with we proposed navigation data, and we
indicate them as LLaVA-Nav and LLaMA-VID-Nav. All baselines are carefully
tuned and promoted, and the details can be found in supplemental material.

Considering the extreme computation cost such as promoting history in GPT-4V
and LLaVA, we randomly sample 100 episodes as a sub-split of VLN-CE R2R Val-
Unseen sub-split (the ID of sampled episodes can be found in the supplemental
material for reproducing). The results can be found in Table III. We find that
the methods without training on proposed navigation data suffer a very poor
performance. Especially, Emu, LLaVA and LLaMA-VID frequently output action-
irrelevant answers, such as environment description or navigation comments,
making the agent very hard to navigate. A more powerful model GPT-4V, with
carefully in-context prompt [104, 36], can relatively stable output valid
actions, showcasing a lower performance at $5\%$ SR. After tuning with the
proposed navigation data, we find significant improvements in the LLaVA-Nav
and LLaMA-VID-Nav, which demonstrate the importance of the collected
navigation data. Besides, training on navigation data leads to better
instruction-following ability, making the valid actions answer ratio from
$0\%$ to $91.3\%$ for LLaMA-VID-Nav.

TABLE III:  Comparison of LLMs on 100 sampled VLN-CE R2R Val-Unseen split.
Here, the ’-’ indicates the corresponding method does not predict valid
language actions for execution.

| VLN-CE R2R Val-Unseen Sub-Split
---|---
TL | NE$\downarrow$ | OS$\uparrow$ | SR$\uparrow$ | SPL$\uparrow$
GPT-4V [98] | 5.79 | 11.4 | 10.00 | 5.00 | 3.11
Emu [84] | - | - | - | - | -
LLaVA [58] | - | - | - | - | -
LLaMA-VID [55] | - | - | - | - | -
LLaVA-Nav | 5.05 | 7.82 | 14.0 | 10.0 | 9.43
LLaMA-VID-Nav | 10.7 | 8.73 | 40.0 | 29.0 | 27.5
Ours | 8.02 | 5.52 | 45.0 | 38.0 | 35.4

TABLE IV:  Comparing on real-world environments on four diverse scenes
(Meeting Room, Office, Lab and Lounge). Simple I.F. and Complex I.F. indicate
the simple instruction following and complex instruction following tasks,
respectively.

| Meeting Room | Office | Lab | Lounge
---|---|---|---|---
| Simple I.F. | Complex I.F. | Simple I.F. | Complex I.F. | Simple I.F. | Complex I.F. | Simple I.F. | Complex I.F.
| SR$\uparrow$ | NE$\downarrow$ | SR$\uparrow$ | NE | SR$\uparrow$ | NE$\downarrow$ | SR$\uparrow$ | NE$\downarrow$ | SR$\uparrow$ | NE$\downarrow$ | SR$\downarrow$ | NE$\downarrow$ | SR$\uparrow$ | NE$\downarrow$ | SR$\uparrow$ | NE$\downarrow$
Seq2Seq [43] | 4% | 4.45 | 0% | 7.21 | 0% | 4.28 | 0% | 6.92 | 0% | 4.58 | 0% | 6.61 | 0% | 5.95 | 0% | 6.82
CMA [43] | 0% | 4.27 | 0% | 7.30 | 8% | 4.62 | 0% | 5.71 | 4% | 4.35 | 0% | 5.67 | 0% | 4.63 | 0% | 5.46
WS-MGMap [16] | 52% | 1.18 | 24% | 2.20 | 60% | 0.96 | 20% | 2.94 | 44% | 1.85 | 12% | 3.18 | 48% | 1.66 | 32% | 2.88
Ours | 92% | 0.55 | 56% | 0.98 | 84% | 0.63 | 48% | 0.71 | 76% | 0.83 | 40% | 1.89 | 88% | 0.72 | 44% | 1.37

###  V-C Comparison on Real-world Environment.

To further evaluate the generalizability of methods in more challenging
situations, we conduct an extensive experiment in real-world environments.
Here, We select two widely used baselines Seq2Seq and CMA, and a competitive
method MS-MGMap (on Table I). Each method is tested on four diverse
environments with two types of instructions of different difficulties (25
simple instructions and 25 complex instructions for each scene). For a
detailed description of real-world experiments please refer to Sec. V-A and
supplemental material. The results are presented in Table IV. Here, we find
that our method showcases a significant improvement to all baseline methods.
The end-to-end methods, Seq2Seq and CMA, suffer from extremely poor
performance, which we believe is the sim-to-real gap between VLN-CE and the
real world in terms of depth and color domain. Compared to end-to-end methods,
the map-based method, WS-MGMap, shows better performance by leveraging an
ongoing semantics map. With careful parameters and algorithm adjustment, the
map-based methods are widely regarded as a potentially robust strategy in the
real world [100, 72]. Nevertheless, our method, by exploring the generability
of VLM in VLN, is able to complete most instructions (about $84$) and
considerably complex instructions (about $48\%$), with only the requirement of
RGB videos.

Figure 4: Visual results of our method on forward instructions in real-world
environments. from top to bottom are instructions, third-person trajectories,
and robot perspective videos. Figure 5: Visual results of our method on
rotation instruction following in real-world environments. Our method can
accurately park the robot near a specified target, despite initially being
oriented towards a similarly classified object (a chair).

To further investigate the generalizability of NaVid under different
instructions, we showcase partial visual results of real-world experiments
(Table IV), that require NaVid to follow instructions with very specific
landmarks and actions. In Fig. 4, we give two similar instructions to drive
the robot to move forward under different stop conditions. We find that in
both two cases, the robot can follow the instructions to move forward and
successfully stop close to the given stop conditions. This demonstrates that
NaVid has the ability to understand the instructions and give correct actions
for the robot to execute. Besides, we conduct a more challenging experiment by
asking the robot to follow complex instructions. The visual results can be
found in Fig. 5, where the robot is required to turn orientation and move
forward to stop near the given target. These instructions are particularly
challenging because the robot is facing an object that shares the same
semantic category as the target (chair). Nevertheless, our method is able to
follow the instructions by turning the robot to the target orientation and
stopping near the correct targets. The videos of these cases can be found in
the attached supplemental material.

We provide more visual results of our method in Fig. 6 in real-world
environments. More visual results can be found in the supplemental material.

Figure 6: Visual results of our method in a real-world environment. In each
real environment example, from top to bottom are instruction, robot
perspective video, and third-person view video.

###  V-D Ablation Studies.

TABLE V: Ablation study on both training strategy and architecture.

| VLN-CE R2R Val-Unseen
---|---
| Type | TL | NE$\downarrow$ | OS$\uparrow$ | SR$\uparrow$ | SPL$\uparrow$
Training Strategy | No co-training | 6.76 | 6.33 | 30.8 | 24.7 | 23.6
No instruction reasoning sample | 9.46 | 6.51 | 46.7 | 31.1 | 29.1
No non-oracle navigation sample | 8.73 | 5.82 | 46.4 | 34.2 | 32.0
Architecture | No [NAV] | 8.71 | 5.62 | 48.1 | 35.9 | 33.5
No [HIS] and [OBS] | 8.45 | 5.56 | 46.2 | 35.7 | 33.4
2 observation tokens per frame | 9.01 | 8.13 | 30.4 | 23.9 | 20.5
Waypoints prediction | 13.65 | 10.8 | 11.5 | 0.00 | 0.00
Ours | Full pipeline | 7.63 | 5.47 | 49.1 | 37.4 | 35.9

To verify the effectiveness of each component in our method, we perform
ablation studies on both training strategy and network architecture in Table
V. Through the experiment, we find that the co-tuning data is critical to the
performance. This proved that lack of co-tuning data may cause the large
foundation to lose generalizability. As expected, the instruction reasoning
samples and Dagger nav-action samples show improvement. This inspires us that
collecting more navigation-related data could further boost the performance of
our method, which may be the follow-up direction of VLM for VLN. In the
architecture ablation studies, the task identifier token [NAV] and observation
identifier token [HIS] and [OBS] demonstrate improvement, which may be because
of the benefits to help the VLM to understand the VLN task and corresponding
observations (Sec. IV-A). Waypoint prediction variant replaces the
$\mathcal{A}_{\text{action}}$ with continuous location and orientation (see
supplemental material for detailed implementation). However, the results
demonstrate that direct output continuous location and orientation cause
extreme challenges to VLMs, making VLMs difficult to learn navigation skills.

Figure 7: Performance analysis of our method on the VLN task across different
data scales, showing the impact of data volume on SR, OS, SPL, and NE during
pre-dagger and post-dagger training phases.

We also conduct breakdown experiments to verify the importance of the
navigation data in Fig. 7. Here, we split the training into two phases, pre-
dagger (0 to 330k) and post-dagger (330k to 510k). For pre-dagger, we randomly
sample the navigation data for training. Here, we find that insufficient data
(less than 280k) may lead to a slow improvement in terms of SR, OS, SPL and
NE. However, when the data increases to 330k, there is an obvious performance
boost, indicating the model starts to master the VLN task. For the post-dagger
phase, the improvements become minimal. The key reason is that the dagger on
R2R train split does not provide sufficient diverse environments or
instruction information for the VLM to learn.

##  VI Conclusion

In this paper, we propose a video-based VLM method, NaVid, for the vision-and-
language navigation task. NaVid achieves SOTA navigation performance without
relying on odometers, depth sensors, or maps. Specifically, we extend a video-
based VLM model to enable to encoding of both history and current observation
information, enabled by self-defined special tokens. To learn the vision-
instruction following ability, we collect 490k action planning samples from
both R2R (320k) and Dagger (180k) and instruction reasoning samples (10k). The
extensive experiments in simulator environments demonstrate that our method
can achieve SOTA-level performance with only monocular videos as input.
Besides, we deploy the NaVid in real-world environments, showing
generalizability to conduct VLN tasks in real worlds.

Limitation and Future work. Considering the significant parameter scale of
NaVid, substantial computational resources are required for its training and
inference. Furthermore, the constrained memory capacity of GPUs restricts the
number of observation tokens, thereby inhibiting the ability of NaVid to
handle longer instructions effectively. In future research, we would like to
further explore enhancements in the efficiency of NaVid and investigate its
applicability across embodied AI tasks.

## References

  * Abhishek Kadian et al. [2019] Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra.  Are We Making Real Progress in Simulated Environments? Measuring the Sim2Real Gap in Embodied Visual Navigation.  In _arXiv:1912.06321_ , 2019.
  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * An et al. [2022] Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao.  Bevbert: Topo-metric map pre-training for language-guided navigation.  _arXiv preprint arXiv:2212.04385_ , 2022.
  * An et al. [2023] Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, and Liang Wang.  Etpnav: Evolving topological planning for vision-language navigation in continuous environments.  _arXiv preprint arXiv:2304.03047_ , 2023.
  * Anderson et al. [2018a] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al.  On evaluation of embodied navigation agents.  _arXiv preprint arXiv:1807.06757_ , 2018a.
  * Anderson et al. [2018b] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al.  On evaluation of embodied navigation agents.  _arXiv preprint arXiv:1807.06757_ , 2018b.
  * Anderson et al. [2018c] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3674–3683, 2018c.
  * Anderson et al. [2021a] Peter Anderson, Ayush Shrivastava, Joanne Truong, Arjun Majumdar, Devi Parikh, Dhruv Batra, and Stefan Lee.  Sim-to-real transfer for vision-and-language navigation.  In _Conference on Robot Learning_ , pages 671–681. PMLR, 2021a.
  * Anderson et al. [2021b] Peter Anderson, Ayush Shrivastava, Joanne Truong, Arjun Majumdar, Devi Parikh, Dhruv Batra, and Stefan Lee.  Sim-to-real transfer for vision-and-language navigation.  In _Conference on Robot Learning_ , pages 671–681. PMLR, 2021b.
  * Blattmann et al. [2023] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  _arXiv preprint arXiv:2311.15127_ , 2023.
  * Brohan et al. [2023] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  _arXiv preprint arXiv:2307.15818_ , 2023.
  * Chang et al. [2017] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  In _2017 International Conference on 3D Vision (3DV)_ , pages 667–676. IEEE, 2017.
  * Chen et al. [2019] Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi.  Touchdown: Natural language navigation and spatial reasoning in visual street environments.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12538–12547, 2019.
  * Chen et al. [2024] Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan-Yee K Wong.  Mapgpt: Map-guided prompting for unified vision-and-language navigation.  _arXiv preprint arXiv:2401.07314_ , 2024.
  * Chen et al. [2021a] Kevin Chen, Junshen K Chen, Jo Chuang, Marynel Vázquez, and Silvio Savarese.  Topological planning with transformers for vision-and-language navigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11276–11286, 2021a.
  * Chen et al. [2022a] Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng, Thomas H Li, Mingkui Tan, and Chuang Gan.  Weakly-supervised multi-granularity map learning for vision-and-language navigation.  _arXiv preprint arXiv:2210.07506_ , 2022a.
  * Chen et al. [2023] Peihao Chen, Xinyu Sun, Hongyan Zhi, Runhao Zeng, Thomas H. Li, Gaowen Liu, Mingkui Tan, and Chuang Gan.  A2nav: Action-aware zero-shot robot navigation by exploiting vision-and-language ability of foundation models.  _ArXiv_ , abs/2308.07997, 2023.
  * Chen et al. [2021b] Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev.  History aware multimodal transformer for vision-and-language navigation.  _Advances in Neural Information Processing Systems_ , 34:5834–5847, 2021b.
  * Chen et al. [2022b] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev.  Think global, act local: Dual-scale graph transformer for vision-and-language navigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16537–16547, 2022b.
  * Chen et al. [2020] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In _European conference on computer vision_ , pages 104–120. Springer, 2020.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_ , 2023.
  * Cui and Trinkle [2021] Jinda Cui and Jeff Trinkle.  Toward next-generation learned robot manipulation.  _Science robotics_ , 6(54):eabd9461, 2021.
  * Dai et al. [2023] W Dai, J Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023.  _arXiv preprint arXiv:2305.06500_ , 2, 2023.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Driess et al. [2023] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ , 2023.
  * Fried et al. [2018] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell.  Speaker-follower models for vision-and-language navigation.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Fu et al. [2024] Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao.  Drive like a human: Rethinking autonomous driving with large language models.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 910–919, 2024.
  * Fu et al. [2020] Tsu-Jui Fu, Xin Eric Wang, Matthew F Peterson, Scott T Grafton, Miguel P Eckstein, and William Yang Wang.  Counterfactual vision-and-language navigation via adversarial path sampler.  In _European Conference on Computer Vision_ , pages 71–86. Springer, 2020.
  * Georgakis et al. [2022] Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Daniilidis.  Cross-modal map learning for vision and language navigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15460–15470, 2022.
  * Gu et al. [2022] Jing Gu, Eliana Stefani, Qi Wu, Jesse Thomason, and Xin Wang.  Vision-and-language navigation: A survey of tasks, methods, and future directions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7606–7623, 2022.
  * Guhur et al. [2021] Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid.  Airbert: In-domain pretraining for vision-and-language navigation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1634–1643, 2021.
  * Hao et al. [2020] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao.  Towards learning a generic agent for vision-and-language navigation via pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13137–13146, 2020.
  * Hong et al. [2020] Yicong Hong, Cristian Rodriguez, Yuankai Qi, Qi Wu, and Stephen Gould.  Language and visual entity relationship graph for agent navigation.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Hong et al. [2021] Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould.  A recurrent vision-and-language bert for navigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1643–1653, June 2021.
  * Hong et al. [2022] Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould.  Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15439–15449, 2022.
  * Hu et al. [2023] Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao.  Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning.  _arXiv preprint arXiv:2311.17842_ , 2023.
  * Huang et al. [2023] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al.  Inner monologue: Embodied reasoning through planning with language models.  In _Conference on Robot Learning_ , pages 1769–1782. PMLR, 2023.
  * Irshad et al. [2021] Muhammad Zubair Irshad, Niluthpol Chowdhury Mithun, Zachary Seymour, Han-Pang Chiu, Supun Samarasekera, and Rakesh Kumar.  Sasra: Semantically-aware spatio-temporal reasoning agent for vision-and-language navigation in continuous environments.  _arXiv preprint arXiv:2108.11945_ , 2021.
  * Kamath et al. [2022] Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Jason Baldridge, and Zarana Parekh.  A new path: Scaling vision-and-language navigation with synthetic instructions and imitation learning.  _arXiv preprint arXiv:2210.03112_ , 2022.
  * Ke et al. [2019] Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa.  Tactical rewind: Self-correction via backtracking in vision-and-language navigation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6741–6749, 2019.
  * Kearns [1991] Steven M Kearns.  Extending regular expressions with context operators and parse extraction.  _Software: Practice and Experience_ , 21(8):787–804, 1991.
  * Krantz and Lee [2022] Jacob Krantz and Stefan Lee.  Sim-2-sim transfer for vision-and-language navigation in continuous environments.  In _European Conference on Computer Vision_ , pages 588–603. Springer, 2022.
  * Krantz et al. [2020a] Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee.  Beyond the nav-graph: Vision-and-language navigation in continuous environments.  In _European Conference on Computer Vision_ , pages 104–120. Springer, 2020a.
  * Krantz et al. [2020b] Jacob Krantz, Erik Wijmans, Arjun Majundar, Dhruv Batra, and Stefan Lee.  Beyond the nav-graph: Vision and language navigation in continuous environments.  In _European Conference on Computer Vision (ECCV)_ , 2020b.
  * Krantz et al. [2021] Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets.  Waypoint models for instruction-guided navigation in continuous environments.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15162–15171, 2021.
  * Krantz et al. [2023] Jacob Krantz, Shurjo Banerjee, Wang Zhu, Jason Corso, Peter Anderson, Stefan Lee, and Jesse Thomason.  Iterative vision-and-language navigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14921–14930, 2023.
  * Ku et al. [2020a] Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge.  Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4392–4412, 2020a.
  * Ku et al. [2020b] Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge.  Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4392–4412, 2020b.
  * Kuang et al. [2024] Yuxuan Kuang, Hai Lin, and Meng Jiang.  Openfmnav: Towards open-set zero-shot object navigation via vision-language foundation models.  _arXiv preprint arXiv:2402.10670_ , 2024.
  * Li et al. [2023a] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023a.
  * Li et al. [2019a] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_ , 2019a.
  * Li et al. [2023b] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al.  Vision-language foundation models as effective robot imitators.  _arXiv preprint arXiv:2311.01378_ , 2023b.
  * Li et al. [2019b] Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah Smith, and Yejin Choi.  Robust navigation with language pretraining and stochastic sampling.  _arXiv preprint arXiv:1909.02244_ , 2019b.
  * Li et al. [2020] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _European Conference on Computer Vision_ , pages 121–137. Springer, 2020.
  * Li et al. [2023c] Yanwei Li, Chengyao Wang, and Jiaya Jia.  Llama-vid: An image is worth 2 tokens in large language models.  _arXiv preprint arXiv:2311.17043_ , 2023c.
  * Liang et al. [2023] Xiwen Liang, Liang Ma, Shanshan Guo, Jianhua Han, Hang Xu, Shikui Ma, and Xiaodan Liang.  Mo-vln: A multi-task benchmark for open-set zero-shot vision-and-language navigation.  _arXiv preprint arXiv:2306.10322_ , 2023.
  * Lin et al. [2023] Jinzhou Lin, Han Gao, Rongtao Xu, Changwei Wang, Li Guo, and Shibiao Xu.  The development of llms for embodied navigation.  _arXiv preprint arXiv:2311.00530_ , 2023.
  * Liu et al. [2023a] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  _arXiv preprint arXiv:2310.03744_ , 2023a.
  * Liu et al. [2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  In _NeurIPS_ , 2023b.
  * Long et al. [2023] Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong.  Discuss before moving: Visual language navigation via multi-expert discussions.  _arXiv preprint arXiv:2309.11382_ , 2023.
  * Ma et al. [2019] Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong.  Self-monitoring navigation agent via auxiliary progress estimation.  _arXiv preprint arXiv:1901.03035_ , 2019.
  * Macenski et al. [2020a] Steve Macenski, Francisco Martín, Ruffin White, and Jonatan Ginés Clavero.  The marathon 2: A navigation system.  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 2718–2725. IEEE, 2020a.
  * Macenski et al. [2020b] Steven Macenski, Francisco Martin, Ruffin White, and Jonatan Ginés Clavero.  The marathon 2: A navigation system.  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020b.
  * Majumdar et al. [2020] Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra.  Improving vision-and-language navigation with image-text pairs from the web.  In _European Conference on Computer Vision_ , pages 259–274. Springer, 2020.
  * Pan et al. [2023] Bowen Pan, Rameswar Panda, SouYoung Jin, Rogerio Feris, Aude Oliva, Phillip Isola, and Yoon Kim.  Langnav: Language as a perceptual representation for navigation.  _arXiv preprint arXiv:2310.07889_ , 2023.
  * Park and Kim [2023] Sang-Min Park and Young-Gab Kim.  Visual language navigation: A survey and open challenges.  _Artificial Intelligence Review_ , 56(1):365–427, 2023.
  * Qi et al. [2020a] Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, and Qi Wu.  Object-and-action aware model for visual language navigation.  In _European Conference on Computer Vision_ , pages 303–317. Springer, 2020a.
  * Qi et al. [2020b] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel.  Reverie: Remote embodied visual referring expression in real indoor environments.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9982–9991, 2020b.
  * Qiao et al. [2022] Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu.  Hop: History-and-order aware pre-training for vision-and-language navigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15418–15427, 2022.
  * Qiao et al. [2023] Yanyuan Qiao, Yuankai Qi, Zheng Yu, Jing Liu, and Qi Wu.  March in chat: Interactive prompting for remote embodied referring expression.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15758–15767, 2023.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Ramakrishnan et al. [2022] Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, and Kristen Grauman.  Poni: Potential functions for objectgoal navigation with interaction-free learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18890–18900, 2022.
  * Raychaudhuri et al. [2021] Sonia Raychaudhuri, Saim Wani, Shivansh Patel, Unnat Jain, and Angel X Chang.  Language-aligned waypoint (law) supervision for vision-and-language navigation in continuous environments.  _arXiv preprint arXiv:2109.15207_ , 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ross et al. [2011] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 627–635. JMLR Workshop and Conference Proceedings, 2011.
  * Savva et al. [2019] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al.  Habitat: A platform for embodied ai research.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9339–9347, 2019.
  * Schumann et al. [2023] Raphael Schumann, Wanrong Zhu, Weixi Feng, Tsu-Jui Fu, Stefan Riezler, and William Yang Wang.  Velma: Verbalization embodiment of llm agents for vision and language navigation in street view.  _arXiv preprint arXiv:2307.06082_ , 2023.
  * Shah et al. [2022] Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine.  Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action.  _arXiv preprint arXiv:2207.04429_ , 2022.
  * Shah et al. [2023] Dhruv Shah, Błażej Osiński, Sergey Levine, et al.  Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action.  In _Conference on Robot Learning_ , pages 492–504. PMLR, 2023.
  * Shridhar et al. [2022] Mohit Shridhar, Lucas Manuelli, and Dieter Fox.  Cliport: What and where pathways for robotic manipulation.  In _Conference on Robot Learning_ , pages 894–906. PMLR, 2022.
  * Song et al. [2023] Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su.  Llm-planner: Few-shot grounded planning for embodied agents with large language models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2998–3009, 2023.
  * Su et al. [2019] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  _arXiv preprint arXiv:1908.08530_ , 2019.
  * Sun et al. [2023a] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  _arXiv preprint arXiv:2303.15389_ , 2023a.
  * Sun et al. [2023b] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.  Generative pretraining in multimodality.  _arXiv preprint arXiv:2307.05222_ , 2023b.
  * Suomalainen et al. [2022] Markku Suomalainen, Yiannis Karayiannidis, and Ville Kyrki.  A survey of robot manipulation in contact.  _Robotics and Autonomous Systems_ , 156:104224, 2022.
  * Tan and Bansal [2019] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5100–5111, 2019.
  * Tan et al. [2019] Hao Tan, Licheng Yu, and Mohit Bansal.  Learning to navigate unseen environments: Back translation with environmental dropout.  In _Proceedings of NAACL-HLT_ , pages 2610–2621, 2019.
  * Thomason et al. [2020] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer.  Vision-and-dialog navigation.  In _Conference on Robot Learning_ , pages 394–406, 2020.
  * Tsai et al. [2023] Yao-Hung Hubert Tsai, Vansh Dhar, Jialu Li, Bowen Zhang, and Jian Zhang.  Multimodal large language model for visual navigation.  _arXiv preprint arXiv:2310.08669_ , 2023.
  * Vasudevan et al. [2021] Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool.  Talk2nav: Long-range vision-and-language navigation with dual attention and spatial memory.  _International Journal of Computer Vision_ , 129:246–266, 2021.
  * Wang et al. [2019] Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang.  Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6629–6638, 2019.
  * Wang et al. [2023a] Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang.  Gridmm: Grid memory map for vision-and-language navigation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15625–15636, 2023a.
  * Wang et al. [2023b] Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao.  Scaling data generation in vision-and-language navigation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 12009–12020, 2023b.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ , 2022.
  * Wu et al. [2023] Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong.  Unleashing large-scale video generative pre-training for visual robot manipulation.  _arXiv preprint arXiv:2312.13139_ , 2023.
  * Wu et al. [2022] Siying Wu, Xueyang Fu, Feng Wu, and Zheng-Jun Zha.  Cross-modal semantic alignment pre-training for vision-and-language navigation.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 4233–4241, 2022.
  * Xu et al. [2023] Chengguang Xu, Hieu T Nguyen, Christopher Amato, and Lawson LS Wong.  Vision and language navigation in the real world via online visual language mapping.  _arXiv preprint arXiv:2310.10822_ , 2023.
  * Yang et al. [2023] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang.  The dawn of lmms: Preliminary explorations with gpt-4v (vision).  _arXiv preprint arXiv:2309.17421_ , 9(1):1, 2023.
  * Zeng et al. [2021] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al.  Transporter networks: Rearranging the visual world for robotic manipulation.  In _Conference on Robot Learning_ , pages 726–747. PMLR, 2021.
  * Zhang et al. [2023a] Jiazhao Zhang, Liu Dai, Fanpeng Meng, Qingnan Fan, Xuelin Chen, Kai Xu, and He Wang.  3d-aware object goal navigation via simultaneous exploration and identification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6672–6682, 2023a.
  * Zhang et al. [2023b] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023b.
  * Zhang and Pless [2004] Qilong Zhang and Robert Pless.  Extrinsic calibration of a camera and laser range finder (improves camera calibration).  In _2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566)_ , volume 3, pages 2301–2306. IEEE, 2004.
  * Zheng et al. [2023] Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang.  Towards learning a generalist model for embodied navigation.  _arXiv preprint arXiv:2312.02010_ , 2023.
  * Zhou et al. [2023] Gengze Zhou, Yicong Hong, and Qi Wu.  Navgpt: Explicit reasoning in vision-and-language navigation with large language models.  _arXiv preprint arXiv:2305.16986_ , 2023.

The supplemental material is organized as follows:

  * •

We introduce the real-world experiment setup, including the indoor scenes
(Sec. A-A), instructions (Sec. A-B) and robot setup (Sec. A-C).

  * •

Sec. A-D introduces more details about the baselines.

  * •

Sec. A-E reports more quantitative experiments of NaVid.

  * •

Sec. B reports more visual results of NaVid.

##  Appendix A Real-world Experiments Setup

We conduct real-world experiments following previous vision-and-language
reports [1], primarily focusing on indoor scenes, instructions, and robot
setup.

###  A-A Indoor Scenes.

We have chosen four challenging indoor environments, including Meeting Room,
Office, Lab, and Lounge. These environments are crowded with objects of
diverse styles and are subject to various lighting conditions. Visualizations
of these environments are provided in Fig. 8.

Figure 8: Indoor environments used in real-world experiments.

###  A-B Instructions.

There are two types of instructions: (1) simple instructions, which require
the agent to navigate to a single robot landmark and stop; (2) complex
instructions, which require the agent to follow a series of instructions. All
instructions are designed based on the objects present in the indoor scenes.

We list some examples of instructions used in the experiments. Here are some
examples of simple instructions:

  * •

Walk to the plant.

  * •

Move towards the white box then stop.

  * •

Turn right and walk forward to the door then stop.

Here are some examples of complex instructions:

  * •

Turn around and go straight to the wall then turn right to the corner then
stop.

  * •

Go straight to the chair, then turn left/right to the stairs and stop.

  * •

Move forward to the wall, then turn left. Walk through the door, face the
chair, and then stop.

###  A-C Robot setup.

We provide a detailed description of our real robot, and a visualization of it
can be found in Fig 9. Our robot is based on a Turtlebot 4, and we utilize the
Azure Kinect DK to capture RGB and Depth images (our method only utilizes RGB
images). Additionally, we deploy a PRLIDAR A1M8 Lidar on top of the robot to
capture a 1D laser point cloud, which is utilized by the lidar-odometry
algorithm to obtain location and orientation [63].

Figure 9: Real-world robot setup.

Based on the above-mentioned robot, we design a pipeline for vision-and-
language navigation with NaVid (Fig. 10). NaVid is deployed on a server with
an A100 GPU, which receives compressed images from the robot and sends back
parsed commands through the Internet. The robot receives commands, such as
”Turn left” or ”Move forward”, and drives the robot to execute the actions.
During movement, the robot consistently tracks motion to confirm that rotation
or forward movement is aligned with the commands.

Figure 10: Pipeline of our navigation system. Figure 11: Visualization of
real-world experiments.

###  A-D Baseline details

LLaVA[59] and LLaMA-VID[55] are evaluated in Table III of the main paper. We
directly use the models and weights from the openly released
repositories222https://github.com/dvlab-research/LLaMA-
VID333https://github.com/haotian-liu/LLaVA. However, we find that neither
method is able to consistently produce valid action outputs. To address this,
we adopt prompt techniques following [104, 60], incorporate CoT [94] and
utilize language-described historical information (we also use LLaVA). Despite
these efforts, most responses are related to scene descriptions and navigation
skills, with valid navigation answers being rare.

LLaVA-Nav is a variant model trained using our navigation data. Since LLaVA
can only process images and text, we follow existing work [104] and utilize
text to describe the historical trajectory. Due to the token number
limitation, we uniformly sample keyframes, including the first and last
frames, and utilize LLaVA itself to describe these keyframes. We adopt the
same output format as NaVid.

Waypoint-NaVid is a variant of NaVid that directly outputs a valid waypoint
(location and orientation) for the next action. These waypoints are defined by
the shortest path to follow the trajectory. For each prediction, we sample a
valid waypoint within the field of view of the current observation. Despite
using the same inputs as NaVid, waypoint prediction demonstrates extremely
poor performance with a success rate of only $16\%$ (where success is defined
as predictions with less than 30cm distance error and less than 30 degrees
error). This leads to frequent failures during navigation by the agent.

###  A-E Quantitative Experiments

We conduct additional experiments to comprehensively evaluate our method,
including single-step prediction and trajectory summary.

Single-step prediction. The sample for training NaVid is composed of a
sequence of images and a step prediction of the next movements. The single-
step prediction results can directly reflect the quality of robot learning.
After training on all navigation samples (as described in the main paper), we
evaluate the model on the new samples created from R2R Val-Uneen. We evaluate
the success rate (correct prediction /number of samples), stop success rate
(correct stop/number of stop samples), angle error (average angle degree error
of correct rotation prediction) and distance error (average distance error of
correct rotation prediction). Here, our method achieves $69.4\%$ success rate,
$52.1\%$ stop success rate, $6.82$ (degree) angle error and $16.7$
(centimetre) average distance error. We find this single-step performance
leads to a $37.4\%$ success rate under the full vision-and-language task. The
reason behind this is that the instruction following trajectory can be
completed with different sequences of sing-step actions. Nevertheless, the
model can not exactly follow the oracle trajectories, it can follow
instructions with sub-optimal trajectories.

Trajectory summary. As mentioned in the main paper (Sec. IV-C), the training
samples include 10k instruction reasoning samples. Even the 10k samples are
relatively low for a large language model to fully master the skill, but this
indicates our model is able to do the trajectory summary task. Therefore, we
test our model on diverse trajectories, including both real-world and
simulator environments (shown in Fig. 12 and Fig. 13). We observe that our
model can briefly describe the trajectory, demonstrating that our model can
understand its motions.

##  Appendix B Visual Experiments

We propose extensive visual experiments of our method on real-world (Fig. 11)
simulator environments. Fig. 14 and Fig. 15 for VLN-CE R2R dataset and Fig. 16
and Fig. 17 for VLN-CE RxR dataset. More visual results please refer to the
attached video.

Figure 12: (1/2) Caption results visualization. Figure 13: (2/2) Caption
results visualization. Figure 14: (1/2) R2R data visualization. Figure 15:
(2/2) R2R data visualization. Figure 16: (1/2) RxR data visualization. Figure
17: (2/2) RxR data visualization.