performing fine-tuning of the pretrained language model during training with
deep offline RL algorithms.

Advances in model architecture (e.g. transformer) for foundation models allow
the model to effectively model and predict sequences. To harness the power of
these models, some recent studies investigate exploiting these architectures
for sequence modeling in RL problems. Reid et al. [200] explore the potential
of leveraging the sequence modeling formulation of reinforcement learning and
examine the transferability of pretrained sequence models across different
domains, such as vision and language. They specifically focus on the
effectiveness of fine-tuning these pretrained models on offline RL tasks,
including control and games. In addition to investigating the transferability
of pretrained sequence models, the authors propose techniques to enhance the
transfer of knowledge between these domains. These techniques aim to improve
the adaptability and performance of the pretrained models when applied to new
tasks or domains.

High-level task planning using LLMs is demonstrated in embodied AI
environments. Huang et al. [68] propose employing pretrained Language Models
(LMs) as zero-shot planners. The approach is evaluated in the VirtualHome
[129] environment. In this work, first, an autoregressive LLM such as GPT-3
[2] or Codex [201] is quarried to generate action plans for high-level tasks.
Some of these action plans might not be executable by the agent due to
ambiguity in language or referring to objects that are not present or grounded
in the environment. So, to select the admissible action plans, admissible
environment actions, and generated actions by the causal LLM are embedded
using a BERT-style LM. Then for each admissible environment action, its
semantic distance to the generated action is computed using cousin similarity.

Chain of thought reasoning and action generation are proposed for embodied
agents as well. ReAct [202] combines reasoning (e.g. chain of thought) and
acting (e.g. sequence of action generation) within LLM. Reasoning traces
enhance the model’s ability to deduce, monitor, and revise action plans, along
with managing exceptions effectively. Actions facilitate interaction with
external resources, like knowledge bases or environments, enabling it to
acquire supplementary information. ReAct showcases its proficiency across a
wide array of language and decision-making tasks, including question-answering
and fact verification. It enhances interpretability and trust for users by
transparently illustrating the process through which it searches for evidence
and formulates conclusions. Unlike prior methods that depend on a singular
chain-of-thought, ReAct engages with a Wikipedia API for pertinent information
retrieval and belief updating. This strategy effectively mitigates the issues
commonly associated with chain-of-thought reasoning, such as hallucination and
error propagation.

VPT [72] presents video pretraining in which the agent learns to act by
watching unlabeled online videos. It is shown that an inverse dynamic model
can be trained with a small labeled dataset and the model can be used to label
a huge unlabeled data of the internet. Videos of people who have played
Minecraft are used to train an embodied AI agent to play Minecraft. The model
exhibits zero-shot performance and can be fine-tuned for more complex skills
using imitation learning or reinforcement learning. The VPT model is trained
with a standard behavioral cloning loss (9) (negative log-likelihood) while
the actions are drawn from the inverse dynamic model.

###  V-A Generalist AI

A long-standing challenge in robotics research is deploying robots or embodied
AI agents in a variety of non-factory real-world applications, performing a
range of tasks. To make generalist robots that can operate in diverse
environments with diverse tasks, some researchers have proposed generative
simulators for robot learning. For example, Generative Agents [203] discusses
how generative agents can produce realistic imitations of human behavior for
interactive applications, creating a miniature community of agents similar to
those found in games like The Sims. The authors connect their architecture
with the ChatGPT large language model to create a game environment with 25
agents. The study includes two evaluations, a controlled evaluation and an
end-to-end evaluation, which demonstrate the causal effects of the various
components of their architecture. Xian et al. [204], authors propose a fully
automated generative pipeline, known as a generative simulation for robot
learning, which utilizes models to generate diverse tasks, scenes, and
training guidance on a large scale. This approach can facilitate the scaling
up of low-level skill learning, ultimately leading to a foundational model for
robotics that empowers generalist robots.

An alternative method for developing generalist AI involves using
generalizable multi-modal representations. Gato [154] is a generalist agent
that works as a multi-modal, multi-task, multi-embodiment generalist policy.
Using the same neural network with the same set of weights, Gato can sense and
act with different embodiments in various environments across different tasks.
Gato can play Atari, chat, caption images, stack blocks with a real robot arm,
navigate in a 3D simulated environment, and more. Gato is trained on 604
different tasks with various modalities, observations, and actions. In this
setting, language acts as a common grounding across different embodiments.
Gato has 1.2B parameters and is trained offline in a supervised way.
Positioned at the confluence of representation learning and reinforcement
learning (RL), RRL [205] learns behaviors directly from proprioceptive inputs.
By harnessing pre-trained visual representations, RRL is able to learn from
visual inputs, which typically pose challenges in conventional RL settings.

###  V-B Simulators

High-quality simulators or benchmarks are crucial for robotics development.
Hence, we put the “simulator” section here to highlight its essential role. To
facilitate generalization from simulation to the real world, Gibson [206]
emphasizes real-world perception for embodied agents. To bridge the gap
between simulation and real-world, iGibson [146] and BEHAVIOR-1K [207] further
support the simulation of a more diverse set of household tasks and reach high
levels of simulation realism. As a simulation platform for research in
Embodied AI, Habitat [208] consists of Habitat-Sim and Habitat-API. Habitat-
Sim can achieve several thousand frames per second (fps) running single-
threaded. Rather than modeling into low-level physics, Habitat-Lab [147], is a
high-level library for embodied AI, giving a modular framework for end-to-end
development. It facilitates the definition of embodied AI tasks, such as
navigation, interaction, instruction following, and question answering.
Additionally, it enables the configuration of embodied agents, encompassing
their physical form, sensors, and capabilities. The library supports various
training methodologies for these agents, including imitation learning,
reinforcement learning, and traditional non-learning approaches like the
SensePlanAct pipelines. Furthermore, it provides standard metrics for
evaluating agent performance across these tasks. In line with this, the recent
release of Habitat 3.0 [209] further expands these capabilities.

Similarly, RoboTHOR [210] serves as a platform for the development and
evaluation of embodied AI agents, offering environments in both simulated and
physical settings. Currently, RoboTHOR includes a training and validation set
comprising 75 simulated scenes. Additionally, there are 14 scenes each for
test-dev and test-standard in the simulation, with corresponding physical
counterparts. Key features of RoboTHOR include its reconfigurability and
benchmarking capabilities. The physical environments are constructed using
modular, movable components, enabling the creation of diverse scene layouts
and furniture configurations in a single physical area. Another simulator,
VirtualHome [129], models complex activities that occur in a typical
household. It supports program descriptions for a variety of activities that
happen in people’s homes. Huang et al. [33] use VirtualHome to evaluate the
robot planning ability with language models. These simulators have the
potential to be applied for evaluating LLMs on robotics tasks.

##  VI Challenges and Future Directions

In this section, we examine challenges related to integrating foundation
models into robotics settings. We also explore potential future avenues to
address some of these challenges.

###  VI-A Overcoming Data Scarcity in Training Foundation Models for Robotics

One main challenge is that compared to the internet-scale text and image data
that large models are trained on, robotic-specific data is scarce. We discuss
various techniques to overcome data scarcity. For example, to scale up robot
learning, some recent works suggest the use of play data instead of expert
data for imitation learning. Another technique is data augmentation using in-
painting techniques.

####  VI-A1 Scaling Robot Learning Using Unstructured Play Data and Unlabeled
Videos of Humans

Language-conditioned learning such as language-conditioned behavioral cloning,
or language-conditioned affordance learning requires having access to large
annotated datasets. To scale up learning, in Play-LMP [26], the authors
suggest using teleoperated human-provided play data instead of fully annotated
expert demonstrations. Play data is unstructured, unlabeled, cheap to collect,
but rich. Collecting play data does not require scene staging, task
segmenting, or resetting to an initial state. Also, in MimicPlay [118] a goal-
conditioned trajectory generation model is trained based on human-play data.
The play data includes unlabeled video sequences of humans interacting with
the environment with their hands. Recently works such as [125] have shown a
very small percentage (as little as 1%) of language-annotated data is needed
to train a visuo-lingual affordance model for robot manipulation tasks.

####  VI-A2 Data Augmentation using Inpainting

Collecting robotics data requires the robot to interact with the real physical
world. This data collection process can be associated with significant costs
and potential safety concerns. One way to tackle this challenge is to use
generative AI such as text-to-image diffusion models for data augmentation.
For example, ROSIE (Scaling Robot Learning with Semantically Imagined
Experience) [176] presents a diffusion-based data augmentation. Given a robot
manipulation dataset, they use inpainting to create various unseen objects,
backgrounds, and distractors with textual guidance. One important challenge
for these methods is developing inpainting strategies that can generate
sufficient semantically and visually diverse data, while at the same time
ensuring that this data is physically feasible and accurate. For instance,
using inpainting to modify an image of an object within a robot’s gripper may
result in an image with a physically unrealistic grasp, leading to poor
downstream training performance. Additional investigation into generative
foundation models that are evaluated not only for visual quality but also for
physical realism may improve the generality of these methods.

####  VI-A3 Overcoming 3D Data Scarcity for Training 3D Foundation Models

Currently, multi-modal Vision-and-Language Models (VLMs) can analyze 2D
images, but they lack a connection to the 3D world, which encompasses 3D
spatial relationships, 3D planning, 3D affordances, and more. The primary
obstacle in developing foundational 3D VLM models lies in the scarcity of 3D
data, especially data that is paired with language descriptions. As discussed,
language-driven perception tasks such as language-driven 3D scene
representation, language-driven 3D scene editing, language-driven 3D scene or
shape generation, language-driven 3D classification, and affordance prediction
require access to 3D data or multi-view images with camera matrices which are
not readily available data types. New datasets or data generation methods need
to be created in the future to overcome data scarcity in the 3D domain.

####  VI-A4 Synthetic Data Generation via High-Fidelity Simulation

High-fidelity simulation via gaming engines can provide an efficient means to
collect data, especially to solve multimodal and 3D perception tasks on
robots. For example, TartanAir [211], a dataset for robot navigation tasks,
was collected in [212] with the presence of moving objects, changing light,
and various weather conditions. By collecting data in simulations, it was
possible to obtain multi-modal sensor data and precise ground truth labels
such as the stereo RGB image, depth image, segmentation, optical flow, camera
poses, and LiDAR point cloud. A large number of environments were set up with
various styles and scenes, covering challenging viewpoints and diverse motion
patterns that are difficult to achieve by using physical data collection
platforms. An extension TartanAir-V2 (https://tartanair.org) furthers the
dataset by incorporating additional environments and modalities, such as
fisheye, panoramas, and pinholes, with arbitrary camera intrinsic and
rotations.

####  VI-A5 Data Augmentation using VLMs

Data augmentation can be provided using Visual-Language Models (VLMs). In DIAL
[213], Data-driven Instruction Augmentation for Language-conditioned control
is introduced. DIAL uses VLM to label offline datasets for language-
conditioned policy learning. DIAL performs instruction augmentation using VLMs
to weakly relabel offline control datasets. DIAL consists of three steps 1)
Contrastive fine-tuning of a VLM such as CLIP [4] on a small robot
manipulation dataset of trajectories with crowd-sourced annotation, 2)
producing new instruction labels by using the fine-tuned VLM to score
relevancy of crowd-sourced annotations against a larger dataset of
trajectories, 3) training a language-conditioned policy using behavior cloning
on both, the original and re-annotated dataset.

####  VI-A6 Robot Physical Skills are Limited to Distribution of Skills

One key limitation of the existing robot transformers and other related works
in robotics is that robot physical skills are limited to the distribution of
skills observed within the robot data. Using these transformers, the robot
lacks the capability to generate new movements. To address this constraint, an
approach involves using motion data from videos that humans performing various
tasks. The inherent motion information within these videos can then be
employed to facilitate the acquisition of physical skills in robotics.

###  VI-B Real Time Performance (High Inference Time of Foundation Models)

Another bottleneck for deploying foundation models on robots is the inference
time of these models. In Table II, the inference time for some of these models
is reported. As seen, the inference time for some of the models still needs to
be improved for reliable real-time deployment of the robotic systems. As real-
time capability is an essential requirement for any robotic system, more
research needs to be performed to improve the computational efficiency of
foundation models.

Furthermore, foundation models are most often stored and run in remote data
centers, and accessed through APIs that require network connectivity. Many
foundation models (e.g., the GPT models, the Dall-E models) can only be
accessed this way, while others are usually accessed this way, but can also be
downloaded and run locally with sufficient local computing power (such as SAM
[59], LLaMA [214], and DINOv2 [107]). Given this cloud-service paradigm, the
latencies and service times in response to an API call for a foundation model
depend on the underlying network over which the data is routed and the data
center where the computation takes place—factors that are beyond the control
of a robot. So network reliability should be taken into account before
integrating a foundation model into a robot’s autonomy stack.

For some robotics domains reliance on the network and 3rd party computing may
not be a safe or realistic operating paradigm. In autonomous driving,
autonomous aircraft, search and rescue or emergency response applications, and
defense applications the robot cannot rely on network connectivity for time-
critical perception or control computations. One option is to have a safe
fall-back mode that relies on classical autonomy tools using only local
computation, that can take over if access to the cloud is interrupted for some
reason. Another potential longer-term solution for network-free autonomy is
the distillation of large foundation models into smaller-sized specialized
models that run on onboard robot hardware. Some recent work has attempted this
approach (though without an explicit link to robotics) [215]. Such distilled
models would likely give up some aspect of the full model, e.g. restricting
operation to a certain limited context, in exchange for smaller size and
faster compute. This could be an interesting future direction for bringing the
power of foundation models to safety-critical robotics systems.

###  VI-C Limitations in Multimodal Representation

Multimodal interaction implicitly assumes that the modality is tokenizable and
can be standardized into input sequences without losing information. The
Multimodal models provide information sharing between multiple modalities and
are some variation of multimodal transformers with cross-modal attention
between every pair of inputs. In multimodal representation learning, it is
assumed that cross-modal interactions and the dimension of heterogeneity
between different modalities can all be captured by simple embeddings. In
other words, a simple embedding is assumed to be sufficient to identify the
modality or for example, how different language is from vision. In the realm
of multimodal representation learning, the question of whether a single
multimodal model can accommodate all modalities remains an open challenge.

Additionally, when paired data between a modality and text is available one
can embed that modality into text directly. In robotics applications there are
some modalities for which sufficient data is not available and to be able to
align them with other modalities, they need to be first converted to other
modalities and then be used. For example, 3D point cloud data has various
applications in robotics but training a foundation model using this type of
data is challenging since data is scarce and is not aligned with text. So, one
way to overcome this challenge is first converting this 3D point cloud data to
other modalities such as images and subsequently images to text as the
secondary step of alignment. Then they can be used in foundation model
training. As another example, in Socratic models [194], each modality, whether
visual or auditory, is initially translated into language, after which
language models attempt to respond to these modalities.

###  VI-D Uncertainty Quantification

How can we provide assurances on the reliability of foundation models when
they are deployed in potentially safety-critical robotics applications [188]?
Current foundation models such as LLMs often _hallucinate_ , i.e., produce
outputs that are factually incorrect, logically inconsistent, or physically
infeasible. While such failures may be acceptable in applications where the
outputs from the model can be checked by a human in real-time (e.g., as is
often the case for LLM-based conversational agents), they are not acceptable
when deploying autonomous robots that use the outputs of foundation models in
order to act in human-centered environments. Rigorous uncertainty
quantification is a key step toward addressing this challenge and safely
integrating foundation models into robotic systems. Below, we highlight
challenges and recent progress in uncertainty quantification for foundation
models in robotics.

####  VI-D1 Instance-Level Uncertainty Quantification

How can we quantify the uncertainty in the output of a foundation model for a
_particular_ input? As an example, consider the problem of image
classification; given a particular image, one may quantify uncertainty in the
output by producing a _set_ of object labels that the model is uncertain among
or a _distribution_ over object labels. Instance-level uncertainty
quantification can inform the robot’s decisions at runtime. For example, if an
image classification model running on an autonomous vehicle produces a
prediction set $\\{\texttt{Pedestrian, Bicyclist}\\}$ representing that it is
uncertain whether a particular agent is a pedestrian or a bicyclist, the
autonomous vehicle can take actions that consider both possibilities.

####  VI-D2 Distribution-Level Uncertainty Quantification

How can we quantify the uncertainty in the correctness of a foundation model
that will be deployed on a _distribution_ of possible future inputs? For the
problem of image classification, one may want to compute or bound the
probability of errors over the distribution of inputs that a robot may
encounter when deployed. Distribution-level uncertainty quantification allows
us to decide whether a given model is sufficiently reliable to deploy in our
target distribution of scenarios. For example, we may want to collect
additional data or fine-tune the model if the computed probability of error is
too high.

####  VI-D3 Calibration

In order to be useful, estimates of uncertainty (both at the instance-level
and distribution level) should be _calibrated_. If we perform instance-level
uncertainty quantification using prediction sets, calibration asks for the
prediction set to contain the true label with a user-specified probability
(e.g., $95\%$) over future inputs. If instance-level uncertainty is quantified
using a distribution over outputs, it should be the case that outputs that are
assigned confidence $p$ are in fact correct with probability $p$ over future
inputs. Similarly, distribution-level uncertainty estimates should bound the
true probability of errors when encountering inputs from the target
distribution.

We highlight a subtle, but important, point that is often overlooked when
performing uncertainty quantification in robotics: it can be crucial to pay
attention to the distinction between Frequentist and Bayesian interpretations
of probabilities. In many robotics contexts — particularly safety-critical
ones — the desired interpretation is often Frequentist in nature. For example,
if we produce a bound $\epsilon$ for the probability of collision of an
autonomous vehicle, this should bound the actual observed rate of collisions
when the vehicle is deployed. Bayesian techniques (e.g., Gaussian processes or
Bayesian ensembles) do not necessarily produce estimates of uncertainty that
are calibrated in this Frequentist sense (since the estimates depend on the
specific prior that is used to produce the estimates). Trusting the resulting
uncertainty estimates may lead one astray if the goal is to provide
statistical guarantees on the safety or performance of the robotic system when
it is deployed.

####  VI-D4 Distribution Shift

An important challenge in performing calibrated uncertainty quantification is
_distribution shift_. A foundation model trained on a particular distribution
of inputs may not produce calibrated estimates of uncertainty when deployed on
a different distribution for a downstream task. A more subtle cause of
distribution shift in robotics arises from _closed-loop_ deployment of a
model. For example, imagine an autonomous vehicle that chooses actions using
the output of a perception system that relies on a pretrained foundation
model; since the robot’s actions influence future states and observations, the
distribution of inputs the perception system receives can be potentially very
different from the one it was trained on.

####  VI-D5 Case Study: Uncertainty Quantification for Language-Instructed
Robots

Recently, there has been exciting progress in performing rigorous uncertainty
quantification for language-instructed robots [216]. This work proposes an
approach called KnowNo for endowing language-instructed robots with the
ability to _know when they don’t know_ and to ask for help or clarification
from humans in order to resolve uncertainty. KnowNo performs both instance-
level and distribution-level uncertainty quantification in a calibrated manner
using the theory of _conformal prediction_. In particular, given a language
instruction (and a description of the robot’s environment generated using its
sensors), conformal prediction is used to generate a prediction set of
candidate actions. If this set is a singleton, the robot executes the
corresponding action; otherwise, the robot seeks help from a human by asking
them to choose an action from the generated set. Using conformal prediction,
KnowNo ensures that asking for help in this manner results in a statistically
guaranteed level of task success (i.e., distribution-level uncertainty
quantification). KnowNo tackles potential challenges with distribution shift
by collecting a small amount of calibration data from the target distribution
of environments, tasks, and language instructions, and using this as part of
the conformal prediction calibration procedure. While KnowNo serves as an
example of calibrated instance-level and distribution-level uncertainty
quantification for LLMs, future research should also explore assessing and
ensuring the reliability of various other foundation models, such as vision-
language models, vision-navigation models, and vision-language-action models,
commonly employed in robotics. In addition, exploring how Bayesian uncertainty
quantification techniques (e.g., ensembling [217, 218]) can be combined with
approaches such as conformal prediction to produce calibrated estimates of
instance-level and distribution-level uncertainty is a promising direction.

###  VI-E Safety Evaluation

The problem of _safety evaluation_ is closely related to uncertainty
quantification. How can we rigorously _test_ for the safety of a foundation
model-based robotic system (i) before deployment, (ii) as the model is updated
during its lifecycle, and (iii) as the robot operates in its target
environments? We highlight challenges and research opportunities related to
these problems below.

####  VI-E1 Pre-deployment safety tests

Rigorous pre-deployment testing is crucial for ensuring the safety of any
robotic system. However, this can be particularly challenging for robots that
incorporate foundation models. First, foundation models are trained on vast
amounts of data; thus, a rigorous testing procedure should ensure that test
scenarios were not seen by the model during training. Second, foundation
models often commit errors in ways that are hard to predict _a priori_ ; thus,
tests need to cover a diverse enough range of scenarios to uncover flaws.
Third, foundation models such as LLMs are often used to produce open-ended
outputs (e.g., a plan for a robot described in natural language). The
correctness of such outputs can be challenging to evaluate in an automated
manner if these outputs are evaluated in isolation from the entire system.

The deployment cycle of current foundation models (in non-robotics
applications) involves thorough _red-teaming_ by human evaluators [3, 219].
Recent work has also considered partially automating this process by using
foundation models themselves to perform red-teaming [220, 221]. Developing
ways to perform red-teaming (both by humans and in a partially automated way)
for foundation models in robotics is an exciting direction for future
research.

In addition to evaluating the foundation model in isolation, it is also
critical to assess the safety of the end-to-end robotic system. Simulation can
play a critical role here, and already does so for current field-deployed
systems such as autonomous vehicles [222, 223]. The primary challenges are to
ensure that (i) the simulator has high enough fidelity for results to
meaningfully transfer to the real world, and (ii) test scenarios (manually
specified, replicated from real-world scenarios, or automatically generated
via adversarial methods [224]) are representative of real-world scenarios and
are diverse enough to expose flaws in the underlying foundation models. In
addition, finding ways to augment large-scale simulation-based testing with
smaller-scale real-world testing is an important direction for future work. We
emphasize the need for performing such testing throughout the lifecycle of a
field-deployed robotic system, especially as updates are made to different
components (which may interact in unpredictable ways with foundation models).

####  VI-E2 Runtime monitoring and out-of-distribution detection

In addition to performing rigorous testing _offline_ , robots with foundation
model-based components should also perform _runtime monitoring_. This can take
the form of _failure prediction_ in a given scenario, which can allow the
robot to deploy a safety-preserving fallback policy [225, 226, 227, 228, 229].
Alternately, the robot can perform _out-of-distribution (OOD) detection_ using
experiences collected from a small batch of scenarios in a novel distribution
[230, 231, 232, 233]; this can potentially trigger the robot to cease its
operations and collect additional training data in the novel distribution in
order to re-train its policy. Developing techniques that perform runtime
monitoring and OOD detection with statistical guarantees on false
positive/negative error rates in a data-efficient manner remains an important
research direction.

###  VI-F Using Existing Foundation Models as Plug-and-Play or Building New
Foundation Models for Robotics

To incorporate foundation models into robotics, either the existing pretrained
large models can be employed as plug-and-play or new foundation models can be
built using robotics data. Using foundation models as plug-and-play refers to
integrating foundation models into various applications without the need for
extensive customization. A large body of the existing literature on foundation
models in robotics is centered around the use of foundation models from other
domains such as language or vision as plug-and-play. The plug-and-play
approach simplifies and facilitates the integration of recent AI advances into
the robotics domain. While employing these models as plug-and-play offers a
convenient way to harness the power of AI and provide rapid implementation,
versatility, and scalability, they are not always customized to specific
applications. When specific domain expertise is needed, it is necessary to
build a foundation model from scratch or fine-tune the existing models.
Building a foundation model from scratch is resource-intensive and demands
significant computational power. However, it provides fine-grained control
over the architecture, training parameters, and overall behavior.

###  VI-G High Variability in Robotic Settings

Another challenge is the high variability in robotic settings. Robot platforms
are inherently diverse with different physical characteristics,
configurations, and capabilities. Real-world environments that robots operate
in are also diverse and uncertain with a wide range of variations. Due to all
these variabilities, robotic solutions are usually tailored to specific robot
platforms with specific layouts, environments, and objects for specific tasks.
These solutions are not generalizable across various embodiments,
environments, or tasks. Hence, to build general-purpose pretrained robotic
foundation models, a key factor is to pre-train large models that are task-
agnostic, cross-embodiment, and open-ended and capture diverse robotic data.
In ROSIE [176] a diverse dataset is generated for robot learning by performing
inpainting of various unseen objects, backgrounds, and distractors with
semantic textual guidance. To overcome variability in robotic settings and
improve generalization, another solution as ViNT [137] presents is to train
foundation models on diverse robotic data across various embodiments. RT-X
[46] also investigates the possibility of training large cross-embodied
robotic models in the domain of robotic manipulation. RT-X is trained using a
multi-embodiment dataset which is created by collecting data from different
robot platforms collected through a collaboration between 21 institutions,
demonstrating 160266 tasks. RT-X demonstrates transfer across embodiment
improves robot capabilities by employing experience from diverse robotic
platforms.

###  VI-H Benchmarking and Reproducibility in Robotics Settings

Another significant obstacle in incorporating foundation models into robotics
research is the necessary reliance on real-world hardware experiments. This
creates challenges for reproducibility, as replicating results obtained from
hardware experiments may necessitate access to the exact equipment employed.
Conversely, many recent works have relied on non-physics-based simulators
(e.g., ignoring or greatly simplifying contact physics in gasping) that
instead focus on high-level, long-term tasks and visual environment models.
Examples of this class of simulators are common and include many of the
simulators described above in Sec. V. For example the Gibson family of
simulators [206, 146], the Habitat family [147, 208, 209], RobotTHOR [210],
and VirtualHome [129] all neglect low-level physics in favor of simulating
higher level tasks with high visual fidelity. This leads to a large sim-to-
real gap and introduces variability in real-world performance based on how
low-level planning and control modules handle the true physics of the
scenario. Even when physics-based simulators are used (e.g., PyBullet or
MuJoCo), the absence of standardized simulation settings, computing
environments, and a persistent sim-to-real gap impede efforts to benchmark and
compare performance across various research endeavors. A combination of open
hardware, benchmarking in physics-based simulators, and promoting transparency
in experimental and simulation setups can significantly alleviate the
challenges associated with reproducibility in the integration of foundation
models into robotics research. These practices contribute to the development
of a more robust and collaborative research ecosystem within the field.

##  VII Conclusion

Through examination of the recent literature, we have surveyed the diverse and
promising applications of foundation models in robotics. We have delved into
how these models have enhanced the capabilities of robots in areas such as
decision-making, planning and control, and perception. We also discussed the
literature on embodied AI and generalist AI, with an eye toward opportunities
for roboticists to extend the concepts in that research field to real-world
robotic applications. Generalization, zero-shot capabilities, multimodal
capabilities, and scalability of foundation models have the potential to
transform robotics. However, as we navigate through this paradigm shift in
incorporating foundation models in robotics applications, it is imperative to
recognize the challenges and potential risks that must be addressed in future
research. Data scarcity in robotics applications, high variability in robotics
settings, uncertainty quantification, safety evaluation, and real-time
performance remain significant concerns that demand future research. We have
delved into some of these challenges and have discussed potential avenues for
improvement.

## Acknowledgments

The first author was supported on an ASEE e-Fellows postdoctoral fellowship.
J.T. and S.T. were partially supported by NSF Graduate Research Fellowships.
This project was also partially supported by DARPA project HR001120C0107 and
by a gift from Meta. We are grateful for this support. Anirudha Majumdar was
supported by the NSF CAREER Award [$\\#$2044149] and the Office of Naval
Research [N00014-23-1-2148].

## References

  * [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  NeurIPS, 33:1877–1901, 2020.
  * [3] OpenAI.  GPT-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, ICML, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 18–24 Jul 2021.
  * [5] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In ICML, pages 8821–8831. PMLR, 2021.
  * [6] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence.  PaLM-E: An embodied multimodal language model.  In arXiv preprint arXiv:2303.03378, 2023.
  * [7] Jiankai Sun, De-An Huang, Bo Lu, Yun-Hui Liu, Bolei Zhou, and Animesh Garg.  PlaTe: Visually-grounded planning with transformers in procedural tasks.  IEEE Robotics and Automation Letters, 7(2):4924–4930, 2022.
  * [8] Jianing Qiu, Lin Li, Jiankai Sun, Jiachuan Peng, Peilun Shi, Ruiyang Zhang, Yinzhao Dong, Kyle Lam, Frank P.-W. Lo, Bo Xiao, Wu Yuan, Ningli Wang, Dong Xu, and Benny Lo.  Large ai models in health informatics: Applications, challenges, and the future.  IEEE Journal of Biomedical and Health Informatics, 27(12):6074–6087, 2023.
  * [9] Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al.  Reasoning with foundation models: Concepts, methodologies, and outlook.  In Zenodo preprint 10.5281/zenodo.10298866, 2023.
  * [10] Dingyuan Zhang, Dingkang Liang, Hongcheng Yang, Zhikang Zou, Xiaoqing Ye, Zhe Liu, and Xiang Bai.  SAM3D: Zero-shot 3D object detection via segment anything model.  arXiv preprint arXiv:2306.02245, 2023.
  * [11] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan.  3D-LLM: Injecting the 3D world into large language models.  arXiv preprint arXiv:2307.12981, 2023.
  * [12] William Chen, Siyi Hu, Rajat Talak, and Luca Carlone.  Leveraging large language models for robot 3D scene understanding.  arXiv preprint arXiv:2209.05629, 2022.
  * [13] Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans.  Foundation models for decision making: Problems, methods, and opportunities.  arXiv preprint arXiv:2303.04129, 2023.
  * [14] Zhao Mandi, Homanga Bharadhwaj, Vincent Moens, Shuran Song, Aravind Rajeswaran, and Vikash Kumar.  CACTI: A framework for scalable multi-task multi-scene visual imitation learning.  arXiv preprint arXiv:2212.05711, 2022.
  * [15] Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever, Markus Wulfmeier, Nicolas Heess, and Martin Riedmiller.  Towards a unified agent with foundation models.  In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023.
  * [16] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh.  Reward design with language models.  In ICLR, 2023.
  * [17] Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Mguni, Yali Du, and Jun Wang.  ChessGPT: Bridging policy learning and language modeling.  arXiv preprint arXiv:2306.09200, 2023.
  * [18] Yifan Du, Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen.  Zero-shot visual question answering with language model feedback.  arXiv preprint arXiv:2305.17006, 2023.
  * [19] Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al.  AnnoLLM: Making large language models to be better crowdsourced annotators.  arXiv preprint arXiv:2303.16854, 2023.
  * [20] Xuan Xiao, Jiahang Liu, Zhipeng Wang, Yanmin Zhou, Yong Qi, Qian Cheng, Bin He, and Shuo Jiang.  Robot learning in the era of foundation models: A survey.  arXiv preprint arXiv:2311.14379, 2023.
  * [21] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  ACM Computing Surveys, 55(9):1–35, 2023.
  * [22] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in vision: A survey.  ACM Computing Surveys, 54(10s):1–41, 2022.
  * [23] Muning Wen, Runji Lin, Hanjing Wang, Yaodong Yang, Ying Wen, Luo Mai, Jun Wang, Haifeng Zhang, and Weinan Zhang.  Large sequence models for sequential decision-making: a survey.  Frontiers of Computer Science, 17(6):176349, 2023.
  * [24] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  arXiv preprint arXiv:2108.07258, 2021.
  * [25] Mohit Shridhar, Lucas Manuelli, and Dieter Fox.  CLIPort: What and where pathways for robotic manipulation.  In CoRL, pages 894–906. PMLR, 2022.
  * [26] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet.  Learning latent plans from play.  In CoRL, pages 1113–1132. PMLR, 2020.
  * [27] Mohit Shridhar, Lucas Manuelli, and Dieter Fox.  Perceiver-Actor: A multi-task transformer for robotic manipulation.  In CoRL, pages 785–799. PMLR, 2023.
  * [28] Corey Lynch and Pierre Sermanet.  Language conditioned imitation learning over unstructured data.  Robotics: Science and Systems, 2021.
  * [29] Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang.  Language-driven representation learning for robotics.  In RSS, 2023.
  * [30] Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei Zhang.  Human-timescale adaptation in an open-ended task space.  In ICML, 2023.
  * [31] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta.  R3M: A universal visual representation for robot manipulation.  arXiv preprint arXiv:2203.12601, 2022.
  * [32] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al.  Do as I can, not as I say: Grounding language in robotic affordances.  In CoRL, pages 287–318. PMLR, 2023.
  * [33] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter.  Inner monologue: Embodied reasoning through planning with language models.  In arXiv preprint arXiv:2207.05608, 2022.
  * [34] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei.  VoxPoser: Composable 3D value maps for robotic manipulation with language models.  In CoRL, 2023.
  * [35] Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell.  Zero-shot reward specification via grounded natural language.  In ICML, pages 14743–14752. PMLR, 2022.
  * [36] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang.  VIP: Towards universal visual reward and representation via value-implicit pre-training.  In ICLR, 2023.
  * [37] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman.  LIV: Language-image representations and rewards for robotic control.  In ICML, 2023.
  * [38] Suraj Nair, Eric Mitchell, Kevin Chen, brian ichter, Silvio Savarese, and Chelsea Finn.  Learning language-conditioned robot behavior from offline data and crowd-sourced annotation.  In CoRL, pages 1303–1315. PMLR, 08–11 Nov 2022.
  * [39] Yongchao Chen, Rujul Gandhi, Yang Zhang, and Chuchu Fan.  NL2TL: Transforming natural languages to temporal logics using large language models.  arXiv preprint arXiv:2305.07766, 2023.
  * [40] Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan.  AutoTAMP: Autoregressive task and motion planning with llms as translators and checkers.  arXiv preprint arXiv:2306.06531, 2023.
  * [41] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg.  ProgPrompt: Generating situated robot task plans using large language models.  In ICRA, pages 11523–11530. IEEE, 2023.
  * [42] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng.  Code as Policies: Language model programs for embodied control.  In ICRA, pages 9493–9500. IEEE, 2023.
  * [43] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor.  ChatGPT for Robotics: Design principles and model abilities.  Technical Report MSR-TR-2023-8, Microsoft, 2023.
  * [44] Anthony Brohan, Noah Brown, Justice Carbajal, and et al.  RT-1: Robotics transformer for real-world control at scale.  In RSS, 2023.
  * [45] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R Sanketi, Grecia Salazar, Michael S Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J Joshi, Alex Irpan, brian ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han.  RT-2: Vision-language-action models transfer web knowledge to robotic control.  In CoRL, 2023.
  * [46] Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al.  Open X-Embodiment: Robotic learning datasets and RT-X models.  arXiv preprint arXiv:2310.08864, 2023.
  * [47] Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Frujeri, Shuhang Chen, and Ashish Kapoor.  PACT: Perception-action causal transformer for autoregressive robotics pretraining.  In IROS. IEEE, 2023.
  * [48] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik.  Masked visual pre-training for motor control.  arXiv preprint arXiv:2203.06173, 2022.
  * [49] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell.  Real-world robot learning with masked visual pre-training.  In CoRL, pages 416–426. PMLR, 2023.
  * [50] Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Sai Vemprala, and Rogerio Bonatti.  LATTE: LAnguage Trajectory TransformEr.  In ICRA, pages 7287–7294. IEEE, 2023.
  * [51] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al.  Simple open-vocabulary object detection with vision transformers.  In ECCV, pages 728–755. Springer, 2022.
  * [52] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.  Grounded language-image pre-training.  In CVPR, pages 10965–10975, 2022.
  * [53] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.
  * [54] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li.  PointCLIP: Point cloud understanding by CLIP.  In CVPR, pages 8552–8562, 2022.
  * [55] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.  Point-BERT: Pre-training 3D point cloud transformers with masked point modeling.  In CVPR, pages 19313–19322, 2022.
  * [56] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese.  ULIP: Learning unified representation of language, image and point cloud for 3D understanding.  arXiv preprint arXiv:2212.05171, 2022.
  * [57] Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese.  ULIP-2: Towards scalable multimodal pre-training for 3d understanding.  arXiv preprint arXiv:2305.08275, 2023.
  * [58] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl.  Language-driven semantic segmentation.  In ICLR, 2022.
  * [59] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick.  Segment anything.  In ICCV, pages 4015–4026, October 2023.
  * [60] Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang.  Fast segment anything.  arXiv preprint arXiv:2306.12156, 2023.
  * [61] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung Ho Bae, Seungkyu Lee, and Choong Seon Hong.  Faster segment anything: Towards lightweight sam for mobile applications.  arXiv preprint arXiv:2306.14289, 2023.
  * [62] Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng.  Track anything: Segment anything meets videos.  arXiv preprint arXiv:2304.11968, 2023.
  * [63] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao.  CLIP-NeRF: Text-and-image driven manipulation of neural radiance fields.  In CVPR, pages 3835–3844, 2022.
  * [64] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik.  LERF: Language embedded radiance fields.  In ICCV, pages 19729–19739, 2023.
  * [65] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann.  Decomposing NeRF for editing via feature field distillation.  In NeurIPS, 2022.
  * [66] Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, and Sifei Liu.  Affordance diffusion: Synthesizing hand-object interactions.  In CVPR, 2023.
  * [67] Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak.  Affordances from human videos as a versatile representation for robotics.  In CVPR, 2023.
  * [68] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In ICML, 2022.
  * [69] Takuma Yoneda, Jiading Fang, Peng Li, Huanyu Zhang, Tianchong Jiang, Shengjie Lin, Ben Picker, David Yunis, Hongyuan Mei, and Matthew R. Walter.  Statler: State-maintaining language models for embodied reasoning.  arXiv preprint arXiv:2306.17840, 2023.
  * [70] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo.  EmbodiedGPT: Vision-language pre-training via embodied chain of thought.  arXiv preprint arXiv:2305.15021, 2023.
  * [71] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar.  MineDojo: Building open-ended embodied agents with internet-scale knowledge.  In NeurIPS Datasets and Benchmarks Track, 2022.
  * [72] Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune.  Video PreTraining (VPT): Learning to act by watching unlabeled online videos.  In NeurIPS, 2022.
  * [73] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  arXiv preprint arXiv: Arxiv-2305.16291, 2023.
  * [74] Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas.  Guiding pretraining in reinforcement learning with large language models.  In ICML, pages 8657–8677. PMLR, 23–29 Jul 2023.
  * [75] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In ACL, 2016.
  * [76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  NeurIPS, 2017.
  * [77] Daniel Dugas.  The gpt-3 architecture, on a napkin.  [Online; accessed 28-November-2023].
  * [78] Wikipedia.  GPT-3.  [Online; accessed 28-November-2023].
  * [79] John Thickstun.  The Transformer Model in Equations.  [Online; accessed 28-November-2023].
  * [80] George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung.  Time series analysis: forecasting and control.  John Wiley & Sons, 2015.
  * [81] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  https://openai.com/research/language-unsupervised, 2018.
  * [82] Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer.  Generating Wikipedia by summarizing long sequences.  In ICLR, 2018.
  * [83] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz.  Contrastive learning of medical visual representations from paired images and text.  In MLHC, 2022.
  * [84] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * [85] Kihyuk Sohn.  Improved deep metric learning with multi-class N-pair loss objective.  In NeurIPS, 2016.
  * [86] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In CVPR, 2018.
  * [87] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [88] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with CLIP latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [89] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In ICML, 2015.
  * [90] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  In NeurIPS, 2019.
  * [91] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In NeurIPS, 2020.
  * [92] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI Blog, 2019.
  * [93] Hector Levesque, Ernest Davis, and Leora Morgenstern.  The Winograd schema challenge.  In KR, 2012.
  * [94] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  arXiv preprint arXiv:1804.07461, 2018.
  * [95] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  LLaMA: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [96] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  PaLM: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [97] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang.  GLM-130B: An open bilingual pre-trained model.  In ICLR, 2023.
  * [98] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [99] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al.  A survey on vision transformer.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  * [100] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in vision: A survey.  ACM Computing Surveys, 2022.
  * [101] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In CVPR, 2022.
  * [102] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al.  PaLI: A jointly-scaled multilingual language-image model.  In NeurIPS, 2022.
  * [103] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al.  Scaling vision transformers to 22 billion parameters.  In ICML, 2023.
  * [104] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al.  PaLI-X: On scaling up a multilingual vision and language model.  arXiv preprint arXiv:2305.18565, 2023.
  * [105] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [106] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  arXiv preprint arXiv:1512.03385, 2015.
  * [107] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  DINOv2: Learning robust visual features without supervision.  arXiv preprint arXiv:2304.07193, 2023.
  * [108] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In ICML, 2022.
  * [109] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu.  CLIP2: Contrastive language-image-point pretraining from real-world point cloud data.  In CVPR, 2023.
  * [110] Lewei Yao, Runhu Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  FILIP: Fine-grained interactive language-image pre-training.  In ICLR, 2022.
  * [111] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He.  Scaling language-image pre-training via masking.  In CVPR, 2023.
  * [112] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821–8831. PMLR, 18–24 Jul 2021.
  * [113] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In CVPR, pages 16000–16009, 2022.
  * [114] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, pages 8748–8763, 2021.
  * [115] Ge Yuying, Macaluso Annabella, Erran Li Li, Luo Ping, and Wang Xiaolong.  Policy adaptation from foundation model feedback.  In CVPR, 2023.
  * [116] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee.  Transporter networks: Rearranging the visual world for robotic manipulation.  In CoRL, 2020.
  * [117] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [118] Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar.  MimicPlay: Long-horizon imitation learning by watching human play.  In CoRL, 2023.
  * [119] Rutav Shah, Roberto Martín-Martín, and Yuke Zhu.  MUTEX: Learning unified policies from multimodal task specifications.  In CoRL, 2023.
  * [120] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel.  RL^2: Fast reinforcement learning via slow reinforcement learning.  arXiv preprint arXiv:1611.02779, 2016.
  * [121] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov.  Transformer-XL: Attentive language models beyond a fixed-length context.  In ACL, 2019.
  * [122] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michaël Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin Dalibard, and Wojciech Marian Czarnecki.  Open-ended learning leads to generally capable agents.  arXiv preprint arXiv:2107.12808, 2021.
  * [123] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Scaling egocentric vision: The EPIC-KITCHENS dataset.  In ECCV, 2018.
  * [124] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg.  Text2Motion: From natural language instructions to feasible plans.  Autonomous Robots. Special Issue: Large Language Models in Robotics, 2023.
  * [125] Oier Mees, Jessica Borja-Diaz, and Wolfram Burgard.  Grounding language with visual affordances over unstructured data.  In ICRA, pages 11576–11582. IEEE, 2023.
  * [126] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  * [127] Junning Huang, Sirui Xie, Jiankai Sun, Qiurui Ma, Chunxiao Liu, Dahua Lin, and Bolei Zhou.  Learning a decision module by imitating driver’s control behaviors.  In CoRL, pages 1–10. PMLR, 2021.
  * [128] Jiankai Sun, Hao Sun, Tian Han, and Bolei Zhou.  Neuro-symbolic program search for autonomous driving decision module design.  In CoRL, pages 21–30. PMLR, 2021.
  * [129] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba.  VirtualHome: Simulating household activities via programs.  In CVPR, pages 8494–8502, 2018.
  * [130] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui.  A survey for in-context learning.  arXiv preprint arXiv:2301.00234, 2022.
  * [131] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  NeurIPS, 35:24824–24837, 2022.
  * [132] Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng.  Large language models as general pattern machines.  In arXiv preprint arXiv:2307.04721, 2023.
  * [133] Zhiwei Jia, Fangchen Liu, Vineet Thumuluri, Linghao Chen, Zhiao Huang, and Hao Su.  Chain-of-thought predictive control.  arXiv preprint arXiv:2304.00776, 2023.
  * [134] Yanchao Sun, Shuang Ma, Ratnesh Madaan, Rogerio Bonatti, Furong Huang, and Ashish Kapoor.  SMART: Self-supervised multi-task pretraining with control transformers.  In ICLR, 2023.
  * [135] Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra.  Improving vision-and-language navigation with image-text pairs from the web.  In ECCV, pages 259–274. Springer, 2020.
  * [136] Dhruv Shah, Błażej Osiński, Sergey Levine, et al.  LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action.  In CoRL, pages 492–504. PMLR, 2023.
  * [137] Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, and Sergey Levine.  ViNT: A Foundation Model for Visual Navigation.  In CoRL, 2023.
  * [138] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard.  Audio visual language maps for robot navigation.  arXiv preprint arXiv:2303.07522, 2023.
  * [139] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  NetVLAD: CNN architecture for weakly supervised place recognition.  In CVPR, pages 5297–5307, 2016.
  * [140] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In CVPR Deep Learning for Visual SLAM Workshop, 2018.
  * [141] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel.  AudioCLIP: Extending clip to image, text and audio.  In ICASSP, pages 976–980. IEEE, 2022.
  * [142] Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot.  Navigating to objects in the real world.  arXiv preprint arXiv:2212.00922, 2023.
  * [143] Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song.  CoWs on pasture: Baselines and benchmarks for language-driven zero-shot object navigation.  In CVPR, pages 23171–23181, 2023.
  * [144] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3D: Learning from RGB-D data in indoor environments.  In 3DV, pages 667–676, 2017.
  * [145] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  In CVPR, pages 3674–3683, 2018.
  * [146] Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Elliott Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese.  iGibson 2.0: Object-centric simulation for robot learning of everyday household tasks.  In CoRL, 2021.
  * [147] Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al.  Habitat 2.0: Training home assistants to rearrange their habitat.  In NeurIPS, 2021.
  * [148] Yu Bangguo, Kasaei Hamidreza, and Cao Ming.  LL3MVN: Leveraging large language models for visual target navigation.  arXiv preprint arXiv:2304.05501, 2023.
  * [149] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A robustly optimized BERT pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [150] Junting Chen, Guohao Li, Suryansh Kumar, Bernard Ghanem, and Fisher Yu.  How to not train your dragon: Training-free embodied object goal navigation with semantic frontiers.  arXiv preprint arXiv:2305.16925, 2023.
  * [151] Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, Zsolt Kira, Manolis Savva, Angel Chang, Devendra Singh Chaplot, Dhruv Batra, Roozbeh Mottaghi, Yonatan Bisk, and Chris Paxton.  Homerobot: Open vocabulary mobile manipulation.  arXiv preprint arXiv:2306.11565, 2023.
  * [152] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan.  VIMA: General robot manipulation with multimodal prompts.  In ICML, 2023.
  * [153] Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al.  RoboCat: A self-improving foundation agent for robotic manipulation.  arXiv preprint arXiv:2306.11706, 2023.
  * [154] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al.  A generalist agent.  arXiv preprint arXiv:2205.06175, 2022.
  * [155] Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, and Chris Paxton.  StructDiffusion: Language-guided creation of physically-valid structures using unseen objects.  In RSS, 2023.
  * [156] Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, Chelsea Finn, and Karol Hausman.  Open-world object manipulation using pre-trained vision-language model.  arXiv preprint arXiv:2303.00905, 2023.
  * [157] Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns.  DALL-E-Bot: Introducing web-scale diffusion models to robotics.  IEEE Robotics and Automation Letters, 8(7):3956–3963, 2023.
  * [158] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In ICCV, pages 2980–2988, 2017.
  * [159] Tay Yi, Dehghani Mostafa, Tran Vinh Q., Garcia Xavier, Wei Jason, Wang Xuezhi, Chung Hyung Won, Shakeri Siamak, Bahri Dara, Schuster Tal, Steven Zheng Huaixiu, Zhou Denny, Houlsby Neil, and Donald Metzler.  UL2: Unifying language learning paradigms.  arXiv preprint arXiv:2205.05131, 2023.
  * [160] Shuang Ma, Sai Vemprala, Wenshan Wang, Jayesh K Gupta, Yale Song, Daniel McDufft, and Ashish Kapoor.  COMPASS:: Contrastive multimodal pretraining for autonomous systems.  In IROS, pages 1000–1007. IEEE, 2022.
  * [161] Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency.  Multimodal machine learning: A survey and taxonomy.  IEEE Trans. Pattern Anal. Mach. Intell., 41(2):423–443, feb 2019.
  * [162] Minghua Liu, Yinhao Zhu, Hong Cai, Shizhong Han, Zhan Ling, Fatih Porikli, and Hao Su.  Partslip: Low-shot part segmentation for 3d point clouds via pretrained image-language models.  In CVPR, pages 21736–21746, 2023.
  * [163] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  ShapeNet: An information-rich 3D model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [164] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In ICCV, pages 12179–12188, 2021.
  * [165] Ho Kei Cheng and Alexander G Schwing.  XMem: Long-term video object segmentation with an atkinson-shiffrin memory model.  In ECCV, pages 640–658. Springer, 2022.
  * [166] Qiuhong Shen, Xingyi Yang, and Xinchao Wang.  Anything-3D: Towards single-view anything reconstruction in the wild.  arXiv preprint arXiv:2304.10261, 2023.
  * [167] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [168] Jiankai Sun, Yan Xu, Mingyu Ding, Hongwei Yi, Chen Wang, Jingdong Wang, Liangjun Zhang, and Mac Schwager.  NeRF-Loc: Transformer-based object localization within neural radiance fields.  IEEE Robotics and Automation Letters, 8(8):5244–5250, 2023.
  * [169] Jiankai Sun, Jianing Qiu, Chuanyang Zheng, John Tucker, Javier Yu, and Mac Schwager.  Aria-NeRF: Multimodal egocentric view synthesis.  arXiv preprint arXiv:2311.06455, 2023.
  * [170] Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur Szlam.  CLIP-Fields: Weakly supervised semantic fields for robotic memory.  In RSS, 2023.
  * [171] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard.  Visual language maps for robot navigation.  In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 10608–10615. IEEE, 2023.
  * [172] Huy Ha and Shuran Song.  Semantic abstraction: Open-world 3D scene understanding from 2D vision-language models.  In CoRL, 2022.
  * [173] Jianglong Ye, Naiyan Wang, and Xiaolong Wang.  FeatureNeRF: Learning generalizable nerfs by distilling pre-trained vision foundation models.  arXiv preprint arXiv:2303.12786, 2023.
  * [174] Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi.  Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representations.  In 3DV, pages 443–453. IEEE, 2022.
  * [175] Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, Leonidas Guibas, Hao Su, and Kyle Genova.  Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision.  In CVPR, pages 8274–8284, 2023.
  * [176] Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Brian Ichter, Karol Hausman, and Fei Xia.  Scaling robot learning with semantically imagined experience.  In arXiv preprint arXiv:2302.11550, 2023.
  * [177] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al.  Imagen editor and editbench: Advancing and evaluating text-guided image inpainting.  In CVPR, pages 18359–18369, 2023.
  * [178] Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar.  GenAug: Retargeting behaviors to unseen situations via generative augmentation.  In RSS, 2023.
  * [179] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann.  Neural Descriptor Fields: SE(3)-equivariant object representations for manipulation.  In ICRA, 2022.
  * [180] William Shen, Ge Yang, Alan Yu, Jensen Wong, Leslie Pack Kaelbling, and Phillip Isola.  Distilled feature fields enable few-shot manipulation.  In CoRL, 2023.
  * [181] Walter Goodwin, Ioannis Havoutis, and Ingmar Posner.  You only look at one: Category-level object representations for pose estimation from a single example.  In CoRL, 2022.
  * [182] Walter Goodwin, Sagar Vaze, Ioannis Havoutis, and Ingmar Posner.  Zero-shot category-level object pose estimation.  In ECCV, 2022.
  * [183] Shinji Umeyama.  Least-squares estimation of transformation parameters between two point patterns.  IEEE Transactions on Pattern Analysis & Machine Intelligence, 13(04):376–380, 1991.
  * [184] Jiankai Sun, Lantao Yu, Pinqian Dong, Bo Lu, and Bolei Zhou.  Adversarial inverse reinforcement learning with self-attention dynamics model.  IEEE Robotics and Automation Letters, 6(2):1880–1886, 2021.
  * [185] Jiankai Sun, Shreyas Kousik, David Fridovich-Keil, and Mac Schwager.  Connected autonomous vehicle motion planning with video predictions from smart, self-supervised infrastructure.  arXiv preprint arXiv:2309.07504, 2023.
  * [186] Jiankai Sun, Shreyas Kousik, David Fridovich-Keil, and Mac Schwager.  Self-supervised traffic advisors: Distributed, multi-view traffic prediction for smart cities.  In IEEE ITSC, 2022.
  * [187] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine.  Planning with diffusion for flexible behavior synthesis.  In ICML, 2022.
  * [188] Jiankai Sun, Yiqi Jiang, Jianing Qiu, Parth Talpur Nobel, Mykel Kochenderfer, and Mac Schwager.  Conformal prediction for uncertainty-aware planning with diffusion dynamics model.  In NeurIPS, 2023.
  * [189] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and D. Erhan.  Phenaki: Variable length video generation from open domain textual description.  In ICLR, 2023.
  * [190] Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn.  RoboNet: Large-scale multi-robot learning.  In CoRL, 2019.
  * [191] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado.  GAIA-1: A generative world model for autonomous driving.  arXiv preprint arXiv:2309.17080, 2023.
  * [192] Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schuurmans, and Pieter Abbeel.  Learning universal policies via text-guided video generation.  In NeurIPS, 2023.
  * [193] Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Kaelbling, Andy Zeng, and Jonathan Tompson.  Video language planning.  arXiv preprint arXiv:2310.10625, 2023.
  * [194] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence.  Socratic Models: Composing zero-shot multimodal reasoning with language.  arXiv, 2022.
  * [195] Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill, and Rob Fergus.  Collaborating with language models for embodied reasoning.  In Second Workshop on Language and Reinforcement Learning, 2022.
  * [196] Herman A. Engelbrecht and Gregor Schiele.  Transforming minecraft into a research platform.  In IEEE CCNC, 2014.
  * [197] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai.  Ghost in the Minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory.  arXiv preprint arXiv:2305.17144, 2023.
  * [198] Ziqi Zhang, Yile Wang, Yue Zhang, and Donglin Wang.  Can offline reinforcement learning help natural language understanding?  arXiv preprint arXiv:2212.03864, 2022.
  * [199] Denis Tarasov, Vladislav Kurenkov, and Sergey Kolesnikov.  Prompts and pre-trained language models for offline reinforcement learning.  In ICLR Workshop on Generalizable Policy Learning in Physical World, 2022.
  * [200] Machel Reid, Yutaro Yamada, and Shixiang Shane Gu.  Can Wikipedia help offline reinforcement learning?  arXiv preprint arXiv:2201.12122, 2022.
  * [201] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  arXiv preprint arXiv:2107.03374, 2021.
  * [202] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.  ReAct: Synergizing reasoning and acting in language models.  In ICLR, 2023.
  * [203] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.  Generative Agents: Interactive simulacra of human behavior.  In ACM Symposium on User Interface Software and Technology, 2023.
  * [204] Zhou Xian, Theophile Gervet, Zhenjia Xu, Yi-Ling Qiao, Tsun-Hsuan Wang, and Yian Wang.  Towards generalist robots: A promising paradigm via generative simulation.  arXiv preprint arXiv:2305.10455, 2023.
  * [205] Rutav Shah and Vikash Kumar.  RRL: Resnet as representation for reinforcement learning.  In ICML, 2021.
  * [206] Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese.  Gibson Env: real-world perception for embodied agents.  In CVPR, 2018.
  * [207] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei.  BEHAVIOR-1K: A benchmark for embodied AI with 1,000 everyday activities and realistic simulation.  In CoRL, 2022.
  * [208] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra.  Habitat: A Platform for Embodied AI Research.  In ICCV, 2019.
  * [209] Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, et al.  Habitat 3.0: A co-habitat for humans, avatars and robots.  arXiv preprint arXiv:2310.13724, 2023.
  * [210] Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al.  RoboTHOR: An open simulation-to-real embodied AI platform.  In CVPR, 2020.
  * [211] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer.  TartanAir: A dataset to push the limits of visual SLAM.  In IROS, 2020.
  * [212] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor.  AirSim: High-fidelity visual and physical simulation for autonomous vehicles.  In Field and Service Robotics, 2017.
  * [213] Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson.  Robotic skill acquisition via instruction augmentation with vision-language models.  In RSS, 2023.
  * [214] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  LLaMA: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [215] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han.  AWQ: Activation-aware weight quantization for llm compression and acceleration.  arXiv preprint arXiv:2306.00978, 2023.
  * [216] Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majumdar.  Robots that ask for help: Uncertainty alignment for large language model planners.  In CoRL, 2023.
  * [217] Young-Jin Park, Hao Wang, Shervin Ardeshir, and Navid Azizan.  Representation reliability and its impact on downstream tasks.  arXiv preprint arXiv:2306.00206, 2023.
  * [218] Meiqi Sun, Wilson Yan, Pieter Abbeel, and Igor Mordatch.  Quantifying uncertainty in foundation models via ensembles.  In NeurIPS Workshop on Robustness in Sequence Modeling, 2022.
  * [219] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.  arXiv preprint arXiv:2209.07858, 2022.
  * [220] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving.  Red teaming language models with language models.  In EMNLP, 2022.
  * [221] Shengbang Tong, Erik Jones, and Jacob Steinhardt.  Mass-producing failures of multimodal systems with language models.  arXiv preprint arXiv:2306.12105, 2023.
  * [222] Kristofer D Kusano, Kurt Beatty, Scott Schnelle, Francesca Favaro, Cam Crary, and Trent Victor.  Collision avoidance testing of the waymo automated driving system.  arXiv preprint arXiv:2212.08148, 2022.
  * [223] Nick Webb, Dan Smith, Christopher Ludwick, Trent Victor, Qi Hommes, Francesca Favaro, George Ivanov, and Tom Daniel.  Waymo’s safety methodologies and safety readiness determinations.  arXiv preprint arXiv:2011.00054, 2020.
  * [224] Wenhao Ding, Chejian Xu, Mansur Arief, Haohong Lin, Bo Li, and Ding Zhao.  A survey on safety-critical driving scenario generation—a methodological perspective.  IEEE ITSC, 2023.
  * [225] Rachel Luo, Shengjia Zhao, Jonathan Kuck, Boris Ivanovic, Silvio Savarese, Edward Schmerling, and Marco Pavone.  Sample-efficient safety assurances using conformal prediction.  In WAFR, 2022.
  * [226] Alec Farid, David Snyder, Allen Z Ren, and Anirudha Majumdar.  Failure prediction with statistical guarantees for vision-based robot control.  arXiv preprint arXiv:2202.05894, 2022.
  * [227] Alec Farid, Sushant Veer, Boris Ivanovic, Karen Leung, and Marco Pavone.  Task-relevant failure detection for trajectory predictors in autonomous vehicles.  In CoRL, 2022.
  * [228] Kai-Chieh Hsu, Allen Z Ren, Duy P Nguyen, Anirudha Majumdar, and Jaime F Fisac.  Sim-to-Lab-to-Real: Safe reinforcement learning with shielding and generalization guarantees.  Artificial Intelligence, 314:103811, 2023.
  * [229] Kai-Chieh Hsu, Haimin Hu, and Jaime Fernández Fisac.  The safety filter: A unified view of safety-critical control in autonomous systems.  arXiv preprint arXiv:2309.05837, 2023.
  * [230] Alec Farid, Sushant Veer, and Anirudha Majumdar.  Task-driven out-of-distribution detection with statistical guarantees for robot learning.  In CoRL, 2021.
  * [231] Ido Greenberg and Shie Mannor.  Detecting rewards deterioration in episodic reinforcement learning.  In ICML, 2021.
  * [232] Feiyang Cai and Xenofon Koutsoukos.  Real-time out-of-distribution detection in learning-enabled cyber-physical systems.  In ICCPS, 2020.
  * [233] Rohan Sinha, Apoorva Sharma, Somrita Banerjee, Thomas Lew, Rachel Luo, Spencer M Richards, Yixiao Sun, Edward Schmerling, and Marco Pavone.  A system-level view on out-of-distribution data in robotics.  arXiv preprint arXiv:2212.14020, 2022.