# LANCAR: Leveraging Language for Context-Aware
Robot Locomotion in Unstructured Environments

Chak Lam Shek1∗, Xiyang Wu1∗, Dinesh Manocha2, Pratap Tokekar2, and Amrit
Singh Bedi2 ∗ Denotes equal contribution1 Authors are with the Department of
Electrical and Computer Engineering, University of Maryland, College Park, MD,
USA {cshek1<EMAIL_ADDRESS>Authors are with the Department of Computer
Science, University of Maryland, College Park, MD, USA {dmanocha,
<EMAIL_ADDRESS>
###### Abstract

Robotic locomotion is a challenging task, especially in unstructured terrains.
In practice, the optimal locomotion policy can be context-dependent by using
the contextual information of encountered terrains in decision-making. Humans
can interpret the environmental context for robots, but the ambiguity of human
language makes it challenging to use in robot locomotion directly. In this
paper, we propose a novel approach, LANCAR, that introduces a context
translator that works with reinforcement learning (RL) agents for context-
aware locomotion. Our formulation allows a robot to interpret the contextual
information from environments generated by human observers or Vision-Language
Models (VLM) with Large Language Models (LLM) and use this information to
generate contextual embeddings. We incorporate the contextual embeddings with
the robot’s internal environmental observations as the input to the RL agent’s
decision neural network. We evaluate LANCAR with contextual information in
varying ambiguity levels and compare its performance using several alternative
approaches. Our experimental results demonstrate that our approach exhibits
good generalizability and adaptability across diverse terrains, by achieving
at least 10% of performance improvement in episodic reward over baselines. The
experiment video can be found at the following link:
https://raaslab.org/projects/LLM_Context_Estimation/.

##  I Introduction

Reinforcement Learning (RL) is prevalent in robotics, impacting manipulation
[1], navigation [2], and locomotion [3] tasks. RL agents learn optimal actions
by interacting with environments. However, developing unified learning RL
agents that can work robustly in diverse conditions is a critical challenge,
as optimal policies in diverse conditions could diverge a lot [4]. Consider
the case when computing robust locomotion policies for a legged robot platform
that can operate in an unstructured environment with various terrains. A
standard RL-based approach does not always lead to good performance in this
scenario, as policy parameters trained on a specific terrain may not translate
to another [5]. One possible approach could be letting the robot agent model
all environmental properties as a part of its state and then learn a policy
that works in all conditions. However, this method is not feasible due to the
sensor limitations in fields of view or ranges. The incapability of sensors in
detecting certain vital factors can downgrade the performance, e.g. a robot
might slip on loose soil without detecting its looseness. Policy
generalization and adaptation across diverse terrains is still an open problem
[6].

To address these issues, many prior works attempt to extract contextual
information from environments from graph-like structures [7] or autoencoders
[8] to assist decision-making. However, those methods lack the reasoning and
inference ability to handle complicated terrains. A natural idea in these
scenarios is to cooperate with humans to interpret the environmental
contextual information to robots through their comprehensive sensing and
better reasoning abilities. For instance, humans can correlate wet grassland
with high damping upon observation, a connection robots struggle to make.
However, the ambiguity of human language prevents this reasoning ability from
being directly used by robots [9], as many similar sentences can be
interpreted differently. As a result, it is challenging for robots to make use
of human-provided contextual information.

Figure 1: Task Description. We consider two robot learning approaches. The
first existing approach (TOP) is when the robot moves over diverse terrains
with a trained policy without any contextual information. Given the complexity
of the terrains, robots may face difficulties in developing a generalized
policy to address all types of terrains, leading to the failure of its
ultimate policy. Our proposed approach (BOTTOM) has the robot moving over
diverse terrains with our trained policy and contextual information from human
observers or visual-language models. Robots convert this interpreted
contextual information into embeddings with LLM. With the extra contextual
information added to robots’ own observations, robots could develop better
policies with a better understanding of the environment.

The recent success of the Large Language Model (LLMs) and their ability to
perform chain-of-thought [10], logic reasoning [11], and common sense
reasoning [12] is a promising approach to address these problems. An
interesting line of work is to incorporate LLM within RL frameworks so that an
RL agent can use LLM to assist its learning process and make it more sample-
efficient. Several prior studies have made attempts along this line, such as
using LLMs to predict the reward functions necessary for RL [13] or providing
control inputs for robots [14]. These approaches, though intriguing, still do
not exploit the full potential of the reasoning abilities of LLM. We believe
that LLMs are better suited as intermediaries, serving as interfaces to
translate human language into formats that are more accessible to RL agents.
This prevents human instructions from dominating and interrupting the
decision-making process of the RL agent, which may cause performance
degradation.

Main Contribution: In this paper, we investigate the possibility of utilizing
LLMs to interpret contextual information from environments (through their
reasoning ability) to help RL agents perform robot locomotion tasks.
Specifically, we study a quadruped robot navigating various terrains with a
human observer helping to interpret the context. The context refers to terrain
properties that the robot might not directly perceive. Fig. 1 gives an
overview of our approach. In Scenario $1$, a robot traverses various terrains
without any contextual information. Given the complexity of the terrains
encountered, robots may fail to develop a generalized policy for all terrains.
In Scenario $2$, the robot traverses the same set of diverse terrains but
receives contextual information from human observers111In many cases vision-
language models may also be a useful surrogate to provide contextual
information that is visual, like “You are entering a grassland right after the
rain” or “You are walking on a dry rocky road under the sun”. Robots use an
LLM-based translator to generate the embedding representing contextual
information from the human interpretation, resulting in better decision-making
as a supplement to robots’ own observations.

Our approach leverages LLMs’ generality in interpreting human languages. By
translating the languages into index or contextual embeddings, the
interpretation helps to address the ambiguity within human languages and makes
robots adapt to diverse terrains with generalized control policies by taking
advantage of the collaboration with human observers.

We justify the advantage of our approach, LANCAR, using the spot-mini-mini
[15] robot simulator. Results demonstrate that LANCAR outperforms the baseline
no-context and indexing approaches, as in most of the cases, LANCAR has at
least 10% of performance improvement. We summarize our main contributions in
this work as follows.

  * •

We propose a novel approach, LANCAR, that incorporates LLMs into RL in robot
decision-making that enables robots to utilize external contextual information
from human observers or VLMs and generate a more robust and generalized RL
policy.

  * •

We propose an LLM-based contextual information translator module that
interprets _high-level_ , ambiguous, human language contextual information of
environments into contextual information embedding accessible for RL agents
with the reasoning ability of LLMs.

  * •

We evaluate LANCAR and several alternative approaches with three case studies
in _low-level_ , _high-level_ , and VLM-interpreted contextual information. We
validate the efficacy of LANCAR in policy generalizability and adaptability
across diverse terrains.

##  II Related Works

Navigation in Complex Environments. Reliable robot locomotion and navigation
in a complex environment is a long-lasting challenging task, as robots must
learn an adaptive policy across diverse terrain [16]. NAUTS [17] proposes an
approach that makes robots adaptive to off-road diverse terrain with a
negotiation process among various navigational policies. VINet [18] uses a
novel navigation-based labeling scheme for terrain classification and
generalization on both known and unknown surfaces. Ada-Nav [2] presents a
novel approach that adaptively tunes policy evaluation trajectory lengths with
policy entropy and evaluates this approach in both simulated and real-world
outdoor environments. Though many approaches listed above reveal good
performance in practice, they mainly train and execute their approach in a
terrain dataset with a limited number of terrains. Many of them depend on a
semantic approach in terrain adaptation, which may constrain their ability to
generalization in real-world diverse terrains.

Human-robot Collaboration. Human-robot collaboration investigates interaction
strategies between humans and robots, especially in unstructured environments
[19]. With the emergence of LLM, human-robot collaboration has received a
boost as robots can take advantage of human knowledge and common sense
reasoning through LLMs. Ren et al. [20] propose an approach that allows robots
to seek help from humans with the assistance of LLM. SayTap [21] uses foot
contact patterns as the interface between human commands in natural language
and a locomotion controller that outputs _low-level_ commands. RE-Move [22]
uses human-language instructions to help robots avoid obstacles. LM-Nav [23]
uses LLM and VLM in object detection for robots’ navigation tasks. These prior
works provide insights into the potential in applying LLM in robot control
tasks. However, they do not directly address the challenge of policy
generalization across contexts, which is the focus of this paper.

LLMs in Robotics and RL. Large Language Model (LLM) [24] and Vision-Language
Model (VLM) [25] reveal their ability of In-context Learning (ICL) through
zero-shot or few-shot examples given within the context [26]. This has
stimulated progress in vision-and-language navigation [27]. RT-2 [28, 29]
allows manipulators to use the Internet-scale data from the VLMs in their
decision-making by taking the action output sequence as another language.
Bucker et al. [30, 31] use LLMs to allow human language to improve the
manipulator trajectories. Mees et al. [32] utilize LLM to decompose the _high-
level_ tasks into sub-tasks for robot to execute. Fu et al. [33] use LLMs as a
driving assistant in autonomous driving tasks. For reinforcement learning,
prior works have explored using LLMs in determining reward values [13] and
policy explainability in human-AI interaction [34]. However, we notice that
there is no prior attempt to use LLM to understand the environments’
observation space and use this inference in RL agents, while our approach
explores this idea within our framework.

##  III Methodology

###  III-A Problem Formation

We model the problem as an extension of a Partially Observable Markov Decision
Process (POMDP), specifically as an implicit POMDP [35]. An implicit POMDP is
specified by a tuple,
$\left\langle\mathcal{S},\mathcal{A},\mathcal{O},\Omega,\mathcal{Z},\mathcal{F},\mathcal{T},\mathcal{R},\gamma\right\rangle$,
where the state space, $\mathcal{S}=\mathcal{S}_{ex}\cup\mathcal{S}_{im}$, is
composed of both explicitly observable states $\mathcal{S}_{ex}$ and
implicitly observable states $\mathcal{S}_{im}$. The explicitly observable
states are those environmental states directly observable from the agent’s
onboard sensors. The agent’s observation space is $\mathcal{O}$. The
observation function is given by
$\Omega:\mathcal{S}_{ex}\rightarrow\mathcal{O}$. The implicitly observable
states are the contextual information in the environment that cannot be
detected directly by the robots but still affect robots’ policies.
$\mathcal{Z}$ denotes the embedding of the contextual information from the
implicitly observable states $\mathcal{S}_{im}$, while the mapping function
between the two is $\mathcal{F}:\mathcal{S}_{im}\rightarrow\mathcal{Z}$.
Nevertheless, the implicitly observable states (i.e. contextual information)
can still be inferred by robots through reasoning over visual perception or
tactile sensing or through human language feedback. In this work, our primary
focus is to recover $\mathcal{S}_{im}$ using contextual information given in
natural language.

The action space $\mathcal{A}$ represents the agent’s feasible actions. The
transition function
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$ characterizes
the dynamics of the robot within the environment. The reward function
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ quantifies the
reward of the agent’s actions. $\gamma$ is the discounted factor. The agent’s
policy $\pi$ is given by
$\pi:\mathcal{O}\times\mathcal{Z}\rightarrow\Delta(\mathcal{A})$, while
$\Delta(\mathcal{A})$ represents the probability distribution over the action
space. We formulate our problem as a finite horizon optimization. The
objective is to find an optimal policy $\pi^{*}$ that maximizes the expected
cumulative reward

$\displaystyle\pi^{*}=\arg\max_{\pi}\mathbb{E}_{\pi\sim\\{s_{t},a_{t}\\}_{t=0}^{H-1}}\sum_{t=0}^{H-1}\gamma^{t}R(s_{t},a_{t})$
(1)

where $H$ is the length of the episode.

Figure 2: Context-Aware Reinforcement Learning Robot Locomotion. Our framework
introduces a context translator aside from the standard RL framework. For the
environment with diverse terrains, the agent gets the explicitly observable
state as the observation, and the human observer (or VLM) perceives the
context information as the implicitly observable state. The human observer (or
VLM) interprets the contextual information to the LLM translator. The LLM
translator extracts the environmental properties from the contextual
information and generates the contextual embedding, which is concatenated with
the observations as the input for RL agents. RL agents produce the action
using their control policies given the context-aware inputs and execute the
action in the environment.

###  III-B Human-Robot Collaboration Framework

We introduce a human-robot collaboration framework, LANCAR, as depicted in
Figure 2. To recover the contextual information from environments, we
introduce the LLM-based context translator module in addition to the standard
RL agent. When robots traverse in environments with diverse terrains at time
$t$, robots observe the environment’s explicitly observable states
$s_{ex}^{t}$, and the human observer or VLM interprets the implicitly
observable states $s_{im}$ (i.e. contextual information). Here, we assume that
the contextual information is consistent within one episode so that $s_{im}$
is fixed. The human observer or VLM provides qualitative descriptions or
captions of the contextual information to the LLM translator. The LLM
translator extracts the environmental properties from the contextual
information and generates the contextual embedding $z$, which is concatenated
with the observations $o_{t}$ as the input for RL agents. RL agents produce
the action $a_{t}$ using their control policies $\pi$ given the context-aware
inputs and execute the action in the environment. This framework is designed
to be compatible with various RL methods, offering flexibility in its
implementation.

The framework is designed delicately to adapt humans’ assistance to enhance
agents’ performance. While it is hypothesized that well-trained agents are
better suited to produce a sequence of continuous decisions, direct human
control over such well-trained agents may disrupt the decision-making process,
potentially leading to degraded performance. On the other hand, human-provided
descriptions translated into state estimates over $\mathcal{S}_{im}$ can serve
as valuable assistance, enabling the agent to improve its overall performance.

###  III-C LLM-based Context Translator

The LLM plays a central role in our framework by utilizing prompts to convert
human-interpreted or VLM-perceived contextual information from the environment
into contextual embeddings accessible to RL agents. We design our context
translator module based on In-context Learning (ICL), which allows us to
utilize the reasoning ability of LLM through zero-shot or few-shot examples.
ICL provides us with an interpretable interface to communicate with the LLM
without any training procedure that imitates human reasoning and decision-
making process [26]. In our framework, we provide descriptive sentences of
contextual information to LLM, along with prompts with examples of inputs and
outputs that LLM may encounter during training.

An example prompt is presented in Figure 3. It consists of several sections.
In the first two sections, we provide our task descriptions and a detailed
description of each environmental property of interest as contextual
information. To map qualitative descriptions of contextual information into
embeddings, we provide a set of multiple-choice questions in the last section
of the prompt. Each question is related to a single environmental property and
LLM must choose among several pre-defined qualitative descriptive words. The
answers are mapped into concatenated one-hot vectors, as contextual embeddings
generated by LLM to provide to RL agents. For example, if the contextual
information describes the terrain with two properties, saying This terrain has
very low friction and very high damping. very low friction maps into an one-
hot vector $[1,0,0,0]$ and very low friction maps into another one-hot vector
$[0,0,0,1]$, then the contextual embedding of this terrain is
$[1,0,0,0,0,0,0,1]$.

We note that our contextual approach, leveraging human-generated prompts and
responses, enables the LLM to effectively bridge the gap between natural
language descriptions and actionable state information, a key aspect of our
framework’s success in recovering contextual information from unobservable
states of environments.

Figure 3: An Example Prompt for LANCAR. The prompt for LANCAR includes four
parts. The first is the _high-level_ task description for LLM. The second is
the details and intuitive examples of terrain properties of interest. The
third part is the examples given for in-context learning; the input given here
is a _low-level_ context of terrain, and the outputs are determined by
question-answering of multi-choice problems. The last part is the actual input
to LLM expecting that gets the embedding.

###  III-D Reinforcement Learning Agent

In this paper, we employ Augmented Random Search (ARS) [36], as the
reinforcement learning algorithm for the robot control agent. Both ARS and its
ancestor approach, BRS, use the finite difference approach, which approximates
the actual gradient value through derivative sampled in $2N$ directions and
updates the network parameters by perturbing policy parameters within the
range of $[-\delta,+\delta]$ to assess resulting rewards within that range,
while $\delta$ is randomly generated from a normal distribution. Compared with
BRS, ARS further improves the performance of RL policies by normalization and
using top-performing directions to update the network parameters. In addition,
ARS uses a linear policy, instead of a non-linear policy like the neural
network, to simplify the RL algorithms.

##  IV Empirical Results and Discussion

In the experiments, we aim to answer the following two questions regarding
performance and policy generalization. The first question is: Does external
contextual information improve the performance of the agent when the agent is
operating in diverse conditions? To answer this question, we designed a series
of experiments and compared our framework with alternative approaches that
include or exclude contextual information. In these experiments, each context
is a different terrain. The second question we investigate is: Is the LLM
model effective when the given input is _high-level_ , open-ended, and
ambiguous in retrieving contextual information and thereby in robot
locomotion?. To answer this question, we use _low-level_ , precise, and
organized human interpretation of contextual information in training but use
_high-level_ , vague, and unorganized contextual information from actual human
observers in evaluation.

We use GPT-4 [37] as our LLM model. As an advanced case study, we also apply
VLM to generate image captions of scenes observed by the robot to act as an
automated proxy for human language. We consider that the contextual
information generated by VLM in this circumstance imitates the actual
operation scenario of robots in the real world.

###  IV-A Environments

We use a quadruped robot locomotion simulator, spot-mini-mini [15], built in
PyBullet [38]. The mission of the robot is to move straight as far as possible
within a limited time. We set the time length of an episode to be $5,000$
steps. The observable state space is given by the touch sensors and includes
the values of joint angles, velocities, and torques for all motors. The state
is $16$–dimensional. The extra context, depending on the training method
covered in Section IV-B is also provided to the agent. The action space is the
desired joint angle for each of the fourteen joints. The reward function is
given by the robot’s traveling distance $d_{x}$ with respect to $x$, the
penalty $d_{y}$ for deviating from the $y$ axis, and its instantaneous posture
$p$. The reward function is defined as $J=d_{x}+0.03d_{y}+10p$. For the ARS
agent, the learning rate is $0.03$. The number of samples for $\delta$ is
$16$. The noise amplitude applied in the exploration is $0.05$.

Given the large number of training examples, it is impractical to manually
generate contextual information each time. Instead, we use LLM to
automatically generate contextual information from the terrains during
training as well. In the training phase, we ask the LLM to generate detailed,
_low-level_ qualitative descriptions of pre-defined properties of the terrain.
Specifically, we first randomly generate samples with parameter values
quantitatively describing properties given in Table I of the simulated
environment. We then design a prompt that provides sampled parameter values
for training environments to the LLM. We describe the value of these
properties with qualitative words: None, Very Low, Low, Medium, High and Very
High. Given the true parameter values, we ask the LLM to generate a _low-
level_ contextual description. A sample description generated during training
is: This environment has no restitution when collision, low friction, very
high stiffness level, and no damping.

TABLE I: Properties for Training Terrains Property | Value Range
---|---
Height-Field | True/False
Restitution | $[0,0.2]$
Lateral / Horizontal Friction | $[0,1]$
Rolling Friction | $[2\times 10^{4},1.6\times 10^{5}]$
Stiffness | $[0,1]$
Damping Coefficient | $[0,0.5]$

In the evaluation phase, we conduct three case study experiments with
increasing difficulty levels, i.e., increasing vagueness of the contextual
information provided to the LLM to examine the reasoning ability of our
approach. Specifically, we evaluate the following three types of contexts (in
increasing order of open-endedness):

####  IV-A1 Low-Level Context

The contextual information provided by human observers during evaluation gives
detailed qualitative descriptions of environmental properties, the same as
those given in the training phase. Three evaluation terrain descriptions we
used are provided in Table II.

TABLE II: Low-Level Contextual Information for Case Study ID | Contextual Information
---|---
A |  This environment has no restitution when collision, very high friction, and no damping.
B |  This environment has no restitution when collision, very low friction, and no damping.
C |  This environment has high restitution when collision, very high friction, and very high damping.

####  IV-A2 High-Level Context

The contextual description provided by human observers is _high-level_ , open-
ended, vague, and descriptive of the environmental conditions, rather than the
environmental properties. Three evaluation terrain descriptions we used are
provided in Table III.

TABLE III: High-Level Contextual Information for Case Study ID | Contextual Information
---|---
D |  The spot is walking on a grassland after last night’s rain, though it’s sunny now.
E |  The spot is walking on a mountain road covered by ice. It’s snowy now.
F |  The spot is walking on the beach near the sea under the sun.

####  IV-A3 VLM-Interpreted Context

Instead of a human providing the description to the LLM, we use a VLM instead.
The VLM first generates captions given an image representing the scene. The
contextual description generated by the VLM depicts the visual observations at
a _high level_. This caption is then provided to the LLM to generate the
embedding. Since the contextual information undergoes a two-stage
interpretation before generating the embedding, the distortion in
understanding the contextual information may contribute to a large deviation
between the perceived and actual contextual information, which leads to a much
more challenging scenario in evaluation. In this paper, we use BLIP [39] as
our VLM model to generate the image captions. The image inputs and the
generated captions we used are provided in Figure 4. From image captions, we
find that contextual information captured by the current version of VLM is
always distorted by irrelevant information, like misinterpreting the head of
spot robot shown in both Fig. 4(a) and 4(b) as yellow suitcase and is taken as
the subject in image captions, or emphasizing on some minor information over
terrain information in captioning, like the shadow in Fig. 4(a). All these
factors raise more challenges in contextual information interpretation in our
framework.

(a) Context G. Someone is taking a picture of a yellow suitcase in the grass.

(b) Context H. There is a yellow suitcase sitting on the sidewalk next to
another sidewalk.

Figure 4: VLM-interpreted Contextual Information for Case Study.

###  IV-B Approaches Compared

We conduct a series of experiments on our approach, LANCAR, and some baseline
approaches, to evaluate the effect of the usage and design of contextual
information embedding strategies. We evaluate the following approaches:

####  IV-B1 No-Context

The RL agent only uses environmental observation in their decision-making. No
contextual information is used. The decision does not rely on the LLM output.
It will be used as the baseline of the experiment.

####  IV-B2 Indexing

The context is encoded as a one-hot vector. The RL agent labels all terrains
encountered during training with a unique index. During evaluation, the output
of the LLM is converted into a one-hot vector which is then combined with the
environmental observation as input to the RL agent. The one-hot vector labels
the $i$-th element of the vector as one for the $i$-th terrain.

####  IV-B3 LANCAR

This is the approach we propose in this paper. The LLM generates contextual
embeddings by interpreting human observers or VLM in the way presented in
Section III-C, and the RL agent incorporates contextual embeddings with
environmental observation in their decision-making. The contextual embeddings
are represented by a vector composed of six one-hot vectors. Each one-hot
vector quantifies properties in Table I into six intervals.

###  IV-C Results

####  IV-C1 Case Study: Low-Level Contextual Description

The _low-level_ contextual descriptions provided in evaluation are given in
Table II. Context A is the normal terrain, Context B has low friction, and
Context C has high damping. In general, low friction and high damping are more
challenging for RL.

Table IV shows the evaluation results in terms of episodic reward over all
three _low-level_ contexts. We find that LANCAR outperforms the other two
approaches. The indexing approach has slightly worse performance than LANCAR
in general, but its performance is not stable across all terrains, as its
episodic reward is close to LANCAR in Context C (high damping), but much worse
in Context B (low friction). This indicates its limited adaptation ability
over variation in the terrain and contextual information.

TABLE IV:  Evaluation Results on Episodic Reward for Low-Level Context.

Context ID No-Context ($\times 10^{3}$) Indexing ($\times 10^{3}$) LANCAR
($\times 10^{3}$) A $25.661\pm 0.475$ $28.083\pm 1.056$ $\mathbf{31.886\pm
0.433}$ B $10.612\pm 1.720$ $10.551\pm 1.700$ $\mathbf{34.919\pm 0.340}$ C
$29.853\pm 0.443$ $34.210\pm 0.411$ $\mathbf{34.915\pm 0.352}$

####  IV-C2 Case Study: High-Level Contextual Description

Our second case study uses _high-level_ contextual descriptions given in Table
III. This is a harder problem than the _low-level_ description as it relies on
the reasoning ability of the LLM translator. We denote Context D as the
contextual information for Moist Grassland, Context E for Snowy Mountain Road,
and Context F for Sunny Beach. All three terrains are more difficult than
those given in the _low-level_ context case study, in terms of surface
stiffness, damping, and friction.

Table V shows the evaluation results in terms of episodic reward over all
three _high-level_ contextual descriptions. We find that LANCAR and indexing
have close performance in Context D (Moist Grassland) and Context F (Sunny
Beach), while indexing performs the worst in Context E (Snowy Mountain Road),
marking its failure in addressing low friction terrains. The No-Context
baseline, on the other hand, has a stable performance over all contexts but is
consistently worse than LANCAR.

TABLE V: Evaluation Results on Episodic Reward for High-Level Context.

Context ID No-Context ($\times 10^{3}$) Indexing ($\times 10^{3}$) LANCAR
($\times 10^{3}$) D $29.592\pm 0.367$ $\mathbf{33.737\pm 0.563}$ $33.725\pm
0.414$ E $25.726\pm 1.732$ $8.718\pm 1.988$ $\mathbf{30.332\pm 2.130}$ F
$29.821\pm 0.434$ $34.294\pm 0.397$ $\mathbf{34.917\pm 0.272}$

####  IV-C3 Case Study: VLM-Interpreted Contextual Description

The last case study uses VLM-Interpreted contextual information given in
Figure 4 by presenting images taken by robots in the real world to generate
contextual information. We design this case study to simulate circumstances
that robots may encounter in the real world where there is a no human to
provide contextual information, but instead, it must be obtained
automatically. We denote Context G as Dry Grassland, Context H as Stiff Road.
Due to the ambiguity in the terrain images acquired in the outdoor
unstructured environments, image captions generated by VLM could be greatly
distorted and noisy, which sets up a higher demand for LLM translators to
extract key information and reason with incomplete contextual information, so
we take this case study as the hardest one.

Table VI shows the evaluation results in terms of episodic reward over all
three VLM-Interpreted contexts. We find that LANCAR and indexing have close
performance in Context H (Stiff Road), but there is a clear gap between their
performance in Context G (Dry Grassland). We find that Stiff Road terrain is
more like Context A (the normal terrain) given in the _low-level_ context case
study, which is low in its difficulty level. As a result, it is acceptable for
indexing and LANCAR to have similar performance. Notably, LANCAR has a better
performance in Dry Grassland, indicating its better generalization ability
across diverse terrains.

TABLE VI: Evaluation Results on Episodic Reward for Context Interpreted by
VLM.

Context ID No-Context ($\times 10^{3}$) Indexing ($\times 10^{3}$) LANCAR
($\times 10^{3}$) G $29.913\pm 0.568$ $31.942\pm 0.828$ $\mathbf{33.112\pm
0.583}$ H $29.788\pm 0.391$ $\mathbf{34.142\pm 0.368}$ $\mathbf{34.142\pm
0.441}$

###  IV-D Discussion

Our experiments consistently demonstrated that the context-aware methods
(indexing and LANCAR) outperformed the baseline no-context method. Between the
two context-aware methods, LANCAR consistently achieved the best results due
to its adaptability to the set of diverse scenarios. The indexing method still
performs well in most of the cases, as well as LANCAR, except for the poor
performance in specific cases. However, when it fails, the indexing method
performs much worse. We suspect this is due to the LLM incorrectly classifying
the terrain which leads to the RL agent applying vastly different policy
parameters. This is not an issue with LANCAR, since we do not explicitly
identify the context. Rather, we represent the context as an embedding. As
such, even if the predicted embedding during evaluation is slightly different
than _ground truth_ , it has a marginally smaller effect on the performance of
the RL agent.

##  V Conclusion

This paper proposes a novel approach that allows human observers and VLM to
interpret the implicit contextual information in the environment and use LLM
to translate this information into contextual embeddings that could be
understood by RL agents, RL agents concatenate their own observation with
these contextual embeddings in their decision-making. Results validate the
efficacy of LANCAR in policy generalizability and adaptability across diverse
terrains over three case studies using contextual information with different
ambiguity levels. In our future work, we intend to develop an advanced
approach that incorporates a VLM and an LLM to interpret environmental
contextual information by generating more precise captions of observed images
of robots and refining the reasoning abilities in these image captions [40].
In addition, we will try to investigate a mechanism that makes robots more
adaptive when transitioning from one context to another within the same
episode. We hope this contributes to a more robust and adaptive strategy in
real-world robot locomotion tasks.

## References

  * [1] H. Nguyen and H. La, “Review of deep reinforcement learning for robot manipulation,” in 2019 Third IEEE International Conference on Robotic Computing (IRC), pp. 590–595, IEEE, 2019.
  * [2] B. Patel, K. Weerakoon, W. A. Suttle, A. Koppel, B. M. Sadler, A. S. Bedi, and D. Manocha, “Ada-nav: Adaptive trajectory-based sample efficient policy learning for robotic navigation,” arXiv preprint arXiv:2306.06192, 2023.
  * [3] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis, “Rloc: Terrain-aware legged locomotion using reinforcement learning and optimal control,” IEEE Transactions on Robotics, vol. 38, no. 5, pp. 2908–2927, 2022.
  * [4] S. Josef and A. Degani, “Deep reinforcement learning for safe local planning of a ground vehicle in unknown rough terrain,” IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 6748–6755, 2020.
  * [5] X. Xiao, Z. Wang, Z. Xu, B. Liu, G. Warnell, G. Dhamankar, A. Nair, and P. Stone, “Appl: Adaptive planner parameter learning,” Robotics and Autonomous Systems, vol. 154, p. 104132, 2022.
  * [6] T. Li, Y. Zhang, C. Zhang, Q. Zhu, J. sheng, W. Chi, C. Zhou, and L. Han, “Learning terrain-adaptive locomotion with agile behaviors by imitating animals,” 2023.
  * [7] J. Liang, Z. Wang, Y. Cao, J. Chiun, M. Zhang, and G. A. Sartoretti, “Context-aware deep reinforcement learning for autonomous robotic navigation in unknown area,” in 7th Annual Conference on Robot Learning, 2023.
  * [8] H. Karnan, E. Yang, D. Farkash, G. Warnell, J. Biswas, and P. Stone, “Self-supervised terrain representation learning from unconstrained robot experience,” in ICRA2023 Workshop on Pretraining for Robotics (PT4R), 2023.
  * [9] H. Ahn, S. Choi, N. Kim, G. Cha, and S. Oh, “Interactive text2pickup networks for natural language-based human–robot collaboration,” IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3308–3315, 2018.
  * [10] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837, 2022.
  * [11] A. Creswell, M. Shanahan, and I. Higgins, “Selection-inference: Exploiting large language models for interpretable logical reasoning,” arXiv preprint arXiv:2205.09712, 2022.
  * [12] M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant, “Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 346–361, 2021.
  * [13] M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh, “Reward design with language models,” arXiv preprint arXiv:2303.00001, 2023.
  * [14] S. Mirchandani, F. Xia, P. Florence, B. Ichter, D. Driess, M. G. Arenas, K. Rao, D. Sadigh, and A. Zeng, “Large language models as general pattern machines,” arXiv preprint arXiv:2307.04721, 2023.
  * [15] M. Rahme, I. Abraham, M. Elwin, and T. Murphey, “Spotminimini: Pybullet gym environment for gait modulation with bezier curves,” 2020.
  * [16] M. Elnoor, A. J. Sathyamoorthy, K. Weerakoon, and D. Manocha, “Pronav: Proprioceptive traversability estimation for autonomous legged robot navigation in outdoor environments,” arXiv preprint arXiv:2307.09754, 2023.
  * [17] S. Siva, M. Wigness, J. G. Rogers, L. Quang, and H. Zhang, “Nauts: Negotiation for adaptation to unstructured terrain surfaces,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1733–1740, IEEE, 2022.
  * [18] T. Guan, R. Song, Z. Ye, and L. Zhang, “Vinet: Visual and inertial-based terrain classification and adaptive navigation over unknown terrain,” in 2023 IEEE international conference on robotics and automation (ICRA), pp. 4106–4112, IEEE, 2023.
  * [19] A. Ajoudani, A. M. Zanchettin, S. Ivaldi, A. Albu-Schäffer, K. Kosuge, and O. Khatib, “Progress and prospects of the human–robot collaboration,” Autonomous Robots, vol. 42, pp. 957–975, 2018.
  * [20] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, et al., “Robots that ask for help: Uncertainty alignment for large language model planners,” arXiv preprint arXiv:2307.01928, 2023.
  * [21] Y. Tang, W. Yu, J. Tan, H. Zen, A. Faust, and T. Harada, “Saytap: Language to quadrupedal locomotion,” arXiv preprint arXiv:2306.07580, 2023.
  * [22] S. Chakraborty, K. Weerakoon, P. Poddar, P. Tokekar, A. S. Bedi, and D. Manocha, “Re-move: An adaptive policy design approach for dynamic environments via language-based feedback,” arXiv preprint arXiv:2303.07622, 2023.
  * [23] D. Shah, B. Osiński, S. Levine, et al., “Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,” in Conference on Robot Learning, pp. 492–504, PMLR, 2023.
  * [24] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  * [25] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,” International Journal of Computer Vision, vol. 130, no. 9, pp. 2337–2348, 2022.
  * [26] Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey for in-context learning,” arXiv preprint arXiv:2301.00234, 2022.
  * [27] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3674–3683, 2018.
  * [28] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al., “Rt-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817, 2022.
  * [29] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al., “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818, 2023.
  * [30] A. Bucker, L. Figueredo, S. Haddadinl, A. Kapoor, S. Ma, and R. Bonatti, “Reshaping robot trajectories using natural language commands: A study of multi-modal data alignment using transformers,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 978–984, IEEE, 2022.
  * [31] A. Bucker, L. Figueredo, S. Haddadin, A. Kapoor, S. Ma, S. Vemprala, and R. Bonatti, “Latte: Language trajectory transformer,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7287–7294, IEEE, 2023.
  * [32] O. Mees, J. Borja-Diaz, and W. Burgard, “Grounding language with visual affordances over unstructured data,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11576–11582, IEEE, 2023.
  * [33] D. Fu, X. Li, L. Wen, M. Dou, P. Cai, B. Shi, and Y. Qiao, “Drive like a human: Rethinking autonomous driving with large language models,” arXiv preprint arXiv:2307.07162, 2023.
  * [34] H. Hu and D. Sadigh, “Language instructed reinforcement learning for human-ai coordination,” arXiv preprint arXiv:2304.07297, 2023.
  * [35] D. Ghosh, J. Rahme, A. Kumar, A. Zhang, R. P. Adams, and S. Levine, “Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability,” Advances in Neural Information Processing Systems, vol. 34, pp. 25502–25515, 2021.
  * [36] H. Mania, A. Guy, and B. Recht, “Simple random search provides a competitive approach to reinforcement learning,” arXiv preprint arXiv:1803.07055, 2018.
  * [37] OpenAI, “Gpt-4 technical report,” 2023.
  * [38] E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning.” http://pybullet.org, 2016–2021.
  * [39] J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International Conference on Machine Learning, pp. 12888–12900, PMLR, 2022.
  * [40] J. Guo, J. Li, D. Li, A. M. H. Tiong, B. Li, D. Tao, and S. C. Hoi, “From images to textual prompts: Zero-shot vqa with frozen large language models,” arXiv preprint arXiv:2212.10846, 2022.