# Search Beyond Queries: Training Smaller Language Models for Web Interactions
via Reinforcement Learning

Moghis Fereidouni  University of KentuckyUSA and  A.B. Siddique  University of
KentuckyUSA

(2018)

###### Abstract.

Traditional search systems focus on query formulation for effective results
but face challenges in scenarios such as product searches where crucial
product details (size, color) remain concealed until users visit specific
product pages. This highlights the need for intelligent web navigation agents
capable of formulating queries and navigating web pages according to users’
high-level intents. In response to this need, this work introduces a Grounded
Language Agent for Intelligent Web Interactions, called $\mathsf{GLAINTEL}$.
Drawing upon advancements in language modeling and reinforcement learning,
$\mathsf{GLAINTEL}$ investigates the efficacy of transformer-based models in
enhancing the search capabilities of interactive web environments. Given the
_dynamic action space_ for each state in web navigation, $\mathsf{GLAINTEL}$
employs the Flan-T5 architecture and incorporates language modeling and value
estimation heads. This work focuses on training smaller language models as
agents across various scenarios, systematically evaluating the impact of human
demonstrations on the training process. Specifically, we investigate scenarios
where no human demonstrations are available and subsequently assess the
effective utilization of such demonstrations. We also explore unsupervised
domain adaptation for situations where demonstrations are confined to a
specific domain. Experimental evaluations across diverse setups demonstrate
the effectiveness of training agents in unsupervised settings, outperforming
in-context learning-based approaches that employ larger models with up to 540
billion parameters. Surprisingly, behavioral cloning-based methods that
straightforwardly use human demonstrations do not outperform unsupervised
learning-based methods. Additionally, combining human demonstrations with
Reinforcement Learning-based training yields results comparable to models
utilizing GPT-4. The code is available at: Anonymous GitHub Repository.

Product Search, Web Navigation, Reinforcement Learning.

††copyright: acmcopyright††journalyear: 2018††doi: XXXXXXX.XXXXXXX††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Traditional search systems have predominantly concentrated on effective query
formulation techniques to enhance the retrieval and ranking of results.
However, real-world search scenarios, such as product search on websites like
Amazon, introduce additional complexities, where vital product features, such
as available colors and sizes, remain unknown until the user navigates to the
specific product web page. Conventional search systems operate under the
assumption of immediate access to all relevant information. However, this
reliance on complete data undermines their effectiveness in producing useful
ranking of search results. Consequently, users are left with the tedious task
of manually discovering and evaluating products. To address these limitations,
there is a critical need for an intelligent web navigation agent. This agent,
driven by user intent expressed in natural language, should not only formulate
effective queries but also seamlessly navigate web pages, explore product
features, and fulfill user requirements.

While research on web navigation agents is not a novel endeavor, the majority
of existing work in the web navigation domain either deals with a limited
number of transition and action spaces or faces challenges in scaling up. Some
studies concentrate solely on single classification tasks (Pasupat et al.,
2018; Mazumder and Riva, 2020; Su et al., 2017) or interactions involving only
a restricted number of pages in each episode (Shi et al., 2017). On the other
hand, alternative approaches propose tasks with longer horizons but are
confined to following hyperlinks for web navigation (Nogueira and Cho, 2016)
or necessitate human-in-the-loop feedback due to the absence of an automated
reward function (Nakano et al., 2021).

The transformer-based Large Language Models (LLMs) ( GPT-3, BERT) have
demonstrated their proficiency in tasks such as text classification,
information extraction, and question answering (Bommasani et al., 2021; Brown
et al., 2020; Vaswani et al., 2017; Raffel et al., 2019; Radford et al.,
2019). Similarly, reinforcement learning (RL) has evolved as a powerful
paradigm for training intelligent agents to navigate complex environments
(Huang et al., 2022b; Ahn et al., 2022; Liang et al., 2023). Moreover, recent
research highlights the capabilities of agents powered by LLMs. For example,
agents utilizing GPT-4 can engage in the exploration of the virtual world in
Minecraft, acquire a diverse set of composable skills, and exhibit exceptional
proficiency in playing the game (Wang et al., 2023). The exceptional amount of
world knowledge, often derived from vast text datasets, opens up possibilities
for developing LLM-assisted intelligent web navigation agents capable of
navigating and interacting with web pages in a more human-like manner.

Figure 1. Overview of $\mathsf{GLAINTEL}$: The unsupervised learning phase is
at the core of $\mathsf{GLAINTEL}$. Our agent employs the Flan-T5 architecture
and incorporates a language modeling head to adapt to a dynamic action space.
Additionally, the values head enhances precise value estimations, ensuring
robust training via Reinforcement Learning.

Despite their remarkable capabilities, off-the-shelf pre-trained LLMs face
challenges in grounding and aligning themselves in interactive web
environments (Mahowald et al., 2023). This limitation hampers their functional
competence without additional customization, particularly in real-world
product search scenarios. For instance, effective query formulation involves
the agent operating over a huge action space. Navigating through diverse web
pages poses additional challenges, requiring strategic exploration due to the
presence of different actions on each page (i.e., dynamic action space). This
complexity prevents the straightforward utilization of an action head on top
of LLM. Moreover, the challenge extends to preserving long-term memory
capabilities, which are crucial for comparing items or backtracking during the
search process. Last but not least, employing LLMs with billion-scale
parameters, such as GPT-4, incurs substantial costs. Therefore, we focus on
training agents that leverage smaller LLMs for product search in interactive
web environments. Specifically, we address the following research questions.

  * •

_RQ1: Effectiveness of Unsupervised Learning:_ Can LLM-based agents learn to
address effective query generation and web page exploration challenges in the
context of product search with no human demonstrations?

  * •

_RQ2: Impact of Human Demonstrations:_ Can incorporating human demonstrations
facilitate LLM-based agents to achieve improved overall performance in the
product search? How to effectively leverage human demonstrations for training
agents?

  * •

_RQ3: Unsupervised Domain Adaptation:_ Can LLM-based agents generalize to new
unseen product categories where no human demonstrations are available?

To systematically answer the above questions, we introduce
$\mathsf{GLAINTEL}$, a Grounded Language Agent designed for Intelligent Web
Interactions, leveraging the WebShop environment (Yao et al., 2022) – a
simulated yet realistic e-commerce web platform featuring 1.18 million real-
world products and 12,087 crowd-sourced natural language intents. Our approach
involves utilizing smaller LLMs (i.e., Flan-T5) as agent policies to generate
actions responding to natural language intents in the interactive environment.
Given a user’s intent specifying a product requirement, $\mathsf{GLAINTEL}$
formulates queries, navigates diverse web pages, and executes various actions
to identify, customize, and purchase the desired product. This process entails
perceiving the outcomes of these actions and continually grounding and
updating its knowledge with new observations. Building upon the recent
successes of using RL to fine-tune LLMs for natural language generation tasks,
we investigate the functional grounding of LLMs through RL (Stiennon et al.,
2020; Ouyang et al., 2022; Ramamurthy et al., 2022). Our approach utilizes the
Flan-T5 architecture and employs a language modeling head to accommodate a
dynamic action space and introduce an additional value head. The user’s goal
and observation are sequentially input into the model at each step. First, we
obtain the input representation for every potential action token.
Subsequently, we compute the normalized joint probability for each action
conditioned on the user goal and observation. Following the estimation of each
action’s probability, we apply a softmax function over these probabilities and
sample an action according to this distribution. To fine-tune the agent, we
employ the Proximal Policy Optimization (PPO) algorithm (Dhariwal et al.,
2017). Figure 1 provides an overview of $\mathsf{GLAINTEL}$.

Based on our empirical study, we demonstrate that training smaller LLMs (780
million parameters) in the unsupervised setting (i.e., no human
demonstrations) can outperform in-context learning methods (Sridhar et al.,
rint) that rely on models with up to 540 billion parameters. To quantify the
impact of human supervision, we utilized 1010 human demonstrations for
training supervised learning models using behavior cloning (BC) (Pomerleau,
1989). Our findings indicate that incorporating human demonstrations through
straightforward BC does not produce superior results when compared to the
unsupervised RL-based PPO algorithm. Furthermore, our investigations reveal
that leveraging human demonstrations through BC and then further training the
agent with PPO in the unsupervised setting leads to the best results.
Remarkably, this approach achieves results comparable to the method (Ma et
al., 2023) that utilizes GPT-4. In the unsupervised domain adaptation (UDA)
experiment, we observe that incorporating human demonstrations from a single
category enables the agent to generalize to new product categories where no
human demonstrations are available. Additionally, we present experiments on a
real website, eBay, for product search and conduct a comprehensive ablation
study.

##  2\. Preliminaries

###  2.1. Problem Formulation

Given a user intent in natural language, the agent’s goal is to buy the most
appropriate product that fulfills the user’s intent. We formulate the task as
a goal-augmented Partially Observable Markov Decision Process
$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\mathcal{G},\mathcal{O},\gamma)$,
where $\mathcal{S}$ is a set of states $\mathbf{s}\in\mathcal{S}$;
$\mathcal{A}\subset\mathcal{V}^{N}$ represents action space sampled from LLM’s
vocabulary $\mathcal{V}$ of size $N$; $\mathcal{G}\subset\mathcal{V}^{N}$
denotes the goal space;
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\mapsto\mathcal{S}$ is the transition
function;
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\times\mathcal{G}\mapsto\mathbb{R}$
characterizes the goal-conditioned reward function; $\mathcal{O}$ is a set of
observations $\mathbf{o}\in\mathcal{O}$ (i.e., web page state visible to
agent); $\gamma$ is the discount factor. We employ the language model’s head
(i.e., distribution over the vocabulary) to accommodate the dynamic action
space. This formulation also facilitates directly computing the log
probabilities of each action
$\mathbf{a}_{i}=(w_{0},\cdots,w_{|\mathbf{a}_{i}|})$ sampled from a dynamic
action space given the agent’s goal $g\in\mathcal{G}$ and observation
$\mathbf{o}$.

###  2.2. Pre-trained Large Language Models

Pre-trained LLMs have proven to be highly effective in a variety of downstream
tasks due to their extensive training on large datasets. More recently
autoregressive sequence models (Brown et al., 2020) have gained popularity,
leveraging the decoder stack from the transformer architecture (Vaswani et
al., 2017). The key concept behind autoregressive models is the factorization
of any joint distribution over a sequence, denoted as $x=(x_{1},...x_{L})$, in
the following autoregressive manner:

$p(x)=\Pi_{i=1}^{L}p(x_{i}|x_{<i}).$

Using this factorization, the process of estimating the density $p(x)$ is
simplified to the learning of individual conditional factors
$p(x_{i}|x_{<i})$, which can be easily parameterized using a transformer
model. The associated objective function, denoted as
$\mathcal{L}_{\mathrm{LM}}(p)$, can be expressed as:

$\mathcal{L}_{\mathrm{LM}}(p)=\mathbb{E}_{x\sim\mathcal{D}}\left[\sum_{i=1}^{L}-\log
p(x_{i}|x_{<i})\right].$

In this work, we employ instruction-finetuned LLM, specifically the Flan-T5
(Chung et al., 2022). In the following, we provide an overview of the T5 model
and outline the instruction-finetuning process.

T5 model. The T5 model (Raffel et al., 2019) follows the encoder-decoder
framework, aligning with the original transformer architecture (Vaswani et
al., 2017). In the self-supervised pre-training of the T5 model, the objective
influenced by masked language modeling and word dropout techniques (Bowman et
al., 2015) is adopted. This objective entails masking 15 percent of the input
tokens, where consecutive spans of eliminated tokens are replaced with a
sentinel token. The model is then trained to predict the original tokens that
were replaced with these sentinel tokens.

Instruction Finetuning. Instruction finetuning is a technique designed to
customize LLMs, enabling them to proficiently execute specific tasks guided by
explicit natural language instructions (Chung et al., 2022). The instruction
finetuned variant of T5, called Flan-T5, underwent finetuning using a dataset
that encompassed a total of 1836 diverse tasks. This dataset was curated by
consolidating information from previous studies, incorporating contributions
from T0-SF (Sanh et al., 2022), Muffin (Wei et al., 2022), NIV2 (Wang et al.,
2022), and chain of thought (CoT). Flan-T5 has shown state-of-the-art
performance across a range of challenging benchmark such as MMLU (Hendrycks et
al., 2021), BBH (Srivastava et al., 2023), TyDiQA (Clark et al., 2020), and
MGSM (Shi et al., 2022).

In this work, we employ Flan-T5 as the backbone model and further customize it
for our task. Specifically, we use the publicly released checkpoints of the
Flan-T5 model 111Checkpoints: https://github.com/google-
research/t5x/blob/main/docs/models.md#flan-t5-checkpoints.

###  2.3. Reinforcement Learning

Reinforcement learning is an extensively studied area in the domain of
unsupervised learning. RL algorithms, such as PPO, have been shown to empower
an agent to learn effective decision-making skills in a defined environment,
ultimately resulting in the desired outcomes. To optimize such an agent,
methods utilizing policy gradients commonly entail the computation of a
gradient estimator, which is subsequently incorporated into a stochastic
gradient ascent algorithm. A common optimization objective for enhancing the
policy $\pi$ involves maximizing the expected reward $r\in\mathbb{R}$ over
selected action $\mathbf{a}$ given current state $\mathbf{s}$. The PPO
algorithm introduces a clipped surrogate objective along with a penalty on the
KL divergence. Unlike trust region policy optimization algorithms (Schulman et
al., 2015), where the Kullback–Leibler (KL) divergence is imposed as a hard
constraint, PPO modifies the objective function using a penalty term
associated with KL divergence. The policy update in PPO, at step $t$, is given
by:

$\theta_{t+1}=\arg\max_{\theta}\mathbb{E}_{\mathbf{s},\mathbf{a}\sim\pi_{\theta_{t}}}[\mathcal{L}(\mathbf{s},\mathbf{a},\theta_{t},\theta)]$

where $\mathbf{s}$ represents the state and $\mathbf{s}$ denotes the action.

In our work, we utilize the PPO algorithm (Dhariwal et al., 2017) for updating
the policy of the agent. This algorithm has demonstrated scalability,
particularly in the context of LLMs, as well as efficiency in handling data
and robustness without necessitating excessive hyperparameter tuning
(Bellemare et al., 2013).

##  3\. Proposed Agent: $\mathsf{GLAINTEL}$

Our backbone model is Flan-T5, serving as the core architecture, with the
integration of the language modeling head and value head on top of the model.
Our proposed agent, $\mathsf{GLAINTEL}$, is adaptable to training across
various setups: (_i_) no human demonstrations are available for supervision,
(_ii_) limited human demonstrations in a single domain are available, and
(_iii_) human demonstrations are accessible. Furthermore, our objective
extends to quantifying the impact of human demonstrations and exploring
effective strategies to leverage them for enhancing overall performance. In
the following, we detail the specifics of the training and inference phases.
The inclusion or exclusion of these phases is contingent upon the particular
research question under consideration.

###  3.1. Optional Phase One: Supervised Training

The human demonstrations can serve as mappings from states to actions,
facilitating the process of supervised learning. Techniques such as imitation
learning or behavioral cloning (BC) (Pomerleau, 1989) can be employed to fine-
tune the policy $\pi$ by minimizing the following loss over a dataset
$\mathcal{D}$ comprising human demonstrations:

$\mathcal{L}(\pi)=\mathbb{E}_{(s,a)\sim\mathcal{D}}[-\log\pi(a|s)].$

The above formulation can be adapted to incorporate the interaction history
$\pi(\mathbf{a}_{t}|\mathbf{s}_{t},\tau_{<t})$, where $\tau_{<t}$ refers to
the interaction trajectory leading up to time $t$. Subsequently, this
formulation readily extends to utilize LLMs, such as Flan-T5, to learn an
optimal policy where the encoder encodes the history of observations
$(\mathbf{s}_{t},\tau_{<t})$ and the decoder generates the next action
$\mathbf{a}_{t}$ as:

$\mathcal{L}_{\text{LLM}}(\pi)=\mathbb{E}_{\tau\sim\mathcal{D}}[\sum_{t=0}^{L}-\log\pi(\mathbf{a}_{t}|\tau_{<t},\mathbf{s}_{t})].$

Building upon the recent works in return-conditioned supervised learning
(Brandfonbrener et al., 2022; Paster et al., 2022; Yang et al., 2022), we
introduce an additional conditioning variable denoted as $g\in\mathcal{G}$
(representing the agent’s goal). This variable captures overall trajectory-
level information, to steer the model’s attention toward the overall goal.
Moreover, in practical implementation, we depend on observations denoted as
$\mathbf{o}$ (representing the web page visible to the agent) instead of the
actual state $\mathbf{s}$. Our finalized formulation for this phase can be
expressed as follows:

$\mathcal{L}_{\mathrm{LLM}}(\pi)=\mathbb{E}_{\tau\sim\mathcal{D}}[\sum_{t=0}^{L}-\log\pi(\mathbf{a}_{t}|\tau_{<t},\mathbf{o}_{t},g)].$

Our training in this phase leverages the human demonstrations dataset
developed as part of the WebShop environment (Yao et al., 2022), which is
publicly accessible 222Human demonstrations dataset:
https://github.com/princeton-nlp/WebShop/tree/master/baseline_models/data.
Specifically, the loss is computed based on the predicted action (’click [Buy
Now]’) and the corresponding ground truth action (’click [Back to Search]’)
associated with a given user goal.

The training of this phase can be skipped or chosen based on the availability
and feasibility of acquiring human demonstrations, which can be resource-
intensive. In our approach to address RQ1 (Effectiveness of Unsupervised
Learning), we intentionally omit this phase. We limit the human demonstration
data to a single category, while we focus on RQ3 (Unsupervised Domain
Adaptation). Conversely, to investigate RQ2 (Impact of Human Demonstrations),
we utilize the entirety of the available training data for the supervised
training phase.

###  3.2. Phase Two: Unsupervised Training

The unsupervised learning phase, which forms the core of the proposed agent
$\mathsf{GLAINTEL}$, operates without any human demonstrations. This phase is
designed to autonomously learn and adapt without relying on expert-guided
examples. The objective of the agent is to learn a policy
$\pi:\mathcal{O}\times\mathcal{G}\mapsto\mathbb{P}(\mathcal{A})$ that
optimizes the expected discounted cumulative rewards for a given goal $g$. In
this work, we leverage PPO algorithm for training, which simultaneously learns
a policy $\hat{\pi}$ and a value function
$\hat{V}:\mathcal{O}\times\mathcal{G}\mapsto\mathbb{R}$ approximating to the
true value
$V(\mathbf{s},g)=\mathbb{E}_{\mathbf{a}\sim\hat{\pi}(\mathcal{O}(\mathbf{s}),g)}\bigl{[}\mathcal{R}(\mathbf{s},\mathbf{a},g)+\gamma
V(\mathcal{T}(\mathbf{s},\mathbf{a}),g)\bigr{]}$. We can calculate the
probability of each action $\mathbf{a}_{i}\in\mathcal{A}$ using the likelihood
computed by the LLM, expressed as:
$\hat{\pi}(\mathbf{a}_{i}|\mathbf{o},g)=P(\mathbf{a}_{i}|g)$. That is, the
likelihood of choosing each action is calculated based on the probability
distributions associated with the tokens that make up the action. This
approach ties the action probabilities directly to the distributions of the
individual tokens involved in constructing the action. To approximate the
value $V$, we incorporate a multilayer perception (MLP) with a single output
on top of the last layer of the LLM. Specifically, we employ the language
modeling head (i.e., distribution over the vocabulary) to directly compute the
log probabilities of each action
$\mathbf{a}_{i}=\\{w_{0},\cdots,w_{|\mathbf{a}_{i}|}\\}$ from the dynamic
action space given the agent’s goal $g\in\mathcal{G}$ and observation
$\mathbf{o}_{t}$ at time $t$ as follows:

$P(\mathbf{a}_{i})=\frac{1}{|a_{i}|}\sum^{|a_{i}|}_{k=0}\log P_{\texttt{LM-
head}}(w_{k}|g,\mathbf{o}_{t},w_{<k}).$

Subsequently, employing the softmax operation, we calculate a probability
distribution over the action space $\mathcal{A}$ as follows:

$P(\mathbf{a}_{i}|g)=\frac{e^{P(\mathbf{a}_{i})}}{\sum_{\mathbf{a}_{k}\in\mathcal{A}}e^{P(\mathbf{a}_{k})}}.$

It is crucial to highlight that actions comprise multiple tokens. Furthermore,
the range of possible actions can vary substantially depending on the current
state, introducing complexity in the actions that the agent may undertake.
This phase is mandatory regardless of whether training is conducted in the
optional first phase.

###  3.3. Phase Three: Inference

In the inference phase, various decoding techniques for action selection can
be employed, each carrying its own set of advantages and disadvantages. Given
the well-established nature of these techniques in the literature, we omit
details for brevity, focusing on key insights only. Greedy decoding, chosen
for action selection, has a drawback as it tends to trap the agent in loops,
ultimately resulting in suboptimal overall performance. Conversely, opting for
top-p sampling can yield a higher success rate, as it provides a theoretical
tradeoff between sampling and greedy decoding. However, the process of
determining the optimal values for p can be time-intensive.

To address these issues, we turn to the Epsilon-Greedy algorithm for action
selection during inference. In particular, at a step $t$, the greedy will
choose the action with the highest probability, while the epsilon will sample
based on the probability distribution across the action space. This method
excels in achieving both a higher success rate and an enhanced overall
performance, all while avoiding the issue of getting stuck in loops. It is
worth noting that a judiciously chosen, small value for epsilon has been
employed in our work, eliminating the need for an exhaustive search.

###  3.4. Implementation Details

Our implementation operates on a client-server architecture, with the training
scripts serving as the client and communicating requests to LLM servers.
Specifically, a master server manages these requests, distributing them across
multiple LLM servers. Once each LLM server completes its computations, the
master server consolidates the results and sends them back to the training
script. Furthermore, we use vertical model parallelism, enabling the
parallelization of individual LLMs across multiple GPUs. In our experiments,
we utilized a single LLM, Flan-T5-Large, with 780 million parameters. This
model was parallelized across 4 Nvidia V100 32GB GPUs. We incorporated the
last two observations as the model input and an encoder context size of 1024.

To train the agent using the human demonstrations, we used the Trainer library
provided by Huggingface 333Trainer:
https://huggingface.co/docs/transformers/main_classes/trainer. We employed the
Adam optimizer, and for the remaining hyperparameter values, refer to Table 1.
In our unsupervised learning phase, we leverage the PPO algorithm, and the
complete values of hyperparameters can be found in Table 2.

Table 1. Supervised Learning Hyperparameters. Hyperparameter | Value
---|---
Number of Epochs | 10
Learning Rate | $2\times 10^{-5}$
Warmup Steps | 100
Weight Decay | 0.01
Batch Size | 32
Adam Optimizer Epsilon | $10^{-8}$
Adam Optimizer $\beta_{1}$ | 0.9
Adam Optimizer $\beta_{2}$ | 0.999
Table 2. Unsupervised Learning Hyperparameters. Hyperparameter | Value
---|---
# of collected transitions between two updates | 640 (16 × 40)
Number of epochs per update | 1
Batch Size | 8
Learning Rate | $10^{-6}$
Adam Optimizer Epsilon | $10^{-5}$
Adam Optimizer $\beta_{1}$ | 0.9
Adam Optimizer $\beta_{2}$ | 0.999
Discount Factor | 0.99
Lambda for Generalized Advantage Estimate | 0.99
Entropy Loss Coefficient | 0.01
Value Loss Coefficient | 0.5
Maximum Gradient Norm | 0.5
Clipping Epsilon | 0.2

##  4\. Experimental Setup

###  4.1. WebShop Environment and Demonstrations

Webshop (Yao et al., 2022) is a simulated web-based interactive environment
with 1.18 million real-world products and 12,087 crowd-sourced text
instructions. The goal of the agent is to buy a product with specific
attributes and options given natural language instruction. The environment
contains 5 different categories: (_i_) Garden, (_ii_) Fashion, (_iii_) Beauty,
(_iv_) Electronics, and (_v_) Grocery. These categories exhibit significant
dissimilarities, particularly in terms of possessing nearly exclusive
attributes. For instance, as illustrated in Table 3, a substantial 95.9% of
Fashion’s attributes are unique to its category.

Additionally, we also used a human demonstration dataset in the optional phase
one training. This dataset is created by asking humans to demonstrate how they
would query a product and then take different steps in the Webshop environment
to buy a product with desired options and attributes. The human demonstration
dataset encompasses a total of 1010 distinct trajectories, distributed across
categories as follows: 211 trajectories in the Garden category, 217 in
Fashion, 224 in Beauty, 169 in Electronics, and 189 in Grocery.

Table 3. Detail about Webshop Environment. Category | # Attributes | % Unique Attributes
---|---|---
Beauty | 143 | 85.3%
Garden | 133 | 87.2%
Grocery | 117 | 92.3%
Electronics | 141 | 91.4%
Fashion | 173 | 95.9%

###  4.2. Evaluation Methodology

Reward. In alignment with the Webshop (Yao et al., 2022) environment, we
assign a final reward $r\in[0,1]$ to the agent after it completes a purchase
at the concluding step of an episode. Specifically, the reward is determined
by how closely the purchased product matches the specific attributes and
options mentioned in the user instructions. The reward is outlined as follows:

$r=r_{\text{type}}\cdot\frac{|U_{\text{att}}\cap
Y_{\text{att}}|+|U_{\text{opt}}\cap Y_{\text{opt}}|+1[y_{\text{price}}\leq
u_{\text{price}}]}{|U_{\text{att}}|+|U_{\text{opt}}|+1}$

The reward incorporates three main components: $U_{\text{att}}$,
$U_{\text{opt}}$, and $u_{\text{price}}$, representing a set of attributes, a
set of options, and the price set down in the user’s instruction.
Correspondingly, $Y_{\text{att}}$, $Y_{\text{opt}}$, and $y_{\text{price}}$
denote the set of attributes, the set of options, and the actual price of the
purchased product by the agent. Additionally, $r_{\text{type}}$ functions as a
text-matching heuristic, assigning a lower reward when the purchased product
and the targeted product in the user instruction have similar attributes and
options while being different types of products. Interested readers are
referred to the WebShop environment (Yao et al., 2022).

Evaluation Metrics. Two evaluation metrics are computed using the rewards
obtained from the episodes: (_i_) the Score and (_ii_) the Success Rate. The
Score metric represents the average reward across all test episodes multiplied
by 100, while the Success rate metric measures the percentage of test episodes
in which the full reward (1 out of 1) was attained. Given that our inference
step incorporates sampling, the reported Score and Success Rate metrics are
averaged by running the model four times.

Table 4. Results from methods in the WebShop environment that do not rely on human demonstration data. Approach | Name | Model | Parameters | Score | Success Rate
---|---|---|---|---|---
Zero Shot | Random | - | - | 33.74 | 6.80
Rulews 1 | - | - | 45.60 | 9.60
ZSL-Flan-T5 | Flan-T5-large | 780 Million | 41.10 | 10.30
In-context Learning | Act 2 | PaLM | 540 Billion | 62.30 | 30.10
ASH 4 | CODE-DAVINCI-002 | N/A | 56.70 | 30.20
ReAct 2 | PaLM | 540 Billion | 66.60 | 40.00
| AskAct 3 | Llama-2 | 70 Billion | 68.60 | 42.20
RL-based Method | PGws 1 | BART, BERT | 516 Million | 52.50 | 11.20
DRRN | GRU | 1.2 Million | 46.87 | 11.73
RNNws 1 | GRU | 5 Million | 55.20 | 17.60
PPO500K (Ours) | Flan-T5-large | 780 Million | 68.19 | 38.55
PPO1M (Ours) | Flan-T5-large | 780 Million | 72.13 | 42.55
Human | - | - | - | 82.10 | 59.60
Results are taken from published research: 1 from (Yao et al., 2022), 2 from
(Yao et al., 2023), 3 from (Kim et al., 2023), and 4 from (Sridhar et al.,
rint).

###  4.3. Competing Methods

:

WebShop Baselines (Yao et al., 2022): We consider the following baselines from
the WebShop paper: (_i_) rule-based (called Rulews), (_ii_) behavioral
cloning-based supervised learning (called BCws), (_iii_) two reinforcement
learning-based models, one employing a transformer text encoder (called PGws)
and the other using RNN (called RNNws), and (_iv_) hybrid method (called BC +
PG).

:

DRRN (He et al., 2016): The deep reinforcement relevance network (DRRN) is a
classic RL baseline. It employs distinct neural networks to transform state
and action into embedding vectors. Subsequently, an interaction function
(inner product) is utilized to compute the Q-function value for the given
state-action pair.

:

Act and ReAct (Yao et al., 2023): The ReAct method is an in-context learning
approach that combines verbal reasoning and action execution using LLMs to
address diverse general tasks. In the context of the WebShop environment,
ReAct prompts introduce a layer of reasoning in each step, which helps the
agent in making decisions on what to explore, when to make a purchase, and
which options to select.

:

WebGUM (Furuta et al., 2023): The WebGUM (Web navigation via Grounded
Understanding Models) is an instruction-finetuned model that undergoes
additional training on human demonstrations for web navigation, aiming to
transfer its performance into multimodal settings.

:

ASH Prompting (Sridhar et al., rint): The Actor-Summarizer-Hierarchical (ASH)
method comprises two primary components: (_i_) Summarizer and (_ii_) Actor.
The Summarize component provides a concise observation representation by
retaining only the pertinent information while discarding extraneous content.
Subsequently, the Actor component utilizes this condensed observation to
generate the next action.

:

PIX2ACT (Shaw et al., 2023): The PIX2ACT builds upon the Pix2Struct model
(Lindenberger et al., 2021), utilizing an image transformer encoder along with
a text transformer decoder. This method operates uniquely by taking a
screenshot of the environment as its input and offers a low-level action space
encompassing mouse and keyboard actions.

:

LASER (Ma et al., 2023): The LASER (LLM Agent with State-Space Exploration), a
GPT-4-based method, transforms an interactive decision-making task into state
space exploration. This method categorizes all possible observations an agent
might encounter during the task into a finite set of predetermined states. The
agent then moves between these states by executing a defined set of actions
specific to each state.

:

Prospector (Kim et al., 2023): The Prospector employs two distinct approaches:
the AskAct method and the Trajectory Ranking (TR) method. In the AskAct
method, self-asking steps are incorporated into the few-shot demonstrations to
accurately extract appropriate actions from the LLMs. On the other hand, in
the TR method, the LLMs generate diverse trajectories, after which the most
rewarding trajectory is selected using a reward prediction model.

##  5\. Results

###  5.1. Quantitative Analysis

Table 5. Results from methods in the WebShop environment that use human demonstration data. Approach | Name | Model | Parameters | Score | Success Rate
---|---|---|---|---|---
Behavioral Cloning | PIX2ACT 3 | Pix2Struct | 282 Million | 46.70 | NR
BCws 1 | BART, BERT | 516 Million | 59.90 | 29.10
BCour | Flan-T5-large | 780 Million | 66.56 | 37.05
WebGUM 2 | Flan-T5-XL | 3 Billion | 67.50 | 45.00
Hybrid Methods | BC + PG 1 | BART, BERT | 516 Million | 62.40 | 28.70
AskAct + TR (Prospector) 4 | Llama-2, FLAN-T5-XL | 70 + 3 Billion | 70.20 | 43.60
BC + PPO500K ($\mathsf{GLAINTEL}$500K) | Flan-T5-large | 780 Million | 74.60 | 46.95
| BC + PPO1M ($\mathsf{GLAINTEL}$1M) | Flan-T5-large | 780 Million | 76.87 | 49.60
Human | - | - | - | 82.10 | 59.60
Results are taken from published research: 1 from (Yao et al., 2022), 2 from
(Furuta et al., 2023), 3 from (Shaw et al., 2023), and 4 from (Kim et al.,
2023).
Table 6. Comparison of the Best Models. Approach | Name | Model | Parameters | Score | Success Rate
---|---|---|---|---|---
RL-based Method | PPO1M | Flan-T5-large | 780 Million | 72.12 | 42.55
Hybrid Method | BC + PPO1M ($\mathsf{GLAINTEL}$1M) | Flan-T5-large | 780 Million | 76.87 | 49.6
Unsupervised Domain Adaptation | UDA1M | Flan-T5-large | 780 Million | 74.69 | 46.42
State-Space Exploration | LASER(Ma et al., 2023) | GPT-4-0613 | N/A | 75.6 | 50.0
Human | - | - | - | 82.1 | 59.6
Table 7. The results of unsupervised domain adaptation and single domain methods in the WebShop environment. Approach $\xrightarrow[]{}$ | Single Domain Behavioral Cloning | Unsupervised Domain Adaptation
---|---|---
PPO Adaptation Configs $\xrightarrow[]{}$ | No PPO (SDBC) | PPO for 500k steps (UDA500K) | PPO for 1M steps (UDA1M)
Single-domain Supervision $\downarrow$ | Score | Success Rate | Score | Success Rate | Score | Success Rate
Fine-tuned on Beauty | 64.23 | 31.41 | 73.99 | 45.80 | 74.49 | 45.85
Fine-tuned on Garden | 64.79 | 34.76 | 73.97 | 44.7 | 75.27 | 47.5
Fine-tuned on Grocery | 61.80 | 27.50 | 73.83 | 45.75 | 74.91 | 47.60
Fine-tuned on Electronics | 62.03 | 30.97 | 73.46 | 45.25 | 74.41 | 44.5
Fine-tuned on Fashion | 62.54 | 31.60 | 73.37 | 44.45 | 74.36 | 46.65
Average $\xrightarrow[]{}$ | 63.07 | 31.24 | 73.72 | 45.19 | 74.68 | 46.42

RQ1: Effectiveness of Unsupervised Learning. In Table 4, we systematically
evaluate the performance of various methods that do not use human
demonstrations for training. Starting with RL-based models, the PPO-trained
model with 1 million steps (PPO1M) emerges as the top performer, achieving a
commendable score of 72.13 and a success rate of 42.55. Significantly, these
results surpass those obtained by alternative RL-based approaches, namely
PG${ws}$, DRRN, and RNNws, underscoring the superior efficacy of the PPO
methodology. In-context learning methods, the AskAct stands out with the most
impressive results. However, even the best-performing AskAct, 70 billion
parameters, fails to outperform a smaller model fine-tuned in an unsupervised
setting with PPO (PPO1M). Specifically, in terms of percentage improvements,
the PPO-trained model with 1 million steps outperforms the AskAct by 5.18% on
the score metric and approximately 0.83% on the success rate metric. This
pattern persists when comparing the ReAct, with 540 billion parameters, to the
PPO1M model. This observation suggests that the fine-tuning of small models
using RL can yield superior performance compared to in-context learning
methods relying on models with billions of parameters. In addition to RL-based
and in-context learning methods, Table 4 includes zero-shot learning methods,
which exhibit the poorest performance. We also present zero-shot Flan-T5 (ZSL-
Flan-T5) to quantify the role of unsupervised training.

RQ2: Impact of Human Demonstrations. Table 5 presents the results of various
methods incorporating human demonstration. In the supervised setting, WebGum
emerges as the top performer, utilizing the behavioral cloning technique and
leveraging the Flan-T5-XL model with 3 billion parameters. It achieves a score
of 67.5 and a success rate of 45.0. We also present the results of the
Flan-T5-large model (BCour), fine-tuned with 780 million parameters. Both
these models outperform the PIX2ACT and BC${ws}$ models, which utilize BART
and BERT architectures. This notable superiority underscores the effectiveness
of language models fine-tuned with instructions. Turning to hybrid methods,
$\mathsf{GLAINTEL}$500K, $\mathsf{GLAINTEL}$1M, and BC + PG models initially
undergo refinement through human demonstrations in a supervised setting,
followed by additional fine-tuning in an unsupervised setting using RL. In
contrast, Prospector employs the AskAct method (in-context learning) and a
reward prediction model, choosing the most rewarding trajectory through
supervised learning. Among these approaches, our method $\mathsf{GLAINTEL}$1M
stands out, further fine-tuned using PPO over 1 million steps, achieving
remarkable performance. It attains an exceptional Score of 76.87, coupled with
a Success Rate of 49.6. Notably, our approach surpasses all other hybrid and
behavioral cloning methods in both score and success rate metrics.

_Effective Utilization of Human Demonstrations._ In comparing two variants of
the Flan-T5-large model, as presented in Table 5 and Table 4, we focused on
one fine-tuned in a supervised setting with human demonstrations (referred to
as BCour in Table 5) and another fine-tuned exclusively with PPO for 1 million
steps in an unsupervised setting (referred to as PPO1M in Table 4).
Surprisingly, the unsupervised model (PPO1M) demonstrated an 8.36% higher
score and a 14.84% higher success rate compared to the supervised model. This
outcome suggests that relying on human demonstrations does not always lead to
superior results. Moreover, when the supervised model is subjected to further
training with PPO, it produces the best results.

_Comparison between the Best Models._ We present the results from the best
models in Table 6. Notably, our approach $\mathsf{GLAINTEL}$1M, undergoing
additional refinement with PPO over 1 million steps following behavioral
cloning, has attained a state-of-the-art score (i.e., 76.87) surpassing all
other models. Surprisingly, our model, based on the comparatively modest
Flan-T5-Large with 780 million parameters, has outperformed the LASER method,
which relies on the latest GPT-4 model, on Score metric, while achieving a
comparable performance on Success Rate (49.6 vs 50.0). These findings strongly
suggest that a model, when further fine-tuned with PPO after supervised
training, can deliver superior results, even with a relatively smaller model
size.

Table 8. Results of Zero-shot simulation-to-real experiment on eBay. Approach | Name | Model | Parameters | Score | Success Rate
---|---|---|---|---|---
Hybrid Method | BC + PG | BART, BERT | 516 Million | 59.25 | 24
Hybrid Method | BC + PPO1M ($\mathsf{GLAINTEL}$1M) | Flan-T5-large | 780 Million | 78.35 | 53
State-Space Exploration | LASER | GPT-4-0613 | N/A | 83.55 | 56

RQ3: Unsupervised Domain Adaptation. The Single Domain Behavioral Cloning
(SDBC) approach involves fine-tuning a Flan-T5-large model in a supervised
setting using demonstrations specific to a particular domain (Beauty).
Subsequently, without undergoing additional refinement for other domains, the
model is directly tested using the WebShop environment encompassing all
domains. In contrast, Unsupervised Domain Adaptation (UDA) takes the
Flan-T5-large model fine-tuned in a single domain and further refines it
across all domains using PPO in the unsupervised setting. Table 7 presents two
versions of UDA: UDA500K and UDA1M, both refined through PPO for 500,000 and 1
million steps, respectively. As depicted in Table 7, both UDA methods exhibit
superior performance in terms of Score and Success Rate metrics when compared
to the corresponding metrics of SDBC. This superiority is evident not only on
a domain-specific basis but also when considering the average performance
across domains. In particular, concerning the average performance across
domains, UDA1M surpasses SDBC by 18.4% in the Score and 48.5% in the Success
Rate metrics. This emphasizes the crucial role of unsupervised PPO refinement
and its impact on enhancing overall performance.

_Effective Utilization of Human Demonstrations in a Single Domain._ To compare
the UDA results with RL-based ones, we can refer to Table 7 and Table 4. Upon
examination, it becomes evident that the UDA500K model outperforms the PPO500K
in terms of both Score and Success Rate metrics. Similarly, UDA1M surpassed
PPO1M, displaying the superior Score and Success Rate metrics. Specifically,
the UDA1M model achieves a 3.5% higher Score and a 9.09% higher Success Rate
compared to the PPO1M model. Likewise, the UDA500K model attained an 8.1%
higher Score and a 17.2% higher Success Rate compared to the PPO500K model.
These findings indicate that incorporating single-domain human demonstration
supervision significantly enhances the model’s capacity for more effective
fine-tuning during unsupervised training with PPO. This approach outperforms
models that lack any supervised training, highlighting the value of leveraging
human demonstrations in the adaptation process.

Learning Curve during PPO Fine-tuning. In Figure 2, the learning curves of
Score and Success Rate metrics during PPO fine-tuning are illustrated for
various methodologies: the UDA, the hybrid ($\mathsf{GLAINTEL}$) (BC + PPO),
and the RL-based PPO. Both the hybrid method and the unsupervised domain
adaptation method demonstrate higher sample efficiency compared to the
unsupervised method. This aligns with expectations, considering that both the
hybrid method and the unsupervised domain adaptation method underwent some
level of supervised training before RL fine-tuning – a contrast to the RL-
based unsupervised method, which did not.

Figure 2. Learning curves of different methodologies: Unsupervised Domain
Adaptation (UDA), Hybrid (BC + PPO) ($\mathsf{GLAINTEL}$), and RL-based
Unsupervised (PPO).

###  5.2. Results on Real Website: eBay

We also conduct limited evaluations on a real website: eBay. For this
experiment, we assess the performance of three methods: (_i_) our best model
($\mathsf{GLAINTEL}$1M), (_ii_) a state-of-the-art GPT-4-based method LASER,
and (_iii_) the WebShop baseline (BC + PG). It is important to highlight that
we did not perform any fine-tuning. Following (Yao et al., 2022), we randomly
sampled 100 user instructions and inputted them into these three methods. As
presented in Table 8, our method $\mathsf{GLAINTEL}$1M significantly
outperformed the WebShop baseline (BC + PG) by 32.23% in the Score metric and
by 120.83% in the Success Rate metric. Moreover, although LASER, utilizing
GPT-4, has slightly higher Score and Success Rate metrics compared to our
model $\mathsf{GLAINTEL}$1M, we are confident that $\mathsf{GLAINTEL}$1M can
achieve comparable or even superior results by enabling of unsupervised
training using PPO. Additionally, it is worth noting that our approach
utilizes a 780 million parameter model, which is significantly smaller than
GPT-4, not to mention the costs associated with using GPT-4.

###  5.3. Ablation Study

Figure 3. Hybrid setting: BC + PPO: Flan-T5 is more sample efficient than T5 model. Table 9. Ablation Study (T5 vs Flan-T5) Configs $\xrightarrow[]{}$ | SL (one cat) + PPO (500k) | PPO (500k)
---|---|---
Model $\downarrow$ | Score | Success Rate | Score | Success Rate
Flan-T5 | 73.72 | 45.19 | 68.18 | 38.55
T5 | 71.85 | 43.10 | 52.07 | 25.35

Flan-T5 vs T5. The results, as presented in Table 9, demonstrate that adopting
the Flan-T5-Large model instead of T5-Large leads to a substantial improvement
of 30.93% in the Score and a remarkable 52.07% increase in the Success Rate in
the unsupervised setting (PPO). Furthermore, in the domain adaptation
scenario, we observed a 2.60% Score enhancement and a 4.85% improvement in the
Success Rate. Moreover, Figure 3 demonstrates that employing the Flan-T5 model
over the T5 model results in better sample efficiency. Specifically, both
Score and Success Rate metrics exhibit faster growth during PPO fine-tuning in
the Flan-T5 model compared to the T5 model. This outcome was anticipated as
the Flan-T5 model enjoys the advantage of being fine-tuned on user
instructions, a benefit not shared by the T5 model.

Table 10. Ablation Study (2 observations vs 1 observation) Configs $\xrightarrow[]{}$ | SL (all cats) | SL + PPO (500k)
---|---|---
| Score | Success Rate | Score | Success Rate
2 observations | 66.55 | 37.05 | 74.60 | 46.95
1 observation | 60.20 | 27.20 | 65.29 | 33.60

2 Observations vs 1 Observation. As demonstrated in Table 10, combining the
present observation state with the preceding observation state to create a
historical context and subsequently providing the model with this new
observation containing both leads to a notable 10.54% boost in the Score and a
remarkable 36.21% improvement in Success Rate in the supervised setting. This
substantial enhancement is equally observable in the context of the hybrid
method (SL + PPO) where the supervised training is coupled with unsupervised
training (PPO), resulting in a significant 14.26% increase in the Score and an
impressive 39.73% improvement in Success Rate. Additionally, during the
training, we noticed that employing a historical context (having the current
and last observations) as input enhances the sample efficiency for the agent
compared to using just one observation. Specifically, Score and Success Rate
metrics show a swifter increase with fewer steps when leveraging two
observations (historical context) as input, while the progression is notably
slower when utilizing only a single (or current) observation.

Comparison of Decoding Methods. In Table 11, we compare the performance of
four different decoding methods: (_i_) Epsioln-Greedy algorithm (with epsilon
value of 0.2), (_ii_) Sampling with top_p (with top_p = 0.8 and top_k =
0.0),(_iii_) Sampling with no top_p and no top_k, and (_iv_) Argmax. These
results are determined by averaging the results achieved from models trained
with different techniques and settings, including RL and UDA, among others.
These results show that, on average, the Epsilon-Greedy algorithm consistently
attains the best results during inference, with a Score of 68.23 and a Success
Rate of 39.29. Following closely, the nucleus sampling (top_p) method has
lower Scores and Success Rates of 66.25 and 37.32, respectively. In the third
position, traditional sampling produces a score of 65.92 and a Success Rate of
36.41. The worst outcomes are associated with the Argmax method, primarily
since Argmax frequently causes the web agent to become stuck in a loop. In
simpler terms, the web agent ends up repeatedly navigating back and forth
between web pages.

Table 11. Ablation Study (Decoding Methods) Comparison | Score | Success Rate
---|---|---
Epsilon-Greedy algorithm | 68.23 | 39.29
Sampling with top_p | 66.25 | 37.32
Sampling | 65.92 | 36.41
Argmax | 57.92 | 35.59

##  6\. Related Work

Fine-tuning LMs with RL and Human Feedback. Fine-tuning LLMs with human
feedback and reinforcement learning has been studied extensively. (Nakano et
al., 2021) developed the WebGPT by fine-tuning the GPT-3 model using behavior
cloning and rejection sampling. Moreover, InstructGPT (Ouyang et al., 2022)
was developed using the three-step approach: supervised fine-tuning, reward
model training, and reinforcement learning via PPO with the help of the
trained reward model. Additionally, the authors in (Stiennon et al., 2020)
fine-tuned a model that may choose a human-preferred summary, they used this
model as a reward function to fine-tune a summarization policy using RL.

Foundation Models for Decision Making. Foundation models possess robust
decision-making capabilities, rendering them invaluable across various
downstream tasks. For instance, recent works (Ahn et al., 2022; Huang et al.,
2022a, b) showcase the application of foundation models in the robotics
domain. Moreover, works (Rawles et al., 2023; Wen et al., 2023; Yan et al.,
2023; Hong et al., 2023) utilize foundation models to intelligently navigate
Android applications. Additionally, the foundation models have been utilized
in gaming contexts († et al.(2022), FAIR; Lee et al., 2022; Reed et al., 2022;
Fan et al., 2022; Wang et al., 2023; Carta et al., 2023).

Web Navigation. Many benchmarks and datasets exist for the training and
assessment of web agents (Yao et al., 2022; Shi et al., 2017; Deng et al.,
2023; Zhou et al., 2023; Liu et al., 2018). Researchers have consequently
proposed diverse web agents and tested their performance on these benchmarks.
The MiniWob++ benchmark is among these benchmarks on which different methods
have been applied. For example, (Humphreys et al., 2022) employed a
combination of reinforcement learning and behavioral cloning, (Furuta et al.,
2023) utilized supervised training on an instruction-fine-tuned LLM, (Liu et
al., 2018) introduced Workflow-guided exploration (WGE), and (Gur et al.,
2018) trained DQN agents (QWeb network and INET network). Additionally, the
Mind2Web benchmark introduced the MindAct model, synergizing the strength of
small and large LLMs (Deng et al., 2023). Additionally, a visual language
model named CogAgent was utilized for the benchmark (Hong et al., 2023). (Zeng
et al., 2023) presented AgentTuning as another notable approach to tackle the
Mind2Web benchmark. Furthermore, considering the Webshop benchmark, various
methodologies have been proposed that use in-context learning (Kim et al.,
2023; Yao et al., 2023; Sridhar et al., rint), supervised learning (Furuta et
al., 2023; Shaw et al., 2023), and RL (Yao et al., 2022). _Nonetheless, no
work has clearly outlined the impact of human demonstrations and the optimal
utilization of available demonstration data. Furthermore, UDA remains
underexplored in current research._

##  7\. Conclusion

We introduce $\mathsf{GLAINTEL}$, a flexible agent designed for training
across diverse product search scenarios, accommodating situations with limited
or no human demonstrations for supervision. We also investigate the optimal
utilization of demonstration data, showing that straightforward supervised
learning approaches, like behavior cloning, do not consistently yield superior
results when using human demonstration data. Through extensive experimental
evaluations in the WebShop environment, we highlight the crucial role of the
unsupervised training phase employing the PPO algorithm. When combined with
supervised learning, this approach achieved results comparable to methods
utilizing GPT-4. Additionally, we explore an underexplored scenario where
demonstration data is confined to a single domain, we employ UDA techniques to
accommodate novel domains. We also present evaluations on a real website,
eBay, to showcase the applicability of $\mathsf{GLAINTEL}$ in real-world
search scenarios.

## References

  * (1)
  * Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022\.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ (2022).
  * Bellemare et al. (2013) Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. 2013.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ 47 (2013), 253–279.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021\.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ (2021).
  * Bowman et al. (2015) Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2015.  Generating sentences from a continuous space.  _arXiv preprint arXiv:1511.06349_ (2015).
  * Brandfonbrener et al. (2022) David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. 2022.  When does return-conditioned supervised learning work for offline reinforcement learning?  _arXiv preprint arXiv:2206.01079_ (2022).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Carta et al. (2023) Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. 2023.  Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning.  arXiv:2302.02662 [cs.LG]
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022\.  Scaling Instruction-Finetuned Language Models.  _CoRR_ abs/2210.11416 (2022).  https://doi.org/10.48550/arXiv.2210.11416
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages.  _Transactions of the Association for Computational Linguistics_ 8 (2020), 454–470.  https://doi.org/10.1162/tacl_a_00317
  * Deng et al. (2023) Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023.  Mind2Web: Towards a Generalist Agent for the Web.  arXiv:2306.06070 [cs.CL]
  * Dhariwal et al. (2017) Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. 2017.  OpenAI Baselines.  https://github.com/openai/baselines.
  * (13) Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. 2022.  Human-level play in the game of ¡i¿Diplomacy¡/i¿ by combining language models with strategic reasoning.  _Science_ 378, 6624 (2022), 1067–1074.  https://doi.org/10.1126/science.ade9097 arXiv:https://www.science.org/doi/pdf/10.1126/science.ade9097
  * Fan et al. (2022) Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. 2022.  MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge. In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.  https://openreview.net/forum?id=rc8o_j8I8PX
  * Furuta et al. (2023) Hiroki Furuta, Ofir Nachum, Kuang-Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur. 2023.  Instruction-Finetuned Foundation Models for Multimodal Web Navigation. In _ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models_.  https://openreview.net/forum?id=oLc9sGOBbc
  * Gur et al. (2018) Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur. 2018.  Learning to Navigate the Web.  arXiv:1812.09195 [cs.LG]
  * He et al. (2016) Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. 2016.  Deep Reinforcement Learning with a Natural Language Action Space. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, 1621–1630.  https://doi.org/10.18653/v1/P16-1153
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring Massive Multitask Language Understanding. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=d7KBjmI3GmQ
  * Hong et al. (2023) Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. 2023.  CogAgent: A Visual Language Model for GUI Agents.  arXiv:2312.08914 [cs.CV]
  * Huang et al. (2022a) Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022a.  Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In _Proceedings of the 39th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 162)_ , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 9118–9147.  https://proceedings.mlr.press/v162/huang22a.html
  * Huang et al. (2022b) Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. 2022b.  Inner Monologue: Embodied Reasoning through Planning with Language Models.  arXiv:2207.05608 [cs.RO]
  * Humphreys et al. (2022) Peter C. Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy P. Lillicrap. 2022.  A data-driven approach for learning to control computers. In _International Conference on Machine Learning_.  https://api.semanticscholar.org/CorpusID:246867455
  * Kim et al. (2023) Byoungjip Kim, Youngsoo Jang, Lajanugen Logeswaran, Geon-Hyeong Kim, Yu Jin Kim, Honglak Lee, and Moontae Lee. 2023.  Prospector: Improving LLM Agents with Self-Asking and Trajectory Ranking. In _NeurIPS 2023 Foundation Models for Decision Making Workshop_.  https://openreview.net/forum?id=YSYbTPbCPD
  * Lee et al. (2022) Kuang-Huei Lee, Ofir Nachum, Mengjiao (Sherry) Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, and Igor Mordatch. 2022.  Multi-Game Decision Transformers. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 27921–27936.  https://proceedings.neurips.cc/paper_files/paper/2022/file/b2cac94f82928a85055987d9fd44753f-Paper-Conference.pdf
  * Liang et al. (2023) Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023.  Code as policies: Language model programs for embodied control. In _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 9493–9500.
  * Lindenberger et al. (2021) Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Marc Pollefeys. 2021.  Pixel-Perfect Structure-From-Motion With Featuremetric Refinement. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 5987–5997.
  * Liu et al. (2018) Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018.  Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration. In _International Conference on Learning Representations (ICLR)_.  https://arxiv.org/abs/1802.08802
  * Ma et al. (2023) Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, and Dong Yu. 2023.  LASER: LLM Agent with State-Space Exploration for Web Navigation.  _arXiv preprint arXiv:2309.08172_ (2023).
  * Mahowald et al. (2023) Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2023.  Dissociating language and thought in large language models: a cognitive perspective.  _arXiv preprint arXiv:2301.06627_ (2023).
  * Mazumder and Riva (2020) Sahisnu Mazumder and Oriana Riva. 2020.  Flin: A flexible natural language interface for web navigation.  _arXiv preprint arXiv:2010.12844_ (2020).
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021\.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_ (2021).
  * Nogueira and Cho (2016) Rodrigo Nogueira and Kyunghyun Cho. 2016.  End-to-end goal-driven web navigation.  _Advances in neural information processing systems_ 29 (2016).
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35 (2022), 27730–27744.
  * Paster et al. (2022) Keiran Paster, Sheila McIlraith, and Jimmy Ba. 2022.  You Can’t Count on Luck: Why Decision Transformers Fail in Stochastic Environments.  _arXiv preprint arXiv:2205.15967_ (2022).
  * Pasupat et al. (2018) Panupong Pasupat, Tian-Shun Jiang, Evan Zheran Liu, Kelvin Guu, and Percy Liang. 2018.  Mapping natural language commands to web elements.  _arXiv preprint arXiv:1808.09132_ (2018).
  * Pomerleau (1989) Dean A Pomerleau. 1989.  _Alvinn: An autonomous land vehicle in a neural network_.  Technical Report. CARNEGIE-MELLON UNIV PITTSBURGH PA ARTIFICIAL INTELLIGENCE AND PSYCHOLOGY ….
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ (2019).
  * Ramamurthy et al. (2022) Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022.  Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization.  _arXiv preprint arXiv:2210.01241_ (2022).
  * Rawles et al. (2023) Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023.  Android in the Wild: A Large-Scale Dataset for Android Device Control.  arXiv:2307.10088 [cs.LG]
  * Reed et al. (2022) Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. 2022.  A Generalist Agent.  arXiv:2205.06175 [cs.AI]
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask Prompted Training Enables Zero-Shot Task Generalization. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=9Vrb9D0WI4
  * Schulman et al. (2015) John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015.  Trust region policy optimization. In _International conference on machine learning_. PMLR, 1889–1897.
  * Shaw et al. (2023) Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. 2023.  From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces.  arXiv:2306.00245 [cs.LG]
  * Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022.  Language Models are Multilingual Chain-of-Thought Reasoners.  arXiv:2210.03057 [cs.CL]
  * Shi et al. (2017) Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. 2017.  World of bits: An open-domain platform for web-based agents. In _International Conference on Machine Learning_. PMLR, 3135–3144.
  * Sridhar et al. (rint) Abishek Sridhar, Robert Lo, Frank F. Xu, Hao Zhu, and Shuyan Zhou. preprint.  Hierarchical Prompting Assists Large Language Model on Web Navigation. In _ArXiv_.
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, and et al Adrià Garriga-Alonso. 2023.  Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.  arXiv:2206.04615 [cs.CL]
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ 33 (2020), 3008–3021.
  * Su et al. (2017) Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, Patrick Pantel, Michael Gamon, and Mark Encarnacion. 2017.  Building natural language interfaces to web apis. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 177–186.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2023) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv:2305.16291_ (2023).
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022.  Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 5085–5109.  https://doi.org/10.18653/v1/2022.emnlp-main.340
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022.  Finetuned Language Models are Zero-Shot Learners. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=gEZrGCozdqR
  * Wen et al. (2023) Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2023.  Empowering LLM to use Smartphone for Intelligent Task Automation.  _ArXiv_ abs/2308.15272 (2023).  https://api.semanticscholar.org/CorpusID:261277501
  * Yan et al. (2023) An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. 2023.  GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation.  _ArXiv_ abs/2311.07562 (2023).  https://api.semanticscholar.org/CorpusID:265149992
  * Yang et al. (2022) Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. 2022.  Dichotomy of control: Separating what you can control from what you cannot.  _arXiv preprint arXiv:2210.13435_ (2022).
  * Yao et al. (2022) Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022.  WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 20744–20757.  https://proceedings.neurips.cc/paper_files/paper/2022/file/82ad13ec01f9fe44c01cb91814fd7b8c-Paper-Conference.pdf
  * Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023.  ReAct: Synergizing Reasoning and Acting in Language Models. In _The Eleventh International Conference on Learning Representations_.  https://openreview.net/forum?id=WE_vluYUL-X
  * Zeng et al. (2023) Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023.  AgentTuning: Enabling Generalized Agent Abilities for LLMs.  arXiv:2310.12823 [cs.CL]
  * Zhou et al. (2023) Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023\.  WebArena: A Realistic Web Environment for Building Autonomous Agents.  _arXiv preprint arXiv:2307.13854_ (2023).